Warp/docs/media.md

Media

PLAN.md Phase 6: the media repository. Clients upload files and get an mxc:// URI back to put in an event; anybody in the room downloads them, or asks for a thumbnail; and media uploaded to another server is fetched from it, over federation, the first time somebody here wants it.

Where things live

Package	What
internal/mediaapi	The component: uploads, reserved uploads, downloads, thumbnails, the cache of other servers' media, and the HTTP helpers both APIs share — headers, parameters, error mapping.
internal/clientapi/media.go	The Client-Server endpoints.
internal/federationapi/media.go	The two endpoints other servers download through.
internal/federation/media.go	Downloading from other servers: the multipart answer, redirects, the fallback for servers older than Matrix 1.11.
internal/storage	MediaStore, the metadata (media_repository, media_thumbnails), and BlobStore, the content.
internal/storage/filestore	The local-filesystem BlobStore.

Metadata and content

The database holds what an mxc:// URI means — who uploaded it, its content type and file name, its size — and the SHA-256 of its content. The content itself is in a BlobStore, named by that hash:

media_store/
  ab/cd/abcd0123…   one file per distinct content
  tmp/              uploads in progress

Naming content by its hash gives three things at once. The same file uploaded twice — a meme going round, the same avatar set on two accounts — is stored once. A thumbnail that turns out to be the original (an image already smaller than the size asked for) is a second row pointing at the same file rather than a copy. And the hash is the ETag, so a client that has the file already gets 304 Not Modified.

An upload is written to tmp/, hashed as it is written, and renamed into place only when it is complete: a reader never sees half a file, and a crash leaves a temporary file and nothing else. Nothing in the database refers to content until it is all there.

The blob store is chosen by the storage driver. SQLite and PostgreSQL use the local filesystem under media.base_path. A driver that brings its own file storage implements storage.BlobProvider, and PocketBase will — its filesystem abstraction is local disk or S3 (PLAN.md section 4) — when that driver is built in Phase 10.

Endpoints

POST /_matrix/media/v3/upload	upload; ?filename= names it
POST /_matrix/media/v1/create	reserve an mxc:// URI to upload to later
PUT /_matrix/media/v3/upload/{server}/{id}	upload to a reserved URI
GET /_matrix/client/v1/media/config (and /_matrix/media/v3/config)	m.upload.size
GET /_matrix/client/v1/media/download/{server}/{id}[/{fileName}]	download; the file name in the path overrides the stored one
GET /_matrix/client/v1/media/thumbnail/{server}/{id}	thumbnail: width, height, method, animated
GET /_matrix/federation/v1/media/download/{id}	another server downloading this server's media
GET /_matrix/federation/v1/media/thumbnail/{id}	and its thumbnails

Every download endpoint takes ?timeout_ms=: how long to wait for the content of a reserved URI that has not been uploaded yet (20 seconds by default, two minutes at most). Past it the answer is 504 M_NOT_YET_UPLOADED.

Authenticated media only

Matrix 1.11 moved downloads under /_matrix/client/v1/media, behind an access token, because the old endpoints let anybody who learned an mxc:// URI read the file — no account, no room membership. Warp serves only the new ones by default. The old unauthenticated /_matrix/media/v3/download and /thumbnail come back with media.legacy_unauthenticated_downloads: true, for clients too old to know better; with it they also honour allow_remote=false.

GET /_matrix/client/versions lists v1.11 and later, which is what tells a client to use the authenticated endpoints. The access token may come in the Authorization header or as ?access_token=, which is what a web client needs for an <img src>.

An access token proves an account, not membership: like every other server, Warp lets any logged-in user download any media whose URI they know. Tying media to the rooms it was sent in is MSC3911, not yet in the spec.

What a download is served with

• Content-Disposition: inline only for the types the spec lists as safe to show in a browser — images, audio, video, plain text, CSS, JSON — and attachment for everything else. An uploaded HTML page is a download, never a page on the homeserver's origin. The file name is RFC 6266-encoded when it is not ASCII.
• Content-Security-Policy: sandbox; default-src 'none'; … so that content a browser is tricked into rendering anyway runs no script.
• X-Content-Type-Options: nosniff so a text/plain upload is not sniffed into HTML.
• Cross-Origin-Resource-Policy: cross-origin so a web client on another origin can embed the images.
• Cache-Control: private, max-age=31536000, immutable: the content of an mxc:// URI never changes, but the endpoint needs a token, so no shared cache may keep it.
• ETag, the content's hash, and Range support, from http.ServeContent.

The uploaded Content-Type is kept as sent when it parses, and replaced by application/octet-stream when it does not.

Reserved uploads

POST /media/v1/create hands out an mxc:// URI before there is anything behind it, so a client can send the event that refers to a large file while the file is still uploading. The reservation belongs to the user who made it, lasts media.pending_upload_expiry (a day), and takes exactly one upload: a second is 409 M_CANNOT_OVERWRITE_MEDIA, an upload by anybody else 403 M_FORBIDDEN. A user may hold media.max_pending_uploads (ten) at once; past that, 429 M_LIMIT_EXCEEDED.

A download of a reserved URI waits for its content — in memory, woken by the upload the moment it completes — for as long as its timeout_ms allows. Lapsed reservations are swept whenever a new one is made, which is the only time their number matters.

Thumbnails

A client asks for a thumbnail of a size; the server answers with one of its own sizes, media.thumbnail_sizes:

width × height	method	for
32 × 32	crop	avatars in lists
96 × 96	crop	avatars
320 × 240	scale	images in the timeline
640 × 480	scale
800 × 600	scale

The answer is the smallest size of the method asked for that is at least as large in both dimensions, or the largest if none is. A method with no configured sizes is answered from the other. Fixing the sizes is what makes thumbnails cacheable at all: a client asking for 33 × 33 and another asking for 34 × 34 share one, and nobody can make the server resize the same image a thousand times by asking for a thousand sizes.

• scale fits the image inside the box and keeps its aspect ratio.
• crop fills the box exactly, cutting the longer side down equally at both ends.
• Neither enlarges. An image smaller than the box is cropped at its own size; one that already fits a scale box is served as it is — the thumbnail row points at the original's content.

Thumbnails are made on first request and kept. JPEG, PNG, GIF and WebP are decoded (WebP through golang.org/x/image); a JPEG thumbnail is a JPEG, anything else a PNG, which keeps transparency. A JPEG's EXIF orientation is honoured, so a phone's portrait photo is not a sideways thumbnail. GIFs are always re-encoded, as a still image: Warp makes no animated thumbnails, and animated=true gets a still one, which the spec allows.

What could hurt the server is bounded. The image header is read first, and anything larger than media.max_thumbnail_pixels (32 megapixels) is refused before a pixel is decoded — a 100 KiB PNG can claim to be 30000 × 30000, and decoding costs four bytes a pixel whatever the file size. At most four thumbnails are made at once, and concurrent requests for the same one share one resize.

Content that cannot be thumbnailed — a PDF, a video, an image that does not decode — is 400 M_UNKNOWN.

Other servers' media

Media whose mxc:// server is another server is fetched from it on first request and kept in the same tables and blob store as local media, under its own server name. An mxc:// URI names immutable content, so the copy never goes stale.

Fetching uses the signed federation client:

1. GET /_matrix/federation/v1/media/download/{id}, whose answer is multipart/mixed: a JSON part (an empty object, for now) and then either the content or a part with a Location header saying where the content is.
2. A server older than Matrix 1.11 answers that with 404 M_UNRECOGNIZED, and is then asked through the deprecated GET /_matrix/media/v3/download/{server}/{id}?allow_remote=false — what every server before 1.11 served media to other servers through. A plain 404 M_NOT_FOUND is not retried: that server knows the endpoint and lacks the media.

A Location is an ordinary web address, often a CDN, fetched without signing. It is the one place another server can make this one fetch an arbitrary URL and hand the answer to a client, so the dialer checks the address it actually connects to — after DNS, on every redirect — and refuses loopback, private, link-local and multicast addresses. A remote server cannot use it to read this server's internal network.

Concurrent requests for the same remote media share one fetch, which runs detached from the request that started it: a client that gives up does not waste the download for the others, and what was fetched is kept. The content is held to media.max_upload_bytes, like an upload.

Thumbnails of remote media come from the cached original when there is one; otherwise the remote server is asked for a thumbnail of the configured size that answers the request, and that is cached under that size. The original is not downloaded just to make a thumbnail.

Another server is served this server's own media only. It is never made to fetch from a third server on another's behalf.

With federation off, nothing is fetched: media that was cached before is still served, and anything else from another server is 403 M_FORBIDDEN with a message saying federation is disabled.

Errors

404 M_NOT_FOUND	no such media, here or on its server
413 M_TOO_LARGE	larger than media.max_upload_bytes, uploaded or fetched; or over the uploader's quota
504 M_NOT_YET_UPLOADED	reserved, not uploaded within timeout_ms
409 M_CANNOT_OVERWRITE_MEDIA	a reserved URI that already has content
403 M_FORBIDDEN	someone else's reservation; or remote media with federation off
429 M_LIMIT_EXCEEDED	too many reservations
400 M_UNKNOWN	cannot make a thumbnail of this
502 M_UNKNOWN	the media's server failed to provide it

Known gaps

• URL previews (/media/v3/preview_url) are not implemented and answer M_UNRECOGNIZED. Fetching arbitrary URLs for clients needs the same network guard as media redirects plus an HTML parser, and is its own piece of work.
• No deletion yet. Nothing removes media: no admin purge, and no retention for the remote cache. Content-addressed storage means a purge has to check that no other row still uses a blob before deleting it. Per-user quotas are in (Phase 9): media.max_user_storage_bytes, and a quota per user through the admin API. An upload over the quota is 413 M_TOO_LARGE. See admin-api.md.
• No quarantine, which belongs with the same admin work.
• No animated thumbnails; animated=true gets a still image.
• No S3 until the PocketBase driver brings its filesystem abstraction; the BlobProvider seam is where it plugs in.