# Media

PLAN.md Phase 6: the media repository. Clients upload files and get an
`mxc://` URI back to put in an event; anybody in the room downloads them,
or asks for a thumbnail; and media uploaded to another server is fetched
from it, over federation, the first time somebody here wants it.

## Where things live

| Package | What |
|---|---|
| `internal/mediaapi` | The component: uploads, reserved uploads, downloads, thumbnails, the cache of other servers' media, and the HTTP helpers both APIs share — headers, parameters, error mapping. |
| `internal/clientapi/media.go` | The Client-Server endpoints. |
| `internal/federationapi/media.go` | The two endpoints other servers download through. |
| `internal/federation/media.go` | Downloading from other servers: the multipart answer, redirects, the fallback for servers older than Matrix 1.11. |
| `internal/storage` | `MediaStore`, the metadata (`media_repository`, `media_thumbnails`), and `BlobStore`, the content. |
| `internal/storage/filestore` | The local-filesystem `BlobStore`. |

## Metadata and content

The database holds what an `mxc://` URI means — who uploaded it, its
content type and file name, its size — and the SHA-256 of its content. The
content itself is in a `BlobStore`, named by that hash:

```
media_store/
  ab/cd/abcd0123…   one file per distinct content
  tmp/              uploads in progress
```

Naming content by its hash gives three things at once. The same file
uploaded twice — a meme going round, the same avatar set on two accounts —
is stored once. A thumbnail that turns out to be the original (an image
already smaller than the size asked for) is a second row pointing at the
same file rather than a copy. And the hash is the `ETag`, so a client that
has the file already gets `304 Not Modified`.

An upload is written to `tmp/`, hashed as it is written, and renamed into
place only when it is complete: a reader never sees half a file, and a
crash leaves a temporary file and nothing else. Nothing in the database
refers to content until it is all there.

The blob store is chosen by the storage driver. SQLite and PostgreSQL use
the local filesystem under `media.base_path`. A driver that brings its own
file storage implements `storage.BlobProvider`, and PocketBase will — its
filesystem abstraction is local disk or S3 (PLAN.md section 4) — when that
driver is built in Phase 10.

## Endpoints

| | |
|---|---|
| `POST /_matrix/media/v3/upload` | upload; `?filename=` names it |
| `POST /_matrix/media/v1/create` | reserve an `mxc://` URI to upload to later |
| `PUT /_matrix/media/v3/upload/{server}/{id}` | upload to a reserved URI |
| `GET /_matrix/client/v1/media/config` (and `/_matrix/media/v3/config`) | `m.upload.size` |
| `GET /_matrix/client/v1/media/download/{server}/{id}[/{fileName}]` | download; the file name in the path overrides the stored one |
| `GET /_matrix/client/v1/media/thumbnail/{server}/{id}` | thumbnail: `width`, `height`, `method`, `animated` |
| `GET /_matrix/federation/v1/media/download/{id}` | another server downloading this server's media |
| `GET /_matrix/federation/v1/media/thumbnail/{id}` | and its thumbnails |

Every download endpoint takes `?timeout_ms=`: how long to wait for the
content of a reserved URI that has not been uploaded yet (20 seconds by
default, two minutes at most). Past it the answer is `504
M_NOT_YET_UPLOADED`.

### Authenticated media only

Matrix 1.11 moved downloads under `/_matrix/client/v1/media`, behind an
access token, because the old endpoints let anybody who learned an
`mxc://` URI read the file — no account, no room membership. Warp serves
only the new ones by default. The old unauthenticated
`/_matrix/media/v3/download` and `/thumbnail` come back with
`media.legacy_unauthenticated_downloads: true`, for clients too old to know
better; with it they also honour `allow_remote=false`.

`GET /_matrix/client/versions` lists v1.11 and later, which is what tells a
client to use the authenticated endpoints. The access token may come in the
`Authorization` header or as `?access_token=`, which is what a web client
needs for an `<img src>`.

An access token proves an account, not membership: like every other server,
Warp lets any logged-in user download any media whose URI they know.
Tying media to the rooms it was sent in is MSC3911, not yet in the spec.

### What a download is served with

- **`Content-Disposition`**: `inline` only for the types the spec lists as
  safe to show in a browser — images, audio, video, plain text, CSS, JSON —
  and `attachment` for everything else. An uploaded HTML page is a
  download, never a page on the homeserver's origin. The file name is
  RFC 6266-encoded when it is not ASCII.
- **`Content-Security-Policy: sandbox; default-src 'none'; …`** so that
  content a browser is tricked into rendering anyway runs no script.
- **`X-Content-Type-Options: nosniff`** so a `text/plain` upload is not
  sniffed into HTML.
- **`Cross-Origin-Resource-Policy: cross-origin`** so a web client on
  another origin can embed the images.
- **`Cache-Control: private, max-age=31536000, immutable`**: the content of
  an `mxc://` URI never changes, but the endpoint needs a token, so no
  shared cache may keep it.
- **`ETag`**, the content's hash, and `Range` support, from
  `http.ServeContent`.

The uploaded `Content-Type` is kept as sent when it parses, and replaced by
`application/octet-stream` when it does not.

## Reserved uploads

`POST /media/v1/create` hands out an `mxc://` URI before there is anything
behind it, so a client can send the event that refers to a large file
while the file is still uploading. The reservation belongs to the user who
made it, lasts `media.pending_upload_expiry` (a day), and takes exactly one
upload: a second is `409 M_CANNOT_OVERWRITE_MEDIA`, an upload by anybody
else `403 M_FORBIDDEN`. A user may hold `media.max_pending_uploads` (ten)
at once; past that, `429 M_LIMIT_EXCEEDED`.

A download of a reserved URI waits for its content — in memory, woken by
the upload the moment it completes — for as long as its `timeout_ms`
allows. Lapsed reservations are swept whenever a new one is made, which is
the only time their number matters.

## Thumbnails

A client asks for a thumbnail of a size; the server answers with one of its
own sizes, `media.thumbnail_sizes`:

| width × height | method | for |
|---|---|---|
| 32 × 32 | crop | avatars in lists |
| 96 × 96 | crop | avatars |
| 320 × 240 | scale | images in the timeline |
| 640 × 480 | scale | |
| 800 × 600 | scale | |

The answer is the smallest size of the method asked for that is at least
as large in both dimensions, or the largest if none is. A method with no
configured sizes is answered from the other. Fixing the sizes is what makes
thumbnails cacheable at all: a client asking for 33 × 33 and another
asking for 34 × 34 share one, and nobody can make the server resize the
same image a thousand times by asking for a thousand sizes.

- **`scale`** fits the image inside the box and keeps its aspect ratio.
- **`crop`** fills the box exactly, cutting the longer side down equally
  at both ends.
- Neither enlarges. An image smaller than the box is cropped at its own
  size; one that already fits a `scale` box is served as it is — the
  thumbnail row points at the original's content.

Thumbnails are made on first request and kept. JPEG, PNG, GIF and WebP are
decoded (WebP through `golang.org/x/image`); a JPEG thumbnail is a JPEG,
anything else a PNG, which keeps transparency. A JPEG's EXIF orientation is
honoured, so a phone's portrait photo is not a sideways thumbnail. GIFs are
always re-encoded, as a still image: Warp makes no animated thumbnails, and
`animated=true` gets a still one, which the spec allows.

What could hurt the server is bounded. The image header is read first, and
anything larger than `media.max_thumbnail_pixels` (32 megapixels) is
refused before a pixel is decoded — a 100 KiB PNG can claim to be 30000 ×
30000, and decoding costs four bytes a pixel whatever the file size. At
most four thumbnails are made at once, and concurrent requests for the same
one share one resize.

Content that cannot be thumbnailed — a PDF, a video, an image that does not
decode — is `400 M_UNKNOWN`.

## Other servers' media

Media whose `mxc://` server is another server is fetched from it on first
request and kept in the same tables and blob store as local media, under
its own server name. An `mxc://` URI names immutable content, so the copy
never goes stale.

Fetching uses the signed federation client:

1. `GET /_matrix/federation/v1/media/download/{id}`, whose answer is
   `multipart/mixed`: a JSON part (an empty object, for now) and then
   either the content or a part with a `Location` header saying where the
   content is.
2. A server older than Matrix 1.11 answers that with `404 M_UNRECOGNIZED`,
   and is then asked through the deprecated
   `GET /_matrix/media/v3/download/{server}/{id}?allow_remote=false` —
   what every server before 1.11 served media to other servers through.
   A plain `404 M_NOT_FOUND` is not retried: that server knows the
   endpoint and lacks the media.

A `Location` is an ordinary web address, often a CDN, fetched without
signing. It is the one place another server can make this one fetch an
arbitrary URL and hand the answer to a client, so the dialer checks the
address it actually connects to — after DNS, on every redirect — and
refuses loopback, private, link-local and multicast addresses. A remote
server cannot use it to read this server's internal network.

Concurrent requests for the same remote media share one fetch, which runs
detached from the request that started it: a client that gives up does not
waste the download for the others, and what was fetched is kept. The
content is held to `media.max_upload_bytes`, like an upload.

Thumbnails of remote media come from the cached original when there is
one; otherwise the remote server is asked for a thumbnail of the
configured size that answers the request, and that is cached under that
size. The original is not downloaded just to make a thumbnail.

Another server is served this server's own media only. It is never made to
fetch from a third server on another's behalf.

With federation off, nothing is fetched: media that was cached before is
still served, and anything else from another server is `403 M_FORBIDDEN`
with a message saying federation is disabled.

## Errors

| | |
|---|---|
| `404 M_NOT_FOUND` | no such media, here or on its server |
| `413 M_TOO_LARGE` | larger than `media.max_upload_bytes`, uploaded or fetched |
| `504 M_NOT_YET_UPLOADED` | reserved, not uploaded within `timeout_ms` |
| `409 M_CANNOT_OVERWRITE_MEDIA` | a reserved URI that already has content |
| `403 M_FORBIDDEN` | someone else's reservation; or remote media with federation off |
| `429 M_LIMIT_EXCEEDED` | too many reservations |
| `400 M_UNKNOWN` | cannot make a thumbnail of this |
| `502 M_UNKNOWN` | the media's server failed to provide it |

## Known gaps

- **URL previews** (`/media/v3/preview_url`) are not implemented and answer
  `M_UNRECOGNIZED`. Fetching arbitrary URLs for clients needs the same
  network guard as media redirects plus an HTML parser, and is its own
  piece of work.
- **No deletion or quotas yet.** Nothing removes media: no admin purge, no
  retention for the remote cache, no per-user quota. Those are the media
  parts of Phase 9's admin work, and content-addressed storage means a
  purge has to check that no other row still uses a blob before deleting
  it.
- **No quarantine**, which belongs with the same admin work.
- **No animated thumbnails**; `animated=true` gets a still image.
- **No S3** until the PocketBase driver brings its filesystem abstraction;
  the `BlobProvider` seam is where it plugs in.