tajniak81andClaude Opus 5.5
92836f8dcd
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
Implements PLAN.md Phase 6: the media repository. Clients upload files, reserve an mxc:// URI and upload to it later, download and ask for thumbnails, all through the authenticated media endpoints of Matrix 1.11; other servers download this server's media through the federation media endpoints, and media uploaded elsewhere is fetched over federation on first use and kept. internal/mediaapi is the component. Content goes into a BlobStore named by its SHA-256, so the same file uploaded twice is stored once and the hash doubles as the ETag; the metadata that maps an mxc:// URI onto a blob is in media_repository and media_thumbnails. The local-filesystem store writes to a temporary file and renames it into place once hashed, so nothing ever refers to half a file. A storage driver can bring its own store through storage.BlobProvider, which is where PocketBase's filesystem abstraction (local or S3) plugs in with Phase 10. Reserved uploads belong to the user who made them, take one upload, lapse after a day, and are capped per user; a download of one waits for its content, woken in memory the moment it arrives, for as long as timeout_ms allows. Thumbnails come from a fixed set of configured sizes, so a thousand requested sizes cannot mean a thousand resizes. They are made on first request and kept, never enlarge, honour a JPEG's EXIF orientation, and point at the original when it already fits. The pixel count is checked from the header before anything is decoded, at most four resizes run at once, and concurrent requests for the same thumbnail share one. Remote media is fetched through the signed client: the multipart/mixed answer of Matrix 1.11, with a fallback to the legacy download endpoint for servers that answer M_UNRECOGNIZED, and Location redirects followed by a dialer that refuses loopback, private and link-local addresses. Concurrent requests share one fetch, detached from the request that started it. With federation off nothing is fetched, and what was cached before is still served. Downloads carry a sandboxing CSP, nosniff, inline Content-Disposition only for the types the spec lists as safe, and private immutable caching. The unauthenticated legacy download endpoints are off by default behind media.legacy_unauthenticated_downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
239 lines
12 KiB
Markdown
239 lines
12 KiB
Markdown
# Media
|
||
|
||
PLAN.md Phase 6: the media repository. Clients upload files and get an
|
||
`mxc://` URI back to put in an event; anybody in the room downloads them,
|
||
or asks for a thumbnail; and media uploaded to another server is fetched
|
||
from it, over federation, the first time somebody here wants it.
|
||
|
||
## Where things live
|
||
|
||
| Package | What |
|
||
|---|---|
|
||
| `internal/mediaapi` | The component: uploads, reserved uploads, downloads, thumbnails, the cache of other servers' media, and the HTTP helpers both APIs share — headers, parameters, error mapping. |
|
||
| `internal/clientapi/media.go` | The Client-Server endpoints. |
|
||
| `internal/federationapi/media.go` | The two endpoints other servers download through. |
|
||
| `internal/federation/media.go` | Downloading from other servers: the multipart answer, redirects, the fallback for servers older than Matrix 1.11. |
|
||
| `internal/storage` | `MediaStore`, the metadata (`media_repository`, `media_thumbnails`), and `BlobStore`, the content. |
|
||
| `internal/storage/filestore` | The local-filesystem `BlobStore`. |
|
||
|
||
## Metadata and content
|
||
|
||
The database holds what an `mxc://` URI means — who uploaded it, its
|
||
content type and file name, its size — and the SHA-256 of its content. The
|
||
content itself is in a `BlobStore`, named by that hash:
|
||
|
||
```
|
||
media_store/
|
||
ab/cd/abcd0123… one file per distinct content
|
||
tmp/ uploads in progress
|
||
```
|
||
|
||
Naming content by its hash gives three things at once. The same file
|
||
uploaded twice — a meme going round, the same avatar set on two accounts —
|
||
is stored once. A thumbnail that turns out to be the original (an image
|
||
already smaller than the size asked for) is a second row pointing at the
|
||
same file rather than a copy. And the hash is the `ETag`, so a client that
|
||
has the file already gets `304 Not Modified`.
|
||
|
||
An upload is written to `tmp/`, hashed as it is written, and renamed into
|
||
place only when it is complete: a reader never sees half a file, and a
|
||
crash leaves a temporary file and nothing else. Nothing in the database
|
||
refers to content until it is all there.
|
||
|
||
The blob store is chosen by the storage driver. SQLite and PostgreSQL use
|
||
the local filesystem under `media.base_path`. A driver that brings its own
|
||
file storage implements `storage.BlobProvider`, and PocketBase will — its
|
||
filesystem abstraction is local disk or S3 (PLAN.md section 4) — when that
|
||
driver is built in Phase 10.
|
||
|
||
## Endpoints
|
||
|
||
| | |
|
||
|---|---|
|
||
| `POST /_matrix/media/v3/upload` | upload; `?filename=` names it |
|
||
| `POST /_matrix/media/v1/create` | reserve an `mxc://` URI to upload to later |
|
||
| `PUT /_matrix/media/v3/upload/{server}/{id}` | upload to a reserved URI |
|
||
| `GET /_matrix/client/v1/media/config` (and `/_matrix/media/v3/config`) | `m.upload.size` |
|
||
| `GET /_matrix/client/v1/media/download/{server}/{id}[/{fileName}]` | download; the file name in the path overrides the stored one |
|
||
| `GET /_matrix/client/v1/media/thumbnail/{server}/{id}` | thumbnail: `width`, `height`, `method`, `animated` |
|
||
| `GET /_matrix/federation/v1/media/download/{id}` | another server downloading this server's media |
|
||
| `GET /_matrix/federation/v1/media/thumbnail/{id}` | and its thumbnails |
|
||
|
||
Every download endpoint takes `?timeout_ms=`: how long to wait for the
|
||
content of a reserved URI that has not been uploaded yet (20 seconds by
|
||
default, two minutes at most). Past it the answer is `504
|
||
M_NOT_YET_UPLOADED`.
|
||
|
||
### Authenticated media only
|
||
|
||
Matrix 1.11 moved downloads under `/_matrix/client/v1/media`, behind an
|
||
access token, because the old endpoints let anybody who learned an
|
||
`mxc://` URI read the file — no account, no room membership. Warp serves
|
||
only the new ones by default. The old unauthenticated
|
||
`/_matrix/media/v3/download` and `/thumbnail` come back with
|
||
`media.legacy_unauthenticated_downloads: true`, for clients too old to know
|
||
better; with it they also honour `allow_remote=false`.
|
||
|
||
`GET /_matrix/client/versions` lists v1.11 and later, which is what tells a
|
||
client to use the authenticated endpoints. The access token may come in the
|
||
`Authorization` header or as `?access_token=`, which is what a web client
|
||
needs for an `<img src>`.
|
||
|
||
An access token proves an account, not membership: like every other server,
|
||
Warp lets any logged-in user download any media whose URI they know.
|
||
Tying media to the rooms it was sent in is MSC3911, not yet in the spec.
|
||
|
||
### What a download is served with
|
||
|
||
- **`Content-Disposition`**: `inline` only for the types the spec lists as
|
||
safe to show in a browser — images, audio, video, plain text, CSS, JSON —
|
||
and `attachment` for everything else. An uploaded HTML page is a
|
||
download, never a page on the homeserver's origin. The file name is
|
||
RFC 6266-encoded when it is not ASCII.
|
||
- **`Content-Security-Policy: sandbox; default-src 'none'; …`** so that
|
||
content a browser is tricked into rendering anyway runs no script.
|
||
- **`X-Content-Type-Options: nosniff`** so a `text/plain` upload is not
|
||
sniffed into HTML.
|
||
- **`Cross-Origin-Resource-Policy: cross-origin`** so a web client on
|
||
another origin can embed the images.
|
||
- **`Cache-Control: private, max-age=31536000, immutable`**: the content of
|
||
an `mxc://` URI never changes, but the endpoint needs a token, so no
|
||
shared cache may keep it.
|
||
- **`ETag`**, the content's hash, and `Range` support, from
|
||
`http.ServeContent`.
|
||
|
||
The uploaded `Content-Type` is kept as sent when it parses, and replaced by
|
||
`application/octet-stream` when it does not.
|
||
|
||
## Reserved uploads
|
||
|
||
`POST /media/v1/create` hands out an `mxc://` URI before there is anything
|
||
behind it, so a client can send the event that refers to a large file
|
||
while the file is still uploading. The reservation belongs to the user who
|
||
made it, lasts `media.pending_upload_expiry` (a day), and takes exactly one
|
||
upload: a second is `409 M_CANNOT_OVERWRITE_MEDIA`, an upload by anybody
|
||
else `403 M_FORBIDDEN`. A user may hold `media.max_pending_uploads` (ten)
|
||
at once; past that, `429 M_LIMIT_EXCEEDED`.
|
||
|
||
A download of a reserved URI waits for its content — in memory, woken by
|
||
the upload the moment it completes — for as long as its `timeout_ms`
|
||
allows. Lapsed reservations are swept whenever a new one is made, which is
|
||
the only time their number matters.
|
||
|
||
## Thumbnails
|
||
|
||
A client asks for a thumbnail of a size; the server answers with one of its
|
||
own sizes, `media.thumbnail_sizes`:
|
||
|
||
| width × height | method | for |
|
||
|---|---|---|
|
||
| 32 × 32 | crop | avatars in lists |
|
||
| 96 × 96 | crop | avatars |
|
||
| 320 × 240 | scale | images in the timeline |
|
||
| 640 × 480 | scale | |
|
||
| 800 × 600 | scale | |
|
||
|
||
The answer is the smallest size of the method asked for that is at least
|
||
as large in both dimensions, or the largest if none is. A method with no
|
||
configured sizes is answered from the other. Fixing the sizes is what makes
|
||
thumbnails cacheable at all: a client asking for 33 × 33 and another
|
||
asking for 34 × 34 share one, and nobody can make the server resize the
|
||
same image a thousand times by asking for a thousand sizes.
|
||
|
||
- **`scale`** fits the image inside the box and keeps its aspect ratio.
|
||
- **`crop`** fills the box exactly, cutting the longer side down equally
|
||
at both ends.
|
||
- Neither enlarges. An image smaller than the box is cropped at its own
|
||
size; one that already fits a `scale` box is served as it is — the
|
||
thumbnail row points at the original's content.
|
||
|
||
Thumbnails are made on first request and kept. JPEG, PNG, GIF and WebP are
|
||
decoded (WebP through `golang.org/x/image`); a JPEG thumbnail is a JPEG,
|
||
anything else a PNG, which keeps transparency. A JPEG's EXIF orientation is
|
||
honoured, so a phone's portrait photo is not a sideways thumbnail. GIFs are
|
||
always re-encoded, as a still image: Warp makes no animated thumbnails, and
|
||
`animated=true` gets a still one, which the spec allows.
|
||
|
||
What could hurt the server is bounded. The image header is read first, and
|
||
anything larger than `media.max_thumbnail_pixels` (32 megapixels) is
|
||
refused before a pixel is decoded — a 100 KiB PNG can claim to be 30000 ×
|
||
30000, and decoding costs four bytes a pixel whatever the file size. At
|
||
most four thumbnails are made at once, and concurrent requests for the same
|
||
one share one resize.
|
||
|
||
Content that cannot be thumbnailed — a PDF, a video, an image that does not
|
||
decode — is `400 M_UNKNOWN`.
|
||
|
||
## Other servers' media
|
||
|
||
Media whose `mxc://` server is another server is fetched from it on first
|
||
request and kept in the same tables and blob store as local media, under
|
||
its own server name. An `mxc://` URI names immutable content, so the copy
|
||
never goes stale.
|
||
|
||
Fetching uses the signed federation client:
|
||
|
||
1. `GET /_matrix/federation/v1/media/download/{id}`, whose answer is
|
||
`multipart/mixed`: a JSON part (an empty object, for now) and then
|
||
either the content or a part with a `Location` header saying where the
|
||
content is.
|
||
2. A server older than Matrix 1.11 answers that with `404 M_UNRECOGNIZED`,
|
||
and is then asked through the deprecated
|
||
`GET /_matrix/media/v3/download/{server}/{id}?allow_remote=false` —
|
||
what every server before 1.11 served media to other servers through.
|
||
A plain `404 M_NOT_FOUND` is not retried: that server knows the
|
||
endpoint and lacks the media.
|
||
|
||
A `Location` is an ordinary web address, often a CDN, fetched without
|
||
signing. It is the one place another server can make this one fetch an
|
||
arbitrary URL and hand the answer to a client, so the dialer checks the
|
||
address it actually connects to — after DNS, on every redirect — and
|
||
refuses loopback, private, link-local and multicast addresses. A remote
|
||
server cannot use it to read this server's internal network.
|
||
|
||
Concurrent requests for the same remote media share one fetch, which runs
|
||
detached from the request that started it: a client that gives up does not
|
||
waste the download for the others, and what was fetched is kept. The
|
||
content is held to `media.max_upload_bytes`, like an upload.
|
||
|
||
Thumbnails of remote media come from the cached original when there is
|
||
one; otherwise the remote server is asked for a thumbnail of the
|
||
configured size that answers the request, and that is cached under that
|
||
size. The original is not downloaded just to make a thumbnail.
|
||
|
||
Another server is served this server's own media only. It is never made to
|
||
fetch from a third server on another's behalf.
|
||
|
||
With federation off, nothing is fetched: media that was cached before is
|
||
still served, and anything else from another server is `403 M_FORBIDDEN`
|
||
with a message saying federation is disabled.
|
||
|
||
## Errors
|
||
|
||
| | |
|
||
|---|---|
|
||
| `404 M_NOT_FOUND` | no such media, here or on its server |
|
||
| `413 M_TOO_LARGE` | larger than `media.max_upload_bytes`, uploaded or fetched |
|
||
| `504 M_NOT_YET_UPLOADED` | reserved, not uploaded within `timeout_ms` |
|
||
| `409 M_CANNOT_OVERWRITE_MEDIA` | a reserved URI that already has content |
|
||
| `403 M_FORBIDDEN` | someone else's reservation; or remote media with federation off |
|
||
| `429 M_LIMIT_EXCEEDED` | too many reservations |
|
||
| `400 M_UNKNOWN` | cannot make a thumbnail of this |
|
||
| `502 M_UNKNOWN` | the media's server failed to provide it |
|
||
|
||
## Known gaps
|
||
|
||
- **URL previews** (`/media/v3/preview_url`) are not implemented and answer
|
||
`M_UNRECOGNIZED`. Fetching arbitrary URLs for clients needs the same
|
||
network guard as media redirects plus an HTML parser, and is its own
|
||
piece of work.
|
||
- **No deletion or quotas yet.** Nothing removes media: no admin purge, no
|
||
retention for the remote cache, no per-user quota. Those are the media
|
||
parts of Phase 9's admin work, and content-addressed storage means a
|
||
purge has to check that no other row still uses a blob before deleting
|
||
it.
|
||
- **No quarantine**, which belongs with the same admin work.
|
||
- **No animated thumbnails**; `animated=true` gets a still image.
|
||
- **No S3** until the PocketBase driver brings its filesystem abstraction;
|
||
the `BlobProvider` seam is where it plugs in.
|