tajniak81andClaude Opus 5.5
487e0f410b
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
Implements PLAN.md Phase 10. Simplified Sliding Sync (MSC4186) is served at the unstable org.matrix.simplified_msc3575 endpoint and advertised in /versions. Lists are sorted by activity and filtered by DM, invite, encryption, room type, space and tag; room subscriptions, required_state with wildcards, $ME and $LAZY, and the to_device, e2ee, account_data, receipts and typing extensions are supported. Per-connection state is kept in memory and an unknown pos answers M_UNKNOWN_POS. Threads and relations: m.relates_to is indexed on roomserver_events (migration 0012), /relations (with recursion) and /threads are served, and every event a timeline read returns carries its m.thread, m.replace and m.reference summaries in unsigned.m.relations. Notifications carry a thread ID (migration 0013): threaded receipts clear only their thread, and sync reports unread_thread_notifications when the filter asks. Spaces: /hierarchy walks a space breadth first, asking other servers about rooms this one is not in through the new federation hierarchy endpoint; room_summary describes one room. /search runs on FTS5 (sqlite, pocketbase) or tsvector (postgres), bounded to what each user may see, with context, profiles, state, grouping and paging. The PocketBase driver is real: the sqlite schema and store on an embedded PocketBase app's database, with media in its filesystem, local or S3 (database.pocketbase.s3). It runs the full storage test suite. Ops: OpenTelemetry tracing over OTLP/HTTP (tracing.*), with server spans per route, component spans, and trace propagation on federation, push and application service calls. New metrics for persisted events, waiting syncs, sliding sync connections and the database pool, with a Grafana dashboard. deploy/ adds a Compose setup (PostgreSQL, Prometheus, Grafana, Jaeger) and a Helm chart; cmd/warp-loadtest loads a server and reports request and delivery latencies. The load tester found a long-standing bug, fixed here: on a server where no account data had been written, every incremental /sync re-sent the push rules and returned at once, and an all-zero token was taken for an initial sync, so long polls became busy loops. The PostgreSQL migrations and search dialect compile but were not run against a live PostgreSQL in this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
244 lines
12 KiB
Markdown
244 lines
12 KiB
Markdown
# Media
|
||
|
||
PLAN.md Phase 6: the media repository. Clients upload files and get an
|
||
`mxc://` URI back to put in an event; anybody in the room downloads them,
|
||
or asks for a thumbnail; and media uploaded to another server is fetched
|
||
from it, over federation, the first time somebody here wants it.
|
||
|
||
## Where things live
|
||
|
||
| Package | What |
|
||
|---|---|
|
||
| `internal/mediaapi` | The component: uploads, reserved uploads, downloads, thumbnails, the cache of other servers' media, and the HTTP helpers both APIs share — headers, parameters, error mapping. |
|
||
| `internal/clientapi/media.go` | The Client-Server endpoints. |
|
||
| `internal/federationapi/media.go` | The two endpoints other servers download through. |
|
||
| `internal/federation/media.go` | Downloading from other servers: the multipart answer, redirects, the fallback for servers older than Matrix 1.11. |
|
||
| `internal/storage` | `MediaStore`, the metadata (`media_repository`, `media_thumbnails`), and `BlobStore`, the content. |
|
||
| `internal/storage/filestore` | The local-filesystem `BlobStore`. |
|
||
|
||
## Metadata and content
|
||
|
||
The database holds what an `mxc://` URI means — who uploaded it, its
|
||
content type and file name, its size — and the SHA-256 of its content. The
|
||
content itself is in a `BlobStore`, named by that hash:
|
||
|
||
```
|
||
media_store/
|
||
ab/cd/abcd0123… one file per distinct content
|
||
tmp/ uploads in progress
|
||
```
|
||
|
||
Naming content by its hash gives three things at once. The same file
|
||
uploaded twice — a meme going round, the same avatar set on two accounts —
|
||
is stored once. A thumbnail that turns out to be the original (an image
|
||
already smaller than the size asked for) is a second row pointing at the
|
||
same file rather than a copy. And the hash is the `ETag`, so a client that
|
||
has the file already gets `304 Not Modified`.
|
||
|
||
An upload is written to `tmp/`, hashed as it is written, and renamed into
|
||
place only when it is complete: a reader never sees half a file, and a
|
||
crash leaves a temporary file and nothing else. Nothing in the database
|
||
refers to content until it is all there.
|
||
|
||
The blob store is chosen by the storage driver. SQLite and PostgreSQL use
|
||
the local filesystem under `media.base_path`. A driver that brings its own
|
||
file storage implements `storage.BlobProvider`, and PocketBase does: media
|
||
goes into PocketBase's filesystem, under `warp_media/` in
|
||
`<data directory>/storage`, or in an S3-compatible bucket when
|
||
`database.pocketbase.s3` is on ([configuration.md](configuration.md)).
|
||
PocketBase's filesystem is written a whole file at a time, so an upload is
|
||
spooled to `<data directory>/.warp_tmp` and hashed on the way, then handed
|
||
over under its hash; `media.base_path` is not used by that driver.
|
||
|
||
## Endpoints
|
||
|
||
| | |
|
||
|---|---|
|
||
| `POST /_matrix/media/v3/upload` | upload; `?filename=` names it |
|
||
| `POST /_matrix/media/v1/create` | reserve an `mxc://` URI to upload to later |
|
||
| `PUT /_matrix/media/v3/upload/{server}/{id}` | upload to a reserved URI |
|
||
| `GET /_matrix/client/v1/media/config` (and `/_matrix/media/v3/config`) | `m.upload.size` |
|
||
| `GET /_matrix/client/v1/media/download/{server}/{id}[/{fileName}]` | download; the file name in the path overrides the stored one |
|
||
| `GET /_matrix/client/v1/media/thumbnail/{server}/{id}` | thumbnail: `width`, `height`, `method`, `animated` |
|
||
| `GET /_matrix/federation/v1/media/download/{id}` | another server downloading this server's media |
|
||
| `GET /_matrix/federation/v1/media/thumbnail/{id}` | and its thumbnails |
|
||
|
||
Every download endpoint takes `?timeout_ms=`: how long to wait for the
|
||
content of a reserved URI that has not been uploaded yet (20 seconds by
|
||
default, two minutes at most). Past it the answer is `504
|
||
M_NOT_YET_UPLOADED`.
|
||
|
||
### Authenticated media only
|
||
|
||
Matrix 1.11 moved downloads under `/_matrix/client/v1/media`, behind an
|
||
access token, because the old endpoints let anybody who learned an
|
||
`mxc://` URI read the file — no account, no room membership. Warp serves
|
||
only the new ones by default. The old unauthenticated
|
||
`/_matrix/media/v3/download` and `/thumbnail` come back with
|
||
`media.legacy_unauthenticated_downloads: true`, for clients too old to know
|
||
better; with it they also honour `allow_remote=false`.
|
||
|
||
`GET /_matrix/client/versions` lists v1.11 and later, which is what tells a
|
||
client to use the authenticated endpoints. The access token may come in the
|
||
`Authorization` header or as `?access_token=`, which is what a web client
|
||
needs for an `<img src>`.
|
||
|
||
An access token proves an account, not membership: like every other server,
|
||
Warp lets any logged-in user download any media whose URI they know.
|
||
Tying media to the rooms it was sent in is MSC3911, not yet in the spec.
|
||
|
||
### What a download is served with
|
||
|
||
- **`Content-Disposition`**: `inline` only for the types the spec lists as
|
||
safe to show in a browser — images, audio, video, plain text, CSS, JSON —
|
||
and `attachment` for everything else. An uploaded HTML page is a
|
||
download, never a page on the homeserver's origin. The file name is
|
||
RFC 6266-encoded when it is not ASCII.
|
||
- **`Content-Security-Policy: sandbox; default-src 'none'; …`** so that
|
||
content a browser is tricked into rendering anyway runs no script.
|
||
- **`X-Content-Type-Options: nosniff`** so a `text/plain` upload is not
|
||
sniffed into HTML.
|
||
- **`Cross-Origin-Resource-Policy: cross-origin`** so a web client on
|
||
another origin can embed the images.
|
||
- **`Cache-Control: private, max-age=31536000, immutable`**: the content of
|
||
an `mxc://` URI never changes, but the endpoint needs a token, so no
|
||
shared cache may keep it.
|
||
- **`ETag`**, the content's hash, and `Range` support, from
|
||
`http.ServeContent`.
|
||
|
||
The uploaded `Content-Type` is kept as sent when it parses, and replaced by
|
||
`application/octet-stream` when it does not.
|
||
|
||
## Reserved uploads
|
||
|
||
`POST /media/v1/create` hands out an `mxc://` URI before there is anything
|
||
behind it, so a client can send the event that refers to a large file
|
||
while the file is still uploading. The reservation belongs to the user who
|
||
made it, lasts `media.pending_upload_expiry` (a day), and takes exactly one
|
||
upload: a second is `409 M_CANNOT_OVERWRITE_MEDIA`, an upload by anybody
|
||
else `403 M_FORBIDDEN`. A user may hold `media.max_pending_uploads` (ten)
|
||
at once; past that, `429 M_LIMIT_EXCEEDED`.
|
||
|
||
A download of a reserved URI waits for its content — in memory, woken by
|
||
the upload the moment it completes — for as long as its `timeout_ms`
|
||
allows. Lapsed reservations are swept whenever a new one is made, which is
|
||
the only time their number matters.
|
||
|
||
## Thumbnails
|
||
|
||
A client asks for a thumbnail of a size; the server answers with one of its
|
||
own sizes, `media.thumbnail_sizes`:
|
||
|
||
| width × height | method | for |
|
||
|---|---|---|
|
||
| 32 × 32 | crop | avatars in lists |
|
||
| 96 × 96 | crop | avatars |
|
||
| 320 × 240 | scale | images in the timeline |
|
||
| 640 × 480 | scale | |
|
||
| 800 × 600 | scale | |
|
||
|
||
The answer is the smallest size of the method asked for that is at least
|
||
as large in both dimensions, or the largest if none is. A method with no
|
||
configured sizes is answered from the other. Fixing the sizes is what makes
|
||
thumbnails cacheable at all: a client asking for 33 × 33 and another
|
||
asking for 34 × 34 share one, and nobody can make the server resize the
|
||
same image a thousand times by asking for a thousand sizes.
|
||
|
||
- **`scale`** fits the image inside the box and keeps its aspect ratio.
|
||
- **`crop`** fills the box exactly, cutting the longer side down equally
|
||
at both ends.
|
||
- Neither enlarges. An image smaller than the box is cropped at its own
|
||
size; one that already fits a `scale` box is served as it is — the
|
||
thumbnail row points at the original's content.
|
||
|
||
Thumbnails are made on first request and kept. JPEG, PNG, GIF and WebP are
|
||
decoded (WebP through `golang.org/x/image`); a JPEG thumbnail is a JPEG,
|
||
anything else a PNG, which keeps transparency. A JPEG's EXIF orientation is
|
||
honoured, so a phone's portrait photo is not a sideways thumbnail. GIFs are
|
||
always re-encoded, as a still image: Warp makes no animated thumbnails, and
|
||
`animated=true` gets a still one, which the spec allows.
|
||
|
||
What could hurt the server is bounded. The image header is read first, and
|
||
anything larger than `media.max_thumbnail_pixels` (32 megapixels) is
|
||
refused before a pixel is decoded — a 100 KiB PNG can claim to be 30000 ×
|
||
30000, and decoding costs four bytes a pixel whatever the file size. At
|
||
most four thumbnails are made at once, and concurrent requests for the same
|
||
one share one resize.
|
||
|
||
Content that cannot be thumbnailed — a PDF, a video, an image that does not
|
||
decode — is `400 M_UNKNOWN`.
|
||
|
||
## Other servers' media
|
||
|
||
Media whose `mxc://` server is another server is fetched from it on first
|
||
request and kept in the same tables and blob store as local media, under
|
||
its own server name. An `mxc://` URI names immutable content, so the copy
|
||
never goes stale.
|
||
|
||
Fetching uses the signed federation client:
|
||
|
||
1. `GET /_matrix/federation/v1/media/download/{id}`, whose answer is
|
||
`multipart/mixed`: a JSON part (an empty object, for now) and then
|
||
either the content or a part with a `Location` header saying where the
|
||
content is.
|
||
2. A server older than Matrix 1.11 answers that with `404 M_UNRECOGNIZED`,
|
||
and is then asked through the deprecated
|
||
`GET /_matrix/media/v3/download/{server}/{id}?allow_remote=false` —
|
||
what every server before 1.11 served media to other servers through.
|
||
A plain `404 M_NOT_FOUND` is not retried: that server knows the
|
||
endpoint and lacks the media.
|
||
|
||
A `Location` is an ordinary web address, often a CDN, fetched without
|
||
signing. It is the one place another server can make this one fetch an
|
||
arbitrary URL and hand the answer to a client, so the dialer checks the
|
||
address it actually connects to — after DNS, on every redirect — and
|
||
refuses loopback, private, link-local and multicast addresses. A remote
|
||
server cannot use it to read this server's internal network.
|
||
|
||
Concurrent requests for the same remote media share one fetch, which runs
|
||
detached from the request that started it: a client that gives up does not
|
||
waste the download for the others, and what was fetched is kept. The
|
||
content is held to `media.max_upload_bytes`, like an upload.
|
||
|
||
Thumbnails of remote media come from the cached original when there is
|
||
one; otherwise the remote server is asked for a thumbnail of the
|
||
configured size that answers the request, and that is cached under that
|
||
size. The original is not downloaded just to make a thumbnail.
|
||
|
||
Another server is served this server's own media only. It is never made to
|
||
fetch from a third server on another's behalf.
|
||
|
||
With federation off, nothing is fetched: media that was cached before is
|
||
still served, and anything else from another server is `403 M_FORBIDDEN`
|
||
with a message saying federation is disabled.
|
||
|
||
## Errors
|
||
|
||
| | |
|
||
|---|---|
|
||
| `404 M_NOT_FOUND` | no such media, here or on its server |
|
||
| `413 M_TOO_LARGE` | larger than `media.max_upload_bytes`, uploaded or fetched; or over the uploader's quota |
|
||
| `504 M_NOT_YET_UPLOADED` | reserved, not uploaded within `timeout_ms` |
|
||
| `409 M_CANNOT_OVERWRITE_MEDIA` | a reserved URI that already has content |
|
||
| `403 M_FORBIDDEN` | someone else's reservation; or remote media with federation off |
|
||
| `429 M_LIMIT_EXCEEDED` | too many reservations |
|
||
| `400 M_UNKNOWN` | cannot make a thumbnail of this |
|
||
| `502 M_UNKNOWN` | the media's server failed to provide it |
|
||
|
||
## Known gaps
|
||
|
||
- **URL previews** (`/media/v3/preview_url`) are not implemented and answer
|
||
`M_UNRECOGNIZED`. Fetching arbitrary URLs for clients needs the same
|
||
network guard as media redirects plus an HTML parser, and is its own
|
||
piece of work.
|
||
- **No deletion yet.** Nothing removes media: no admin purge, and no
|
||
retention for the remote cache. Content-addressed storage means a purge
|
||
has to check that no other row still uses a blob before deleting it.
|
||
Per-user quotas are in (Phase 9): `media.max_user_storage_bytes`, and a
|
||
quota per user through the admin API. An upload over the quota is
|
||
`413 M_TOO_LARGE`. See [admin-api.md](admin-api.md).
|
||
- **No quarantine**, which belongs with the same admin work.
|
||
- **No animated thumbnails**; `animated=true` gets a still image.
|
||
- **S3 only on PocketBase.** The sqlite and postgres drivers keep media on
|
||
local disk; S3 comes with the pocketbase driver's filesystem.
|