Phase 6: media
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
Implements PLAN.md Phase 6: the media repository. Clients upload files, reserve an mxc:// URI and upload to it later, download and ask for thumbnails, all through the authenticated media endpoints of Matrix 1.11; other servers download this server's media through the federation media endpoints, and media uploaded elsewhere is fetched over federation on first use and kept. internal/mediaapi is the component. Content goes into a BlobStore named by its SHA-256, so the same file uploaded twice is stored once and the hash doubles as the ETag; the metadata that maps an mxc:// URI onto a blob is in media_repository and media_thumbnails. The local-filesystem store writes to a temporary file and renames it into place once hashed, so nothing ever refers to half a file. A storage driver can bring its own store through storage.BlobProvider, which is where PocketBase's filesystem abstraction (local or S3) plugs in with Phase 10. Reserved uploads belong to the user who made them, take one upload, lapse after a day, and are capped per user; a download of one waits for its content, woken in memory the moment it arrives, for as long as timeout_ms allows. Thumbnails come from a fixed set of configured sizes, so a thousand requested sizes cannot mean a thousand resizes. They are made on first request and kept, never enlarge, honour a JPEG's EXIF orientation, and point at the original when it already fits. The pixel count is checked from the header before anything is decoded, at most four resizes run at once, and concurrent requests for the same thumbnail share one. Remote media is fetched through the signed client: the multipart/mixed answer of Matrix 1.11, with a fallback to the legacy download endpoint for servers that answer M_UNRECOGNIZED, and Location redirects followed by a dialer that refuses loopback, private and link-local addresses. Concurrent requests share one fetch, detached from the request that started it. With federation off nothing is fetched, and what was cached before is still served. Downloads carry a sandboxing CSP, nosniff, inline Content-Disposition only for the types the spec lists as safe, and private immutable caching. The unauthenticated legacy download endpoints are off by default behind media.legacy_unauthenticated_downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6aaf554336
commit
92836f8dcd
40 files changed
+4729
-63
No files matched your search
@@ -0,0 +1,56 @@
|
||||
-- Media: the metadata of content uploaded here and of content fetched from
|
||||
-- other servers and kept, and of the thumbnails made from either.
|
||||
--
|
||||
-- The content itself is not in the database. It is in the blob store —
|
||||
-- files under media.base_path by default — named by its SHA-256, so the
|
||||
-- same bytes uploaded twice are stored once and a row here only says which
|
||||
-- blob an mxc:// URI refers to.
|
||||
--
|
||||
-- Timestamps are milliseconds since the Unix epoch, as everywhere else in
|
||||
-- Warp.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS media_repository (
|
||||
-- The server name of the mxc:// URI: this server for an upload,
|
||||
-- another for cached remote media.
|
||||
media_origin TEXT NOT NULL,
|
||||
media_id TEXT NOT NULL,
|
||||
-- As uploaded, parameters included: "text/plain; charset=utf-8".
|
||||
content_type TEXT NOT NULL DEFAULT '',
|
||||
upload_name TEXT NOT NULL DEFAULT '',
|
||||
file_size BIGINT NOT NULL DEFAULT 0,
|
||||
-- Lower-case hex SHA-256 of the content; '' while pending.
|
||||
sha256 TEXT NOT NULL DEFAULT '',
|
||||
-- The uploader; '' for remote media.
|
||||
user_id TEXT NOT NULL DEFAULT '',
|
||||
created_at_ms BIGINT NOT NULL,
|
||||
-- An ID handed out by POST /_matrix/media/v1/create whose content has
|
||||
-- not been uploaded yet. It is deleted unused at expires_at_ms.
|
||||
pending BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
expires_at_ms BIGINT NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (media_origin, media_id)
|
||||
);
|
||||
|
||||
-- Counting a user's pending uploads, and sweeping the expired ones.
|
||||
CREATE INDEX IF NOT EXISTS media_repository_pending_idx
|
||||
ON media_repository (pending, user_id, expires_at_ms);
|
||||
|
||||
-- Thumbnails, one per configured size that has been asked for.
|
||||
--
|
||||
-- There is no foreign key to media_repository: the thumbnail of remote
|
||||
-- media is fetched from its server on its own, and the original it was
|
||||
-- made from may never have been fetched here at all.
|
||||
CREATE TABLE IF NOT EXISTS media_thumbnails (
|
||||
media_origin TEXT NOT NULL,
|
||||
media_id TEXT NOT NULL,
|
||||
-- The configured size the thumbnail answers, not the size of the
|
||||
-- image: a "scale" thumbnail keeps the original's aspect ratio.
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
-- 'crop' or 'scale'.
|
||||
method TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
file_size BIGINT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
created_at_ms BIGINT NOT NULL,
|
||||
PRIMARY KEY (media_origin, media_id, width, height, method)
|
||||
);
|
||||
Reference in new issue
Block a user