tajniak81andClaude Opus 5.5
fc49f56339
Implements PLAN.md Phase 7: the server half of end-to-end encryption.
Devices upload identity, one-time and fallback keys; other devices query
and claim them, across servers too, and every one-time key is handed out
exactly once. Cross-signing, signature uploads, device list tracking and
server-side key backup complete the set.
internal/keyserver is the component. Keys are stored in canonical JSON
so that re-uploading the same key is a byte comparison, and signatures
uploaded later are kept apart from the key they sign and merged in on
read, so a key's own JSON stays what its owner uploaded. Device keys must
be for the caller's own device; cross-signing keys must be the
uploader's, carry the right usage and, for self- and user-signing keys,
be signed by the master key. Signatures are verified against the key as
stored. Replacing a master key asks for the password; a first upload or
an upload of the keys already stored does not, as the spec allows. A
signature of another user's master key is shown only to the user who
made it and never federated.
Every write that changes what /keys/query answers moves the device list
stream in the same transaction, one position per device, and wakes the
syncs of everyone sharing a room. The stream is the sixth field of a sync
token; older tokens still parse. The syncapi works out device_lists
changed and left from the stream and from membership moves in the
window, skipping invites and knocks, and reports
device_one_time_keys_count and device_unused_fallback_key_types, the
latter as [] when empty. GET /keys/changes uses the same logic.
Over federation: /user/keys/query, /user/keys/claim and
/user/devices/{userId}, and m.device_list_update and m.signing_key_update
both ways. Remote keys are not cached; an update about a remote user
sharing a room moves them in the stream so local clients re-query.
Device display names are not sent to other servers. With federation off,
remote users are reported in failures at once.
Deleting a device deletes its keys, through a new userapi
DeviceObserver. Key backup keeps the better copy of each session by the
spec's rule, never reuses a deleted version's number, and answers writes
to a stale version with M_WRONG_ROOM_KEYS_VERSION.
Storage is migration 0008 for sqlite and postgres, with a PocketBase
stub. The Complement target sets gain the key, device list, fallback,
cross-signing and backup tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
295 lines
16 KiB
Markdown
295 lines
16 KiB
Markdown
# Warp
|
|
|
|
A [Matrix](https://spec.matrix.org) homeserver written from scratch in Go,
|
|
plus a family of first-party clients. Federation is optional and can be
|
|
switched on and off at runtime.
|
|
|
|
The full design and roadmap live in [PLAN.md](PLAN.md). This README covers
|
|
what exists today.
|
|
|
|
## Status
|
|
|
|
**Phase 7 — end-to-end encryption. Complete.** The server half of E2EE:
|
|
devices upload their identity, one-time and fallback keys, other devices
|
|
query and claim them — across servers too — and every one-time key is
|
|
handed out exactly once. Cross-signing keys are checked to be signed by
|
|
the master key, replacing a master key asks for the password, and
|
|
signatures between devices and users are verified against the keys as
|
|
stored; who a user has verified stays between them and their own devices.
|
|
Device list tracking is real: `/sync` reports whose keys changed and whom
|
|
a client no longer shares a room with, fed by a device list stream that
|
|
moves in the same transaction as the keys and travels between servers as
|
|
`m.device_list_update` and `m.signing_key_update`. Deleting a device takes
|
|
its keys with it. Server-side key backup keeps the better copy of each
|
|
room key by the spec's rule. Known gaps — no cache of other servers' keys,
|
|
no dehydrated devices — are listed in [docs/e2ee.md](docs/e2ee.md).
|
|
|
|
**Phase 6 — media. Complete.** Clients can upload files, reserve an
|
|
`mxc://` URI and upload to it later, download and ask for thumbnails, all
|
|
through the authenticated media endpoints of Matrix 1.11. Content is
|
|
stored once per distinct file, named by its SHA-256, on the local
|
|
filesystem; thumbnails are made on first request from a fixed set of
|
|
sizes, the right way up for a phone's photos, with the pixel count checked
|
|
before anything is decoded. Media from other servers is fetched over
|
|
federation on first use — with the fallback older servers need, and
|
|
redirects that cannot be pointed into a private network — and cached, so
|
|
it is still there with federation switched off. Every download carries the
|
|
headers that keep an uploaded file from running as a page on the
|
|
homeserver's origin. Known gaps — no URL previews, no deletion or quotas
|
|
until Phase 9 — are listed in [docs/media.md](docs/media.md).
|
|
|
|
**Phase 5 — federation. Complete.** Warp talks to other homeservers. It
|
|
joins rooms on other servers and lets their users join its rooms, invites
|
|
across servers with the invitee's server countersigning, rejects remote
|
|
invites (locally, if the inviter is gone), and exchanges events, typing,
|
|
read receipts and to-device messages with them. Events from elsewhere go
|
|
through the spec's full intake — signatures and hashes, the rules against
|
|
their own auth events, against the state before them and against the
|
|
current state, with missing history fetched and soft failure for what
|
|
fails only at the end — and forked rooms are resolved back together. The
|
|
outbound queue persists what it owes each server, retries with backoff,
|
|
and survives restarts. Server discovery, key fetching with an optional
|
|
notary, X-Matrix request signing, server ACLs, and a TLS listener for port
|
|
8448. The switch is real now: turning federation off stops the queue and
|
|
refuses every outbound request, turning it back on resumes from where it
|
|
stopped, and the admin API shows each server's queue and backoff. Known
|
|
gaps — no history backfill from other servers, no partial-state joins, no
|
|
key rotation — are listed in [docs/federation.md](docs/federation.md).
|
|
|
|
**Phase 4 — sync. Complete. Milestone M1.** Warp is now a homeserver a
|
|
client can be used against. `/sync` is real: stream tokens across five
|
|
independent streams, long polling that wakes the moment something concerns
|
|
the client rather than when its timeout expires, incremental deltas, gappy
|
|
timelines with a `prev_batch` to close them, filters, and lazy-loaded
|
|
members. `/messages` pages a room's history and `/context` opens a
|
|
permalink. The streams that are not room events are here too: typing,
|
|
read receipts and read markers, to-device messaging for Phase 7's
|
|
encryption to build on, and presence behind a config switch that is off by
|
|
default. A user who has left a room is now served the room as it was when
|
|
they left rather than refused, and `?at=` on `/members` is honoured. See
|
|
[docs/sync.md](docs/sync.md).
|
|
|
|
**Phase 3 — rooms, local only. Complete.** A client can create a room at any
|
|
supported version, invite, join, leave, kick, ban, send messages and state,
|
|
redact, name a room with an alias and find one in the public directory. Every
|
|
event goes through one pipeline — validate, authorise, resolve, persist,
|
|
emit — so a membership change made by `createRoom`, by `/invite` or by a raw
|
|
`PUT .../state/m.room.member` is judged by the same authorisation rules.
|
|
Nothing talks to another server yet, and every operation that would need to
|
|
is refused with a message that says so. See
|
|
[docs/rooms.md](docs/rooms.md).
|
|
|
|
**Phase 2 — the event core. Complete.** The library every later phase is
|
|
built on: canonical JSON, the event model with its hashes and identifiers,
|
|
the redaction algorithm, event signing, the room version registry, the
|
|
authorisation rules and state resolution v2. Pure library code under
|
|
`server/internal/matrix`, verified against the spec's own test vectors.
|
|
See [docs/event-core.md](docs/event-core.md).
|
|
|
|
**Phase 1 — accounts and sessions. Complete.** A client can register, log
|
|
in, manage its devices and set its profile.
|
|
|
|
Client-Server API:
|
|
|
|
| | |
|
|
|---|---|
|
|
| `POST /register`, `GET /register/available` | sign-up, with `m.login.dummy` and optional registration tokens |
|
|
| `GET`, `POST /login` · `POST /logout`, `/logout/all` · `POST /refresh` | sessions, with `m.login.password` and refresh tokens |
|
|
| `GET /account/whoami` · `GET /capabilities` | who am I, and what this build can do |
|
|
| `GET /devices`, `GET`/`PUT`/`DELETE /devices/{id}` · `POST /delete_devices` | device management; deletion re-asks for the password |
|
|
| `GET /profile/{user}` and its `displayname` / `avatar_url` sub-resources | profiles — reads are public, remote ones asked of their server, writes are your own |
|
|
| `GET`/`PUT /user/{user}/account_data/{type}` and the per-room variant | account data, stored and returned byte for byte |
|
|
| `POST /createRoom` | room versions 3 to 12, presets, `initial_state`, invites, aliases |
|
|
| `POST /join/{roomIdOrAlias}` · `POST /rooms/{id}/join`, `/leave` | joining by ID or alias — on another server too, with federation on — and leaving or rejecting an invite |
|
|
| `POST /rooms/{id}/invite`, `/kick`, `/ban`, `/unban` | membership, decided by the room's power levels; invites reach users of other servers |
|
|
| `PUT /rooms/{id}/send/{type}/{txn}` | messages, idempotent per transaction ID |
|
|
| `GET`/`PUT /rooms/{id}/state[/{type}[/{key}]]` | room state, read whole or one piece at a time |
|
|
| `PUT /rooms/{id}/redact/{event}/{txn}` | redactions, applied only when they take effect |
|
|
| `GET /rooms/{id}/event/{event}` · `/members` (with `?at=`) · `/joined_members` · `/aliases` | reading a room |
|
|
| `GET /joined_rooms` | the rooms you are in |
|
|
| `GET`/`PUT`/`DELETE /directory/room/{alias}` | the alias directory; remote aliases are asked of their server |
|
|
| `GET`/`POST /publicRooms` · `GET`/`PUT /directory/list/room/{id}` | the public room directory, ours or (`?server=`) another server's |
|
|
| `GET /sync` | the event stream: long-polled, incremental, filtered, with lazy-loaded members |
|
|
| `GET /rooms/{id}/messages` · `/context/{event}` | paging a room's history, and one event with what surrounds it |
|
|
| `POST /user/{user}/filter` · `GET /user/{user}/filter/{id}` | filters, stored and returned byte for byte |
|
|
| `PUT /rooms/{id}/typing/{user}` | typing notifications, kept in memory and expired |
|
|
| `POST /rooms/{id}/receipt/{type}/{event}` · `/read_markers` | read receipts, public and private, and the read marker |
|
|
| `PUT /sendToDevice/{type}/{txn}` | device-to-device messaging, which carries E2EE's session keys |
|
|
| `POST /keys/upload`, `/keys/query`, `/keys/claim` · `GET /keys/changes` | device, one-time and fallback keys, local and from other servers |
|
|
| `POST /keys/device_signing/upload`, `/keys/signatures/upload` | cross-signing, with the password re-asked to replace a master key |
|
|
| `/room_keys/version[/{v}]` · `/room_keys/keys[/{room}[/{session}]]` | server-side key backup; see [docs/e2ee.md](docs/e2ee.md) |
|
|
| `GET`/`PUT /presence/{user}/status` | presence — **only while `sync.presence` is on**, and it is off by default |
|
|
| `POST /_matrix/media/v3/upload` · `POST /_matrix/media/v1/create` · `PUT /_matrix/media/v3/upload/{server}/{id}` | uploads, directly or to a reserved URI |
|
|
| `GET /_matrix/client/v1/media/download/…` · `/thumbnail/…` · `/config` | downloads and thumbnails, local and from other servers; see [docs/media.md](docs/media.md) |
|
|
| `GET /_matrix/client/versions` · `/.well-known/matrix/client` | discovery |
|
|
| `/.well-known/matrix/server` | server discovery — **only while federation is on** |
|
|
| `/_matrix/federation/*` · `/_matrix/key/*` | the Server-Server API — **only while federation is on**; see [docs/federation.md](docs/federation.md) |
|
|
| `GET /healthz` · `GET /metrics` | liveness with a database ping, and Prometheus |
|
|
| anything else under `/_matrix/` | `404 M_UNRECOGNIZED`, as the spec requires |
|
|
|
|
Admin API under `/_warp/admin/v1`: the federation and registration
|
|
switches, per-server federation health, user listing and creation, the
|
|
room list and detail, purging a room, removing a user from one, and the
|
|
dashboard counters. See
|
|
[docs/admin-api.md](docs/admin-api.md).
|
|
|
|
Also in place: argon2id passwords, hashed access and refresh tokens,
|
|
per-endpoint rate limiting, YAML + environment configuration, structured
|
|
logging, graceful shutdown, the storage interfaces with a working SQLite and
|
|
PostgreSQL driver (and a PocketBase stub), the embedded migration runner and
|
|
the runtime settings service.
|
|
|
|
Complement, the official integration suite, is wired up for both
|
|
profiles. `make complement` runs the account, room and stream tests with
|
|
federation off — `TestRegistration`, `TestLogin`, `TestDeviceManagement`,
|
|
the Phase 3 room tests and the Phase 4 `TestSync*`, `TestMessages*`,
|
|
`TestContext*`, `TestFilter*`, `TestTyping*`, `TestReceipt*` and
|
|
`TestToDevice*`, the Phase 6 media tests and the Phase 7 key tests.
|
|
`make complement-federation` runs the Phase 5 Server-Server tests and the
|
|
device list tests with federation on, over TLS with Complement's CA.
|
|
|
|
## Quick start
|
|
|
|
Requires Go 1.26 or newer. No cgo, no C compiler, no database server: the
|
|
default SQLite driver is pure Go.
|
|
|
|
```sh
|
|
cp warp.yaml.example warp.yaml
|
|
# edit global.server_name
|
|
|
|
make build
|
|
./bin/warp generate-keys
|
|
./bin/warp create-account --username you --admin
|
|
./bin/warp serve
|
|
```
|
|
|
|
Then:
|
|
|
|
```sh
|
|
curl localhost:8008/healthz
|
|
curl localhost:8008/_matrix/client/versions
|
|
```
|
|
|
|
Registration is off in a fresh install, which is why the first account is
|
|
made from the command line. Turn it on from the admin API when you want
|
|
others to sign up.
|
|
|
|
Without `make`, the same from the `server` directory:
|
|
|
|
```sh
|
|
go build -o ../bin/warp ./cmd/warp
|
|
```
|
|
|
|
## Configuration
|
|
|
|
One YAML file, every field overridable from the environment. See
|
|
[warp.yaml.example](warp.yaml.example) — each setting lists its `WARP_*`
|
|
variable. With no config file at all the server runs on defaults plus the
|
|
environment, which is how the container image is meant to be used:
|
|
|
|
```sh
|
|
docker build -t warp .
|
|
docker run -p 8008:8008 -v warp-data:/data -e WARP_SERVER_NAME=warp.example warp
|
|
```
|
|
|
|
An unknown field in the config file is an error, not a warning: a typo in a
|
|
security-relevant setting should never be silently ignored.
|
|
|
|
## Commands
|
|
|
|
| Command | What it does |
|
|
|---|---|
|
|
| `warp serve` | run the homeserver |
|
|
| `warp migrate [--status]` | apply (or list) pending database migrations |
|
|
| `warp generate-keys [--output path] [--key-id id]` | create the ed25519 signing key |
|
|
| `warp create-account --username name [--admin]` | create a local account, straight in the database |
|
|
| `warp version` | version and compiled-in storage drivers |
|
|
|
|
`create-account` talks to the database rather than to a running server, so
|
|
the first account on a new install can be made before anything is listening
|
|
and without turning registration on. It prompts for the password twice;
|
|
`--password-file` and `--password` are there for scripts.
|
|
|
|
The signing key is written as a single line, the same format Synapse uses:
|
|
`ed25519 <key-id> <unpadded-base64-seed>`. `generate-keys` refuses to
|
|
overwrite an existing key — losing it would invalidate every event this
|
|
server has ever signed.
|
|
|
|
## Federation switch
|
|
|
|
Federation is off in a fresh install. While it is off, the federation
|
|
surface does not answer at all: `/_matrix/federation/*`, `/_matrix/key/*`
|
|
and `/.well-known/matrix/server` return `404 M_UNRECOGNIZED`, exactly as
|
|
they would on a server that has no federation API, and nothing outbound
|
|
runs.
|
|
|
|
The switch is stored in the `server_settings` table and cached in memory,
|
|
so flipping it — `PUT /_warp/admin/v1/federation {"enabled": true}` —
|
|
takes effect on the next request without a restart. Switching it on
|
|
starts the outbound queue, which picks up whatever it still owed other
|
|
servers; switching it off stops the queue mid-flight and makes the
|
|
federation client refuse every request before it opens a connection. Set
|
|
`federation.allow_runtime_toggle: false` to pin the switch to the config
|
|
file.
|
|
|
|
To federate, other servers have to be able to reach this one: either set
|
|
`global.well_known.server_address` to a reverse proxy that terminates TLS,
|
|
or let Warp terminate it with `listen.federation_address: ":8448"` and a
|
|
certificate. See [docs/configuration.md](docs/configuration.md) and
|
|
[docs/federation.md](docs/federation.md).
|
|
|
|
## Development
|
|
|
|
```sh
|
|
make test # unit tests (SQLite)
|
|
make lint # golangci-lint
|
|
make check # vet + test, what CI runs
|
|
make cover # coverage.html
|
|
make complement # the Complement C-S subset, federation off (needs Docker)
|
|
make complement-federation # the Complement federation subset
|
|
|
|
# the storage tests against PostgreSQL as well
|
|
WARP_TEST_POSTGRES_DSN='postgres://warp:warp@localhost/warp_test?sslmode=disable' make test-postgres
|
|
```
|
|
|
|
`make test-race` needs cgo and therefore a C toolchain; CI runs it on
|
|
Linux.
|
|
|
|
## Layout
|
|
|
|
```
|
|
server/ the homeserver (Go module)
|
|
cmd/warp/ the binary: serve, migrate, generate-keys, create-account, version
|
|
internal/
|
|
config/ YAML + env configuration
|
|
logging/ metrics/ slog and Prometheus setup
|
|
httputil/ Matrix error envelope, CORS, router, middleware
|
|
ratelimit/ per-endpoint token buckets
|
|
federation/ the Server-Server protocol: gate, X-Matrix, discovery, client, keys
|
|
federationapi/ federation endpoints, the outbound queue, the switchable Module (docs/federation.md)
|
|
settings/ runtime settings service (the switches)
|
|
userapi/ accounts, passwords, devices, tokens, account data
|
|
roomserver/ rooms: the input pipeline (docs/rooms.md)
|
|
mediaapi/ uploads, downloads, thumbnails, remote media (docs/media.md)
|
|
keyserver/ device keys, one-time keys, cross-signing, key backup (docs/e2ee.md)
|
|
clientapi/ the Client-Server API handlers
|
|
adminapi/ /_warp/admin/v1
|
|
setup/ wiring: storage, routes, graceful shutdown
|
|
matrix/ the event core (docs/event-core.md)
|
|
canonicaljson/ the canonical JSON encoding
|
|
signing/ ed25519 server keys, Signing JSON
|
|
id/ identifier grammars as they appear in events
|
|
roomversion/ the room version registry
|
|
event/ PDUs: hashes, event IDs, redaction, signing
|
|
auth/ the authorisation rules and power levels
|
|
stateres/ state resolution v2
|
|
storage/ interfaces + driver registry
|
|
migrations/ embedded migration runner
|
|
sqlstore/ shared database/sql implementation
|
|
filestore/ media content on the local filesystem
|
|
sqlite/ postgres/ pocketbase/
|
|
test/complement/ the Complement homeserver image and runner
|
|
apps/ clients: ui, admin, web, desktop, mobile (Phase 1+)
|
|
docs/
|
|
```
|