tajniak81andClaude Opus 5.5
fc49f56339
Implements PLAN.md Phase 7: the server half of end-to-end encryption.
Devices upload identity, one-time and fallback keys; other devices query
and claim them, across servers too, and every one-time key is handed out
exactly once. Cross-signing, signature uploads, device list tracking and
server-side key backup complete the set.
internal/keyserver is the component. Keys are stored in canonical JSON
so that re-uploading the same key is a byte comparison, and signatures
uploaded later are kept apart from the key they sign and merged in on
read, so a key's own JSON stays what its owner uploaded. Device keys must
be for the caller's own device; cross-signing keys must be the
uploader's, carry the right usage and, for self- and user-signing keys,
be signed by the master key. Signatures are verified against the key as
stored. Replacing a master key asks for the password; a first upload or
an upload of the keys already stored does not, as the spec allows. A
signature of another user's master key is shown only to the user who
made it and never federated.
Every write that changes what /keys/query answers moves the device list
stream in the same transaction, one position per device, and wakes the
syncs of everyone sharing a room. The stream is the sixth field of a sync
token; older tokens still parse. The syncapi works out device_lists
changed and left from the stream and from membership moves in the
window, skipping invites and knocks, and reports
device_one_time_keys_count and device_unused_fallback_key_types, the
latter as [] when empty. GET /keys/changes uses the same logic.
Over federation: /user/keys/query, /user/keys/claim and
/user/devices/{userId}, and m.device_list_update and m.signing_key_update
both ways. Remote keys are not cached; an update about a remote user
sharing a room moves them in the stream so local clients re-query.
Device display names are not sent to other servers. With federation off,
remote users are reported in failures at once.
Deleting a device deletes its keys, through a new userapi
DeviceObserver. Key backup keeps the better copy of each session by the
spec's rule, never reuses a deleted version's number, and answers writes
to a stale version with M_WRONG_ROOM_KEYS_VERSION.
Storage is migration 0008 for sqlite and postgres, with a PocketBase
stub. The Complement target sets gain the key, device list, fallback,
cross-signing and backup tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
21 KiB
Federation
PLAN.md Phase 5: the Server-Server API. Warp can now join rooms on other servers, have their users join its rooms, exchange events and typing, receipts and to-device messages with them, and do all of it behind a switch that turns it off at runtime — cleanly, with nothing left talking to another server — and back on again, picking up where it left off.
Federation is still off in a fresh install. Nothing here changes what a private homeserver does.
Where things live
| Package | What |
|---|---|
internal/federation |
The protocol, with no knowledge of rooms: X-Matrix request signing, server discovery, the signed HTTP client, the key ring, the gate and the allow/deny policy. |
internal/roomserver |
Rooms across servers: taking in events from elsewhere (federated.go), joining, leaving and inviting across servers in both directions (federated_membership.go), answering other servers' reads (federated_query.go), remote aliases and directories, server ACLs (acl.go). |
internal/federationapi |
Federation as a running component: the inbound endpoints, the outbound queue, the EDU adapter and the Module the switch starts and stops. |
The split follows the dependencies. The roomserver uses the protocol — it
fetches events and asks for joins — and the component uses the roomserver,
so the component cannot live in the protocol package. PLAN.md calls it
federation.Module; it is federationapi.Module, for that reason.
The switch
Three things answer to federation.enabled, and each reads it at the
moment it matters rather than being told:
- The gate. Every route under
/_matrix/federation,/_matrix/keyand/.well-known/matrix/serveris wrapped by it, and answers404 M_UNRECOGNIZEDwhile the switch is off — a server with federation off looks exactly like a server without a federation API. - The client. Every outbound request checks the switch, the allow and deny lists, and that it is not addressed to this server, before it resolves anything or opens a connection. A component that somehow missed the toggle still cannot reach another server.
- The module. The one thing that talks to other servers on its own initiative is the outbound queue, and the module starts and stops it. The settings service calls the module when the switch moves; stopping cancels in-flight sends and waits for them, and leaves whatever was not sent in the database.
Events persisted while the switch is off are not queued. Holding them would
turn switching federation back on after a month into a month of events
delivered in one burst; a remote server that cares about a room fetches
what it missed with /get_missing_events the next time it hears from this
one. What was queued before the switch went off goes out when it comes
back on.
Finding other servers
federation.Resolver implements the spec's discovery algorithm, in its
order: an IP literal or an explicit port is used as is; otherwise
/.well-known/matrix/server may delegate to another name, which is then
resolved the same way; then the _matrix-fed._tcp SRV record, then the
deprecated _matrix._tcp, then port 8448.
The result has two halves and the client keeps them apart. The address to
dial can be an SRV target; the name the TLS certificate must be valid for —
also the Host header — is never the SRV target, only the server name or
the name the well-known document delegated to. The client carries the
dial addresses down to its dialer in the request context and builds the URL
from the other half, so Go's TLS verification checks the right name
without any custom verification code.
Well-known answers are cached for their Cache-Control max-age, between
the spec's bounds (24 hours by default, 48 at most); failures for an hour,
so that a server with no document is not asked for one on every request.
Keys
GET /_matrix/key/v2/server serves this server's key, valid for
federation.key_validity_period (24 hours by default).
Everybody else's keys go through the key ring. A key response is accepted
only if it is about the server asked for and every signature the server
made on it is by a key it lists and verifies — a response citing a key it
does not publish is broken or forged. Keys are stored in
federation_server_keys and cached in memory, and are believed for as long
as the server said, but never for more than a week: the spec's cap.
When a server's own key endpoint cannot be reached, the notaries in
federation.trusted_key_servers are asked, in order; a notary's answer
must carry the notary's signature as well. The default is no notaries:
keys come from the server that owns them or not at all.
Room versions 5 and later require a key to have been valid when the event was sent. A cached key that is not known to cover an event's timestamp is fetched again before the event is refused — the server may have extended it — and a fetch that fails is not retried for a minute, so that a server whose key endpoint is down is not asked on every event it sends.
As a notary, Warp answers for its own keys only. It keeps other servers' keys, not the signed documents they came in, and a notary has to hand out the documents.
Authenticating requests
Every federation endpoint but the key and version endpoints checks the
X-Matrix Authorization header: the signature covers the method, the
request target exactly as it appeared on the request line, the origin, the
destination and the JSON body. The origin is then checked against the
allow and deny lists, and — for anything about a room — against the room's
server ACL.
The parser is fuzzed (FuzzParseXMatrix): a header that parses must
survive being written back out and parsed again unchanged.
Taking in events from other servers
An event created here is built on the room's current state and can only
fail the authorisation rules. An event from somewhere else arrives with
none of that settled. The roomserver follows the spec's procedure, in
federated.go:
- It is what it says it is. It parses under the room's version, it is signed by its sender's server, and its content matches its hash. A hash mismatch with a good signature means somebody redacted it on the way; the redacted form is kept rather than the event rejected.
- It passes the rules against its own auth events. Missing auth
events are fetched with
/event_authand checked themselves, oldest first. - It passes them against the state before it — the state after each
of its prev_events, resolved together. A prev_event this server does not
have is asked for with
/get_missing_events(ten events, between this server's forward extremities and the new event). A gap wider than that is closed by asking for the state at the missing event with/state_ids, fetching whatever of it is missing — one at a time with/event, or all at once with/statepast fifty — and checking all of it. - It passes them against the current state. An event that fails only here is soft-failed: kept, because other events may cite it, but never shown to a client and never a forward extremity. This is what stops a banned user's server from sending events built on the state before the ban.
An event that fails 1 to 3 is rejected and not stored. The /send
response says why, per event.
Outliers
Some events have to be held without the room's state at them: the state
and auth chain a join is handed, auth events fetched to judge something
else, the state fetched across a gap, an invite to a room nobody here is
in, a soft-failed event. These are outliers — a column on
roomserver_events rather than a table of their own, because every read
of a room's current state joins the events table, and the current state of
a room joined over federation is almost all outliers.
An outlier can be cited and read by ID. It has no state snapshot, is never
a forward extremity, and never appears in /sync, /messages or
/context — the timeline queries filter it out. An outlier that arrives
again as an ordinary event stays an outlier: this server used it as state
without its history, and putting the room's head on it now could put the
head behind events that are already here.
The room's head
A Phase 3 room had one forward extremity, because every event was a child of the one before. Federation forks rooms: two servers send at the same moment, each without the other's event. The new extremities are the old ones the event does not descend from, plus the event.
The current state then has to be worked out again. In the common case —
the event follows the room's single extremity — it is one tuple more than
before. Otherwise it is resolved across the new extremities and written
whole. The next event created here cites every extremity, and the fork
closes; TestFederatedJoinAndConversation checks that both servers end up
with one extremity and the same state.
History: backfill
A room joined over federation starts, on this server, at the join: the
join is handed the room's state, not the events that led to it. When a
client pages /messages backwards and the page comes back short at the
bottom of what this server holds, the roomserver asks the servers in the
room for what came before it (GET /backfill) before answering, in
backfill.go:
- Where to start. The prev_events of the room's oldest timeline events (the oldest fifty) that are not themselves in the timeline. A room whose oldest event is its create event has nothing to ask for.
- Whom to ask. The servers of the room's members, joined ones first, in turn, until one sends something that passes. A page asks at most three times before it is returned short.
- What is checked. Steps 1 to 3 of the intake above. The state before
each event comes from the events before it in the same batch, or —
for the oldest ones — from
/state_idsat its prev_events. Step 4 does not apply: soft failure refuses events built on a state the room has moved past, which history always is. An event already held as an outlier is promoted into the timeline. - Where it goes. Below everything. The stream ordering counter only
goes up, so backfilled events are numbered by a second counter that runs
downwards from -1 (
roomserver_sequences.backfill_ordering). Each batch sorts below the one before it, and nothing about the room's head, its current state or its stream top changes: a live/syncreads "after my token" and never reaches below zero, so history reaches clients only through/messages,/contextand the state at a point in it. Zero is never a position, so it can go on meaning "unbounded" in every window; the position just below the event at 1 is -1 (storage.PositionBefore).
Backfill is only asked for when the user may see before their join — a
room whose history visibility is joined or invited has nothing to
show them there — and only by /messages, never by /sync or /context,
which do not wait on another server. With federation off, or with nobody
reachable, the page is what this server holds.
Membership across servers
A server can only sign as itself, and a room's state lives with the servers in it, so each membership change that crosses a server boundary is a short protocol:
- Joining a room this server is not in — not held at all, held only as
an invite, or held from before every local user left — asks a resident
server for a template (
make_join), fills it in, signs it, and hands it back (send_join, v2). The answer is the room's state and auth chain, every event of which is verified and checked against its own auth events before any of it is believed. Candidates are tried in order: the client'sserver_namehints, the server in the room ID (before version 12), the servers of the room's members as last known, and whoever sent the invite. A restricted join comes back countersigned by the server that authorised it. - Leaving a room this server is not in — rejecting an invite, in
practice — is the same with
make_leaveandsend_leave. When no server in the room can be reached the invite is still rejected locally: a leave is built and stored here and nowhere else. An invite a user cannot get rid of because its server is gone is the worse failure. - Inviting a user of another server goes through the ordinary
pipeline, with one extra stage under the room lock: the invite is sent
to the invitee's server to countersign (
/invite, v2), and the copy with both signatures is the one persisted.
And the other side of each:
make_joinbuilds the template from the room's current state and runs the rules over a trial build first, so that a banned user hears so before signing anything. Under the restricted join rule it names a local user to authorise the join, andsend_joincountersigns it.send_joinandsend_leavetake the event through the same intake as any other, and mark it to be relayed: the joining server is not in the room yet and relies on this one to tell everybody else./invitecountersigns an invite to a local user. A room this server is in takes the invite in as an ordinary event. One it is not in gets it as an out-of-band membership — an outlier that is nonetheless the invitee's membership in the room's current state, which is all/syncneeds — and the stripped state the inviter sent along is kept inroomserver_invite_statefor the invite list, until the invite is answered.
The outbound queue
federationapi.Queue sends this server's events and EDUs to everybody who
should have them.
The roomserver's emit stage calls it with the room lock held, so that call
only appends to an in-memory list. A dispatcher drains the list, works out
each event's destinations — the servers with a user joined to the room,
plus, for a membership event, the server of the user it is about, so that
a server whose last user was just kicked hears of it — and writes one row
per destination to federation_queue. An event received from another
server is not sent on, because its origin sends it to everybody itself;
the exception is a relayed join or leave, which goes to everybody but its
origin.
A worker per destination sends the rows in transactions of at most 50 PDUs
and 100 EDUs, and deletes them once the destination has answered. A
failure backs the destination off — ten seconds, doubling to six hours,
with a fifth of jitter — in federation_destinations, which the admin API
reports and can reset. A destination later denied by the allow and deny
lists has its backlog dropped rather than sent.
Rows are deleted only once delivered, so a restart loses nothing that had reached the database: the queue starts a worker for every destination with a backlog. Transaction IDs carry the queue's start time, because a receiving server deduplicates by transaction ID and an ID reused after a restart with different contents would be answered from its cache.
Received transactions are recorded with their response in
federation_inbound_txns, so that a retransmission is answered the same
way rather than processed twice.
EDUs
| EDU | Out | In |
|---|---|---|
m.typing |
a local user's typing, to the room's servers | into the typing tracker, if the user belongs to the sending server |
m.receipt |
a local user's public read receipts — never private ones | into the receipts stream, likewise |
m.direct_to_device |
to-device messages for remote users, one EDU per server | delivered to local devices |
m.device_list_update, m.signing_key_update |
a local user's key changes, to every server sharing a room with them | the remote user is marked changed for local syncs; see e2ee.md |
m.presence |
— | ignored: presence is off by default |
Nothing in an EDU is signed, so the one check there is — the user must belong to the server that sent it — is what keeps a server from making anybody appear to type. EDUs about a room are subject to its server ACL.
The sync API hands EDUs over through syncapi.Outbound, and only for what
a local user did: the same calls are how EDUs from other servers come in,
and passing those on would echo them back.
Server ACLs
m.room.server_acl is enforced against the origin of everything about a
room: its PDUs, its typing and receipts, its joins and leaves, and its
reads of the room's history. Globs match case-insensitively, without the
port; deny wins over allow; an empty allow list allows nobody; and
allow_ip_literals: false refuses servers named by an IP address. An ACL
never applies to the server enforcing it.
What other servers may read
/event, /state, /state_ids, /event_auth, /backfill and
/get_missing_events are answered for a server with a user joined to the
room, or for any server if the room is world-readable — and only if the
room's ACL allows it. Anybody else gets 403 whether or not the event
exists.
/state_ids and /state answer with the state before the event asked
about, which is what every implementation's client side expects; the
asker adds the event itself if it is a state event. An outlier has no
snapshot, and asking for the state at one is a 404.
Endpoints
GET /_matrix/key/v2/server · POST /_matrix/key/v2/query · GET /_matrix/key/v2/query/{server} |
keys; the notary answers for this server only |
GET /_matrix/federation/v1/version |
|
PUT /v1/send/{txnId} |
transactions of PDUs and EDUs |
GET /v1/make_join/{room}/{user} · PUT /v1/send_join, /v2/send_join |
joins |
GET /v1/make_leave/{room}/{user} · PUT /v1/send_leave, /v2/send_leave |
leaves |
PUT /v2/invite/{room}/{event} |
invites |
GET /v1/event/{event} · /v1/state/{room} · /v1/state_ids/{room} · /v1/event_auth/{room}/{event} |
reads |
GET /v1/backfill/{room} · POST /v1/get_missing_events/{room} |
history |
GET /v1/query/directory · /v1/query/profile |
queries |
GET, POST /v1/publicRooms |
the public room directory |
Client-Server endpoints that now reach other servers: /join and
/rooms/{id}/join for a room this server is not in (by ID or by remote
alias), /invite for a remote user, /leave to reject a remote invite,
GET /directory/room/{alias} for a remote alias, /publicRooms?server=,
and GET /profile/{user} for a remote user.
Known gaps
- Backfill only fills in history below the room. A gap higher up — a
live event whose prev_events were closed with
/state_idsrather than fetched — stays a gap. And the end of a server's history is not remembered: a room whose servers cannot supply anything before its oldest event is asked again each time a client scrolls to the top. - Backfilled redactions are not applied to earlier backfilled events;
a server sends a redacted event in its redacted form anyway, but
unsigned.redacted_becauseis not filled in. - No partial-state joins (MSC3706). Joining a room of tens of thousands of members means receiving, verifying and checking its whole state before the join completes, which takes minutes.
- No key rotation. The key file holds one key; old keys are not
served as
old_verify_keys, so rotating the key would invalidate this server's old events for anybody who has not cached the old key. - Rejected events are not remembered. A rejected event is not stored, so if it is sent again it is fetched and judged again.
- Soft-failed events are never re-evaluated, and a redaction whose target arrives later is not applied when the target arrives.
- Room versions 1 and 2 — event IDs carried in the event — are not supported, so the oldest rooms on the network cannot be joined.
- Presence EDUs are ignored on the way in and not sent (presence is off by default).
- Third-party invites across servers (
exchange_third_party_invite) and knocking over federation (make_knock,send_knock) are not implemented.
Testing
| Where | What |
|---|---|
internal/federation |
X-Matrix parsing (fuzzed) and signing, discovery against fake DNS and a fake well-known server, the key ring against a fake key server, an authenticated request end to end over TLS, the client refusing while disabled or denied. |
internal/roomserver/federated_test.go |
Two and three roomservers in one process, wired to each other's federation methods: joins at versions 10–12, conversation and fork merging, fail-over, refused joins, invites with countersignatures, rejection with and without the inviter reachable, missing events, a gap closed with state, forged and soft-failed events, ACLs, remote aliases and directories, the switch. backfill_test.go: a server that joined late scrolling back through the whole room at versions 10–12, the state inside the history, backfill with the switch off and under joined visibility. |
internal/federationapi |
The queue: fan-out and relay rules, backoff and retry, transactions within the limits, a restart with a backlog, the policy, EDUs. |
internal/setup/phase5_test.go |
Two complete servers over HTTPS: a join by remote alias, messages both ways through the queues, a remote profile, federation switched off with a message held and delivered after it is switched back on, the admin view of it; the TLS listener and the module following the switch. |
| Complement | make complement-federation — see server/test/complement. |