Warp/docs/federation.md

Federation

PLAN.md Phase 5: the Server-Server API. Warp can now join rooms on other servers, have their users join its rooms, exchange events and typing, receipts and to-device messages with them, and do all of it behind a switch that turns it off at runtime — cleanly, with nothing left talking to another server — and back on again, picking up where it left off.

Federation is still off in a fresh install. Nothing here changes what a private homeserver does.

Where things live

Package	What
internal/federation	The protocol, with no knowledge of rooms: X-Matrix request signing, server discovery, the signed HTTP client, the key ring, the gate and the allow/deny policy.
internal/roomserver	Rooms across servers: taking in events from elsewhere (federated.go), joining, leaving and inviting across servers in both directions (federated_membership.go), answering other servers' reads (federated_query.go), remote aliases and directories, server ACLs (acl.go).
internal/federationapi	Federation as a running component: the inbound endpoints, the outbound queue, the EDU adapter and the Module the switch starts and stops.

The split follows the dependencies. The roomserver uses the protocol — it fetches events and asks for joins — and the component uses the roomserver, so the component cannot live in the protocol package. PLAN.md calls it federation.Module; it is federationapi.Module, for that reason.

The switch

Three things answer to federation.enabled, and each reads it at the moment it matters rather than being told:

• The gate. Every route under /_matrix/federation, /_matrix/key and /.well-known/matrix/server is wrapped by it, and answers 404 M_UNRECOGNIZED while the switch is off — a server with federation off looks exactly like a server without a federation API.
• The client. Every outbound request checks the switch, the allow and deny lists, and that it is not addressed to this server, before it resolves anything or opens a connection. A component that somehow missed the toggle still cannot reach another server.
• The module. The one thing that talks to other servers on its own initiative is the outbound queue, and the module starts and stops it. The settings service calls the module when the switch moves; stopping cancels in-flight sends and waits for them, and leaves whatever was not sent in the database.

Events persisted while the switch is off are not queued. Holding them would turn switching federation back on after a month into a month of events delivered in one burst; a remote server that cares about a room fetches what it missed with /get_missing_events the next time it hears from this one. What was queued before the switch went off goes out when it comes back on.

Finding other servers

federation.Resolver implements the spec's discovery algorithm, in its order: an IP literal or an explicit port is used as is; otherwise /.well-known/matrix/server may delegate to another name, which is then resolved the same way; then the _matrix-fed._tcp SRV record, then the deprecated _matrix._tcp, then port 8448.

The result has two halves and the client keeps them apart. The address to dial can be an SRV target; the name the TLS certificate must be valid for — also the Host header — is never the SRV target, only the server name or the name the well-known document delegated to. The client carries the dial addresses down to its dialer in the request context and builds the URL from the other half, so Go's TLS verification checks the right name without any custom verification code.

Well-known answers are cached for their Cache-Control max-age, between the spec's bounds (24 hours by default, 48 at most); failures for an hour, so that a server with no document is not asked for one on every request.

Keys

GET /_matrix/key/v2/server serves this server's key, valid for federation.key_validity_period (24 hours by default).

Everybody else's keys go through the key ring. A key response is accepted only if it is about the server asked for and every signature the server made on it is by a key it lists and verifies — a response citing a key it does not publish is broken or forged. Keys are stored in federation_server_keys and cached in memory, and are believed for as long as the server said, but never for more than a week: the spec's cap.

When a server's own key endpoint cannot be reached, the notaries in federation.trusted_key_servers are asked, in order; a notary's answer must carry the notary's signature as well. The default is no notaries: keys come from the server that owns them or not at all.

Room versions 5 and later require a key to have been valid when the event was sent. A cached key that is not known to cover an event's timestamp is fetched again before the event is refused — the server may have extended it — and a fetch that fails is not retried for a minute, so that a server whose key endpoint is down is not asked on every event it sends.

As a notary, Warp answers for its own keys only. It keeps other servers' keys, not the signed documents they came in, and a notary has to hand out the documents.

Authenticating requests

Every federation endpoint but the key and version endpoints checks the X-Matrix Authorization header: the signature covers the method, the request target exactly as it appeared on the request line, the origin, the destination and the JSON body. The origin is then checked against the allow and deny lists, and — for anything about a room — against the room's server ACL.

The parser is fuzzed (FuzzParseXMatrix): a header that parses must survive being written back out and parsed again unchanged.

Taking in events from other servers

An event created here is built on the room's current state and can only fail the authorisation rules. An event from somewhere else arrives with none of that settled. The roomserver follows the spec's procedure, in federated.go:

1. It is what it says it is. It parses under the room's version, it is signed by its sender's server, and its content matches its hash. A hash mismatch with a good signature means somebody redacted it on the way; the redacted form is kept rather than the event rejected.
2. It passes the rules against its own auth events. Missing auth events are fetched with /event_auth and checked themselves, oldest first.
3. It passes them against the state before it — the state after each of its prev_events, resolved together. A prev_event this server does not have is asked for with /get_missing_events (ten events, between this server's forward extremities and the new event). A gap wider than that is closed by asking for the state at the missing event with /state_ids, fetching whatever of it is missing — one at a time with /event, or all at once with /state past fifty — and checking all of it.
4. It passes them against the current state. An event that fails only here is soft-failed: kept, because other events may cite it, but never shown to a client and never a forward extremity. This is what stops a banned user's server from sending events built on the state before the ban.

An event that fails 1 to 3 is rejected and not stored. The /send response says why, per event.

Outliers

Some events have to be held without the room's state at them: the state and auth chain a join is handed, auth events fetched to judge something else, the state fetched across a gap, an invite to a room nobody here is in, a soft-failed event. These are outliers — a column on roomserver_events rather than a table of their own, because every read of a room's current state joins the events table, and the current state of a room joined over federation is almost all outliers.

An outlier can be cited and read by ID. It has no state snapshot, is never a forward extremity, and never appears in /sync, /messages or /context — the timeline queries filter it out. An outlier that arrives again as an ordinary event stays an outlier: this server used it as state without its history, and putting the room's head on it now could put the head behind events that are already here.

The room's head

A Phase 3 room had one forward extremity, because every event was a child of the one before. Federation forks rooms: two servers send at the same moment, each without the other's event. The new extremities are the old ones the event does not descend from, plus the event.

The current state then has to be worked out again. In the common case — the event follows the room's single extremity — it is one tuple more than before. Otherwise it is resolved across the new extremities and written whole. The next event created here cites every extremity, and the fork closes; TestFederatedJoinAndConversation checks that both servers end up with one extremity and the same state.

History: backfill

A room joined over federation starts, on this server, at the join: the join is handed the room's state, not the events that led to it. When a client pages /messages backwards and the page comes back short at the bottom of what this server holds, the roomserver asks the servers in the room for what came before it (GET /backfill) before answering, in backfill.go:

• Where to start. The prev_events of the room's oldest timeline events (the oldest fifty) that are not themselves in the timeline. A room whose oldest event is its create event has nothing to ask for.
• Whom to ask. The servers of the room's members, joined ones first, in turn, until one sends something that passes. A page asks at most three times before it is returned short.
• What is checked. Steps 1 to 3 of the intake above. The state before each event comes from the events before it in the same batch, or — for the oldest ones — from /state_ids at its prev_events. Step 4 does not apply: soft failure refuses events built on a state the room has moved past, which history always is. An event already held as an outlier is promoted into the timeline.
• Where it goes. Below everything. The stream ordering counter only goes up, so backfilled events are numbered by a second counter that runs downwards from -1 (roomserver_sequences.backfill_ordering). Each batch sorts below the one before it, and nothing about the room's head, its current state or its stream top changes: a live /sync reads "after my token" and never reaches below zero, so history reaches clients only through /messages, /context and the state at a point in it. Zero is never a position, so it can go on meaning "unbounded" in every window; the position just below the event at 1 is -1 (storage.PositionBefore).

Backfill is only asked for when the user may see before their join — a room whose history visibility is joined or invited has nothing to show them there — and only by /messages, never by /sync or /context, which do not wait on another server. With federation off, or with nobody reachable, the page is what this server holds.

Membership across servers

A server can only sign as itself, and a room's state lives with the servers in it, so each membership change that crosses a server boundary is a short protocol:

• Joining a room this server is not in — not held at all, held only as an invite, or held from before every local user left — asks a resident server for a template (make_join), fills it in, signs it, and hands it back (send_join, v2). The answer is the room's state and auth chain, every event of which is verified and checked against its own auth events before any of it is believed. Candidates are tried in order: the client's server_name hints, the server in the room ID (before version 12), the servers of the room's members as last known, and whoever sent the invite. A restricted join comes back countersigned by the server that authorised it.
• Leaving a room this server is not in — rejecting an invite, in practice — is the same with make_leave and send_leave. When no server in the room can be reached the invite is still rejected locally: a leave is built and stored here and nowhere else. An invite a user cannot get rid of because its server is gone is the worse failure.
• Inviting a user of another server goes through the ordinary pipeline, with one extra stage under the room lock: the invite is sent to the invitee's server to countersign (/invite, v2), and the copy with both signatures is the one persisted.

And the other side of each:

• make_join builds the template from the room's current state and runs the rules over a trial build first, so that a banned user hears so before signing anything. Under the restricted join rule it names a local user to authorise the join, and send_join countersigns it.
• send_join and send_leave take the event through the same intake as any other, and mark it to be relayed: the joining server is not in the room yet and relies on this one to tell everybody else.
• /invite countersigns an invite to a local user. A room this server is in takes the invite in as an ordinary event. One it is not in gets it as an out-of-band membership — an outlier that is nonetheless the invitee's membership in the room's current state, which is all /sync needs — and the stripped state the inviter sent along is kept in roomserver_invite_state for the invite list, until the invite is answered.

The outbound queue

federationapi.Queue sends this server's events and EDUs to everybody who should have them.

The roomserver's emit stage calls it with the room lock held, so that call only appends to an in-memory list. A dispatcher drains the list, works out each event's destinations — the servers with a user joined to the room, plus, for a membership event, the server of the user it is about, so that a server whose last user was just kicked hears of it — and writes one row per destination to federation_queue. An event received from another server is not sent on, because its origin sends it to everybody itself; the exception is a relayed join or leave, which goes to everybody but its origin.

A worker per destination sends the rows in transactions of at most 50 PDUs and 100 EDUs, and deletes them once the destination has answered. A failure backs the destination off — ten seconds, doubling to six hours, with a fifth of jitter — in federation_destinations, which the admin API reports and can reset. A destination later denied by the allow and deny lists has its backlog dropped rather than sent.

Rows are deleted only once delivered, so a restart loses nothing that had reached the database: the queue starts a worker for every destination with a backlog. Transaction IDs carry the queue's start time, because a receiving server deduplicates by transaction ID and an ID reused after a restart with different contents would be answered from its cache.

Received transactions are recorded with their response in federation_inbound_txns, so that a retransmission is answered the same way rather than processed twice.

EDUs

EDU	Out	In
m.typing	a local user's typing, to the room's servers	into the typing tracker, if the user belongs to the sending server
m.receipt	a local user's public read receipts — never private ones	into the receipts stream, likewise
m.direct_to_device	to-device messages for remote users, one EDU per server	delivered to local devices
m.presence, m.device_list_update	—	ignored: presence is off by default, device lists are Phase 7

Nothing in an EDU is signed, so the one check there is — the user must belong to the server that sent it — is what keeps a server from making anybody appear to type. EDUs about a room are subject to its server ACL.

The sync API hands EDUs over through syncapi.Outbound, and only for what a local user did: the same calls are how EDUs from other servers come in, and passing those on would echo them back.

Server ACLs

m.room.server_acl is enforced against the origin of everything about a room: its PDUs, its typing and receipts, its joins and leaves, and its reads of the room's history. Globs match case-insensitively, without the port; deny wins over allow; an empty allow list allows nobody; and allow_ip_literals: false refuses servers named by an IP address. An ACL never applies to the server enforcing it.

What other servers may read

/event, /state, /state_ids, /event_auth, /backfill and /get_missing_events are answered for a server with a user joined to the room, or for any server if the room is world-readable — and only if the room's ACL allows it. Anybody else gets 403 whether or not the event exists.

/state_ids and /state answer with the state before the event asked about, which is what every implementation's client side expects; the asker adds the event itself if it is a state event. An outlier has no snapshot, and asking for the state at one is a 404.

Endpoints

GET /_matrix/key/v2/server · POST /_matrix/key/v2/query · GET /_matrix/key/v2/query/{server}	keys; the notary answers for this server only
GET /_matrix/federation/v1/version
PUT /v1/send/{txnId}	transactions of PDUs and EDUs
GET /v1/make_join/{room}/{user} · PUT /v1/send_join, /v2/send_join	joins
GET /v1/make_leave/{room}/{user} · PUT /v1/send_leave, /v2/send_leave	leaves
PUT /v2/invite/{room}/{event}	invites
GET /v1/event/{event} · /v1/state/{room} · /v1/state_ids/{room} · /v1/event_auth/{room}/{event}	reads
GET /v1/backfill/{room} · POST /v1/get_missing_events/{room}	history
GET /v1/query/directory · /v1/query/profile	queries
GET, POST /v1/publicRooms	the public room directory

Client-Server endpoints that now reach other servers: /join and /rooms/{id}/join for a room this server is not in (by ID or by remote alias), /invite for a remote user, /leave to reject a remote invite, GET /directory/room/{alias} for a remote alias, /publicRooms?server=, and GET /profile/{user} for a remote user.

Known gaps

• Backfill only fills in history below the room. A gap higher up — a live event whose prev_events were closed with /state_ids rather than fetched — stays a gap. And the end of a server's history is not remembered: a room whose servers cannot supply anything before its oldest event is asked again each time a client scrolls to the top.
• Backfilled redactions are not applied to earlier backfilled events; a server sends a redacted event in its redacted form anyway, but unsigned.redacted_because is not filled in.
• No partial-state joins (MSC3706). Joining a room of tens of thousands of members means receiving, verifying and checking its whole state before the join completes, which takes minutes.
• No key rotation. The key file holds one key; old keys are not served as old_verify_keys, so rotating the key would invalidate this server's old events for anybody who has not cached the old key.
• Rejected events are not remembered. A rejected event is not stored, so if it is sent again it is fetched and judged again.
• Soft-failed events are never re-evaluated, and a redaction whose target arrives later is not applied when the target arrives.
• Room versions 1 and 2 — event IDs carried in the event — are not supported, so the oldest rooms on the network cannot be joined.
• Presence and device list EDUs are ignored on the way in and not sent (presence is off by default; device lists are Phase 7).
• Third-party invites across servers (exchange_third_party_invite) and knocking over federation (make_knock, send_knock) are not implemented.

Testing

Where	What
internal/federation	X-Matrix parsing (fuzzed) and signing, discovery against fake DNS and a fake well-known server, the key ring against a fake key server, an authenticated request end to end over TLS, the client refusing while disabled or denied.
internal/roomserver/federated_test.go	Two and three roomservers in one process, wired to each other's federation methods: joins at versions 10–12, conversation and fork merging, fail-over, refused joins, invites with countersignatures, rejection with and without the inviter reachable, missing events, a gap closed with state, forged and soft-failed events, ACLs, remote aliases and directories, the switch. backfill_test.go: a server that joined late scrolling back through the whole room at versions 10–12, the state inside the history, backfill with the switch off and under joined visibility.
internal/federationapi	The queue: fan-out and relay rules, backoff and retry, transactions within the limits, a restart with a backlog, the policy, EDUs.
internal/setup/phase5_test.go	Two complete servers over HTTPS: a join by remote alias, messages both ways through the queues, a remote profile, federation switched off with a message held and delivered after it is switched back on, the admin view of it; the TLS listener and the module following the switch.
Complement	make complement-federation — see server/test/complement.