Warp/docs/federation.md

# Federation

PLAN.md Phase 5: the Server-Server API. Warp can now join rooms on other
servers, have their users join its rooms, exchange events and typing,
receipts and to-device messages with them, and do all of it behind a switch
that turns it off at runtime — cleanly, with nothing left talking to
another server — and back on again, picking up where it left off.

Federation is still off in a fresh install. Nothing here changes what a
private homeserver does.

## Where things live

| Package | What |
|---|---|
| `internal/federation` | The protocol, with no knowledge of rooms: X-Matrix request signing, server discovery, the signed HTTP client, the key ring, the gate and the allow/deny policy. |
| `internal/roomserver` | Rooms across servers: taking in events from elsewhere (`federated.go`), joining, leaving and inviting across servers in both directions (`federated_membership.go`), answering other servers' reads (`federated_query.go`), remote aliases and directories, server ACLs (`acl.go`). |
| `internal/federationapi` | Federation as a running component: the inbound endpoints, the outbound queue, the EDU adapter and the `Module` the switch starts and stops. |

The split follows the dependencies. The roomserver uses the protocol — it
fetches events and asks for joins — and the component uses the roomserver,
so the component cannot live in the protocol package. PLAN.md calls it
`federation.Module`; it is `federationapi.Module`, for that reason.

## The switch

Three things answer to `federation.enabled`, and each reads it at the
moment it matters rather than being told:

- **The gate.** Every route under `/_matrix/federation`, `/_matrix/key`
  and `/.well-known/matrix/server` is wrapped by it, and answers
  `404 M_UNRECOGNIZED` while the switch is off — a server with federation
  off looks exactly like a server without a federation API.
- **The client.** Every outbound request checks the switch, the allow and
  deny lists, and that it is not addressed to this server, before it
  resolves anything or opens a connection. A component that somehow missed
  the toggle still cannot reach another server.
- **The module.** The one thing that talks to other servers on its own
  initiative is the outbound queue, and the module starts and stops it.
  The settings service calls the module when the switch moves; stopping
  cancels in-flight sends and waits for them, and leaves whatever was not
  sent in the database.

Events persisted while the switch is off are not queued. Holding them would
turn switching federation back on after a month into a month of events
delivered in one burst; a remote server that cares about a room fetches
what it missed with `/get_missing_events` the next time it hears from this
one. What was queued before the switch went off goes out when it comes
back on.

## Finding other servers

`federation.Resolver` implements the spec's discovery algorithm, in its
order: an IP literal or an explicit port is used as is; otherwise
`/.well-known/matrix/server` may delegate to another name, which is then
resolved the same way; then the `_matrix-fed._tcp` SRV record, then the
deprecated `_matrix._tcp`, then port 8448.

The result has two halves and the client keeps them apart. The address to
dial can be an SRV target; the name the TLS certificate must be valid for —
also the `Host` header — is never the SRV target, only the server name or
the name the well-known document delegated to. The client carries the
dial addresses down to its dialer in the request context and builds the URL
from the other half, so Go's TLS verification checks the right name
without any custom verification code.

Well-known answers are cached for their `Cache-Control` max-age, between
the spec's bounds (24 hours by default, 48 at most); failures for an hour,
so that a server with no document is not asked for one on every request.

## Keys

`GET /_matrix/key/v2/server` serves this server's key, valid for
`federation.key_validity_period` (24 hours by default).

Everybody else's keys go through the key ring. A key response is accepted
only if it is about the server asked for and every signature the server
made on it is by a key it lists and verifies — a response citing a key it
does not publish is broken or forged. Keys are stored in
`federation_server_keys` and cached in memory, and are believed for as long
as the server said, but never for more than a week: the spec's cap.

When a server's own key endpoint cannot be reached, the notaries in
`federation.trusted_key_servers` are asked, in order; a notary's answer
must carry the notary's signature as well. The default is no notaries:
keys come from the server that owns them or not at all.

Room versions 5 and later require a key to have been valid when the event
was sent. A cached key that is not known to cover an event's timestamp is
fetched again before the event is refused — the server may have extended
it — and a fetch that fails is not retried for a minute, so that a server
whose key endpoint is down is not asked on every event it sends.

As a notary, Warp answers for its own keys only. It keeps other servers'
keys, not the signed documents they came in, and a notary has to hand out
the documents.

## Authenticating requests

Every federation endpoint but the key and version endpoints checks the
`X-Matrix` Authorization header: the signature covers the method, the
request target exactly as it appeared on the request line, the origin, the
destination and the JSON body. The origin is then checked against the
allow and deny lists, and — for anything about a room — against the room's
server ACL.

The parser is fuzzed (`FuzzParseXMatrix`): a header that parses must
survive being written back out and parsed again unchanged.

## Taking in events from other servers

An event created here is built on the room's current state and can only
fail the authorisation rules. An event from somewhere else arrives with
none of that settled. The roomserver follows the spec's procedure, in
`federated.go`:

1. **It is what it says it is.** It parses under the room's version, it is
   signed by its sender's server, and its content matches its hash. A hash
   mismatch with a good signature means somebody redacted it on the way;
   the redacted form is kept rather than the event rejected.
2. **It passes the rules against its own auth events.** Missing auth
   events are fetched with `/event_auth` and checked themselves, oldest
   first.
3. **It passes them against the state before it** — the state after each
   of its prev_events, resolved together. A prev_event this server does not
   have is asked for with `/get_missing_events` (ten events, between this
   server's forward extremities and the new event). A gap wider than that
   is closed by asking for the state at the missing event with
   `/state_ids`, fetching whatever of it is missing — one at a time with
   `/event`, or all at once with `/state` past fifty — and checking all of
   it.
4. **It passes them against the current state.** An event that fails only
   here is *soft-failed*: kept, because other events may cite it, but never
   shown to a client and never a forward extremity. This is what stops a
   banned user's server from sending events built on the state before the
   ban.

An event that fails 1 to 3 is rejected and not stored. The `/send`
response says why, per event.

### Outliers

Some events have to be held without the room's state at them: the state
and auth chain a join is handed, auth events fetched to judge something
else, the state fetched across a gap, an invite to a room nobody here is
in, a soft-failed event. These are *outliers* — a column on
`roomserver_events` rather than a table of their own, because every read
of a room's current state joins the events table, and the current state of
a room joined over federation is almost all outliers.

An outlier can be cited and read by ID. It has no state snapshot, is never
a forward extremity, and never appears in `/sync`, `/messages` or
`/context` — the timeline queries filter it out. An outlier that arrives
again as an ordinary event stays an outlier: this server used it as state
without its history, and putting the room's head on it now could put the
head behind events that are already here.

### The room's head

A Phase 3 room had one forward extremity, because every event was a child
of the one before. Federation forks rooms: two servers send at the same
moment, each without the other's event. The new extremities are the old
ones the event does not descend from, plus the event.

The current state then has to be worked out again. In the common case —
the event follows the room's single extremity — it is one tuple more than
before. Otherwise it is resolved across the new extremities and written
whole. The next event created here cites every extremity, and the fork
closes; `TestFederatedJoinAndConversation` checks that both servers end up
with one extremity and the same state.

### History: backfill

A room joined over federation starts, on this server, at the join: the
join is handed the room's state, not the events that led to it. When a
client pages `/messages` backwards and the page comes back short at the
bottom of what this server holds, the roomserver asks the servers in the
room for what came before it (`GET /backfill`) before answering, in
`backfill.go`:

- **Where to start.** The prev_events of the room's oldest timeline events
  (the oldest fifty) that are not themselves in the timeline. A room whose
  oldest event is its create event has nothing to ask for.
- **Whom to ask.** The servers of the room's members, joined ones first,
  in turn, until one sends something that passes. A page asks at most
  three times before it is returned short.
- **What is checked.** Steps 1 to 3 of the intake above. The state before
  each event comes from the events before it in the same batch, or —
  for the oldest ones — from `/state_ids` at its prev_events. Step 4 does
  not apply: soft failure refuses events built on a state the room has
  moved past, which history always is. An event already held as an
  outlier is promoted into the timeline.
- **Where it goes.** Below everything. The stream ordering counter only
  goes up, so backfilled events are numbered by a second counter that runs
  downwards from -1 (`roomserver_sequences.backfill_ordering`). Each batch
  sorts below the one before it, and nothing about the room's head, its
  current state or its stream top changes: a live `/sync` reads "after my
  token" and never reaches below zero, so history reaches clients only
  through `/messages`, `/context` and the state at a point in it. Zero is
  never a position, so it can go on meaning "unbounded" in every window;
  the position just below the event at 1 is -1 (`storage.PositionBefore`).

Backfill is only asked for when the user may see before their join — a
room whose history visibility is `joined` or `invited` has nothing to
show them there — and only by `/messages`, never by `/sync` or `/context`,
which do not wait on another server. With federation off, or with nobody
reachable, the page is what this server holds.

## Membership across servers

A server can only sign as itself, and a room's state lives with the
servers in it, so each membership change that crosses a server boundary is
a short protocol:

- **Joining a room this server is not in** — not held at all, held only as
  an invite, or held from before every local user left — asks a resident
  server for a template (`make_join`), fills it in, signs it, and hands it
  back (`send_join`, v2). The answer is the room's state and auth chain,
  every event of which is verified and checked against its own auth events
  before any of it is believed. Candidates are tried in order: the
  client's `server_name` hints, the server in the room ID (before version
  12), the servers of the room's members as last known, and whoever sent
  the invite. A restricted join comes back countersigned by the server that
  authorised it.
- **Leaving a room this server is not in** — rejecting an invite, in
  practice — is the same with `make_leave` and `send_leave`. When no server
  in the room can be reached the invite is still rejected locally: a leave
  is built and stored here and nowhere else. An invite a user cannot get
  rid of because its server is gone is the worse failure.
- **Inviting a user of another server** goes through the ordinary
  pipeline, with one extra stage under the room lock: the invite is sent
  to the invitee's server to countersign (`/invite`, v2), and the copy with
  both signatures is the one persisted.

And the other side of each:

- `make_join` builds the template from the room's current state and runs
  the rules over a trial build first, so that a banned user hears so before
  signing anything. Under the restricted join rule it names a local user to
  authorise the join, and `send_join` countersigns it.
- `send_join` and `send_leave` take the event through the same intake as
  any other, and mark it to be **relayed**: the joining server is not in
  the room yet and relies on this one to tell everybody else.
- `/invite` countersigns an invite to a local user. A room this server is
  in takes the invite in as an ordinary event. One it is not in gets it as
  an out-of-band membership — an outlier that is nonetheless the invitee's
  membership in the room's current state, which is all `/sync` needs — and
  the stripped state the inviter sent along is kept in
  `roomserver_invite_state` for the invite list, until the invite is
  answered.

## The outbound queue

`federationapi.Queue` sends this server's events and EDUs to everybody who
should have them.

The roomserver's emit stage calls it with the room lock held, so that call
only appends to an in-memory list. A dispatcher drains the list, works out
each event's destinations — the servers with a user joined to the room,
plus, for a membership event, the server of the user it is about, so that
a server whose last user was just kicked hears of it — and writes one row
per destination to `federation_queue`. An event received from another
server is not sent on, because its origin sends it to everybody itself;
the exception is a relayed join or leave, which goes to everybody but its
origin.

A worker per destination sends the rows in transactions of at most 50 PDUs
and 100 EDUs, and deletes them once the destination has answered. A
failure backs the destination off — ten seconds, doubling to six hours,
with a fifth of jitter — in `federation_destinations`, which the admin API
reports and can reset. A destination later denied by the allow and deny
lists has its backlog dropped rather than sent.

Rows are deleted only once delivered, so a restart loses nothing that had
reached the database: the queue starts a worker for every destination with
a backlog. Transaction IDs carry the queue's start time, because a
receiving server deduplicates by transaction ID and an ID reused after a
restart with different contents would be answered from its cache.

Received transactions are recorded with their response in
`federation_inbound_txns`, so that a retransmission is answered the same
way rather than processed twice.

## EDUs

| EDU | Out | In |
|---|---|---|
| `m.typing` | a local user's typing, to the room's servers | into the typing tracker, if the user belongs to the sending server |
| `m.receipt` | a local user's public read receipts — never private ones | into the receipts stream, likewise |
| `m.direct_to_device` | to-device messages for remote users, one EDU per server | delivered to local devices |
| `m.device_list_update`, `m.signing_key_update` | a local user's key changes, to every server sharing a room with them | the remote user is marked changed for local syncs; see [e2ee.md](e2ee.md) |
| `m.presence` | — | ignored: presence is off by default |

Nothing in an EDU is signed, so the one check there is — the user must
belong to the server that sent it — is what keeps a server from making
anybody appear to type. EDUs about a room are subject to its server ACL.

The sync API hands EDUs over through `syncapi.Outbound`, and only for what
a local user did: the same calls are how EDUs from other servers come in,
and passing those on would echo them back.

## Server ACLs

`m.room.server_acl` is enforced against the origin of everything about a
room: its PDUs, its typing and receipts, its joins and leaves, and its
reads of the room's history. Globs match case-insensitively, without the
port; deny wins over allow; an empty allow list allows nobody; and
`allow_ip_literals: false` refuses servers named by an IP address. An ACL
never applies to the server enforcing it.

## What other servers may read

`/event`, `/state`, `/state_ids`, `/event_auth`, `/backfill` and
`/get_missing_events` are answered for a server with a user joined to the
room, or for any server if the room is world-readable — and only if the
room's ACL allows it. Anybody else gets `403` whether or not the event
exists.

`/state_ids` and `/state` answer with the state *before* the event asked
about, which is what every implementation's client side expects; the
asker adds the event itself if it is a state event. An outlier has no
snapshot, and asking for the state at one is a `404`.

## Endpoints

| | |
|---|---|
| `GET /_matrix/key/v2/server` · `POST /_matrix/key/v2/query` · `GET /_matrix/key/v2/query/{server}` | keys; the notary answers for this server only |
| `GET /_matrix/federation/v1/version` | |
| `PUT /v1/send/{txnId}` | transactions of PDUs and EDUs |
| `GET /v1/make_join/{room}/{user}` · `PUT /v1/send_join`, `/v2/send_join` | joins |
| `GET /v1/make_leave/{room}/{user}` · `PUT /v1/send_leave`, `/v2/send_leave` | leaves |
| `PUT /v2/invite/{room}/{event}` | invites |
| `GET /v1/event/{event}` · `/v1/state/{room}` · `/v1/state_ids/{room}` · `/v1/event_auth/{room}/{event}` | reads |
| `GET /v1/backfill/{room}` · `POST /v1/get_missing_events/{room}` | history |
| `GET /v1/query/directory` · `/v1/query/profile` | queries |
| `GET`, `POST /v1/publicRooms` | the public room directory |

Client-Server endpoints that now reach other servers: `/join` and
`/rooms/{id}/join` for a room this server is not in (by ID or by remote
alias), `/invite` for a remote user, `/leave` to reject a remote invite,
`GET /directory/room/{alias}` for a remote alias, `/publicRooms?server=`,
and `GET /profile/{user}` for a remote user.

## Known gaps

- **Backfill only fills in history below the room.** A gap higher up — a
  live event whose prev_events were closed with `/state_ids` rather than
  fetched — stays a gap. And the end of a server's history is not
  remembered: a room whose servers cannot supply anything before its
  oldest event is asked again each time a client scrolls to the top.
- **Backfilled redactions are not applied** to earlier backfilled events;
  a server sends a redacted event in its redacted form anyway, but
  `unsigned.redacted_because` is not filled in.
- **No partial-state joins** (MSC3706). Joining a room of tens of
  thousands of members means receiving, verifying and checking its whole
  state before the join completes, which takes minutes.
- **No key rotation.** The key file holds one key; old keys are not
  served as `old_verify_keys`, so rotating the key would invalidate this
  server's old events for anybody who has not cached the old key.
- **Rejected events are not remembered.** A rejected event is not stored,
  so if it is sent again it is fetched and judged again.
- **Soft-failed events are never re-evaluated**, and a redaction whose
  target arrives later is not applied when the target arrives.
- **Room versions 1 and 2** — event IDs carried in the event — are not
  supported, so the oldest rooms on the network cannot be joined.
- **Presence EDUs** are ignored on the way in and not sent (presence is
  off by default).
- **Third-party invites** across servers (`exchange_third_party_invite`)
  and **knocking** over federation (`make_knock`, `send_knock`) are not
  implemented.

## Testing

| Where | What |
|---|---|
| `internal/federation` | X-Matrix parsing (fuzzed) and signing, discovery against fake DNS and a fake well-known server, the key ring against a fake key server, an authenticated request end to end over TLS, the client refusing while disabled or denied. |
| `internal/roomserver/federated_test.go` | Two and three roomservers in one process, wired to each other's federation methods: joins at versions 10–12, conversation and fork merging, fail-over, refused joins, invites with countersignatures, rejection with and without the inviter reachable, missing events, a gap closed with state, forged and soft-failed events, ACLs, remote aliases and directories, the switch. `backfill_test.go`: a server that joined late scrolling back through the whole room at versions 10–12, the state inside the history, backfill with the switch off and under `joined` visibility. |
| `internal/federationapi` | The queue: fan-out and relay rules, backoff and retry, transactions within the limits, a restart with a backlog, the policy, EDUs. |
| `internal/setup/phase5_test.go` | Two complete servers over HTTPS: a join by remote alias, messages both ways through the queues, a remote profile, federation switched off with a message held and delivered after it is switched back on, the admin view of it; the TLS listener and the module following the switch. |
| Complement | `make complement-federation` — see [server/test/complement](../server/test/complement/README.md). |