Phase 5: federation

Implements PLAN.md Phase 5: the Server-Server API. Warp joins rooms on
other servers and lets their users join its rooms, invites across servers
with the invitee's server countersigning, rejects remote invites (locally
if the inviter is unreachable), and exchanges events, typing, receipts and
to-device messages with other homeservers — all behind the runtime switch,
which now starts and stops real outbound traffic.

The protocol lives in internal/federation with no knowledge of rooms:
X-Matrix signing (fuzzed parser), the spec's discovery algorithm keeping
the dial address and the TLS name apart, a signed client that refuses
every request while the switch is off or the policy denies the
destination, and a key ring that caches other servers' keys for at most a
week, with optional notaries.

The roomserver takes events from elsewhere through the spec's intake:
signatures and hashes (a hash mismatch keeps the redacted form), the rules
against the event's own auth events, against the state before it — with
/get_missing_events for small gaps and /state_ids for wide ones — and
against the current state, where a failure is a soft failure. Events held
without their state are outliers: a column on roomserver_events, because
the current state of a room joined over federation is almost all
outliers, and excluded from every timeline query. Forks are resolved at
the room's head. Membership across servers is make_join/send_join,
make_leave/send_leave and countersigned invites, in both directions;
m.room.server_acl is enforced on everything another server sends.
History before a remote join is backfilled when /messages pages past it,
stored at negative stream positions.

internal/federationapi is the running component: the inbound endpoints,
and an outbound queue that persists what it owes each destination, sends
it in transactions of 50 PDUs and 100 EDUs, backs off from ten seconds to
six hours, survives restarts, and drops events persisted while federation
is off rather than delivering a backlog in one burst. Its Module is what
the switch starts and stops.

Also: a TLS listener for port 8448, trusted key servers, key validity and
an outbound CA bundle in the config; per-destination health and backoff
reset in the admin API; remote profiles, aliases and directories in the
Client-Server API; a Complement federation profile. See
docs/federation.md, including its known gaps (no partial-state joins, no
key rotation).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/README.md

@@ -7,6 +7,7 @@
 | [event-core.md](event-core.md) | `internal/matrix`: canonical JSON, events, auth rules, state resolution |
 | [rooms.md](rooms.md) | `internal/roomserver`: the input pipeline, room storage, known gaps |
 | [sync.md](sync.md) | `internal/syncapi`: stream tokens, long polling, filters, the ephemeral streams |
+| [federation.md](federation.md) | the Server-Server API: discovery, keys, taking in events, joins across servers, the outbound queue, the switch |
 | [admin-api.md](admin-api.md) | `/_warp/admin/v1`: the operator API behind the admin panel |
 
 The design and roadmap are in [PLAN.md](../PLAN.md) at the repository root.

docs/admin-api.md

@@ -57,6 +57,43 @@ than offer something that will bounce.
 Editing the allow and deny lists is Phase 9. They are reported now because
 the dashboard has to show which servers a running Warp will talk to.
 
+### `GET /federation/destinations`
+
+Delivery health per remote server: the panel's federation status view.
+Every server this one has queued something for, or has failed to reach,
+with how much it is still owed.
+
+```json
+{
+  "running": true,
+  "destinations": [
+    {
+      "server_name": "matrix.org",
+      "failures": 0,
+      "retry_at_ms": 0,
+      "last_success_ms": 1760600000000,
+      "last_failure_ms": 0,
+      "queued_pdus": 0,
+      "queued_edus": 0
+    }
+  ]
+}
+```
+
+`running` is whether the outbound queue is up — `false` while federation is
+switched off. `failures` counts consecutive failed sends; while it is above
+zero the server is being backed off, and `retry_at_ms` is when the next
+attempt is due. The backoff doubles from ten seconds to six hours.
+
+A server started without a signing key has no federation at all, and both
+destination endpoints answer `404`.
+
+### `POST /federation/destinations/{serverName}/reset`
+
+Forget a server's backoff and try it now — for an operator who knows the
+server is back and does not want to wait out a six-hour retry. Answers
+`{}`.
+
 ### `GET /registration`, `PUT /registration`
 
 The same shape, without the lists, for open registration.

docs/configuration.md

@@ -33,6 +33,9 @@ List-valued variables are comma-separated. A single `-` clears a list:
 | `listen.idle_timeout` | `WARP_IDLE_TIMEOUT` | `120s` | |
 | `listen.max_header_bytes` | `WARP_MAX_HEADER_BYTES` | `1048576` | |
 | `listen.shutdown_timeout` | `WARP_SHUTDOWN_TIMEOUT` | `30s` | Grace period for in-flight requests. |
+| `listen.federation_address` | `WARP_LISTEN_FEDERATION_ADDRESS` | empty | A second listener serving the same API over TLS, usually `:8448`. Empty = none; see *Federation over TLS*. |
+| `listen.tls_cert_file` | `WARP_TLS_CERT_FILE` | empty | PEM certificate chain for the federation listener. Required with it. |
+| `listen.tls_key_file` | `WARP_TLS_KEY_FILE` | empty | PEM key for the federation listener. Required with it. |
 | `database.driver` | `WARP_DB_DRIVER` | `sqlite` | `sqlite`, `postgres`, `pocketbase` (not implemented). |
 | `database.connection_string` | `WARP_DB_CONNECTION_STRING` | SQLite file in the working directory | |
 | `database.max_open_conns` | `WARP_DB_MAX_OPEN_CONNS` | `10` | Ignored by SQLite: writes are serialised onto one connection. |
@@ -57,6 +60,9 @@ List-valued variables are comma-separated. A single `-` clears a list:
 | `federation.allowlist` | `WARP_FEDERATION_ALLOWLIST` | `[]` | Non-empty = federate only with these. |
 | `federation.denylist` | `WARP_FEDERATION_DENYLIST` | `[]` | Always wins over the allowlist. |
 | `federation.allow_runtime_toggle` | `WARP_FEDERATION_ALLOW_RUNTIME_TOGGLE` | `true` | See below. |
+| `federation.trusted_key_servers` | `WARP_FEDERATION_TRUSTED_KEY_SERVERS` | `[]` | Notaries asked for a server's key when it cannot be reached itself, e.g. `matrix.org`. Empty = only ever ask the server. |
+| `federation.key_validity_period` | `WARP_FEDERATION_KEY_VALIDITY_PERIOD` | `24h` | How long other servers may cache this server's key. At most `168h`. |
+| `federation.tls_ca_file` | `WARP_FEDERATION_TLS_CA_FILE` | empty | Extra CA certificates (PEM) for outbound federation, on top of the system roots. |
 | `registration.enabled` | `WARP_REGISTRATION_ENABLED` | `false` | |
 | `registration.allow_runtime_toggle` | `WARP_REGISTRATION_ALLOW_RUNTIME_TOGGLE` | `true` | |
 | `registration.tokens` | `WARP_REGISTRATION_TOKENS` | `[]` | Non-empty = sign-up also needs one of these, as the `m.login.registration_token` stage. |
@@ -145,13 +151,41 @@ Everything else is read once at startup.
 
 While `federation.enabled` is false:
 
-- `/.well-known/matrix/server` answers `404 M_UNRECOGNIZED`, and so will
-  `/_matrix/federation/*` and `/_matrix/key/*` once they exist.
-- Nothing outbound runs — in Phase 0 there is nothing outbound yet; from
-  Phase 5 the transaction queue, key fetcher and remote media fetch stay
-  stopped.
+- `/_matrix/federation/*`, `/_matrix/key/*` and `/.well-known/matrix/server`
+  answer `404 M_UNRECOGNIZED`, exactly as on a server with no federation
+  API. The TLS listener on `listen.federation_address`, if there is one,
+  stays up and answers the same way.
+- Nothing outbound runs. The transaction queue is stopped, and the
+  federation client refuses every request — key fetches, joins, profile
+  and alias lookups — before it opens a connection, so a component that
+  somehow missed the switch still cannot reach another server.
+- Events persisted while it is off are not queued for other servers. A
+  remote server that cares about the room fetches what it missed the next
+  time it hears from this one. What was already queued when the switch
+  went off stays queued, and goes out when it comes back on.
 - Client-Server endpoints that would need a remote server answer
   `M_FORBIDDEN` with a message saying federation is disabled.
 - Events are still signed with the server key, and room and event IDs are
   unchanged. A room created while federation is off federates normally
   once it is switched on.
+
+## Federation over TLS
+
+Other servers reach this one at whatever its server name resolves to: the
+address in `/.well-known/matrix/server`, an SRV record, or the name itself
+on port 8448. Two deployment shapes:
+
+- **Behind a reverse proxy** that terminates TLS — the usual one. Set
+  `global.well_known.server_address` to the proxy's `host:443` and leave
+  `listen.federation_address` empty: the proxy forwards
+  `/_matrix/federation` and `/_matrix/key` to `listen.address` like
+  everything else.
+- **Warp terminates TLS itself**: set `listen.federation_address` to
+  `:8448` with `listen.tls_cert_file` and `listen.tls_key_file`. The
+  certificate must be valid for the server name (or for the delegated
+  name, if the well-known document delegates).
+
+Outbound federation verifies certificates against the system roots, plus
+`federation.tls_ca_file` when it is set — for a private federation whose
+servers share an internal CA. There is no switch to turn verification
+off.

docs/development.md

@@ -147,15 +147,17 @@ far along the server is — PLAN.md section 9 treats it as the primary progress
 metric.
 
 ```sh
-make complement                                   # the current target set
+make complement                                   # the C-S set, federation off
+make complement-federation                        # the federation set
 server/test/complement/run.sh TestLogin           # one test
 COMPLEMENT_DEBUG=1 make complement                # with the container logs
 ```
 
 It needs Docker. The script builds the image, clones Complement into
 `.complement/` (ignored by git) and runs the subset for the phases that are
-done — the account, room and stream tests, with federation off. Extend
-`COMPLEMENT_TESTS` in
+done. `COMPLEMENT_PROFILE` picks the set: `csapi` (the default) runs the
+account, room and stream tests with federation off; `federation` runs the
+Server-Server tests with it on. Extend the sets in
 [run.sh](../server/test/complement/run.sh) as each phase lands, rather than
 switching to the whole suite: a pass count that drops because a later phase
 is missing is noise.
@@ -165,9 +167,9 @@ and the deployment is entirely `WARP_*` environment variables — see
 [entrypoint.sh](../server/test/complement/entrypoint.sh). Two things there
 are worth knowing: rate limiting is off, because the whole suite comes from
 one address and would otherwise spend its time reading `M_LIMIT_EXCEEDED`;
-and port 8448 is exposed but nothing listens on it yet. The federation
-profile needs a TLS listener signed by the CA that Complement mounts at
-`/complement/ca`, which arrives with federation in Phase 5.
+and port 8448 is served over TLS with a certificate the entrypoint has the
+CA Complement mounts at `/complement/ca` sign, the same CA Warp is told to
+trust for its outbound connections.
 
 ## Creating the first account
 

docs/federation.md

@@ -0,0 +1,380 @@
+# Federation
+
+PLAN.md Phase 5: the Server-Server API. Warp can now join rooms on other
+servers, have their users join its rooms, exchange events and typing,
+receipts and to-device messages with them, and do all of it behind a switch
+that turns it off at runtime — cleanly, with nothing left talking to
+another server — and back on again, picking up where it left off.
+
+Federation is still off in a fresh install. Nothing here changes what a
+private homeserver does.
+
+## Where things live
+
+| Package | What |
+|---|---|
+| `internal/federation` | The protocol, with no knowledge of rooms: X-Matrix request signing, server discovery, the signed HTTP client, the key ring, the gate and the allow/deny policy. |
+| `internal/roomserver` | Rooms across servers: taking in events from elsewhere (`federated.go`), joining, leaving and inviting across servers in both directions (`federated_membership.go`), answering other servers' reads (`federated_query.go`), remote aliases and directories, server ACLs (`acl.go`). |
+| `internal/federationapi` | Federation as a running component: the inbound endpoints, the outbound queue, the EDU adapter and the `Module` the switch starts and stops. |
+
+The split follows the dependencies. The roomserver uses the protocol — it
+fetches events and asks for joins — and the component uses the roomserver,
+so the component cannot live in the protocol package. PLAN.md calls it
+`federation.Module`; it is `federationapi.Module`, for that reason.
+
+## The switch
+
+Three things answer to `federation.enabled`, and each reads it at the
+moment it matters rather than being told:
+
+- **The gate.** Every route under `/_matrix/federation`, `/_matrix/key`
+  and `/.well-known/matrix/server` is wrapped by it, and answers
+  `404 M_UNRECOGNIZED` while the switch is off — a server with federation
+  off looks exactly like a server without a federation API.
+- **The client.** Every outbound request checks the switch, the allow and
+  deny lists, and that it is not addressed to this server, before it
+  resolves anything or opens a connection. A component that somehow missed
+  the toggle still cannot reach another server.
+- **The module.** The one thing that talks to other servers on its own
+  initiative is the outbound queue, and the module starts and stops it.
+  The settings service calls the module when the switch moves; stopping
+  cancels in-flight sends and waits for them, and leaves whatever was not
+  sent in the database.
+
+Events persisted while the switch is off are not queued. Holding them would
+turn switching federation back on after a month into a month of events
+delivered in one burst; a remote server that cares about a room fetches
+what it missed with `/get_missing_events` the next time it hears from this
+one. What was queued before the switch went off goes out when it comes
+back on.
+
+## Finding other servers
+
+`federation.Resolver` implements the spec's discovery algorithm, in its
+order: an IP literal or an explicit port is used as is; otherwise
+`/.well-known/matrix/server` may delegate to another name, which is then
+resolved the same way; then the `_matrix-fed._tcp` SRV record, then the
+deprecated `_matrix._tcp`, then port 8448.
+
+The result has two halves and the client keeps them apart. The address to
+dial can be an SRV target; the name the TLS certificate must be valid for —
+also the `Host` header — is never the SRV target, only the server name or
+the name the well-known document delegated to. The client carries the
+dial addresses down to its dialer in the request context and builds the URL
+from the other half, so Go's TLS verification checks the right name
+without any custom verification code.
+
+Well-known answers are cached for their `Cache-Control` max-age, between
+the spec's bounds (24 hours by default, 48 at most); failures for an hour,
+so that a server with no document is not asked for one on every request.
+
+## Keys
+
+`GET /_matrix/key/v2/server` serves this server's key, valid for
+`federation.key_validity_period` (24 hours by default).
+
+Everybody else's keys go through the key ring. A key response is accepted
+only if it is about the server asked for and every signature the server
+made on it is by a key it lists and verifies — a response citing a key it
+does not publish is broken or forged. Keys are stored in
+`federation_server_keys` and cached in memory, and are believed for as long
+as the server said, but never for more than a week: the spec's cap.
+
+When a server's own key endpoint cannot be reached, the notaries in
+`federation.trusted_key_servers` are asked, in order; a notary's answer
+must carry the notary's signature as well. The default is no notaries:
+keys come from the server that owns them or not at all.
+
+Room versions 5 and later require a key to have been valid when the event
+was sent. A cached key that is not known to cover an event's timestamp is
+fetched again before the event is refused — the server may have extended
+it — and a fetch that fails is not retried for a minute, so that a server
+whose key endpoint is down is not asked on every event it sends.
+
+As a notary, Warp answers for its own keys only. It keeps other servers'
+keys, not the signed documents they came in, and a notary has to hand out
+the documents.
+
+## Authenticating requests
+
+Every federation endpoint but the key and version endpoints checks the
+`X-Matrix` Authorization header: the signature covers the method, the
+request target exactly as it appeared on the request line, the origin, the
+destination and the JSON body. The origin is then checked against the
+allow and deny lists, and — for anything about a room — against the room's
+server ACL.
+
+The parser is fuzzed (`FuzzParseXMatrix`): a header that parses must
+survive being written back out and parsed again unchanged.
+
+## Taking in events from other servers
+
+An event created here is built on the room's current state and can only
+fail the authorisation rules. An event from somewhere else arrives with
+none of that settled. The roomserver follows the spec's procedure, in
+`federated.go`:
+
+1. **It is what it says it is.** It parses under the room's version, it is
+   signed by its sender's server, and its content matches its hash. A hash
+   mismatch with a good signature means somebody redacted it on the way;
+   the redacted form is kept rather than the event rejected.
+2. **It passes the rules against its own auth events.** Missing auth
+   events are fetched with `/event_auth` and checked themselves, oldest
+   first.
+3. **It passes them against the state before it** — the state after each
+   of its prev_events, resolved together. A prev_event this server does not
+   have is asked for with `/get_missing_events` (ten events, between this
+   server's forward extremities and the new event). A gap wider than that
+   is closed by asking for the state at the missing event with
+   `/state_ids`, fetching whatever of it is missing — one at a time with
+   `/event`, or all at once with `/state` past fifty — and checking all of
+   it.
+4. **It passes them against the current state.** An event that fails only
+   here is *soft-failed*: kept, because other events may cite it, but never
+   shown to a client and never a forward extremity. This is what stops a
+   banned user's server from sending events built on the state before the
+   ban.
+
+An event that fails 1 to 3 is rejected and not stored. The `/send`
+response says why, per event.
+
+### Outliers
+
+Some events have to be held without the room's state at them: the state
+and auth chain a join is handed, auth events fetched to judge something
+else, the state fetched across a gap, an invite to a room nobody here is
+in, a soft-failed event. These are *outliers* — a column on
+`roomserver_events` rather than a table of their own, because every read
+of a room's current state joins the events table, and the current state of
+a room joined over federation is almost all outliers.
+
+An outlier can be cited and read by ID. It has no state snapshot, is never
+a forward extremity, and never appears in `/sync`, `/messages` or
+`/context` — the timeline queries filter it out. An outlier that arrives
+again as an ordinary event stays an outlier: this server used it as state
+without its history, and putting the room's head on it now could put the
+head behind events that are already here.
+
+### The room's head
+
+A Phase 3 room had one forward extremity, because every event was a child
+of the one before. Federation forks rooms: two servers send at the same
+moment, each without the other's event. The new extremities are the old
+ones the event does not descend from, plus the event.
+
+The current state then has to be worked out again. In the common case —
+the event follows the room's single extremity — it is one tuple more than
+before. Otherwise it is resolved across the new extremities and written
+whole. The next event created here cites every extremity, and the fork
+closes; `TestFederatedJoinAndConversation` checks that both servers end up
+with one extremity and the same state.
+
+### History: backfill
+
+A room joined over federation starts, on this server, at the join: the
+join is handed the room's state, not the events that led to it. When a
+client pages `/messages` backwards and the page comes back short at the
+bottom of what this server holds, the roomserver asks the servers in the
+room for what came before it (`GET /backfill`) before answering, in
+`backfill.go`:
+
+- **Where to start.** The prev_events of the room's oldest timeline events
+  (the oldest fifty) that are not themselves in the timeline. A room whose
+  oldest event is its create event has nothing to ask for.
+- **Whom to ask.** The servers of the room's members, joined ones first,
+  in turn, until one sends something that passes. A page asks at most
+  three times before it is returned short.
+- **What is checked.** Steps 1 to 3 of the intake above. The state before
+  each event comes from the events before it in the same batch, or —
+  for the oldest ones — from `/state_ids` at its prev_events. Step 4 does
+  not apply: soft failure refuses events built on a state the room has
+  moved past, which history always is. An event already held as an
+  outlier is promoted into the timeline.
+- **Where it goes.** Below everything. The stream ordering counter only
+  goes up, so backfilled events are numbered by a second counter that runs
+  downwards from -1 (`roomserver_sequences.backfill_ordering`). Each batch
+  sorts below the one before it, and nothing about the room's head, its
+  current state or its stream top changes: a live `/sync` reads "after my
+  token" and never reaches below zero, so history reaches clients only
+  through `/messages`, `/context` and the state at a point in it. Zero is
+  never a position, so it can go on meaning "unbounded" in every window;
+  the position just below the event at 1 is -1 (`storage.PositionBefore`).
+
+Backfill is only asked for when the user may see before their join — a
+room whose history visibility is `joined` or `invited` has nothing to
+show them there — and only by `/messages`, never by `/sync` or `/context`,
+which do not wait on another server. With federation off, or with nobody
+reachable, the page is what this server holds.
+
+## Membership across servers
+
+A server can only sign as itself, and a room's state lives with the
+servers in it, so each membership change that crosses a server boundary is
+a short protocol:
+
+- **Joining a room this server is not in** — not held at all, held only as
+  an invite, or held from before every local user left — asks a resident
+  server for a template (`make_join`), fills it in, signs it, and hands it
+  back (`send_join`, v2). The answer is the room's state and auth chain,
+  every event of which is verified and checked against its own auth events
+  before any of it is believed. Candidates are tried in order: the
+  client's `server_name` hints, the server in the room ID (before version
+  12), the servers of the room's members as last known, and whoever sent
+  the invite. A restricted join comes back countersigned by the server that
+  authorised it.
+- **Leaving a room this server is not in** — rejecting an invite, in
+  practice — is the same with `make_leave` and `send_leave`. When no server
+  in the room can be reached the invite is still rejected locally: a leave
+  is built and stored here and nowhere else. An invite a user cannot get
+  rid of because its server is gone is the worse failure.
+- **Inviting a user of another server** goes through the ordinary
+  pipeline, with one extra stage under the room lock: the invite is sent
+  to the invitee's server to countersign (`/invite`, v2), and the copy with
+  both signatures is the one persisted.
+
+And the other side of each:
+
+- `make_join` builds the template from the room's current state and runs
+  the rules over a trial build first, so that a banned user hears so before
+  signing anything. Under the restricted join rule it names a local user to
+  authorise the join, and `send_join` countersigns it.
+- `send_join` and `send_leave` take the event through the same intake as
+  any other, and mark it to be **relayed**: the joining server is not in
+  the room yet and relies on this one to tell everybody else.
+- `/invite` countersigns an invite to a local user. A room this server is
+  in takes the invite in as an ordinary event. One it is not in gets it as
+  an out-of-band membership — an outlier that is nonetheless the invitee's
+  membership in the room's current state, which is all `/sync` needs — and
+  the stripped state the inviter sent along is kept in
+  `roomserver_invite_state` for the invite list, until the invite is
+  answered.
+
+## The outbound queue
+
+`federationapi.Queue` sends this server's events and EDUs to everybody who
+should have them.
+
+The roomserver's emit stage calls it with the room lock held, so that call
+only appends to an in-memory list. A dispatcher drains the list, works out
+each event's destinations — the servers with a user joined to the room,
+plus, for a membership event, the server of the user it is about, so that
+a server whose last user was just kicked hears of it — and writes one row
+per destination to `federation_queue`. An event received from another
+server is not sent on, because its origin sends it to everybody itself;
+the exception is a relayed join or leave, which goes to everybody but its
+origin.
+
+A worker per destination sends the rows in transactions of at most 50 PDUs
+and 100 EDUs, and deletes them once the destination has answered. A
+failure backs the destination off — ten seconds, doubling to six hours,
+with a fifth of jitter — in `federation_destinations`, which the admin API
+reports and can reset. A destination later denied by the allow and deny
+lists has its backlog dropped rather than sent.
+
+Rows are deleted only once delivered, so a restart loses nothing that had
+reached the database: the queue starts a worker for every destination with
+a backlog. Transaction IDs carry the queue's start time, because a
+receiving server deduplicates by transaction ID and an ID reused after a
+restart with different contents would be answered from its cache.
+
+Received transactions are recorded with their response in
+`federation_inbound_txns`, so that a retransmission is answered the same
+way rather than processed twice.
+
+## EDUs
+
+| EDU | Out | In |
+|---|---|---|
+| `m.typing` | a local user's typing, to the room's servers | into the typing tracker, if the user belongs to the sending server |
+| `m.receipt` | a local user's public read receipts — never private ones | into the receipts stream, likewise |
+| `m.direct_to_device` | to-device messages for remote users, one EDU per server | delivered to local devices |
+| `m.presence`, `m.device_list_update` | — | ignored: presence is off by default, device lists are Phase 7 |
+
+Nothing in an EDU is signed, so the one check there is — the user must
+belong to the server that sent it — is what keeps a server from making
+anybody appear to type. EDUs about a room are subject to its server ACL.
+
+The sync API hands EDUs over through `syncapi.Outbound`, and only for what
+a local user did: the same calls are how EDUs from other servers come in,
+and passing those on would echo them back.
+
+## Server ACLs
+
+`m.room.server_acl` is enforced against the origin of everything about a
+room: its PDUs, its typing and receipts, its joins and leaves, and its
+reads of the room's history. Globs match case-insensitively, without the
+port; deny wins over allow; an empty allow list allows nobody; and
+`allow_ip_literals: false` refuses servers named by an IP address. An ACL
+never applies to the server enforcing it.
+
+## What other servers may read
+
+`/event`, `/state`, `/state_ids`, `/event_auth`, `/backfill` and
+`/get_missing_events` are answered for a server with a user joined to the
+room, or for any server if the room is world-readable — and only if the
+room's ACL allows it. Anybody else gets `403` whether or not the event
+exists.
+
+`/state_ids` and `/state` answer with the state *before* the event asked
+about, which is what every implementation's client side expects; the
+asker adds the event itself if it is a state event. An outlier has no
+snapshot, and asking for the state at one is a `404`.
+
+## Endpoints
+
+| | |
+|---|---|
+| `GET /_matrix/key/v2/server` · `POST /_matrix/key/v2/query` · `GET /_matrix/key/v2/query/{server}` | keys; the notary answers for this server only |
+| `GET /_matrix/federation/v1/version` | |
+| `PUT /v1/send/{txnId}` | transactions of PDUs and EDUs |
+| `GET /v1/make_join/{room}/{user}` · `PUT /v1/send_join`, `/v2/send_join` | joins |
+| `GET /v1/make_leave/{room}/{user}` · `PUT /v1/send_leave`, `/v2/send_leave` | leaves |
+| `PUT /v2/invite/{room}/{event}` | invites |
+| `GET /v1/event/{event}` · `/v1/state/{room}` · `/v1/state_ids/{room}` · `/v1/event_auth/{room}/{event}` | reads |
+| `GET /v1/backfill/{room}` · `POST /v1/get_missing_events/{room}` | history |
+| `GET /v1/query/directory` · `/v1/query/profile` | queries |
+| `GET`, `POST /v1/publicRooms` | the public room directory |
+
+Client-Server endpoints that now reach other servers: `/join` and
+`/rooms/{id}/join` for a room this server is not in (by ID or by remote
+alias), `/invite` for a remote user, `/leave` to reject a remote invite,
+`GET /directory/room/{alias}` for a remote alias, `/publicRooms?server=`,
+and `GET /profile/{user}` for a remote user.
+
+## Known gaps
+
+- **Backfill only fills in history below the room.** A gap higher up — a
+  live event whose prev_events were closed with `/state_ids` rather than
+  fetched — stays a gap. And the end of a server's history is not
+  remembered: a room whose servers cannot supply anything before its
+  oldest event is asked again each time a client scrolls to the top.
+- **Backfilled redactions are not applied** to earlier backfilled events;
+  a server sends a redacted event in its redacted form anyway, but
+  `unsigned.redacted_because` is not filled in.
+- **No partial-state joins** (MSC3706). Joining a room of tens of
+  thousands of members means receiving, verifying and checking its whole
+  state before the join completes, which takes minutes.
+- **No key rotation.** The key file holds one key; old keys are not
+  served as `old_verify_keys`, so rotating the key would invalidate this
+  server's old events for anybody who has not cached the old key.
+- **Rejected events are not remembered.** A rejected event is not stored,
+  so if it is sent again it is fetched and judged again.
+- **Soft-failed events are never re-evaluated**, and a redaction whose
+  target arrives later is not applied when the target arrives.
+- **Room versions 1 and 2** — event IDs carried in the event — are not
+  supported, so the oldest rooms on the network cannot be joined.
+- **Presence and device list EDUs** are ignored on the way in and not sent
+  (presence is off by default; device lists are Phase 7).
+- **Third-party invites** across servers (`exchange_third_party_invite`)
+  and **knocking** over federation (`make_knock`, `send_knock`) are not
+  implemented.
+
+## Testing
+
+| Where | What |
+|---|---|
+| `internal/federation` | X-Matrix parsing (fuzzed) and signing, discovery against fake DNS and a fake well-known server, the key ring against a fake key server, an authenticated request end to end over TLS, the client refusing while disabled or denied. |
+| `internal/roomserver/federated_test.go` | Two and three roomservers in one process, wired to each other's federation methods: joins at versions 10–12, conversation and fork merging, fail-over, refused joins, invites with countersignatures, rejection with and without the inviter reachable, missing events, a gap closed with state, forged and soft-failed events, ACLs, remote aliases and directories, the switch. `backfill_test.go`: a server that joined late scrolling back through the whole room at versions 10–12, the state inside the history, backfill with the switch off and under `joined` visibility. |
+| `internal/federationapi` | The queue: fan-out and relay rules, backoff and retry, transactions within the limits, a restart with a backlog, the policy, EDUs. |
+| `internal/setup/phase5_test.go` | Two complete servers over HTTPS: a join by remote alias, messages both ways through the queues, a remote profile, federation switched off with a message held and delivered after it is switched back on, the admin view of it; the TLS listener and the module following the switch. |
+| Complement | `make complement-federation` — see [server/test/complement](../server/test/complement/README.md). |

docs/rooms.md

@@ -2,9 +2,9 @@
 
 `internal/roomserver` and the room half of `internal/storage`: PLAN.md
 Phase 3. This is what turns the event core of Phase 2 into rooms a client
-can use — locally. Nothing here talks to another server; federation is
-Phase 5, and every operation that would need it is refused with a message
-that says so rather than a bare `M_FORBIDDEN`.
+can use. Phase 3 built it local only; Phase 5 taught the same pipeline to
+take in events from other servers and to join, leave and invite across
+them, which [federation.md](federation.md) describes.
 
 ## Where the decisions live
 
@@ -65,7 +65,7 @@ With that lock, a Phase 3 room cannot fork, so the resolution path is
 unreachable from outside. It is implemented anyway because the alternative
 is a pipeline that authorises against one arbitrary branch of a fork
 without saying so — and because `stateAfter`, which it needs, is also what
-federation will need in Phase 5.
+federation needs, and since Phase 5 uses.
 
 ### Soft failure
 
@@ -107,7 +107,9 @@ handed out by a counter row inside the write transaction. Neither
 `AUTOINCREMENT` nor a sequence would do: the two engines spell those
 differently, and a PostgreSQL sequence is explicitly not transactional, so a
 rolled-back write would leave a hole that `/sync` would have to be taught to
-skip.
+skip. History backfilled from other servers is numbered by a second counter
+that runs downwards from -1, so that it sorts below everything already
+held; see [federation.md](federation.md#history-backfill).
 
 **State snapshots** are stored per event and deduplicated by a hash of their
 contents. That is what makes a per-event snapshot affordable: every message
@@ -136,10 +138,10 @@ PLAN.md section 3, enforced here:
 
 | Operation | While federation is off |
 |---|---|
-| `/join` a room this server does not have | `403`, naming federation |
-| Inviting `@user:other.example` | `403`, naming federation |
-| Resolving `#alias:other.example` | `403`, naming federation |
-| `publicRooms?server=other.example` | `403`, naming federation |
+| `/join` a room this server is not in | `403`, naming federation; with it on, a join through a server that is |
+| Inviting `@user:other.example` | `403`, naming federation; with it on, the invite is countersigned by their server |
+| Resolving `#alias:other.example` | `403`, naming federation; with it on, the owning server is asked |
+| `publicRooms?server=other.example` | `403`, naming federation; with it on, the other server is asked |
 | Room and event IDs | unchanged — a room created offline federates normally if federation is later turned on |
 | Signing | unchanged — every event is signed with the server key |
 
@@ -147,10 +149,11 @@ The messages are explicit on purpose. `M_FORBIDDEN` alone sends a user to
 their operator and the operator hunting through power levels.
 
 Restricted joins are the interesting case. The rule needs a signature from
-the authorising user's server checked, so it normally needs a key fetcher —
-but while federation is off every room is local, this server's own key is in
-hand, and the rule can be applied in full. A join authorised via a *remote*
-user is refused, like everything else that would need another server.
+the authorising user's server checked. This server's own signatures are
+checked against the key in hand, so a restricted join authorised by a local
+user works whether or not federation is on; one authorised by a remote user
+needs the key ring, and while federation is off it is refused like
+everything else that would need another server.
 
 ## Known gaps
 

docs/sync.md

@@ -71,6 +71,11 @@ The spec says only that the token is opaque, and clients treat it that way.
 It is worth being readable anyway: a token is the first thing in every bug
 report about `/sync`.
 
+The PDU position is the one that can be negative: `/messages` pages back
+into history backfilled from other servers, which is numbered downwards
+from -1 (see [federation.md](federation.md#history-backfill)), and its
+tokens name positions there — `s-12`. The other streams start at zero.
+
 `/messages` and a room's `prev_batch` hand out a PDU-only token. Both are
 about room events alone, and a token from either is a valid `?since=` for
 `/sync`, where the other streams then read as unseen — the conservative
@@ -112,6 +117,12 @@ type TimelineRequest struct {
 Low exclusive and High inclusive, **in both directions**. Direction decides
 only which end the limit trims and which order the store reads in.
 
+Zero is "unbounded" at either end, and is never a position: live events
+count up from 1 and backfilled history counts down from -1. So "is this
+bound set" is `!= 0`, never `> 0`, and the position just below an event is
+`storage.PositionBefore` rather than a subtraction — below 1 is -1, because
+a token of zero handed back as a High bound would read the whole room again.
+
 Spelling the bounds as "from" and "to" instead — which is how the
 Client-Server API spells them — would mean each of them being the low bound
 in one direction and the high bound in the other, which is exactly the
@@ -354,8 +365,8 @@ for a key share is both useless and a leak.
 A message to a user of another server is dropped rather than refused. A
 client encrypting for a room with remote members sends one message per
 device in one request, and failing the request because federation is off
-would stop it from delivering to the local devices as well. Phase 5 routes
-them.
+would stop it from delivering to the local devices as well. With federation
+on, they go to the federation queue as `m.direct_to_device` EDUs instead.
 
 ### Presence
 
@@ -391,7 +402,7 @@ no timestamp and have to decide what that meant.
 |---|---|---|
 | `unread_notifications`, `unread_thread_notifications` | push rules | 8 |
 | `device_lists.changed` / `.left`, `device_one_time_keys_count` | the keyserver | 7 |
-| Remote presence, remote receipts, remote typing | federation | 5 |
+| Remote presence | presence over federation | — |
 | Threads, and threaded receipts beyond storing `thread_id` | threads | 10 |
 | `contains_url`, `event_fields` | nothing; they are parsed and ignored | — |
 | Per-event history visibility | state storage (open question 1) | — |