Phase 4: sync

Implements PLAN.md Phase 4: the read model. /sync, /messages and /context,
the filters that narrow them, and the streams a response carries that are
not room events — typing, receipts, presence and to-device messages. With
this a client can be used against Warp, which is milestone M1.

The one architectural choice is that the syncapi has no copy of the rooms:
it reads them through the roomserver. Dendrite gives its syncapi its own
tables and feeds them from the output stream, which is right for a design
where the two can be separate processes, and Dendrite's could be. Warp is
one process by decision, and there the copy buys nothing and costs the
hardest class of bug a homeserver has — a read model that disagrees with
the room, where a client is shown a message that was rejected and the only
way to find out is to compare two tables. So the roomserver grew a read
model (stream.go, roomsync.go, visibility.go) rather than exporting its
store: reading an event means parsing it under its room's version, and a
second copy of the version registry, the parser and the client projection
would be a second place for them to disagree.

syncapi: stream tokens naming a position in each of five independent
streams, so a client that has seen receipt 7 but only event 4 can say so in
one opaque string. A token with fewer positions than this build writes is
accepted and the rest read as zero, which is what lets a token stored
before a stream existed go on working. The PDU position comes from
MAX(stream_ordering) and not from the counter row the other four use: the
counter is advanced inside the writing transaction, so a token taken from
it would name a position whose event no reader can see yet, and the event
would be skipped for good once that transaction committed.

Long polling goes through a notifier that wakes by room and by user rather
than globally — an event wakes the listeners of its room, a membership
event also wakes the user it names, so an invite reaches a client that is
not in the room yet. A wake-up is only ever a hint: the listener re-reads
the streams and decides for itself what changed, which is what makes it
safe for the notifier to hold no state about what each client has seen. The
listener registers before the read, so an event landing in between leaves
the channel signalled and the sync reads again rather than blocking on a
change it just missed. EventPersisted must not block, because it is called
with the room's write lock still held.

"Something happened" is not "a token moved". A stream can move without this
user being concerned by it, and returning on every such move would turn a
long poll back into a busy loop. The same trap caught presence twice over:
a client says it is still online by syncing, so ?set_presence= makes every
sync a presence write, and a stream that moved each time would wake every
peer, which would sync, which would move it again. SetPresence therefore
reports whether anything a peer would notice actually changed, and a write
that only refreshes the last-active timestamp leaves the stream alone.

Timeline windows are named by their two ends and a direction, Low exclusive
and High inclusive in both directions, rather than by "from" and "to" as
the wire format spells them. Each of those would otherwise be the low bound
in one direction and the high bound in the other, which is exactly the
off-by-one every pagination bug is made of; the handlers translate once and
nothing below them thinks about it again. A sync's timeline is read
backwards and reported forwards, because a client wants the newest events
of the window it missed — a forward read with a limit hands it the oldest
and then calls the rest a gap.

visibility: history visibility is now a span rather than a yes or no, which
is what Phase 3 deferred here. A user who has left a room is served the
room as it was when they left — their state reads, their members and their
timeline — instead of being refused, and is bounded above at their own
membership event so a kicked client cannot go on streaming. Their span is
found from their own member events, so a user who joined, left and rejoined
sees their latest span. ?at= on /members is honoured for the same reason.
What is left is that the span is computed from the room's current
history_visibility where the spec judges each event by the one in force
when it was sent; that is a state read per event and belongs with open
question 1.

Typing is held in memory and nowhere else: notifications expire in seconds,
are worthless across a restart, and arrive one per keystroke-burst per
room, so persisting them would be the busiest table in the database holding
the least valuable data in it. Receipts only ever advance, compared on the
position of the event acknowledged rather than on arrival, because two of a
user's clients read at different speeds and the slower one must not drag
the marker backwards. To-device queues are per device and drained by the
next sync rather than by the one that delivered them, so a client that
crashes in between gets the batch again — the right way round, since a lost
key share leaves a message permanently undecryptable and a duplicate is
ignored. A wildcard addressee is expanded when sent, not stored, because
"every device" means the ones that existed then.

Presence is off by default, which answers PLAN.md open question 3. It is
the noisiest thing in Matrix and the one genuinely fan-out notification
here; a private homeserver gains little from it. With it off the endpoints
refuse rather than disappearing, because a client that gets a 404 from
/presence concludes the server is too old to have it and stops asking.

storage: one migration per driver adding the receipt, to-device, presence
and filter tables with a counter row each, and a stream position on the
existing account data table rather than a parallel one — there is exactly
one current value per (user, room, type), and a second table would be a
second thing to keep in step. Account data is reported in an event list but
is not an event, so it gets its own response type: a room event with an
empty sender and no timestamp would make a client decide what that meant.

Deliberately missing, each named in docs/sync.md: notification counts (they
need push rules, Phase 8), device lists (the keyserver, Phase 7), anything
remote (Phase 5), threads (Phase 10). device_lists is sent as two empty
lists because a client reads a missing one as malformed; the notification
counts are left out entirely, because a zero there is a claim and an absent
field is not.

Tested through HTTP rather than against the builder, because what is worth
pinning down is the response a client sees — which section a room lands in,
whether a token handed back can be handed in again, whether paging
terminates and loses nothing. The storage tests run against both SQL
drivers; the notifier has its own tests for the concurrent case. A live
server was driven end to end, which is what caught the two bugs the unit
tests had missed: a backwards /messages page must be in reverse
chronological order, and `start` must name where the page began rather than
position zero when ?from= was omitted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/sync.md

@@ -0,0 +1,457 @@
+# Sync
+
+`internal/syncapi` and the stream half of `internal/storage`: PLAN.md
+Phase 4. This is the read model — everything a client learns after it has
+logged in. `/sync` for the live stream, `/messages` for a room's history,
+`/context` for one event and what surrounds it, plus the streams that are
+not room events at all: typing, receipts, presence and to-device messages.
+
+With this, Warp is a homeserver a client can be used against. Milestone M1.
+
+## The one architectural choice
+
+**The syncapi has no copy of the rooms.** It reads them through the
+roomserver.
+
+This is a departure from Dendrite, which gives its syncapi its own tables
+and feeds them from the roomserver's output stream. That design is right
+for a system where the two can be separate processes, and Dendrite's was.
+Warp is one process by decision (PLAN.md section 2), and in one process the
+copy buys nothing and costs the hardest class of bug a homeserver has: a
+read model that disagrees with the room. A client would be shown a message
+that was rejected, or not shown one that was accepted, and the only way to
+find out which is to compare two tables.
+
+So the split is:
+
+| Question | Answered by |
+|---|---|
+| What events are in this window of this room? | `roomserver` |
+| What state did the room have at this position? | `roomserver` |
+| May this user see any of it? | `roomserver` — see *Visibility* |
+| Where has this client got to? | `syncapi` — the token |
+| What did it ask to be left out? | `syncapi` — the filter |
+| Who is typing, who has read what, who is online? | `syncapi` |
+| Should this request block, and for how long? | `syncapi` — the notifier |
+
+The roomserver grew a read model for this (`stream.go`, `roomsync.go`,
+`visibility.go`) rather than exporting its store, because reading an event
+means parsing it under its room's version — and a second copy of the room
+version registry, the event parser and the client projection would be a
+second place for them to disagree.
+
+## Stream tokens
+
+A sync response is assembled from five streams that move independently:
+
+| Stream | Moves when | Position from |
+|---|---|---|
+| PDU | an event is persisted | `MAX(stream_ordering)` in `roomserver_events` |
+| receipt | somebody sends a read receipt | `sync_sequences` |
+| to-device | a device is sent a message | `sync_sequences` |
+| account data | a user changes a setting | `sync_sequences` |
+| presence | somebody's presence actually changes | `sync_sequences` |
+
+A client that has seen receipt 7 but only event 4 has to be able to say so
+in the one opaque string the API gives it, so a token names a position in
+each:
+
+```
+s{pdu}_{receipt}_{to_device}_{account_data}_{presence}
+```
+
+Trailing zeroes are left off, so a client that has only ever seen room
+events carries `s12` rather than `s12_0_0_0_0`. A token with fewer
+positions than this build writes is accepted and the missing ones read as
+zero — which is what lets a token stored before a stream existed go on
+working: the client is sent that stream in full once and is in step
+afterwards.
+
+The spec says only that the token is opaque, and clients treat it that way.
+It is worth being readable anyway: a token is the first thing in every bug
+report about `/sync`.
+
+`/messages` and a room's `prev_batch` hand out a PDU-only token. Both are
+about room events alone, and a token from either is a valid `?since=` for
+`/sync`, where the other streams then read as unseen — the conservative
+direction, since the client is sent those streams again rather than having
+them skipped.
+
+### Why the PDU position comes from the events table
+
+Not from a counter row, even though the other four do. The counter is
+advanced *inside* the writing transaction, so it is already at the new
+value while that transaction is still open. A token taken from it would
+name a position whose event no reader can see yet — and the event would be
+skipped for good once the transaction committed.
+
+`MAX(stream_ordering)` is only ever the position of an event that is
+actually visible.
+
+### Why the counters are rows and not sequences
+
+The same reason as the roomserver's (see [rooms.md](rooms.md)): an `UPDATE`
+takes a row lock both engines hold until the transaction ends, so a
+position handed out by a transaction that then rolls back is never used.
+PostgreSQL sequences are explicitly not transactional, and a gap in a
+stream is something every reader of it would have to be taught to skip.
+
+## Windows, and the bounds that name them
+
+Every timeline read is a window named by its two ends and a direction:
+
+```go
+type TimelineRequest struct {
+    Low  int64             // exclusive
+    High int64             // inclusive
+    Direction TimelineDirection
+    Limit int
+}
+```
+
+Low exclusive and High inclusive, **in both directions**. Direction decides
+only which end the limit trims and which order the store reads in.
+
+Spelling the bounds as "from" and "to" instead — which is how the
+Client-Server API spells them — would mean each of them being the low bound
+in one direction and the high bound in the other, which is exactly the
+off-by-one that every pagination bug is made of. The handlers translate the
+client's `from` and `to` into Low and High once, and nothing below them has
+to think about it again.
+
+A position is somewhere a client has got to, not an event it still wants,
+which is why Low is exclusive: a token can be handed straight back without
+re-reading the event it names.
+
+### Why a sync's timeline is read backwards
+
+A client wants the *newest* events of the window it missed — the bottom of
+the room, which is what it is looking at. A forward read with a limit would
+hand it the oldest of the window and then call the rest a gap. So
+`RoomSync` reads backwards from the top of the window and reports the
+result oldest first, which is the order a client renders.
+
+`Timeline.Start` sits one below the oldest event returned and `End` is at
+the newest, so either can be handed back as the opposite bound and the next
+page begins exactly where this one stopped. `prev_batch` is `Start`.
+
+### `limited`, and what `state` means
+
+The spec's rule is that a client which applies `state` and then `timeline`
+ends up with the room's state at the end of the timeline. That gives three
+cases, which `roomserver.syncState` implements:
+
+- **Initial sync, or `?full_state=true`** — the whole state as of the start
+  of the timeline.
+- **Incremental, timeline reaches back to `since`** — nothing. Every state
+  change in between is a timeline event the client is already being given.
+- **Incremental, timeline cut short (a gap)** — what changed between
+  `since` and where the timeline starts. Those changes are not in the
+  timeline and the client would otherwise never see them.
+
+A timeline that reaches back to the room's create event reports no state at
+all: there is nothing before it.
+
+## Visibility
+
+`roomserver.VisibleRange` answers "what span of this room may this user
+read" once per room, rather than filtering event by event. The rules are
+the spec's history visibility:
+
+| `history_visibility` | Who, and from when |
+|---|---|
+| `world_readable` | anybody, all of it |
+| `shared` | anybody who is or has been a member, all of it |
+| `invited` | a member, from their invite onwards |
+| `joined` | a member, from their join onwards |
+
+A user who has left or was banned is bounded **above** at their own
+membership event in every case. That is what stops a kicked user's client
+from going on streaming the room, and it is why the range is a range: their
+view of the room has an end as well as a beginning.
+
+An invitee gets no timeline. They are told enough to decide — the stripped
+state that travels with the invite — and nothing more.
+
+Phase 3 refused a user who had left a room outright and said in a comment
+that serving them the state as of their leave belonged here. It does now:
+`GET /rooms/{id}/state`, `/state/{type}`, `/members` and `/messages` all
+serve a departed member from the state at their departure, through
+`StateAtStream`.
+
+A user's membership history is read from their own member events
+(`UserMembershipEvents`), so a user who joined, left and rejoined sees
+their latest span and not the earlier ones. There is one row per membership
+change of one user in one room, so this stays small however busy the room
+is.
+
+### The known gap
+
+The range is computed from the room's **current** history visibility. The
+spec judges each event by the visibility in force when it was sent, so a
+room that was once `world_readable` and is now `joined` hides history here
+that the spec would show, and the reverse shows history it would hide.
+
+Doing it exactly means resolving the history visibility at every event in
+the window — a state read per event — and belongs with the state storage
+work of PLAN.md open question 1. Until then this follows the setting the
+room's members last agreed on, which is the one they would expect to apply.
+
+## Long polling
+
+`/sync` holds the request open until something happens. Polling the
+database instead would mean every connected client querying every second,
+which is the cost the notifier avoids.
+
+`syncapi.Notifier` is the other half of the roomserver's emit stage — the
+seam Phase 3 left. The same object is the roomserver's `Notifier` and the
+syncapi's wake-up, wired together in `setup.New`.
+
+Waking is **by room and by user**, not globally:
+
+- An event wakes the listeners of its room.
+- A membership event also wakes the listener of the user it names, which is
+  how an invite reaches a client that is not in the room yet and so is not
+  listening for it.
+- Receipts and typing wake a room; to-device, account data and presence
+  wake a user.
+- An event in a room nobody is polling wakes nobody.
+
+A wake-up is only ever a hint. The listener re-reads the streams and works
+out for itself what changed, so a spurious wake-up costs one query and a
+missed one costs at most the sync timeout. That is what makes it safe for
+the notifier to hold no state about what each client has seen.
+
+The listener is registered **before** the streams are read. Anything that
+lands between the two leaves the channel already signalled, so the wait
+returns at once and the sync reads again rather than blocking on a change
+it has just missed.
+
+`EventPersisted` must not block: it is called with the room's write lock
+still held, so anything slow there would serialise the room's writers
+behind the slowest reader of it. It is a non-blocking send onto a one-slot
+channel and nothing else.
+
+### What counts as "something happened"
+
+Not "a token moved". A stream can move without this user being concerned by
+it — somebody else's receipt in a room they are not in — and a sync that
+returned on every such move would turn a long poll back into a busy loop.
+So the test is whether the response has anything in it, with an initial
+sync as the exception: a client that has just logged in needs an answer
+whatever is in it.
+
+The same trap caught presence. A client says it is still online by syncing,
+so `?set_presence=` makes every sync a presence write; a stream that moved
+each time would wake every peer's long poll, which would sync, which would
+move it again. `SetPresence` therefore reports whether anything a peer
+would notice actually changed, and a write that only refreshes the
+last-active timestamp leaves the stream where it was.
+
+## Filters
+
+A filter is uploaded once and referred to by ID on every later sync, which
+is the point of it: the filter a client wants is long and does not change,
+so sending it on every request would be the largest part of every request.
+`?filter=` also accepts an inline filter as JSON.
+
+What Warp acts on:
+
+- `room.timeline.limit`, `types`, `not_types`, `senders`, `not_senders` —
+  pushed into SQL, so a filtered page is a full page rather than a short
+  one that looks like the end of the room. A trailing `*` is the spec's
+  wildcard; a `%` or `_` in a pattern is matched literally.
+- `room.state` — the same fields, applied in Go: a state read is a set of
+  tuples, not a window of a stream, so there is no index to push it to.
+- `room.rooms` / `not_rooms`, at both levels.
+- `room.timeline.lazy_load_members`.
+- `room.include_leave`.
+- `presence` and `account_data` types and senders.
+
+What it parses and does not act on: `event_fields`, `contains_url`,
+`unread_thread_notifications`. A field Warp ignores is **not** an error — a
+server is allowed to return a superset of what a filter asked for, and
+refusing the filter is the one outcome that would stop a client syncing at
+all. `event_format: federation` *is* refused: that is a server's view of an
+event and the Client-Server API has no business handing it out.
+
+A stored filter is returned byte for byte. A client that compares what it
+gets back with what it sent would otherwise see a difference that is not
+there, because the fields this build does not act on would come back
+reordered or missing.
+
+### Lazy loading
+
+`lazy_load_members` drops the member events from a room's state and puts
+back only the senders who appear in the timeline being returned. It is what
+makes a room with ten thousand members openable.
+
+The spec also lets a server leave out members it has already sent a
+particular device, which would need per-device bookkeeping of what each one
+has seen. Warp sends them every time instead: that is the behaviour
+`include_redundant_members: true` asks for, so no client can be surprised
+by it, and it costs one state read rather than a table.
+
+## The streams that are not room events
+
+### Typing
+
+In memory, and nowhere else. Typing notifications expire in seconds, they
+are worthless after a restart — nobody is still typing the message they
+were typing when the server went down — and they arrive one per
+keystroke-burst per room. Writing them to disk would be the busiest table
+in the database holding the least valuable data in it.
+
+The cost: a restart clears them, which a client cannot tell from everybody
+having stopped typing, and a second process would not see them. The second
+only matters when Warp stops being one process, which PLAN.md says is not
+the plan.
+
+They therefore have no stream position. A notification that started and
+stopped between two syncs is simply never reported; one that is live is
+reported on every sync until it expires.
+
+### Receipts
+
+Only the latest receipt of each (user, type, thread) in a room is kept: a
+receipt supersedes the one before it, and `/sync` reports the current one
+rather than a history.
+
+A receipt that would move **backwards** is dropped. Two of a user's clients
+read a room at different speeds, and letting the slower one win would make
+the unread marker jump backwards on the other — so the comparison is on the
+stream position of the event being acknowledged, not on when the receipt
+arrived.
+
+`m.read.private` is filtered on the way out rather than in SQL: the query
+is the same one the public receipts come from, and a `WHERE` clause naming
+the asking user would turn one query per sync into two.
+
+`m.fully_read` is not a receipt at all — it is the read marker, stored as
+room account data. `POST /read_markers` takes all three in one body and
+performs up to three independent writes. They are not atomic and cannot
+usefully be: the marker is account data and the receipts are the receipt
+stream, which are two streams with two positions. A failure partway through
+leaves the earlier writes in place, which is the same state a client would
+reach by sending them separately and losing its connection — and which the
+next call corrects.
+
+### To-device messages
+
+A queue per device, drained by the device acknowledging a later sync token
+rather than by reading it. A device acknowledges the batch it was sent by
+coming back with the token that carried it, so the deletion happens on the
+*next* sync. A client that crashes between the two gets the batch again,
+which is the right way round: a lost key share leaves a message
+permanently undecryptable, and a duplicate one is ignored.
+
+A wildcard addressee (`"*"`) is expanded when the message is sent, not
+stored as a wildcard. The set of a user's devices changes, and a message
+addressed to "every device" means the ones that existed when it was sent —
+storing the wildcard would deliver it to a device created afterwards, which
+for a key share is both useless and a leak.
+
+A message to a user of another server is dropped rather than refused. A
+client encrypting for a room with remote members sends one message per
+device in one request, and failing the request because federation is off
+would stop it from delivering to the local devices as well. Phase 5 routes
+them.
+
+### Presence
+
+Off by default — PLAN.md open question 3, now answered. See
+[configuration.md](configuration.md#presence) for what the switch does.
+
+It is the one genuinely fan-out notification in the component: a user's
+presence change is published to everybody they share a room with, which is
+walked as "their rooms, and the members of each". That is the reason it is
+a switch.
+
+`last_active_ago` is computed on the way out rather than stored, for the
+same reason `age` is on an event: a value written down at write time would
+be wrong by however long the row sat in the database.
+
+## Account data
+
+Account data gained a `stream_pos` column on the existing table rather than
+a parallel one: there is exactly one current value per (user, room, type),
+and a second table would be a second thing to keep in step with it. The
+position is stamped inside the same transaction as the write, so a client
+is never told the stream moved and then handed the old value.
+
+It is reported in an event list but it is **not** an event: it has a type
+and a content and nothing else — no sender, no ID, no room. Warp gives it
+its own response type for that reason. Reporting it as a room event with
+empty fields would be worse: a client would see an event with no sender and
+no timestamp and have to decide what that meant.
+
+## What is deliberately missing
+
+| Thing | Needs | Phase |
+|---|---|---|
+| `unread_notifications`, `unread_thread_notifications` | push rules | 8 |
+| `device_lists.changed` / `.left`, `device_one_time_keys_count` | the keyserver | 7 |
+| Remote presence, remote receipts, remote typing | federation | 5 |
+| Threads, and threaded receipts beyond storing `thread_id` | threads | 10 |
+| `contains_url`, `event_fields` | nothing; they are parsed and ignored | — |
+| Per-event history visibility | state storage (open question 1) | — |
+| Simplified Sliding Sync (MSC4186) | — | 10 |
+
+`device_lists` is sent as two empty lists rather than left out: a client
+reads a missing one as a malformed response, where two empty lists say
+"nothing changed". The notification counts are left out entirely instead,
+because a zero there is a claim — "you have nothing unread" — and an absent
+field is not.
+
+## Endpoints
+
+| Endpoint | Notes |
+|---|---|
+| `GET /sync` | Long-polls. `?since=`, `?filter=`, `?timeout=`, `?full_state=`, `?set_presence=`. |
+| `GET /rooms/{id}/messages` | `?dir=` (`b` default), `?from=`, `?to=`, `?limit=`, `?filter=`. No `end` at the end of the room. |
+| `GET /rooms/{id}/context/{eventId}` | `?limit=`, `?filter=`. Half the budget each side, the odd one backwards. |
+| `POST /user/{userId}/filter` | Returns an opaque ID. |
+| `GET /user/{userId}/filter/{filterId}` | Byte for byte as uploaded. |
+| `PUT /rooms/{id}/typing/{userId}` | Caller only. Requires membership. |
+| `POST /rooms/{id}/receipt/{type}/{eventId}` | `m.read`, `m.read.private`, or `m.fully_read`. |
+| `POST /rooms/{id}/read_markers` | Up to three writes. |
+| `PUT /sendToDevice/{type}/{txnId}` | The transaction ID is accepted and not recorded; see below. |
+| `GET` / `PUT /presence/{userId}/status` | `PUT` is caller only. `403` when presence is off. |
+
+The `sendToDevice` transaction ID is accepted and not used. Its purpose is
+idempotency, and a to-device message is already idempotent in the way that
+matters: a duplicate is a second copy of a key share, which the receiving
+client discards. Recording it would mean a table whose only reader would be
+a retry that costs nothing anyway.
+
+Typing, presence and the receipts are the caller's own claims, not data
+belonging to them, so unlike a profile an admin may not set them for
+somebody else: there is no administrative reason to say that another user
+is typing.
+
+## Testing
+
+| Level | Where |
+|---|---|
+| Tokens and filters | `internal/syncapi/{token,filter}_test.go` — the parsing edges, which is where a client's stored token meets a newer server |
+| Stream reads, the sync store | `internal/storage/sync_test.go`, against every driver |
+| Visibility, windows, summaries | `internal/roomserver/stream_test.go` |
+| The responses a client sees | `internal/clientapi/{sync,messages,ephemeral}_test.go`, through HTTP |
+| Wiring | `internal/setup/phase4_test.go` — the whole stack, to catch an endpoint written but never registered |
+| Integration | Complement: `TestSync*`, `TestMessages*`, `TestContext*`, `TestFilter*`, `TestTyping*`, `TestReceipt*`, `TestToDevice*`, `TestLazyLoading*` |
+
+The HTTP tests are deliberately not unit tests of the builder. What is
+worth pinning down is the response a client sees — which section a room
+lands in, whether a token handed back can be handed in again, whether
+paging terminates and loses nothing — and none of that is visible from
+inside.
+
+## Known cost
+
+A sync over K changed rooms is roughly a constant number of queries per
+room: one `RoomSync` (room row, summary, membership history, timeline,
+state) plus the per-room ephemeral and account-data reads. On an incremental
+sync K is usually zero or one, which is the case that matters. An initial
+sync of a user in fifty rooms is fifty times that, and would be worth a
+batched read if it ever shows up in a profile — the shape of the code does
+not stop it, since `changedRooms` already decides the set up front.