Warp/docs/sync.md

# Sync

`internal/syncapi` and the stream half of `internal/storage`: PLAN.md
Phase 4. This is the read model — everything a client learns after it has
logged in. `/sync` for the live stream, `/messages` for a room's history,
`/context` for one event and what surrounds it, plus the streams that are
not room events at all: typing, receipts, presence and to-device messages.

With this, Warp is a homeserver a client can be used against. Milestone M1.

## The one architectural choice

**The syncapi has no copy of the rooms.** It reads them through the
roomserver.

This is a departure from Dendrite, which gives its syncapi its own tables
and feeds them from the roomserver's output stream. That design is right
for a system where the two can be separate processes, and Dendrite's was.
Warp is one process by decision (PLAN.md section 2), and in one process the
copy buys nothing and costs the hardest class of bug a homeserver has: a
read model that disagrees with the room. A client would be shown a message
that was rejected, or not shown one that was accepted, and the only way to
find out which is to compare two tables.

So the split is:

| Question | Answered by |
|---|---|
| What events are in this window of this room? | `roomserver` |
| What state did the room have at this position? | `roomserver` |
| May this user see any of it? | `roomserver` — see *Visibility* |
| Where has this client got to? | `syncapi` — the token |
| What did it ask to be left out? | `syncapi` — the filter |
| Who is typing, who has read what, who is online? | `syncapi` |
| Should this request block, and for how long? | `syncapi` — the notifier |

The roomserver grew a read model for this (`stream.go`, `roomsync.go`,
`visibility.go`) rather than exporting its store, because reading an event
means parsing it under its room's version — and a second copy of the room
version registry, the event parser and the client projection would be a
second place for them to disagree.

## Stream tokens

A sync response is assembled from seven streams that move independently:

| Stream | Moves when | Position from |
|---|---|---|
| PDU | an event is persisted | `MAX(stream_ordering)` in `roomserver_events` |
| receipt | somebody sends a read receipt | `sync_sequences` |
| to-device | a device is sent a message | `sync_sequences` |
| account data | a user changes a setting | `sync_sequences` |
| presence | somebody's presence actually changes | `sync_sequences` |
| device list | somebody's end-to-end keys change ([e2ee.md](e2ee.md)) | `keyserver_sequences` |
| notification | a user's unread counts in a room move ([push.md](push.md)) | `push_sequences` |

A client that has seen receipt 7 but only event 4 has to be able to say so
in the one opaque string the API gives it, so a token names a position in
each:

```
s{pdu}_{receipt}_{to_device}_{account_data}_{presence}_{device_list}_{notification}
```

Trailing zeroes are left off, so a client that has only ever seen room
events carries `s12` rather than `s12_0_0_0_0_0_0`. The device list and
notification positions came later (Phases 7 and 8) and were added at the
end. A token with fewer
positions than this build writes is accepted and the missing ones read as
zero — which is what lets a token stored before a stream existed go on
working: the client is sent that stream in full once and is in step
afterwards.

The spec says only that the token is opaque, and clients treat it that way.
It is worth being readable anyway: a token is the first thing in every bug
report about `/sync`.

The PDU position is the one that can be negative: `/messages` pages back
into history backfilled from other servers, which is numbered downwards
from -1 (see [federation.md](federation.md#history-backfill)), and its
tokens name positions there — `s-12`. The other streams start at zero.

`/messages` and a room's `prev_batch` hand out a PDU-only token. Both are
about room events alone, and a token from either is a valid `?since=` for
`/sync`, where the other streams then read as unseen — the conservative
direction, since the client is sent those streams again rather than having
them skipped.

### Why the PDU position comes from the events table

Not from a counter row, even though the other four do. The counter is
advanced *inside* the writing transaction, so it is already at the new
value while that transaction is still open. A token taken from it would
name a position whose event no reader can see yet — and the event would be
skipped for good once the transaction committed.

`MAX(stream_ordering)` is only ever the position of an event that is
actually visible.

### Why the counters are rows and not sequences

The same reason as the roomserver's (see [rooms.md](rooms.md)): an `UPDATE`
takes a row lock both engines hold until the transaction ends, so a
position handed out by a transaction that then rolls back is never used.
PostgreSQL sequences are explicitly not transactional, and a gap in a
stream is something every reader of it would have to be taught to skip.

## Windows, and the bounds that name them

Every timeline read is a window named by its two ends and a direction:

```go
type TimelineRequest struct {
    Low  int64             // exclusive
    High int64             // inclusive
    Direction TimelineDirection
    Limit int
}
```

Low exclusive and High inclusive, **in both directions**. Direction decides
only which end the limit trims and which order the store reads in.

Zero is "unbounded" at either end, and is never a position: live events
count up from 1 and backfilled history counts down from -1. So "is this
bound set" is `!= 0`, never `> 0`, and the position just below an event is
`storage.PositionBefore` rather than a subtraction — below 1 is -1, because
a token of zero handed back as a High bound would read the whole room again.

Spelling the bounds as "from" and "to" instead — which is how the
Client-Server API spells them — would mean each of them being the low bound
in one direction and the high bound in the other, which is exactly the
off-by-one that every pagination bug is made of. The handlers translate the
client's `from` and `to` into Low and High once, and nothing below them has
to think about it again.

A position is somewhere a client has got to, not an event it still wants,
which is why Low is exclusive: a token can be handed straight back without
re-reading the event it names.

### Why a sync's timeline is read backwards

A client wants the *newest* events of the window it missed — the bottom of
the room, which is what it is looking at. A forward read with a limit would
hand it the oldest of the window and then call the rest a gap. So
`RoomSync` reads backwards from the top of the window and reports the
result oldest first, which is the order a client renders.

`Timeline.Start` sits one below the oldest event returned and `End` is at
the newest, so either can be handed back as the opposite bound and the next
page begins exactly where this one stopped. `prev_batch` is `Start`.

### `limited`, and what `state` means

The spec's rule is that a client which applies `state` and then `timeline`
ends up with the room's state at the end of the timeline. That gives three
cases, which `roomserver.syncState` implements:

- **Initial sync, or `?full_state=true`** — the whole state as of the start
  of the timeline.
- **Incremental, timeline reaches back to `since`** — nothing. Every state
  change in between is a timeline event the client is already being given.
- **Incremental, timeline cut short (a gap)** — what changed between
  `since` and where the timeline starts. Those changes are not in the
  timeline and the client would otherwise never see them.

A timeline that reaches back to the room's create event reports no state at
all: there is nothing before it.

## Visibility

`roomserver.VisibleRange` answers "what span of this room may this user
read" once per room, rather than filtering event by event. The rules are
the spec's history visibility:

| `history_visibility` | Who, and from when |
|---|---|
| `world_readable` | anybody, all of it |
| `shared` | anybody who is or has been a member, all of it |
| `invited` | a member, from their invite onwards |
| `joined` | a member, from their join onwards |

A user who has left or was banned is bounded **above** at their own
membership event in every case. That is what stops a kicked user's client
from going on streaming the room, and it is why the range is a range: their
view of the room has an end as well as a beginning.

An invitee gets no timeline. They are told enough to decide — the stripped
state that travels with the invite — and nothing more.

Phase 3 refused a user who had left a room outright and said in a comment
that serving them the state as of their leave belonged here. It does now:
`GET /rooms/{id}/state`, `/state/{type}`, `/members` and `/messages` all
serve a departed member from the state at their departure, through
`StateAtStream`.

A user's membership history is read from their own member events
(`UserMembershipEvents`), so a user who joined, left and rejoined sees
their latest span and not the earlier ones. There is one row per membership
change of one user in one room, so this stays small however busy the room
is.

### The known gap

The range is computed from the room's **current** history visibility. The
spec judges each event by the visibility in force when it was sent, so a
room that was once `world_readable` and is now `joined` hides history here
that the spec would show, and the reverse shows history it would hide.

Doing it exactly means resolving the history visibility at every event in
the window — a state read per event — and belongs with the state storage
work of PLAN.md open question 1. Until then this follows the setting the
room's members last agreed on, which is the one they would expect to apply.

## Long polling

`/sync` holds the request open until something happens. Polling the
database instead would mean every connected client querying every second,
which is the cost the notifier avoids.

`syncapi.Notifier` is the other half of the roomserver's emit stage — the
seam Phase 3 left. The same object is the roomserver's `Notifier` and the
syncapi's wake-up, wired together in `setup.New`.

Waking is **by room and by user**, not globally:

- An event wakes the listeners of its room.
- A membership event also wakes the listener of the user it names, which is
  how an invite reaches a client that is not in the room yet and so is not
  listening for it.
- Receipts and typing wake a room; to-device, account data and presence
  wake a user.
- An event in a room nobody is polling wakes nobody.

A wake-up is only ever a hint. The listener re-reads the streams and works
out for itself what changed, so a spurious wake-up costs one query and a
missed one costs at most the sync timeout. That is what makes it safe for
the notifier to hold no state about what each client has seen.

The listener is registered **before** the streams are read. Anything that
lands between the two leaves the channel already signalled, so the wait
returns at once and the sync reads again rather than blocking on a change
it has just missed.

`EventPersisted` must not block: it is called with the room's write lock
still held, so anything slow there would serialise the room's writers
behind the slowest reader of it. It is a non-blocking send onto a one-slot
channel and nothing else.

### What counts as "something happened"

Not "a token moved". A stream can move without this user being concerned by
it — somebody else's receipt in a room they are not in — and a sync that
returned on every such move would turn a long poll back into a busy loop.
So the test is whether the response has anything in it, with an initial
sync as the exception: a client that has just logged in needs an answer
whatever is in it.

"Initial" means the client sent no `?since=`, not that its token is all
zeroes. On a server whose streams have not moved yet, `s0` is every
client's token, and taking it for a first sync made every long poll
return at once. The push rules a first sync carries are sent on the
client's first sync for the same reason, not on any sync whose account
data position is zero: before anybody had written account data, that was
every sync, and each one counted as news.

The same trap caught presence. A client says it is still online by syncing,
so `?set_presence=` makes every sync a presence write; a stream that moved
each time would wake every peer's long poll, which would sync, which would
move it again. `SetPresence` therefore reports whether anything a peer
would notice actually changed, and a write that only refreshes the
last-active timestamp leaves the stream where it was.

## Filters

A filter is uploaded once and referred to by ID on every later sync, which
is the point of it: the filter a client wants is long and does not change,
so sending it on every request would be the largest part of every request.
`?filter=` also accepts an inline filter as JSON.

What Warp acts on:

- `room.timeline.limit`, `types`, `not_types`, `senders`, `not_senders` —
  pushed into SQL, so a filtered page is a full page rather than a short
  one that looks like the end of the room. A trailing `*` is the spec's
  wildcard; a `%` or `_` in a pattern is matched literally.
- `room.state` — the same fields, applied in Go: a state read is a set of
  tuples, not a window of a stream, so there is no index to push it to.
- `room.rooms` / `not_rooms`, at both levels.
- `room.timeline.lazy_load_members`.
- `room.timeline.unread_thread_notifications`: the unread counts split by
  thread; see [push.md](push.md).
- `room.include_leave`.
- `presence` and `account_data` types and senders.

What it parses and does not act on: `event_fields`, `contains_url`. A field Warp ignores is **not** an error — a
server is allowed to return a superset of what a filter asked for, and
refusing the filter is the one outcome that would stop a client syncing at
all. `event_format: federation` *is* refused: that is a server's view of an
event and the Client-Server API has no business handing it out.

A stored filter is returned byte for byte. A client that compares what it
gets back with what it sent would otherwise see a difference that is not
there, because the fields this build does not act on would come back
reordered or missing.

### Lazy loading

`lazy_load_members` drops the member events from a room's state and puts
back only the senders who appear in the timeline being returned. It is what
makes a room with ten thousand members openable.

The spec also lets a server leave out members it has already sent a
particular device, which would need per-device bookkeeping of what each one
has seen. Warp sends them every time instead: that is the behaviour
`include_redundant_members: true` asks for, so no client can be surprised
by it, and it costs one state read rather than a table.

## The streams that are not room events

### Typing

In memory, and nowhere else. Typing notifications expire in seconds, they
are worthless after a restart — nobody is still typing the message they
were typing when the server went down — and they arrive one per
keystroke-burst per room. Writing them to disk would be the busiest table
in the database holding the least valuable data in it.

The cost: a restart clears them, which a client cannot tell from everybody
having stopped typing, and a second process would not see them. The second
only matters when Warp stops being one process, which PLAN.md says is not
the plan.

They therefore have no stream position. A notification that started and
stopped between two syncs is simply never reported; one that is live is
reported on every sync until it expires.

### Receipts

Only the latest receipt of each (user, type, thread) in a room is kept: a
receipt supersedes the one before it, and `/sync` reports the current one
rather than a history.

A receipt that would move **backwards** is dropped. Two of a user's clients
read a room at different speeds, and letting the slower one win would make
the unread marker jump backwards on the other — so the comparison is on the
stream position of the event being acknowledged, not on when the receipt
arrived.

`m.read.private` is filtered on the way out rather than in SQL: the query
is the same one the public receipts come from, and a `WHERE` clause naming
the asking user would turn one query per sync into two.

`m.fully_read` is not a receipt at all — it is the read marker, stored as
room account data. `POST /read_markers` takes all three in one body and
performs up to three independent writes. They are not atomic and cannot
usefully be: the marker is account data and the receipts are the receipt
stream, which are two streams with two positions. A failure partway through
leaves the earlier writes in place, which is the same state a client would
reach by sending them separately and losing its connection — and which the
next call corrects.

### To-device messages

A queue per device, drained by the device acknowledging a later sync token
rather than by reading it. A device acknowledges the batch it was sent by
coming back with the token that carried it, so the deletion happens on the
*next* sync. A client that crashes between the two gets the batch again,
which is the right way round: a lost key share leaves a message
permanently undecryptable, and a duplicate one is ignored.

A wildcard addressee (`"*"`) is expanded when the message is sent, not
stored as a wildcard. The set of a user's devices changes, and a message
addressed to "every device" means the ones that existed when it was sent —
storing the wildcard would deliver it to a device created afterwards, which
for a key share is both useless and a leak.

A message to a user of another server is dropped rather than refused. A
client encrypting for a room with remote members sends one message per
device in one request, and failing the request because federation is off
would stop it from delivering to the local devices as well. With federation
on, they go to the federation queue as `m.direct_to_device` EDUs instead.

### Presence

Off by default — PLAN.md open question 3, now answered. See
[configuration.md](configuration.md#presence) for what the switch does.

It is the one genuinely fan-out notification in the component: a user's
presence change is published to everybody they share a room with, which is
walked as "their rooms, and the members of each". That is the reason it is
a switch.

`last_active_ago` is computed on the way out rather than stored, for the
same reason `age` is on an event: a value written down at write time would
be wrong by however long the row sat in the database.

## Account data

Account data gained a `stream_pos` column on the existing table rather than
a parallel one: there is exactly one current value per (user, room, type),
and a second table would be a second thing to keep in step with it. The
position is stamped inside the same transaction as the write, so a client
is never told the stream moved and then handed the old value.

It is reported in an event list but it is **not** an event: it has a type
and a content and nothing else — no sender, no ID, no room. Warp gives it
its own response type for that reason. Reporting it as a room event with
empty fields would be worse: a client would see an event with no sender and
no timestamp and have to decide what that meant.

## Simplified Sliding Sync

`POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync` is
MSC4186, which Element X and every client on the matrix-rust-sdk use when
`/versions` advertises `org.matrix.simplified_msc3575` — as Warp's does.

A classic initial sync sends every room before a client can draw
anything, which for a large account takes minutes. Sliding sync turns that
round: the client asks for *lists* — "my rooms, most recently active
first, positions 0 to 19" — and for the few pieces of state it needs to
draw each one, and is sent those rooms and a count of the rest. As the
user scrolls the client widens the window; as rooms become active they
move into it and are sent.

The code is `internal/syncapi/sliding.go` and `slidingconn.go`, over a few
roomserver reads in `internal/roomserver/sliding.go`.

### Connections

What a client has been sent depends on what it asked for, so a token
cannot just be a position in each stream. The server remembers, per
connection — one user, one device, one `conn_id` — which rooms it has sent
and up to which position, and `pos` names one of those memories. A room is
sent once in full (`"initial": true`) and then only when it changes; a room
that leaves the window and comes back is not sent again unless something
happened to it.

The memory is in the process and is lost on restart. An unknown `pos` —
expired after 30 minutes idle, from before a restart, or one of the
connection's older positions beyond the last three — is answered with
`400 M_UNKNOWN_POS`, and the client starts a new connection. That is how
the protocol is meant to recover. A request without `pos` starts its
connection over. A device holds at most ten connections; the oldest goes
first.

Request parameters are not sticky: every request says what it wants.

### Lists

A list holds the rooms the user is joined to, invited to or knocking on,
plus rooms they left or were banned from while the connection was open,
which is how a client hears that a room it is showing is gone. Rooms are
sorted by `bump_stamp`, the position of the last message, encrypted
message, sticker, call invite, poll or live location in the room (not
membership changes, reactions or receipts, which would keep a busy room on
top for reasons nobody could see). An invite, a knock or a room the user
has left is bumped by their own membership event, because they cannot see
what happens in it.

| Filter | |
|---|---|
| `is_dm` | rooms in the user's `m.direct` account data |
| `is_invite` | |
| `is_encrypted` | an `m.room.encryption` in the room's state |
| `room_types`, `not_room_types` | the create event's `type`; `null` is an ordinary room |
| `spaces` | children of the named spaces this server is in |
| `tags`, `not_tags` | the room's `m.tag` account data |

`ranges` are inclusive windows into the sorted list. The response gives
each list's `count`, and the rooms in any window, merged with the room
subscriptions.

### Rooms

`room_subscriptions` names rooms directly, whatever list they are or are
not in: the room a client has open. A room in several lists and
subscriptions is sent with the largest `timeline_limit` and the union of
the `required_state`.

`required_state` is a list of `[type, state_key]`. `*` is a wildcard in
either place, `$ME` is the user, and `["m.room.member", "$LAZY"]` is the
members who sent something in the timeline, plus the user. Only those
tuples are read: the room's state is read as a set of keys first and the
event bodies after, so a room with ten thousand members costs one
member event, not ten thousand. On an incremental response it is the
tuples that changed since the room was last sent.

Each room carries `name` and `avatar`, `heroes` (with names and avatars)
when it has no name, `is_dm`, `joined_count`, `invited_count`, the unread
counts, `bump_stamp`, and the `timeline` with `prev_batch` (a `/messages`
token), `limited` and `num_live` (how many of the timeline's events are
new since the previous response). An invite is its `invite_state`, a knock
its `knock_state`, and neither has a timeline.

### Extensions

| Extension | What |
|---|---|
| `to_device` | The device's queued messages, 100 at a time by default. It has a `since` token of its own; coming back with it acknowledges the batch. |
| `e2ee` | `device_lists`, `device_one_time_keys_count`, `device_unused_fallback_key_types`, worked out as for `/sync`. |
| `account_data` | Global, and per room for the rooms in the response's lists and subscriptions — all of it for a room sent in full. |
| `receipts` | Per room, the receipts since the last response; all of them for a room sent in full. |
| `typing` | Per room, the typing list when it changed — including to empty, so a client stops showing somebody typing. |

`lists` and `rooms` narrow the rooms a per-room extension covers. Naming
any list the request has is every room in the windows: the windows are
merged before the extensions run.

### Long polling

A response is returned at once on a new connection; otherwise only when a
room is sent, a list's count moved, or an extension has something to say.
Until then it waits on the same notifier as `/sync`.

## What is deliberately missing

| Thing | Needs |
|---|---|
| Remote presence | presence over federation |
| `contains_url`, `event_fields` | nothing; they are parsed and ignored |
| Per-event history visibility | state storage (open question 1) |

Threads, relations and the summaries every timeline event carries in
`unsigned.m.relations` are in [relations.md](relations.md).

`device_lists` is sent as two empty lists rather than left out: a client
reads a missing one as a malformed response, where two empty lists say
"nothing changed". How the two lists are worked out, and the key counts
that ride along with them, are in [e2ee.md](e2ee.md).

Each joined room carries `unread_notifications` from the push server,
whose counts are the seventh stream — the main timeline's alone, beside an
`unread_thread_notifications` map of each thread's, when the filter sets
`unread_thread_notifications` — and the global account data carries
`m.push_rules`, rendered over the server defaults for a user who never
changed a rule. Both are described in [push.md](push.md). A build with no
push server leaves the counts out entirely rather than sending zeroes,
because a zero is a claim — "you have nothing unread" — and an absent field
is not.

## Endpoints

| Endpoint | Notes |
|---|---|
| `GET /sync` | Long-polls. `?since=`, `?filter=`, `?timeout=`, `?full_state=`, `?set_presence=`. |
| `POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync` | Simplified Sliding Sync; see below. `?pos=`, `?timeout=`, `?set_presence=`. |
| `GET /_matrix/client/v1/rooms/{id}/relations/{eventId}` · `GET /_matrix/client/v1/rooms/{id}/threads` | See [relations.md](relations.md). |
| `POST /search` | See [search.md](search.md). |
| `GET /rooms/{id}/messages` | `?dir=` (`b` default), `?from=`, `?to=`, `?limit=`, `?filter=`. No `end` at the end of the room. |
| `GET /rooms/{id}/context/{eventId}` | `?limit=`, `?filter=`. Half the budget each side, the odd one backwards. |
| `POST /user/{userId}/filter` | Returns an opaque ID. |
| `GET /user/{userId}/filter/{filterId}` | Byte for byte as uploaded. |
| `PUT /rooms/{id}/typing/{userId}` | Caller only. Requires membership. |
| `POST /rooms/{id}/receipt/{type}/{eventId}` | `m.read`, `m.read.private`, or `m.fully_read`. |
| `POST /rooms/{id}/read_markers` | Up to three writes. |
| `PUT /sendToDevice/{type}/{txnId}` | The transaction ID is accepted and not recorded; see below. |
| `GET` / `PUT /presence/{userId}/status` | `PUT` is caller only. `403` when presence is off. |

The `sendToDevice` transaction ID is accepted and not used. Its purpose is
idempotency, and a to-device message is already idempotent in the way that
matters: a duplicate is a second copy of a key share, which the receiving
client discards. Recording it would mean a table whose only reader would be
a retry that costs nothing anyway.

Typing, presence and the receipts are the caller's own claims, not data
belonging to them, so unlike a profile an admin may not set them for
somebody else: there is no administrative reason to say that another user
is typing.

## Testing

| Level | Where |
|---|---|
| Tokens and filters | `internal/syncapi/{token,filter}_test.go` — the parsing edges, which is where a client's stored token meets a newer server |
| Stream reads, the sync store | `internal/storage/sync_test.go`, against every driver |
| Visibility, windows, summaries | `internal/roomserver/stream_test.go` |
| The responses a client sees | `internal/clientapi/{sync,messages,ephemeral}_test.go`, through HTTP |
| Wiring | `internal/setup/phase4_test.go` — the whole stack, to catch an endpoint written but never registered |
| Integration | Complement: `TestSync*`, `TestMessages*`, `TestContext*`, `TestFilter*`, `TestTyping*`, `TestReceipt*`, `TestToDevice*`, `TestLazyLoading*` |

The HTTP tests are deliberately not unit tests of the builder. What is
worth pinning down is the response a client sees — which section a room
lands in, whether a token handed back can be handed in again, whether
paging terminates and loses nothing — and none of that is visible from
inside.

## Known cost

A sync over K changed rooms is roughly a constant number of queries per
room: one `RoomSync` (room row, summary, membership history, timeline,
state) plus the per-room ephemeral and account-data reads. On an incremental
sync K is usually zero or one, which is the case that matters. An initial
sync of a user in fifty rooms is fifty times that, and would be worth a
batched read if it ever shows up in a profile — the shape of the code does
not stop it, since `changedRooms` already decides the set up front.