Warp/docs/sync.md

Sync

internal/syncapi and the stream half of internal/storage: PLAN.md Phase 4. This is the read model — everything a client learns after it has logged in. /sync for the live stream, /messages for a room's history, /context for one event and what surrounds it, plus the streams that are not room events at all: typing, receipts, presence and to-device messages.

With this, Warp is a homeserver a client can be used against. Milestone M1.

The one architectural choice

The syncapi has no copy of the rooms. It reads them through the roomserver.

This is a departure from Dendrite, which gives its syncapi its own tables and feeds them from the roomserver's output stream. That design is right for a system where the two can be separate processes, and Dendrite's was. Warp is one process by decision (PLAN.md section 2), and in one process the copy buys nothing and costs the hardest class of bug a homeserver has: a read model that disagrees with the room. A client would be shown a message that was rejected, or not shown one that was accepted, and the only way to find out which is to compare two tables.

So the split is:

Question	Answered by
What events are in this window of this room?	roomserver
What state did the room have at this position?	roomserver
May this user see any of it?	roomserver — see Visibility
Where has this client got to?	syncapi — the token
What did it ask to be left out?	syncapi — the filter
Who is typing, who has read what, who is online?	syncapi
Should this request block, and for how long?	syncapi — the notifier

The roomserver grew a read model for this (stream.go, roomsync.go, visibility.go) rather than exporting its store, because reading an event means parsing it under its room's version — and a second copy of the room version registry, the event parser and the client projection would be a second place for them to disagree.

Stream tokens

A sync response is assembled from eight streams that move independently:

Stream	Moves when	Position from
PDU	an event is persisted	MAX(stream_ordering) in roomserver_events
receipt	somebody sends a read receipt	sync_sequences
to-device	a device is sent a message	sync_sequences
account data	a user changes a setting	sync_sequences
presence	somebody's presence actually changes	sync_sequences
device list	somebody's end-to-end keys change (e2ee.md)	keyserver_sequences
notification	a user's unread counts in a room move (push.md)	push_sequences
typing	somebody starts or stops typing, or a notification runs out	a counter in memory (see Typing)

A client that has seen receipt 7 but only event 4 has to be able to say so in the one opaque string the API gives it, so a token names a position in each:

s{pdu}_{receipt}_{to_device}_{account_data}_{presence}_{device_list}_{notification}_{typing}

Trailing zeroes are left off, so a client that has only ever seen room events carries s12 rather than s12_0_0_0_0_0_0_0. The device list, notification and typing positions came later and were added at the end. A token with fewer positions than this build writes is accepted and the missing ones read as zero — which is what lets a token stored before a stream existed go on working: the client is sent that stream in full once and is in step afterwards.

The spec says only that the token is opaque, and clients treat it that way. It is worth being readable anyway: a token is the first thing in every bug report about /sync.

The PDU position is the one that can be negative: /messages pages back into history backfilled from other servers, which is numbered downwards from -1 (see federation.md), and its tokens name positions there — s-12. The other streams start at zero.

/messages and a room's prev_batch hand out a PDU-only token. Both are about room events alone, and a token from either is a valid ?since= for /sync, where the other streams then read as unseen — the conservative direction, since the client is sent those streams again rather than having them skipped.

Why the PDU position comes from the events table

Not from a counter row, even though the other four do. The counter is advanced inside the writing transaction, so it is already at the new value while that transaction is still open. A token taken from it would name a position whose event no reader can see yet — and the event would be skipped for good once the transaction committed.

MAX(stream_ordering) is only ever the position of an event that is actually visible.

Why the counters are rows and not sequences

The same reason as the roomserver's (see rooms.md): an UPDATE takes a row lock both engines hold until the transaction ends, so a position handed out by a transaction that then rolls back is never used. PostgreSQL sequences are explicitly not transactional, and a gap in a stream is something every reader of it would have to be taught to skip.

Windows, and the bounds that name them

Every timeline read is a window named by its two ends and a direction:

type TimelineRequest struct {
    Low  int64             // exclusive
    High int64             // inclusive
    Direction TimelineDirection
    Limit int
}

Low exclusive and High inclusive, in both directions. Direction decides only which end the limit trims and which order the store reads in.

Zero is "unbounded" at either end, and is never a position: live events count up from 1 and backfilled history counts down from -1. So "is this bound set" is != 0, never > 0, and the position just below an event is storage.PositionBefore rather than a subtraction — below 1 is -1, because a token of zero handed back as a High bound would read the whole room again.

Spelling the bounds as "from" and "to" instead — which is how the Client-Server API spells them — would mean each of them being the low bound in one direction and the high bound in the other, which is exactly the off-by-one that every pagination bug is made of. The handlers translate the client's from and to into Low and High once, and nothing below them has to think about it again.

A position is somewhere a client has got to, not an event it still wants, which is why Low is exclusive: a token can be handed straight back without re-reading the event it names.

Why a sync's timeline is read backwards

A client wants the newest events of the window it missed — the bottom of the room, which is what it is looking at. A forward read with a limit would hand it the oldest of the window and then call the rest a gap. So RoomSync reads backwards from the top of the window and reports the result oldest first, which is the order a client renders.

Timeline.Start sits one below the oldest event returned and End is at the newest, so either can be handed back as the opposite bound and the next page begins exactly where this one stopped. prev_batch is Start.

limited, and what state means

The spec's rule is that a client which applies state and then timeline ends up with the room's state at the end of the timeline. That gives three cases, which roomserver.syncState implements:

• Initial sync, or ?full_state=true — the whole state as of the start of the timeline.
• Incremental, timeline reaches back to since — nothing. Every state change in between is a timeline event the client is already being given.
• Incremental, timeline cut short (a gap) — what changed between since and where the timeline starts. Those changes are not in the timeline and the client would otherwise never see them.

A timeline that reaches back to the room's create event reports no state at all: there is nothing before it.

A room joined inside the window

To the client, a room the user has just joined is new: it has none of its history and none of its state. So an incremental sync reads such a room the way an initial sync does — the recent history the user may see and the whole state before it — instead of the events since since, which would be the join alone with limited: false, telling the client it has everything.

"Joined inside the window" means the user's latest membership event is in the window and the membership before it was not already join. A display name or avatar change is a new join event too, but resending the room's history for one would be wrong; roomserver.MembershipBefore tells the two apart.

Visibility

roomserver.VisibleRange answers "what span of this room may this user read" once per room, rather than filtering event by event. The rules are the spec's history visibility:

history_visibility	Who, and from when
world_readable	anybody, all of it
shared	anybody who is or has been a member, all of it
invited	a member, from their invite onwards
joined	a member, from their join onwards

A user who has left or was banned is bounded above at their own membership event in every case. That is what stops a kicked user's client from going on streaming the room, and it is why the range is a range: their view of the room has an end as well as a beginning.

An invitee gets no timeline. They are told enough to decide — the stripped state that travels with the invite — and nothing more.

Phase 3 refused a user who had left a room outright and said in a comment that serving them the state as of their leave belonged here. It does now: GET /rooms/{id}/state, /state/{type}, /members and /messages all serve a departed member from the state at their departure, through StateAtStream.

A user's membership history is read from their own member events (UserMembershipEvents), so a user who joined, left and rejoined sees their latest span and not the earlier ones. There is one row per membership change of one user in one room, so this stays small however busy the room is.

The known gap

The range is computed from the room's current history visibility. The spec judges each event by the visibility in force when it was sent, so a room that was once world_readable and is now joined hides history here that the spec would show, and the reverse shows history it would hide.

Doing it exactly means resolving the history visibility at every event in the window — a state read per event — and belongs with the state storage work of PLAN.md open question 1. Until then this follows the setting the room's members last agreed on, which is the one they would expect to apply.

Long polling

/sync holds the request open until something happens. Polling the database instead would mean every connected client querying every second, which is the cost the notifier avoids.

syncapi.Notifier is the other half of the roomserver's emit stage — the seam Phase 3 left. The same object is the roomserver's Notifier and the syncapi's wake-up, wired together in setup.New.

Waking is by room and by user, not globally:

• An event wakes the listeners of its room.
• A membership event also wakes the listener of the user it names, which is how an invite reaches a client that is not in the room yet and so is not listening for it.
• Receipts and typing wake a room; to-device, account data and presence wake a user.
• An event in a room nobody is polling wakes nobody.

A wake-up is only ever a hint. The listener re-reads the streams and works out for itself what changed, so a spurious wake-up costs one query and a missed one costs at most the sync timeout. That is what makes it safe for the notifier to hold no state about what each client has seen.

The listener is registered before the streams are read. Anything that lands between the two leaves the channel already signalled, so the wait returns at once and the sync reads again rather than blocking on a change it has just missed.

EventPersisted must not block: it is called with the room's write lock still held, so anything slow there would serialise the room's writers behind the slowest reader of it. It is a non-blocking send onto a one-slot channel and nothing else.

What counts as "something happened"

Not "a token moved". A stream can move without this user being concerned by it — somebody else's receipt in a room they are not in — and a sync that returned on every such move would turn a long poll back into a busy loop. So the test is whether the response has anything in it, with an initial sync as the exception: a client that has just logged in needs an answer whatever is in it.

"Initial" means the client sent no ?since=, not that its token is all zeroes. On a server whose streams have not moved yet, s0 is every client's token, and taking it for a first sync made every long poll return at once. The push rules a first sync carries are sent on the client's first sync for the same reason, not on any sync whose account data position is zero: before anybody had written account data, that was every sync, and each one counted as news.

The same trap caught presence. A client says it is still online by syncing, so ?set_presence= makes every sync a presence write; a stream that moved each time would wake every peer's long poll, which would sync, which would move it again. SetPresence therefore reports whether anything a peer would notice actually changed, and a write that only refreshes the last-active timestamp leaves the stream where it was.

Filters

A filter is uploaded once and referred to by ID on every later sync, which is the point of it: the filter a client wants is long and does not change, so sending it on every request would be the largest part of every request. ?filter= also accepts an inline filter as JSON.

What Warp acts on:

• room.timeline.limit, types, not_types, senders, not_senders — pushed into SQL, so a filtered page is a full page rather than a short one that looks like the end of the room. A trailing * is the spec's wildcard; a % or _ in a pattern is matched literally.
• room.state — the same fields, applied in Go: a state read is a set of tuples, not a window of a stream, so there is no index to push it to.
• room.rooms / not_rooms, at both levels.
• room.timeline.lazy_load_members.
• room.timeline.unread_thread_notifications: the unread counts split by thread; see push.md.
• room.include_leave.
• presence and account_data types and senders.

What it parses and does not act on: event_fields, contains_url. A field Warp ignores is not an error — a server is allowed to return a superset of what a filter asked for, and refusing the filter is the one outcome that would stop a client syncing at all. event_format: federation is refused: that is a server's view of an event and the Client-Server API has no business handing it out.

A stored filter is returned byte for byte. A client that compares what it gets back with what it sent would otherwise see a difference that is not there, because the fields this build does not act on would come back reordered or missing.

Lazy loading

lazy_load_members drops the member events from a room's state and puts back only the senders who appear in the timeline being returned. It is what makes a room with ten thousand members openable.

The spec also lets a server leave out members it has already sent a particular device, which would need per-device bookkeeping of what each one has seen. Warp sends them every time instead: that is the behaviour include_redundant_members: true asks for, so no client can be surprised by it, and it costs one state read rather than a table.

The streams that are not room events

Typing

In memory, and nowhere else. Typing notifications expire in seconds, they are worthless after a restart — nobody is still typing the message they were typing when the server went down — and they arrive one per keystroke-burst per room. Writing them to disk would be the busiest table in the database holding the least valuable data in it.

The cost: a restart clears them, which a client cannot tell from everybody having stopped typing, and a second process would not see them. The second only matters when Warp stops being one process, which PLAN.md says is not the plan.

Their stream position is a counter in the tracker that moves on every change, with each room remembering the position of its last one. A sync reports a room's typing list when it changed inside the window — including to empty, which is how a client hears that somebody stopped — and not again while it stays the same. Reporting a live notification on every sync instead would make every long poll return at once while anybody typed: a busy loop that ends in the rate limiter.

The counter starts at zero again after a restart, so a client's typing position is compared for difference, never for order: one the tracker has never reached is from before the restart, and every room it knows of counts as changed. A notification that started and stopped between two syncs is reported as the empty list, which is what the client already had.

Receipts

Only the latest receipt of each (user, type, thread) in a room is kept: a receipt supersedes the one before it, and /sync reports the current one rather than a history.

A receipt that would move backwards is dropped. Two of a user's clients read a room at different speeds, and letting the slower one win would make the unread marker jump backwards on the other — so the comparison is on the stream position of the event being acknowledged, not on when the receipt arrived.

m.read.private is filtered on the way out rather than in SQL: the query is the same one the public receipts come from, and a WHERE clause naming the asking user would turn one query per sync into two.

m.fully_read is not a receipt at all — it is the read marker, stored as room account data. POST /read_markers takes all three in one body and performs up to three independent writes. They are not atomic and cannot usefully be: the marker is account data and the receipts are the receipt stream, which are two streams with two positions. A failure partway through leaves the earlier writes in place, which is the same state a client would reach by sending them separately and losing its connection — and which the next call corrects.

To-device messages

A queue per device, drained by the device acknowledging a later sync token rather than by reading it. A device acknowledges the batch it was sent by coming back with the token that carried it, so the deletion happens on the next sync. A client that crashes between the two gets the batch again, which is the right way round: a lost key share leaves a message permanently undecryptable, and a duplicate one is ignored.

A wildcard addressee ("*") is expanded when the message is sent, not stored as a wildcard. The set of a user's devices changes, and a message addressed to "every device" means the ones that existed when it was sent — storing the wildcard would deliver it to a device created afterwards, which for a key share is both useless and a leak.

A message to a user of another server is dropped rather than refused. A client encrypting for a room with remote members sends one message per device in one request, and failing the request because federation is off would stop it from delivering to the local devices as well. With federation on, they go to the federation queue as m.direct_to_device EDUs instead.

Presence

Off by default — PLAN.md open question 3, now answered. See configuration.md for what the switch does.

It is the one genuinely fan-out notification in the component: a user's presence change is published to everybody they share a room with, which is walked as "their rooms, and the members of each". That is the reason it is a switch.

last_active_ago is computed on the way out rather than stored, for the same reason age is on an event: a value written down at write time would be wrong by however long the row sat in the database.

Account data

Account data gained a stream_pos column on the existing table rather than a parallel one: there is exactly one current value per (user, room, type), and a second table would be a second thing to keep in step with it. The position is stamped inside the same transaction as the write, so a client is never told the stream moved and then handed the old value.

It is reported in an event list but it is not an event: it has a type and a content and nothing else — no sender, no ID, no room. Warp gives it its own response type for that reason. Reporting it as a room event with empty fields would be worse: a client would see an event with no sender and no timestamp and have to decide what that meant.

Simplified Sliding Sync

POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync is MSC4186, which Element X and every client on the matrix-rust-sdk use when /versions advertises org.matrix.simplified_msc3575 — as Warp's does.

A classic initial sync sends every room before a client can draw anything, which for a large account takes minutes. Sliding sync turns that round: the client asks for lists — "my rooms, most recently active first, positions 0 to 19" — and for the few pieces of state it needs to draw each one, and is sent those rooms and a count of the rest. As the user scrolls the client widens the window; as rooms become active they move into it and are sent.

The code is internal/syncapi/sliding.go and slidingconn.go, over a few roomserver reads in internal/roomserver/sliding.go.

Connections

What a client has been sent depends on what it asked for, so a token cannot just be a position in each stream. The server remembers, per connection — one user, one device, one conn_id — which rooms it has sent and up to which position, and pos names one of those memories. A room is sent once in full ("initial": true) and then only when it changes; a room that leaves the window and comes back is not sent again unless something happened to it.

The memory is in the process and is lost on restart. An unknown pos — expired after 30 minutes idle, from before a restart, or one of the connection's older positions beyond the last three — is answered with 400 M_UNKNOWN_POS, and the client starts a new connection. That is how the protocol is meant to recover. A request without pos starts its connection over. A device holds at most ten connections; the oldest goes first.

Request parameters are not sticky: every request says what it wants.

Lists

A list holds the rooms the user is joined to, invited to or knocking on, plus rooms they left or were banned from while the connection was open, which is how a client hears that a room it is showing is gone. Rooms are sorted by bump_stamp, the position of the last message, encrypted message, sticker, call invite, poll or live location in the room (not membership changes, reactions or receipts, which would keep a busy room on top for reasons nobody could see). An invite, a knock or a room the user has left is bumped by their own membership event, because they cannot see what happens in it.

Filter
is_dm	rooms in the user's m.direct account data
is_invite
is_encrypted	an m.room.encryption in the room's state
room_types, not_room_types	the create event's type; null is an ordinary room
spaces	children of the named spaces this server is in
tags, not_tags	the room's m.tag account data

ranges are inclusive windows into the sorted list. The response gives each list's count, and the rooms in any window, merged with the room subscriptions.

Rooms

room_subscriptions names rooms directly, whatever list they are or are not in: the room a client has open. A room in several lists and subscriptions is sent with the largest timeline_limit and the union of the required_state.

required_state is a list of [type, state_key]. * is a wildcard in either place, $ME is the user, and ["m.room.member", "$LAZY"] is the members who sent something in the timeline, plus the user. Only those tuples are read: the room's state is read as a set of keys first and the event bodies after, so a room with ten thousand members costs one member event, not ten thousand. On an incremental response it is the tuples that changed since the room was last sent.

Each room carries name and avatar, heroes (with names and avatars) when it has no name, is_dm, joined_count, invited_count, the unread counts, bump_stamp, and the timeline with prev_batch (a /messages token), limited and num_live (how many of the timeline's events are new since the previous response). An invite is its invite_state, a knock its knock_state, and neither has a timeline.

Extensions

Extension	What
to_device	The device's queued messages, 100 at a time by default. It has a since token of its own; coming back with it acknowledges the batch.
e2ee	device_lists, device_one_time_keys_count, device_unused_fallback_key_types, worked out as for /sync.
account_data	Global, and per room for the rooms in the response's lists and subscriptions — all of it for a room sent in full.
receipts	Per room, the receipts since the last response; all of them for a room sent in full.
typing	Per room, the typing list when it changed — including to empty, so a client stops showing somebody typing.

lists and rooms narrow the rooms a per-room extension covers. Naming any list the request has is every room in the windows: the windows are merged before the extensions run.

Long polling

A response is returned at once on a new connection; otherwise only when a room is sent, a list's count moved, or an extension has something to say. Until then it waits on the same notifier as /sync.

What is deliberately missing

Thing	Needs
Remote presence	presence over federation
contains_url, event_fields	nothing; they are parsed and ignored
Per-event history visibility	state storage (open question 1)

Threads, relations and the summaries every timeline event carries in unsigned.m.relations are in relations.md.

device_lists is sent as two empty lists rather than left out: a client reads a missing one as a malformed response, where two empty lists say "nothing changed". How the two lists are worked out, and the key counts that ride along with them, are in e2ee.md.

Each joined room carries unread_notifications from the push server, whose counts are the seventh stream — the main timeline's alone, beside an unread_thread_notifications map of each thread's, when the filter sets unread_thread_notifications — and the global account data carries m.push_rules, rendered over the server defaults for a user who never changed a rule. Both are described in push.md. A build with no push server leaves the counts out entirely rather than sending zeroes, because a zero is a claim — "you have nothing unread" — and an absent field is not.

Endpoints

Endpoint	Notes
GET /sync	Long-polls. ?since=, ?filter=, ?timeout=, ?full_state=, ?set_presence=.
POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync	Simplified Sliding Sync; see below. ?pos=, ?timeout=, ?set_presence=.
GET /_matrix/client/v1/rooms/{id}/relations/{eventId} · GET /_matrix/client/v1/rooms/{id}/threads	See relations.md.
POST /search	See search.md.
GET /rooms/{id}/messages	?dir= (b default), ?from=, ?to=, ?limit=, ?filter=. No end at the end of the room.
GET /rooms/{id}/context/{eventId}	?limit=, ?filter=. Half the budget each side, the odd one backwards.
POST /user/{userId}/filter	Returns an opaque ID.
GET /user/{userId}/filter/{filterId}	Byte for byte as uploaded.
PUT /rooms/{id}/typing/{userId}	Caller only. Requires membership.
POST /rooms/{id}/receipt/{type}/{eventId}	m.read, m.read.private, or m.fully_read.
POST /rooms/{id}/read_markers	Up to three writes.
PUT /sendToDevice/{type}/{txnId}	The transaction ID is accepted and not recorded; see below.
GET / PUT /presence/{userId}/status	PUT is caller only. 403 when presence is off.

The sendToDevice transaction ID is accepted and not used. Its purpose is idempotency, and a to-device message is already idempotent in the way that matters: a duplicate is a second copy of a key share, which the receiving client discards. Recording it would mean a table whose only reader would be a retry that costs nothing anyway.

Typing, presence and the receipts are the caller's own claims, not data belonging to them, so unlike a profile an admin may not set them for somebody else: there is no administrative reason to say that another user is typing.

Testing

Level	Where
Tokens and filters	internal/syncapi/{token,filter}_test.go — the parsing edges, which is where a client's stored token meets a newer server
Stream reads, the sync store	internal/storage/sync_test.go, against every driver
Visibility, windows, summaries	internal/roomserver/stream_test.go
The responses a client sees	internal/clientapi/{sync,messages,ephemeral}_test.go, through HTTP
Wiring	internal/setup/phase4_test.go — the whole stack, to catch an endpoint written but never registered
Integration	Complement: TestSync*, TestMessages*, TestContext*, TestFilter*, TestTyping*, TestReceipt*, TestToDevice*, TestLazyLoading*

The HTTP tests are deliberately not unit tests of the builder. What is worth pinning down is the response a client sees — which section a room lands in, whether a token handed back can be handed in again, whether paging terminates and loses nothing — and none of that is visible from inside.

Known cost

A sync over K changed rooms is roughly a constant number of queries per room: one RoomSync (room row, summary, membership history, timeline, state) plus the per-room ephemeral and account-data reads. On an incremental sync K is usually zero or one, which is the case that matters. An initial sync of a user in fifty rooms is fifty times that, and would be worth a batched read if it ever shows up in a profile — the shape of the code does not stop it, since changedRooms already decides the set up front.