CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
CI / Admin panel (push) Canceled after 0s
CI / Web client (push) Canceled after 0s
apps/web: a Matrix client for the browser on Vue 3, Tailwind and matrix-js-sdk, built on @warp/ui. Like the admin panel it is an optional, separate service: warp-web embeds the built app, serves /config.json and proxies only /_matrix/client and /_matrix/media to the homeserver (or, with -public-homeserver, lets the browser call it directly). Sign-in and sign-up with registration tokens, room list, timeline with back-pagination, text, images, video and files over authenticated media, local echo with sent/read/failed states, receipts, typing, invites, creating, joining and leaving rooms, room info, settings, phone layout. End-to-end encryption through the Rust crypto: encrypted messages and files, emoji verification, key backup setup and restore. apps/ui: WRoomItem, WMessageBubble and WComposer from the brand package, and the icons the client needs. syncapi fixes found with the client: - A room joined inside the sync window was sent as the join alone with limited: false, so clients thought they had the whole room. It is now read like an initial sync; a profile change is not mistaken for a join. - A live typing notification was reported on every sync, so every long poll returned at once while anybody typed. Typing now has a position in the sync token and is reported when it changes, including the empty list when the last person stops. Also: Dockerfile, a `web` compose profile, Makefile targets, a CI job with a proxy smoke test, docs. PLAN.md moves out of the repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
623 lines
30 KiB
Markdown
623 lines
30 KiB
Markdown
# Sync
|
|
|
|
`internal/syncapi` and the stream half of `internal/storage`: PLAN.md
|
|
Phase 4. This is the read model — everything a client learns after it has
|
|
logged in. `/sync` for the live stream, `/messages` for a room's history,
|
|
`/context` for one event and what surrounds it, plus the streams that are
|
|
not room events at all: typing, receipts, presence and to-device messages.
|
|
|
|
With this, Warp is a homeserver a client can be used against. Milestone M1.
|
|
|
|
## The one architectural choice
|
|
|
|
**The syncapi has no copy of the rooms.** It reads them through the
|
|
roomserver.
|
|
|
|
This is a departure from Dendrite, which gives its syncapi its own tables
|
|
and feeds them from the roomserver's output stream. That design is right
|
|
for a system where the two can be separate processes, and Dendrite's was.
|
|
Warp is one process by decision (PLAN.md section 2), and in one process the
|
|
copy buys nothing and costs the hardest class of bug a homeserver has: a
|
|
read model that disagrees with the room. A client would be shown a message
|
|
that was rejected, or not shown one that was accepted, and the only way to
|
|
find out which is to compare two tables.
|
|
|
|
So the split is:
|
|
|
|
| Question | Answered by |
|
|
|---|---|
|
|
| What events are in this window of this room? | `roomserver` |
|
|
| What state did the room have at this position? | `roomserver` |
|
|
| May this user see any of it? | `roomserver` — see *Visibility* |
|
|
| Where has this client got to? | `syncapi` — the token |
|
|
| What did it ask to be left out? | `syncapi` — the filter |
|
|
| Who is typing, who has read what, who is online? | `syncapi` |
|
|
| Should this request block, and for how long? | `syncapi` — the notifier |
|
|
|
|
The roomserver grew a read model for this (`stream.go`, `roomsync.go`,
|
|
`visibility.go`) rather than exporting its store, because reading an event
|
|
means parsing it under its room's version — and a second copy of the room
|
|
version registry, the event parser and the client projection would be a
|
|
second place for them to disagree.
|
|
|
|
## Stream tokens
|
|
|
|
A sync response is assembled from eight streams that move independently:
|
|
|
|
| Stream | Moves when | Position from |
|
|
|---|---|---|
|
|
| PDU | an event is persisted | `MAX(stream_ordering)` in `roomserver_events` |
|
|
| receipt | somebody sends a read receipt | `sync_sequences` |
|
|
| to-device | a device is sent a message | `sync_sequences` |
|
|
| account data | a user changes a setting | `sync_sequences` |
|
|
| presence | somebody's presence actually changes | `sync_sequences` |
|
|
| device list | somebody's end-to-end keys change ([e2ee.md](e2ee.md)) | `keyserver_sequences` |
|
|
| notification | a user's unread counts in a room move ([push.md](push.md)) | `push_sequences` |
|
|
| typing | somebody starts or stops typing, or a notification runs out | a counter in memory (see [Typing](#typing)) |
|
|
|
|
A client that has seen receipt 7 but only event 4 has to be able to say so
|
|
in the one opaque string the API gives it, so a token names a position in
|
|
each:
|
|
|
|
```
|
|
s{pdu}_{receipt}_{to_device}_{account_data}_{presence}_{device_list}_{notification}_{typing}
|
|
```
|
|
|
|
Trailing zeroes are left off, so a client that has only ever seen room
|
|
events carries `s12` rather than `s12_0_0_0_0_0_0_0`. The device list,
|
|
notification and typing positions came later and were added at the
|
|
end. A token with fewer
|
|
positions than this build writes is accepted and the missing ones read as
|
|
zero — which is what lets a token stored before a stream existed go on
|
|
working: the client is sent that stream in full once and is in step
|
|
afterwards.
|
|
|
|
The spec says only that the token is opaque, and clients treat it that way.
|
|
It is worth being readable anyway: a token is the first thing in every bug
|
|
report about `/sync`.
|
|
|
|
The PDU position is the one that can be negative: `/messages` pages back
|
|
into history backfilled from other servers, which is numbered downwards
|
|
from -1 (see [federation.md](federation.md#history-backfill)), and its
|
|
tokens name positions there — `s-12`. The other streams start at zero.
|
|
|
|
`/messages` and a room's `prev_batch` hand out a PDU-only token. Both are
|
|
about room events alone, and a token from either is a valid `?since=` for
|
|
`/sync`, where the other streams then read as unseen — the conservative
|
|
direction, since the client is sent those streams again rather than having
|
|
them skipped.
|
|
|
|
### Why the PDU position comes from the events table
|
|
|
|
Not from a counter row, even though the other four do. The counter is
|
|
advanced *inside* the writing transaction, so it is already at the new
|
|
value while that transaction is still open. A token taken from it would
|
|
name a position whose event no reader can see yet — and the event would be
|
|
skipped for good once the transaction committed.
|
|
|
|
`MAX(stream_ordering)` is only ever the position of an event that is
|
|
actually visible.
|
|
|
|
### Why the counters are rows and not sequences
|
|
|
|
The same reason as the roomserver's (see [rooms.md](rooms.md)): an `UPDATE`
|
|
takes a row lock both engines hold until the transaction ends, so a
|
|
position handed out by a transaction that then rolls back is never used.
|
|
PostgreSQL sequences are explicitly not transactional, and a gap in a
|
|
stream is something every reader of it would have to be taught to skip.
|
|
|
|
## Windows, and the bounds that name them
|
|
|
|
Every timeline read is a window named by its two ends and a direction:
|
|
|
|
```go
|
|
type TimelineRequest struct {
|
|
Low int64 // exclusive
|
|
High int64 // inclusive
|
|
Direction TimelineDirection
|
|
Limit int
|
|
}
|
|
```
|
|
|
|
Low exclusive and High inclusive, **in both directions**. Direction decides
|
|
only which end the limit trims and which order the store reads in.
|
|
|
|
Zero is "unbounded" at either end, and is never a position: live events
|
|
count up from 1 and backfilled history counts down from -1. So "is this
|
|
bound set" is `!= 0`, never `> 0`, and the position just below an event is
|
|
`storage.PositionBefore` rather than a subtraction — below 1 is -1, because
|
|
a token of zero handed back as a High bound would read the whole room again.
|
|
|
|
Spelling the bounds as "from" and "to" instead — which is how the
|
|
Client-Server API spells them — would mean each of them being the low bound
|
|
in one direction and the high bound in the other, which is exactly the
|
|
off-by-one that every pagination bug is made of. The handlers translate the
|
|
client's `from` and `to` into Low and High once, and nothing below them has
|
|
to think about it again.
|
|
|
|
A position is somewhere a client has got to, not an event it still wants,
|
|
which is why Low is exclusive: a token can be handed straight back without
|
|
re-reading the event it names.
|
|
|
|
### Why a sync's timeline is read backwards
|
|
|
|
A client wants the *newest* events of the window it missed — the bottom of
|
|
the room, which is what it is looking at. A forward read with a limit would
|
|
hand it the oldest of the window and then call the rest a gap. So
|
|
`RoomSync` reads backwards from the top of the window and reports the
|
|
result oldest first, which is the order a client renders.
|
|
|
|
`Timeline.Start` sits one below the oldest event returned and `End` is at
|
|
the newest, so either can be handed back as the opposite bound and the next
|
|
page begins exactly where this one stopped. `prev_batch` is `Start`.
|
|
|
|
### `limited`, and what `state` means
|
|
|
|
The spec's rule is that a client which applies `state` and then `timeline`
|
|
ends up with the room's state at the end of the timeline. That gives three
|
|
cases, which `roomserver.syncState` implements:
|
|
|
|
- **Initial sync, or `?full_state=true`** — the whole state as of the start
|
|
of the timeline.
|
|
- **Incremental, timeline reaches back to `since`** — nothing. Every state
|
|
change in between is a timeline event the client is already being given.
|
|
- **Incremental, timeline cut short (a gap)** — what changed between
|
|
`since` and where the timeline starts. Those changes are not in the
|
|
timeline and the client would otherwise never see them.
|
|
|
|
A timeline that reaches back to the room's create event reports no state at
|
|
all: there is nothing before it.
|
|
|
|
### A room joined inside the window
|
|
|
|
To the client, a room the user has just joined is new: it has none of its
|
|
history and none of its state. So an incremental sync reads such a room
|
|
the way an initial sync does — the recent history the user may see and the
|
|
whole state before it — instead of the events since `since`, which would
|
|
be the join alone with `limited: false`, telling the client it has
|
|
everything.
|
|
|
|
"Joined inside the window" means the user's latest membership event is in
|
|
the window *and* the membership before it was not already `join`. A
|
|
display name or avatar change is a new join event too, but resending the
|
|
room's history for one would be wrong; `roomserver.MembershipBefore` tells
|
|
the two apart.
|
|
|
|
## Visibility
|
|
|
|
`roomserver.VisibleRange` answers "what span of this room may this user
|
|
read" once per room, rather than filtering event by event. The rules are
|
|
the spec's history visibility:
|
|
|
|
| `history_visibility` | Who, and from when |
|
|
|---|---|
|
|
| `world_readable` | anybody, all of it |
|
|
| `shared` | anybody who is or has been a member, all of it |
|
|
| `invited` | a member, from their invite onwards |
|
|
| `joined` | a member, from their join onwards |
|
|
|
|
A user who has left or was banned is bounded **above** at their own
|
|
membership event in every case. That is what stops a kicked user's client
|
|
from going on streaming the room, and it is why the range is a range: their
|
|
view of the room has an end as well as a beginning.
|
|
|
|
An invitee gets no timeline. They are told enough to decide — the stripped
|
|
state that travels with the invite — and nothing more.
|
|
|
|
Phase 3 refused a user who had left a room outright and said in a comment
|
|
that serving them the state as of their leave belonged here. It does now:
|
|
`GET /rooms/{id}/state`, `/state/{type}`, `/members` and `/messages` all
|
|
serve a departed member from the state at their departure, through
|
|
`StateAtStream`.
|
|
|
|
A user's membership history is read from their own member events
|
|
(`UserMembershipEvents`), so a user who joined, left and rejoined sees
|
|
their latest span and not the earlier ones. There is one row per membership
|
|
change of one user in one room, so this stays small however busy the room
|
|
is.
|
|
|
|
### The known gap
|
|
|
|
The range is computed from the room's **current** history visibility. The
|
|
spec judges each event by the visibility in force when it was sent, so a
|
|
room that was once `world_readable` and is now `joined` hides history here
|
|
that the spec would show, and the reverse shows history it would hide.
|
|
|
|
Doing it exactly means resolving the history visibility at every event in
|
|
the window — a state read per event — and belongs with the state storage
|
|
work of PLAN.md open question 1. Until then this follows the setting the
|
|
room's members last agreed on, which is the one they would expect to apply.
|
|
|
|
## Long polling
|
|
|
|
`/sync` holds the request open until something happens. Polling the
|
|
database instead would mean every connected client querying every second,
|
|
which is the cost the notifier avoids.
|
|
|
|
`syncapi.Notifier` is the other half of the roomserver's emit stage — the
|
|
seam Phase 3 left. The same object is the roomserver's `Notifier` and the
|
|
syncapi's wake-up, wired together in `setup.New`.
|
|
|
|
Waking is **by room and by user**, not globally:
|
|
|
|
- An event wakes the listeners of its room.
|
|
- A membership event also wakes the listener of the user it names, which is
|
|
how an invite reaches a client that is not in the room yet and so is not
|
|
listening for it.
|
|
- Receipts and typing wake a room; to-device, account data and presence
|
|
wake a user.
|
|
- An event in a room nobody is polling wakes nobody.
|
|
|
|
A wake-up is only ever a hint. The listener re-reads the streams and works
|
|
out for itself what changed, so a spurious wake-up costs one query and a
|
|
missed one costs at most the sync timeout. That is what makes it safe for
|
|
the notifier to hold no state about what each client has seen.
|
|
|
|
The listener is registered **before** the streams are read. Anything that
|
|
lands between the two leaves the channel already signalled, so the wait
|
|
returns at once and the sync reads again rather than blocking on a change
|
|
it has just missed.
|
|
|
|
`EventPersisted` must not block: it is called with the room's write lock
|
|
still held, so anything slow there would serialise the room's writers
|
|
behind the slowest reader of it. It is a non-blocking send onto a one-slot
|
|
channel and nothing else.
|
|
|
|
### What counts as "something happened"
|
|
|
|
Not "a token moved". A stream can move without this user being concerned by
|
|
it — somebody else's receipt in a room they are not in — and a sync that
|
|
returned on every such move would turn a long poll back into a busy loop.
|
|
So the test is whether the response has anything in it, with an initial
|
|
sync as the exception: a client that has just logged in needs an answer
|
|
whatever is in it.
|
|
|
|
"Initial" means the client sent no `?since=`, not that its token is all
|
|
zeroes. On a server whose streams have not moved yet, `s0` is every
|
|
client's token, and taking it for a first sync made every long poll
|
|
return at once. The push rules a first sync carries are sent on the
|
|
client's first sync for the same reason, not on any sync whose account
|
|
data position is zero: before anybody had written account data, that was
|
|
every sync, and each one counted as news.
|
|
|
|
The same trap caught presence. A client says it is still online by syncing,
|
|
so `?set_presence=` makes every sync a presence write; a stream that moved
|
|
each time would wake every peer's long poll, which would sync, which would
|
|
move it again. `SetPresence` therefore reports whether anything a peer
|
|
would notice actually changed, and a write that only refreshes the
|
|
last-active timestamp leaves the stream where it was.
|
|
|
|
## Filters
|
|
|
|
A filter is uploaded once and referred to by ID on every later sync, which
|
|
is the point of it: the filter a client wants is long and does not change,
|
|
so sending it on every request would be the largest part of every request.
|
|
`?filter=` also accepts an inline filter as JSON.
|
|
|
|
What Warp acts on:
|
|
|
|
- `room.timeline.limit`, `types`, `not_types`, `senders`, `not_senders` —
|
|
pushed into SQL, so a filtered page is a full page rather than a short
|
|
one that looks like the end of the room. A trailing `*` is the spec's
|
|
wildcard; a `%` or `_` in a pattern is matched literally.
|
|
- `room.state` — the same fields, applied in Go: a state read is a set of
|
|
tuples, not a window of a stream, so there is no index to push it to.
|
|
- `room.rooms` / `not_rooms`, at both levels.
|
|
- `room.timeline.lazy_load_members`.
|
|
- `room.timeline.unread_thread_notifications`: the unread counts split by
|
|
thread; see [push.md](push.md).
|
|
- `room.include_leave`.
|
|
- `presence` and `account_data` types and senders.
|
|
|
|
What it parses and does not act on: `event_fields`, `contains_url`. A field Warp ignores is **not** an error — a
|
|
server is allowed to return a superset of what a filter asked for, and
|
|
refusing the filter is the one outcome that would stop a client syncing at
|
|
all. `event_format: federation` *is* refused: that is a server's view of an
|
|
event and the Client-Server API has no business handing it out.
|
|
|
|
A stored filter is returned byte for byte. A client that compares what it
|
|
gets back with what it sent would otherwise see a difference that is not
|
|
there, because the fields this build does not act on would come back
|
|
reordered or missing.
|
|
|
|
### Lazy loading
|
|
|
|
`lazy_load_members` drops the member events from a room's state and puts
|
|
back only the senders who appear in the timeline being returned. It is what
|
|
makes a room with ten thousand members openable.
|
|
|
|
The spec also lets a server leave out members it has already sent a
|
|
particular device, which would need per-device bookkeeping of what each one
|
|
has seen. Warp sends them every time instead: that is the behaviour
|
|
`include_redundant_members: true` asks for, so no client can be surprised
|
|
by it, and it costs one state read rather than a table.
|
|
|
|
## The streams that are not room events
|
|
|
|
### Typing
|
|
|
|
In memory, and nowhere else. Typing notifications expire in seconds, they
|
|
are worthless after a restart — nobody is still typing the message they
|
|
were typing when the server went down — and they arrive one per
|
|
keystroke-burst per room. Writing them to disk would be the busiest table
|
|
in the database holding the least valuable data in it.
|
|
|
|
The cost: a restart clears them, which a client cannot tell from everybody
|
|
having stopped typing, and a second process would not see them. The second
|
|
only matters when Warp stops being one process, which PLAN.md says is not
|
|
the plan.
|
|
|
|
Their stream position is a counter in the tracker that moves on every
|
|
change, with each room remembering the position of its last one. A sync
|
|
reports a room's typing list when it changed inside the window — including
|
|
to empty, which is how a client hears that somebody stopped — and not
|
|
again while it stays the same. Reporting a live notification on every sync
|
|
instead would make every long poll return at once while anybody typed: a
|
|
busy loop that ends in the rate limiter.
|
|
|
|
The counter starts at zero again after a restart, so a client's typing
|
|
position is compared for difference, never for order: one the tracker has
|
|
never reached is from before the restart, and every room it knows of
|
|
counts as changed. A notification that started and stopped between two
|
|
syncs is reported as the empty list, which is what the client already had.
|
|
|
|
### Receipts
|
|
|
|
Only the latest receipt of each (user, type, thread) in a room is kept: a
|
|
receipt supersedes the one before it, and `/sync` reports the current one
|
|
rather than a history.
|
|
|
|
A receipt that would move **backwards** is dropped. Two of a user's clients
|
|
read a room at different speeds, and letting the slower one win would make
|
|
the unread marker jump backwards on the other — so the comparison is on the
|
|
stream position of the event being acknowledged, not on when the receipt
|
|
arrived.
|
|
|
|
`m.read.private` is filtered on the way out rather than in SQL: the query
|
|
is the same one the public receipts come from, and a `WHERE` clause naming
|
|
the asking user would turn one query per sync into two.
|
|
|
|
`m.fully_read` is not a receipt at all — it is the read marker, stored as
|
|
room account data. `POST /read_markers` takes all three in one body and
|
|
performs up to three independent writes. They are not atomic and cannot
|
|
usefully be: the marker is account data and the receipts are the receipt
|
|
stream, which are two streams with two positions. A failure partway through
|
|
leaves the earlier writes in place, which is the same state a client would
|
|
reach by sending them separately and losing its connection — and which the
|
|
next call corrects.
|
|
|
|
### To-device messages
|
|
|
|
A queue per device, drained by the device acknowledging a later sync token
|
|
rather than by reading it. A device acknowledges the batch it was sent by
|
|
coming back with the token that carried it, so the deletion happens on the
|
|
*next* sync. A client that crashes between the two gets the batch again,
|
|
which is the right way round: a lost key share leaves a message
|
|
permanently undecryptable, and a duplicate one is ignored.
|
|
|
|
A wildcard addressee (`"*"`) is expanded when the message is sent, not
|
|
stored as a wildcard. The set of a user's devices changes, and a message
|
|
addressed to "every device" means the ones that existed when it was sent —
|
|
storing the wildcard would deliver it to a device created afterwards, which
|
|
for a key share is both useless and a leak.
|
|
|
|
A message to a user of another server is dropped rather than refused. A
|
|
client encrypting for a room with remote members sends one message per
|
|
device in one request, and failing the request because federation is off
|
|
would stop it from delivering to the local devices as well. With federation
|
|
on, they go to the federation queue as `m.direct_to_device` EDUs instead.
|
|
|
|
### Presence
|
|
|
|
Off by default — PLAN.md open question 3, now answered. See
|
|
[configuration.md](configuration.md#presence) for what the switch does.
|
|
|
|
It is the one genuinely fan-out notification in the component: a user's
|
|
presence change is published to everybody they share a room with, which is
|
|
walked as "their rooms, and the members of each". That is the reason it is
|
|
a switch.
|
|
|
|
`last_active_ago` is computed on the way out rather than stored, for the
|
|
same reason `age` is on an event: a value written down at write time would
|
|
be wrong by however long the row sat in the database.
|
|
|
|
## Account data
|
|
|
|
Account data gained a `stream_pos` column on the existing table rather than
|
|
a parallel one: there is exactly one current value per (user, room, type),
|
|
and a second table would be a second thing to keep in step with it. The
|
|
position is stamped inside the same transaction as the write, so a client
|
|
is never told the stream moved and then handed the old value.
|
|
|
|
It is reported in an event list but it is **not** an event: it has a type
|
|
and a content and nothing else — no sender, no ID, no room. Warp gives it
|
|
its own response type for that reason. Reporting it as a room event with
|
|
empty fields would be worse: a client would see an event with no sender and
|
|
no timestamp and have to decide what that meant.
|
|
|
|
## Simplified Sliding Sync
|
|
|
|
`POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync` is
|
|
MSC4186, which Element X and every client on the matrix-rust-sdk use when
|
|
`/versions` advertises `org.matrix.simplified_msc3575` — as Warp's does.
|
|
|
|
A classic initial sync sends every room before a client can draw
|
|
anything, which for a large account takes minutes. Sliding sync turns that
|
|
round: the client asks for *lists* — "my rooms, most recently active
|
|
first, positions 0 to 19" — and for the few pieces of state it needs to
|
|
draw each one, and is sent those rooms and a count of the rest. As the
|
|
user scrolls the client widens the window; as rooms become active they
|
|
move into it and are sent.
|
|
|
|
The code is `internal/syncapi/sliding.go` and `slidingconn.go`, over a few
|
|
roomserver reads in `internal/roomserver/sliding.go`.
|
|
|
|
### Connections
|
|
|
|
What a client has been sent depends on what it asked for, so a token
|
|
cannot just be a position in each stream. The server remembers, per
|
|
connection — one user, one device, one `conn_id` — which rooms it has sent
|
|
and up to which position, and `pos` names one of those memories. A room is
|
|
sent once in full (`"initial": true`) and then only when it changes; a room
|
|
that leaves the window and comes back is not sent again unless something
|
|
happened to it.
|
|
|
|
The memory is in the process and is lost on restart. An unknown `pos` —
|
|
expired after 30 minutes idle, from before a restart, or one of the
|
|
connection's older positions beyond the last three — is answered with
|
|
`400 M_UNKNOWN_POS`, and the client starts a new connection. That is how
|
|
the protocol is meant to recover. A request without `pos` starts its
|
|
connection over. A device holds at most ten connections; the oldest goes
|
|
first.
|
|
|
|
Request parameters are not sticky: every request says what it wants.
|
|
|
|
### Lists
|
|
|
|
A list holds the rooms the user is joined to, invited to or knocking on,
|
|
plus rooms they left or were banned from while the connection was open,
|
|
which is how a client hears that a room it is showing is gone. Rooms are
|
|
sorted by `bump_stamp`, the position of the last message, encrypted
|
|
message, sticker, call invite, poll or live location in the room (not
|
|
membership changes, reactions or receipts, which would keep a busy room on
|
|
top for reasons nobody could see). An invite, a knock or a room the user
|
|
has left is bumped by their own membership event, because they cannot see
|
|
what happens in it.
|
|
|
|
| Filter | |
|
|
|---|---|
|
|
| `is_dm` | rooms in the user's `m.direct` account data |
|
|
| `is_invite` | |
|
|
| `is_encrypted` | an `m.room.encryption` in the room's state |
|
|
| `room_types`, `not_room_types` | the create event's `type`; `null` is an ordinary room |
|
|
| `spaces` | children of the named spaces this server is in |
|
|
| `tags`, `not_tags` | the room's `m.tag` account data |
|
|
|
|
`ranges` are inclusive windows into the sorted list. The response gives
|
|
each list's `count`, and the rooms in any window, merged with the room
|
|
subscriptions.
|
|
|
|
### Rooms
|
|
|
|
`room_subscriptions` names rooms directly, whatever list they are or are
|
|
not in: the room a client has open. A room in several lists and
|
|
subscriptions is sent with the largest `timeline_limit` and the union of
|
|
the `required_state`.
|
|
|
|
`required_state` is a list of `[type, state_key]`. `*` is a wildcard in
|
|
either place, `$ME` is the user, and `["m.room.member", "$LAZY"]` is the
|
|
members who sent something in the timeline, plus the user. Only those
|
|
tuples are read: the room's state is read as a set of keys first and the
|
|
event bodies after, so a room with ten thousand members costs one
|
|
member event, not ten thousand. On an incremental response it is the
|
|
tuples that changed since the room was last sent.
|
|
|
|
Each room carries `name` and `avatar`, `heroes` (with names and avatars)
|
|
when it has no name, `is_dm`, `joined_count`, `invited_count`, the unread
|
|
counts, `bump_stamp`, and the `timeline` with `prev_batch` (a `/messages`
|
|
token), `limited` and `num_live` (how many of the timeline's events are
|
|
new since the previous response). An invite is its `invite_state`, a knock
|
|
its `knock_state`, and neither has a timeline.
|
|
|
|
### Extensions
|
|
|
|
| Extension | What |
|
|
|---|---|
|
|
| `to_device` | The device's queued messages, 100 at a time by default. It has a `since` token of its own; coming back with it acknowledges the batch. |
|
|
| `e2ee` | `device_lists`, `device_one_time_keys_count`, `device_unused_fallback_key_types`, worked out as for `/sync`. |
|
|
| `account_data` | Global, and per room for the rooms in the response's lists and subscriptions — all of it for a room sent in full. |
|
|
| `receipts` | Per room, the receipts since the last response; all of them for a room sent in full. |
|
|
| `typing` | Per room, the typing list when it changed — including to empty, so a client stops showing somebody typing. |
|
|
|
|
`lists` and `rooms` narrow the rooms a per-room extension covers. Naming
|
|
any list the request has is every room in the windows: the windows are
|
|
merged before the extensions run.
|
|
|
|
### Long polling
|
|
|
|
A response is returned at once on a new connection; otherwise only when a
|
|
room is sent, a list's count moved, or an extension has something to say.
|
|
Until then it waits on the same notifier as `/sync`.
|
|
|
|
## What is deliberately missing
|
|
|
|
| Thing | Needs |
|
|
|---|---|
|
|
| Remote presence | presence over federation |
|
|
| `contains_url`, `event_fields` | nothing; they are parsed and ignored |
|
|
| Per-event history visibility | state storage (open question 1) |
|
|
|
|
Threads, relations and the summaries every timeline event carries in
|
|
`unsigned.m.relations` are in [relations.md](relations.md).
|
|
|
|
`device_lists` is sent as two empty lists rather than left out: a client
|
|
reads a missing one as a malformed response, where two empty lists say
|
|
"nothing changed". How the two lists are worked out, and the key counts
|
|
that ride along with them, are in [e2ee.md](e2ee.md).
|
|
|
|
Each joined room carries `unread_notifications` from the push server,
|
|
whose counts are the seventh stream — the main timeline's alone, beside an
|
|
`unread_thread_notifications` map of each thread's, when the filter sets
|
|
`unread_thread_notifications` — and the global account data carries
|
|
`m.push_rules`, rendered over the server defaults for a user who never
|
|
changed a rule. Both are described in [push.md](push.md). A build with no
|
|
push server leaves the counts out entirely rather than sending zeroes,
|
|
because a zero is a claim — "you have nothing unread" — and an absent field
|
|
is not.
|
|
|
|
## Endpoints
|
|
|
|
| Endpoint | Notes |
|
|
|---|---|
|
|
| `GET /sync` | Long-polls. `?since=`, `?filter=`, `?timeout=`, `?full_state=`, `?set_presence=`. |
|
|
| `POST /_matrix/client/unstable/org.matrix.simplified_msc3575/sync` | Simplified Sliding Sync; see below. `?pos=`, `?timeout=`, `?set_presence=`. |
|
|
| `GET /_matrix/client/v1/rooms/{id}/relations/{eventId}` · `GET /_matrix/client/v1/rooms/{id}/threads` | See [relations.md](relations.md). |
|
|
| `POST /search` | See [search.md](search.md). |
|
|
| `GET /rooms/{id}/messages` | `?dir=` (`b` default), `?from=`, `?to=`, `?limit=`, `?filter=`. No `end` at the end of the room. |
|
|
| `GET /rooms/{id}/context/{eventId}` | `?limit=`, `?filter=`. Half the budget each side, the odd one backwards. |
|
|
| `POST /user/{userId}/filter` | Returns an opaque ID. |
|
|
| `GET /user/{userId}/filter/{filterId}` | Byte for byte as uploaded. |
|
|
| `PUT /rooms/{id}/typing/{userId}` | Caller only. Requires membership. |
|
|
| `POST /rooms/{id}/receipt/{type}/{eventId}` | `m.read`, `m.read.private`, or `m.fully_read`. |
|
|
| `POST /rooms/{id}/read_markers` | Up to three writes. |
|
|
| `PUT /sendToDevice/{type}/{txnId}` | The transaction ID is accepted and not recorded; see below. |
|
|
| `GET` / `PUT /presence/{userId}/status` | `PUT` is caller only. `403` when presence is off. |
|
|
|
|
The `sendToDevice` transaction ID is accepted and not used. Its purpose is
|
|
idempotency, and a to-device message is already idempotent in the way that
|
|
matters: a duplicate is a second copy of a key share, which the receiving
|
|
client discards. Recording it would mean a table whose only reader would be
|
|
a retry that costs nothing anyway.
|
|
|
|
Typing, presence and the receipts are the caller's own claims, not data
|
|
belonging to them, so unlike a profile an admin may not set them for
|
|
somebody else: there is no administrative reason to say that another user
|
|
is typing.
|
|
|
|
## Testing
|
|
|
|
| Level | Where |
|
|
|---|---|
|
|
| Tokens and filters | `internal/syncapi/{token,filter}_test.go` — the parsing edges, which is where a client's stored token meets a newer server |
|
|
| Stream reads, the sync store | `internal/storage/sync_test.go`, against every driver |
|
|
| Visibility, windows, summaries | `internal/roomserver/stream_test.go` |
|
|
| The responses a client sees | `internal/clientapi/{sync,messages,ephemeral}_test.go`, through HTTP |
|
|
| Wiring | `internal/setup/phase4_test.go` — the whole stack, to catch an endpoint written but never registered |
|
|
| Integration | Complement: `TestSync*`, `TestMessages*`, `TestContext*`, `TestFilter*`, `TestTyping*`, `TestReceipt*`, `TestToDevice*`, `TestLazyLoading*` |
|
|
|
|
The HTTP tests are deliberately not unit tests of the builder. What is
|
|
worth pinning down is the response a client sees — which section a room
|
|
lands in, whether a token handed back can be handed in again, whether
|
|
paging terminates and loses nothing — and none of that is visible from
|
|
inside.
|
|
|
|
## Known cost
|
|
|
|
A sync over K changed rooms is roughly a constant number of queries per
|
|
room: one `RoomSync` (room row, summary, membership history, timeline,
|
|
state) plus the per-room ephemeral and account-data reads. On an incremental
|
|
sync K is usually zero or one, which is the case that matters. An initial
|
|
sync of a user in fifty rooms is fifty times that, and would be worth a
|
|
batched read if it ever shows up in a profile — the shape of the code does
|
|
not stop it, since `changedRooms` already decides the set up front.
|