Phase 4: sync

Implements PLAN.md Phase 4: the read model. /sync, /messages and /context,
the filters that narrow them, and the streams a response carries that are
not room events — typing, receipts, presence and to-device messages. With
this a client can be used against Warp, which is milestone M1.

The one architectural choice is that the syncapi has no copy of the rooms:
it reads them through the roomserver. Dendrite gives its syncapi its own
tables and feeds them from the output stream, which is right for a design
where the two can be separate processes, and Dendrite's could be. Warp is
one process by decision, and there the copy buys nothing and costs the
hardest class of bug a homeserver has — a read model that disagrees with
the room, where a client is shown a message that was rejected and the only
way to find out is to compare two tables. So the roomserver grew a read
model (stream.go, roomsync.go, visibility.go) rather than exporting its
store: reading an event means parsing it under its room's version, and a
second copy of the version registry, the parser and the client projection
would be a second place for them to disagree.

syncapi: stream tokens naming a position in each of five independent
streams, so a client that has seen receipt 7 but only event 4 can say so in
one opaque string. A token with fewer positions than this build writes is
accepted and the rest read as zero, which is what lets a token stored
before a stream existed go on working. The PDU position comes from
MAX(stream_ordering) and not from the counter row the other four use: the
counter is advanced inside the writing transaction, so a token taken from
it would name a position whose event no reader can see yet, and the event
would be skipped for good once that transaction committed.

Long polling goes through a notifier that wakes by room and by user rather
than globally — an event wakes the listeners of its room, a membership
event also wakes the user it names, so an invite reaches a client that is
not in the room yet. A wake-up is only ever a hint: the listener re-reads
the streams and decides for itself what changed, which is what makes it
safe for the notifier to hold no state about what each client has seen. The
listener registers before the read, so an event landing in between leaves
the channel signalled and the sync reads again rather than blocking on a
change it just missed. EventPersisted must not block, because it is called
with the room's write lock still held.

"Something happened" is not "a token moved". A stream can move without this
user being concerned by it, and returning on every such move would turn a
long poll back into a busy loop. The same trap caught presence twice over:
a client says it is still online by syncing, so ?set_presence= makes every
sync a presence write, and a stream that moved each time would wake every
peer, which would sync, which would move it again. SetPresence therefore
reports whether anything a peer would notice actually changed, and a write
that only refreshes the last-active timestamp leaves the stream alone.

Timeline windows are named by their two ends and a direction, Low exclusive
and High inclusive in both directions, rather than by "from" and "to" as
the wire format spells them. Each of those would otherwise be the low bound
in one direction and the high bound in the other, which is exactly the
off-by-one every pagination bug is made of; the handlers translate once and
nothing below them thinks about it again. A sync's timeline is read
backwards and reported forwards, because a client wants the newest events
of the window it missed — a forward read with a limit hands it the oldest
and then calls the rest a gap.

visibility: history visibility is now a span rather than a yes or no, which
is what Phase 3 deferred here. A user who has left a room is served the
room as it was when they left — their state reads, their members and their
timeline — instead of being refused, and is bounded above at their own
membership event so a kicked client cannot go on streaming. Their span is
found from their own member events, so a user who joined, left and rejoined
sees their latest span. ?at= on /members is honoured for the same reason.
What is left is that the span is computed from the room's current
history_visibility where the spec judges each event by the one in force
when it was sent; that is a state read per event and belongs with open
question 1.

Typing is held in memory and nowhere else: notifications expire in seconds,
are worthless across a restart, and arrive one per keystroke-burst per
room, so persisting them would be the busiest table in the database holding
the least valuable data in it. Receipts only ever advance, compared on the
position of the event acknowledged rather than on arrival, because two of a
user's clients read at different speeds and the slower one must not drag
the marker backwards. To-device queues are per device and drained by the
next sync rather than by the one that delivered them, so a client that
crashes in between gets the batch again — the right way round, since a lost
key share leaves a message permanently undecryptable and a duplicate is
ignored. A wildcard addressee is expanded when sent, not stored, because
"every device" means the ones that existed then.

Presence is off by default, which answers PLAN.md open question 3. It is
the noisiest thing in Matrix and the one genuinely fan-out notification
here; a private homeserver gains little from it. With it off the endpoints
refuse rather than disappearing, because a client that gets a 404 from
/presence concludes the server is too old to have it and stops asking.

storage: one migration per driver adding the receipt, to-device, presence
and filter tables with a counter row each, and a stream position on the
existing account data table rather than a parallel one — there is exactly
one current value per (user, room, type), and a second table would be a
second thing to keep in step. Account data is reported in an event list but
is not an event, so it gets its own response type: a room event with an
empty sender and no timestamp would make a client decide what that meant.

Deliberately missing, each named in docs/sync.md: notification counts (they
need push rules, Phase 8), device lists (the keyserver, Phase 7), anything
remote (Phase 5), threads (Phase 10). device_lists is sent as two empty
lists because a client reads a missing one as malformed; the notification
counts are left out entirely, because a zero there is a claim and an absent
field is not.

Tested through HTTP rather than against the builder, because what is worth
pinning down is the response a client sees — which section a room lands in,
whether a token handed back can be handed in again, whether paging
terminates and loses nothing. The storage tests run against both SQL
drivers; the notifier has its own tests for the concurrent case. A live
server was driven end to end, which is what caught the two bugs the unit
tests had missed: a backwards /messages page must be in reverse
chronological order, and `start` must name where the page began rather than
position zero when ?from= was omitted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

server/internal/clientapi/clientapi.go

@@ -23,6 +23,7 @@ import (
 	"github.com/dariuszjaniak/warp/server/internal/ratelimit"
 	"github.com/dariuszjaniak/warp/server/internal/roomserver"
 	"github.com/dariuszjaniak/warp/server/internal/settings"
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
 	"github.com/dariuszjaniak/warp/server/internal/userapi"
 )
 
@@ -32,6 +33,7 @@ type ClientAPI struct {
 	clientBaseURL string
 	users         *userapi.API
 	rooms         *roomserver.API
+	sync          *syncapi.API
 	settings      *settings.Service
 	federation    *federation.Gate
 	registration  config.Registration
@@ -48,6 +50,12 @@ type ClientAPI struct {
 	limitRegister   *ratelimit.Limiter
 	limitRoomCreate *ratelimit.Limiter
 	limitJoin       *ratelimit.Limiter
+
+	// limitSync is deliberately generous. A client is in /sync all the
+	// time, and a long poll that is refused leaves it reconnecting in a
+	// loop — the one place where rate limiting makes the load worse rather
+	// than better.
+	limitSync *ratelimit.Limiter
 }
 
 // Options configure the component.
@@ -70,6 +78,12 @@ type Options struct {
 	// createRoom it cannot honour.
 	Rooms *roomserver.API
 
+	// Sync is the syncapi component. Like the roomserver it is optional,
+	// and for the same reason: without it /sync, /messages, /context and
+	// the ephemeral endpoints answer M_UNRECOGNIZED rather than
+	// pretending there is a stream to read.
+	Sync *syncapi.API
+
 	// Settings is the runtime-settings service; the registration switch is
 	// read from it on every registration attempt.
 	Settings *settings.Service
@@ -118,6 +132,7 @@ func New(opts Options) (*ClientAPI, error) {
 		clientBaseURL: opts.ClientBaseURL,
 		users:         opts.Users,
 		rooms:         opts.Rooms,
+		sync:          opts.Sync,
 		settings:      opts.Settings,
 		federation:    opts.Federation,
 		registration:  opts.Registration,
@@ -137,10 +152,12 @@ func New(opts Options) (*ClientAPI, error) {
 		// reads.
 		c.limitRoomCreate = ratelimit.New(bucket(opts.RateLimit.RoomCreate))
 		c.limitJoin = ratelimit.New(bucket(opts.RateLimit.Join))
+		c.limitSync = ratelimit.New(bucket(opts.RateLimit.Sync))
 	} else {
 		c.limitDefault, c.limitLogin, c.limitRegister =
 			ratelimit.Disabled(), ratelimit.Disabled(), ratelimit.Disabled()
 		c.limitRoomCreate, c.limitJoin = ratelimit.Disabled(), ratelimit.Disabled()
+		c.limitSync = ratelimit.Disabled()
 	}
 	return c, nil
 }
@@ -204,9 +221,45 @@ func (c *ClientAPI) RegisterRoutes(r *httputil.Router) {
 	c.authed(r, "PUT "+v3+"/user/{userID}/rooms/{roomID}/account_data/{dataType}",
 		c.limitDefault, c.handlePutRoomAccountData)
 
-	c.authed(r, "GET "+v3+"/sync", c.limitDefault, c.handleSync)
-
 	c.registerRoomRoutes(r)
+	c.registerSyncRoutes(r)
+}
+
+// registerSyncRoutes adds the endpoints the syncapi serves, if there is
+// one.
+//
+// Like the room routes they are left unregistered rather than stubbed in a
+// build without the component: a client that gets M_UNRECOGNIZED from
+// /sync knows the server does not have it, where an empty success would
+// have it believe nothing was happening.
+func (c *ClientAPI) registerSyncRoutes(r *httputil.Router) {
+	if c.sync == nil {
+		return
+	}
+
+	c.authed(r, "GET "+v3+"/sync", c.limitSync, c.handleSync)
+	c.authed(r, "GET "+v3+"/rooms/{roomID}/messages", c.limitDefault, c.handleMessages)
+	c.authed(r, "GET "+v3+"/rooms/{roomID}/context/{eventID}", c.limitDefault, c.handleContext)
+
+	// Filters.
+	c.authed(r, "POST "+v3+"/user/{userID}/filter", c.limitDefault, c.handlePostFilter)
+	c.authed(r, "GET "+v3+"/user/{userID}/filter/{filterID}", c.limitDefault, c.handleGetFilter)
+
+	// The ephemeral streams. Typing goes in the sync limiter's class
+	// rather than the default one: a client sends one per keystroke-burst,
+	// and it is the chattiest write in the API.
+	c.authed(r, "PUT "+v3+"/rooms/{roomID}/typing/{userID}", c.limitSync, c.handlePutTyping)
+	c.authed(r, "POST "+v3+"/rooms/{roomID}/receipt/{receiptType}/{eventID}",
+		c.limitSync, c.handlePostReceipt)
+	c.authed(r, "POST "+v3+"/rooms/{roomID}/read_markers", c.limitDefault, c.handlePostReadMarkers)
+
+	// To-device messaging. A client encrypting for a busy room sends one
+	// of these per device, so it shares the sync budget too.
+	c.authed(r, "PUT "+v3+"/sendToDevice/{eventType}/{txnID}", c.limitSync, c.handleSendToDevice)
+
+	// Presence.
+	c.authed(r, "GET "+v3+"/presence/{userID}/status", c.limitDefault, c.handleGetPresence)
+	c.authed(r, "PUT "+v3+"/presence/{userID}/status", c.limitSync, c.handlePutPresence)
 }
 
 // registerRoomRoutes adds the room endpoints, if there is a roomserver.

server/internal/clientapi/clientapi_test.go

@@ -19,6 +19,7 @@ import (
 	"github.com/dariuszjaniak/warp/server/internal/settings"
 	"github.com/dariuszjaniak/warp/server/internal/storage"
 	_ "github.com/dariuszjaniak/warp/server/internal/storage/drivers"
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
 	"github.com/dariuszjaniak/warp/server/internal/userapi"
 )
 
@@ -34,8 +35,10 @@ type testServer struct {
 	handler  http.Handler
 	users    *userapi.API
 	rooms    *roomserver.API
+	syncAPI  *syncapi.API
 	settings *settings.Service
 	fedGate  *federation.Gate
+	db       storage.Database
 }
 
 // newTestServer builds a server. tweak adjusts the options before the
@@ -84,22 +87,45 @@ func newTestServer(t *testing.T, tweak func(*clientapi.Options)) *testServer {
 	if err != nil {
 		t.Fatalf("signing.Generate: %v", err)
 	}
+	// The notifier is the seam between the two: the roomserver's emit
+	// stage and the syncapi's wake-up are the same object, exactly as in
+	// a real server.
+	notifier := syncapi.NewNotifier()
 	rooms, err := roomserver.New(roomserver.Options{
 		ServerName:        testServerName,
 		Store:             db.Rooms(),
 		Profiles:          users,
 		SigningKey:        key,
 		FederationEnabled: settingsService.FederationEnabled,
+		Notifier:          notifier,
 		Logger:            slog.New(slog.DiscardHandler),
 	})
 	if err != nil {
 		t.Fatalf("roomserver.New: %v", err)
 	}
 
+	// Presence is on in tests and off in a default config: the tests are
+	// the only thing that exercises it, and a switch nothing turns on is
+	// a switch nothing tests.
+	sync, err := syncapi.New(syncapi.Options{
+		ServerName:      testServerName,
+		Rooms:           rooms,
+		Accounts:        users,
+		Store:           db.Sync(),
+		Notifier:        notifier,
+		PresenceEnabled: true,
+		Logger:          slog.New(slog.DiscardHandler),
+	})
+	if err != nil {
+		t.Fatalf("syncapi.New: %v", err)
+	}
+	t.Cleanup(sync.Close)
+
 	opts := clientapi.Options{
 		ServerName: testServerName,
 		Users:      users,
 		Rooms:      rooms,
+		Sync:       sync,
 		Settings:   settingsService,
 		Federation: gate,
 		Logger:     slog.New(slog.DiscardHandler),
@@ -120,8 +146,10 @@ func newTestServer(t *testing.T, tweak func(*clientapi.Options)) *testServer {
 		handler:  router,
 		users:    users,
 		rooms:    rooms,
+		syncAPI:  sync,
 		settings: settingsService,
 		fedGate:  gate,
+		db:       db,
 	}
 }
 

server/internal/clientapi/ephemeral.go

@@ -0,0 +1,269 @@
+package clientapi
+
+import (
+	"encoding/json"
+	"net/http"
+	"time"
+
+	"github.com/dariuszjaniak/warp/server/internal/httputil"
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
+	"github.com/dariuszjaniak/warp/server/internal/userapi"
+)
+
+// The endpoints behind the streams that are not room events: typing,
+// receipts, read markers, to-device messages and presence.
+//
+// They are all writes, and all of them are writes a client makes constantly
+// — a typing notification per keystroke-burst, a receipt per message read.
+// None of them creates an event, which is why none of them goes through the
+// roomserver.
+
+// typingRequest is the PUT /rooms/{roomID}/typing/{userID} body.
+type typingRequest struct {
+	Typing bool `json:"typing"`
+
+	// Timeout is in milliseconds, and is only meaningful when Typing is
+	// true.
+	Timeout int64 `json:"timeout"`
+}
+
+// handlePutTyping implements
+// PUT /_matrix/client/v3/rooms/{roomID}/typing/{userID}.
+func (c *ClientAPI) handlePutTyping(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	roomID, err := pathRoomID(r)
+	if err != nil {
+		return err
+	}
+	if err := requireOwnUserID(auth, r.PathValue("userID")); err != nil {
+		return err
+	}
+
+	var req typingRequest
+	if err := httputil.DecodeJSON(w, r, &req); err != nil {
+		return err
+	}
+	if req.Timeout < 0 {
+		return httputil.InvalidParam("timeout must not be negative")
+	}
+
+	if err := c.sync.SetTyping(r.Context(), syncapi.TypingRequest{
+		RoomID:  roomID,
+		User:    auth.UserID,
+		Typing:  req.Typing,
+		Timeout: time.Duration(req.Timeout) * time.Millisecond,
+	}); err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, struct{}{})
+	return nil
+}
+
+// receiptRequest is the POST /rooms/{roomID}/receipt/{receiptType}/{eventID}
+// body. Every field is optional; the only one defined is the thread.
+type receiptRequest struct {
+	ThreadID string `json:"thread_id"`
+}
+
+// handlePostReceipt implements
+// POST /_matrix/client/v3/rooms/{roomID}/receipt/{receiptType}/{eventID}.
+func (c *ClientAPI) handlePostReceipt(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	roomID, err := pathRoomID(r)
+	if err != nil {
+		return err
+	}
+	receiptType := r.PathValue("receiptType")
+	if receiptType == "" {
+		return httputil.MissingParam("A receipt type is required")
+	}
+	eventID := r.PathValue("eventID")
+	if eventID == "" {
+		return httputil.MissingParam("An event ID is required")
+	}
+
+	// The body is optional: a client marking a room read sends none.
+	var req receiptRequest
+	if _, err := httputil.DecodeOptionalJSON(w, r, &req); err != nil {
+		return err
+	}
+
+	if err := c.sync.SetReceipt(r.Context(), syncapi.ReceiptRequest{
+		RoomID:      roomID,
+		User:        auth.UserID,
+		ReceiptType: receiptType,
+		EventID:     eventID,
+		ThreadID:    req.ThreadID,
+	}); err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, struct{}{})
+	return nil
+}
+
+// readMarkersRequest is the POST /rooms/{roomID}/read_markers body.
+type readMarkersRequest struct {
+	FullyRead   string `json:"m.fully_read"`
+	Read        string `json:"m.read"`
+	ReadPrivate string `json:"m.read.private"`
+}
+
+// handlePostReadMarkers implements
+// POST /_matrix/client/v3/rooms/{roomID}/read_markers.
+func (c *ClientAPI) handlePostReadMarkers(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	roomID, err := pathRoomID(r)
+	if err != nil {
+		return err
+	}
+
+	var req readMarkersRequest
+	if err := httputil.DecodeJSON(w, r, &req); err != nil {
+		return err
+	}
+	if req.FullyRead == "" && req.Read == "" && req.ReadPrivate == "" {
+		return httputil.MissingParam(
+			"At least one of m.fully_read, m.read or m.read.private is required")
+	}
+
+	if err := c.sync.SetReadMarkers(r.Context(), syncapi.ReadMarkersRequest{
+		RoomID:      roomID,
+		User:        auth.UserID,
+		FullyRead:   req.FullyRead,
+		Read:        req.Read,
+		ReadPrivate: req.ReadPrivate,
+	}); err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, struct{}{})
+	return nil
+}
+
+// sendToDeviceRequest is the PUT /sendToDevice/{eventType}/{txnID} body.
+type sendToDeviceRequest struct {
+	Messages map[string]map[string]json.RawMessage `json:"messages"`
+}
+
+// handleSendToDevice implements
+// PUT /_matrix/client/v3/sendToDevice/{eventType}/{txnID}.
+//
+// The transaction ID is accepted and not used. Its purpose is idempotency,
+// and a to-device message is already idempotent in the way that matters: a
+// duplicate is a second copy of a key share, which the receiving client
+// discards. Recording the transaction would mean a table whose only reader
+// would be a retry that costs nothing anyway.
+func (c *ClientAPI) handleSendToDevice(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	eventType := r.PathValue("eventType")
+	if eventType == "" {
+		return httputil.MissingParam("An event type is required")
+	}
+	if r.PathValue("txnID") == "" {
+		return httputil.MissingParam("A transaction ID is required")
+	}
+
+	var req sendToDeviceRequest
+	if err := httputil.DecodeJSON(w, r, &req); err != nil {
+		return err
+	}
+
+	if err := c.sync.SendToDevice(r.Context(), syncapi.SendToDeviceRequest{
+		Sender:    auth.UserID,
+		EventType: eventType,
+		Messages:  req.Messages,
+	}); err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, struct{}{})
+	return nil
+}
+
+// presenceRequest is the PUT /presence/{userID}/status body.
+type presenceRequest struct {
+	Presence string `json:"presence"`
+
+	// StatusMsg is a pointer so that a request which does not mention it
+	// can be told from one that clears it.
+	StatusMsg *string `json:"status_msg"`
+}
+
+// presenceResponse is what GET /presence/{userID}/status returns.
+type presenceResponse struct {
+	Presence string `json:"presence"`
+
+	StatusMsg string `json:"status_msg,omitempty"`
+
+	// LastActiveAgo is milliseconds since the user was last seen, left out
+	// for a user nothing is known about.
+	LastActiveAgo *int64 `json:"last_active_ago,omitempty"`
+
+	// CurrentlyActive is only meaningful while the user is online.
+	CurrentlyActive bool `json:"currently_active,omitempty"`
+}
+
+// handlePutPresence implements
+// PUT /_matrix/client/v3/presence/{userID}/status.
+func (c *ClientAPI) handlePutPresence(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	if err := requireOwnUserID(auth, r.PathValue("userID")); err != nil {
+		return err
+	}
+
+	var req presenceRequest
+	if err := httputil.DecodeJSON(w, r, &req); err != nil {
+		return err
+	}
+	if req.Presence == "" {
+		return httputil.MissingParam("A presence state is required")
+	}
+
+	if err := c.sync.SetPresence(r.Context(), syncapi.PresenceRequest{
+		User:      auth.UserID,
+		Presence:  req.Presence,
+		StatusMsg: req.StatusMsg,
+	}); err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, struct{}{})
+	return nil
+}
+
+// handleGetPresence implements
+// GET /_matrix/client/v3/presence/{userID}/status.
+//
+// Anybody may ask about anybody: the spec has no visibility rule on
+// presence beyond the server choosing who to publish changes to, and a
+// client showing a room's member list asks about every member.
+func (c *ClientAPI) handleGetPresence(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error {
+	userID := r.PathValue("userID")
+	if userID == "" {
+		return httputil.MissingParam("A user ID is required")
+	}
+
+	state, err := c.sync.Presence(r.Context(), userID)
+	if err != nil {
+		return syncError(err)
+	}
+
+	resp := presenceResponse{
+		Presence:        state.Presence,
+		StatusMsg:       state.StatusMsg,
+		CurrentlyActive: state.Presence == syncapi.PresenceOnline,
+	}
+	if state.LastActiveMS > 0 {
+		ago := max(c.now().UnixMilli()-state.LastActiveMS, 0)
+		resp.LastActiveAgo = &ago
+	}
+	httputil.WriteJSON(w, http.StatusOK, resp)
+	return nil
+}
+
+// requireOwnUserID insists that the user ID in a path is the caller's.
+//
+// Typing and presence are claims about the caller, not data belonging to
+// them, so unlike a profile an admin may not set them for somebody else:
+// there is no administrative reason to say that another user is typing.
+func requireOwnUserID(auth userapi.Authenticated, pathUserID string) error {
+	if pathUserID == "" {
+		return httputil.MissingParam("A user ID is required")
+	}
+	if pathUserID != auth.UserID {
+		return httputil.Forbidden("Cannot act on behalf of another user")
+	}
+	return nil
+}

server/internal/clientapi/ephemeral_test.go

@@ -0,0 +1,362 @@
+package clientapi_test
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+	"testing"
+)
+
+// The streams that are not room events: typing, receipts, read markers,
+// to-device messages and presence.
+
+// ephemeralEvent finds a room's ephemeral EDU of a type in a sync result.
+func ephemeralEvent(result syncResult, roomID, eventType string) (syncEvent, bool) {
+	for _, e := range result.Rooms.Join[roomID].Ephemeral.Events {
+		if e.Type == eventType {
+			return e, true
+		}
+	}
+	return syncEvent{}, false
+}
+
+func TestTypingReachesTheOtherMembers(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	since := ts.sync(bob, "").NextBatch
+
+	expectStatus(t, ts.put(fmt.Sprintf("/_matrix/client/v3/rooms/%s/typing/%s",
+		url.PathEscape(roomID), url.PathEscape(alice.UserID)),
+		alice.AccessToken, map[string]any{"typing": true, "timeout": 30000}), http.StatusOK)
+
+	result := ts.sync(bob, since)
+	typing, ok := ephemeralEvent(result, roomID, "m.typing")
+	if !ok {
+		t.Fatalf("no m.typing in the ephemeral stream: %+v", result.Rooms.Join[roomID].Ephemeral)
+	}
+	users, _ := typing.Content["user_ids"].([]any)
+	if len(users) != 1 || users[0] != alice.UserID {
+		t.Errorf("user_ids = %v, want [%s]", users, alice.UserID)
+	}
+
+	// Stopping is reported as the list without them, not as nothing: a
+	// client replaces what it had.
+	expectStatus(t, ts.put(fmt.Sprintf("/_matrix/client/v3/rooms/%s/typing/%s",
+		url.PathEscape(roomID), url.PathEscape(alice.UserID)),
+		alice.AccessToken, map[string]any{"typing": false}), http.StatusOK)
+
+	stopped := ts.sync(bob, result.NextBatch)
+	if _, ok := ephemeralEvent(stopped, roomID, "m.typing"); ok {
+		t.Error("m.typing was still reported after the notification was withdrawn")
+	}
+}
+
+func TestTypingIsOnlyAboutTheCaller(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+
+	rec := ts.put(fmt.Sprintf("/_matrix/client/v3/rooms/%s/typing/%s",
+		url.PathEscape(roomID), url.PathEscape(bob.UserID)),
+		alice.AccessToken, map[string]any{"typing": true})
+	expectError(t, rec, http.StatusForbidden, "M_FORBIDDEN")
+}
+
+func TestTypingRequiresMembership(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+
+	rec := ts.put(fmt.Sprintf("/_matrix/client/v3/rooms/%s/typing/%s",
+		url.PathEscape(roomID), url.PathEscape(bob.UserID)),
+		bob.AccessToken, map[string]any{"typing": true})
+	expectError(t, rec, http.StatusForbidden, "M_FORBIDDEN")
+}
+
+func TestReceiptReachesTheRoom(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	eventID := ts.sendMessage(alice, roomID, "read me")
+	since := ts.sync(alice, "").NextBatch
+
+	expectStatus(t, ts.post(fmt.Sprintf("/_matrix/client/v3/rooms/%s/receipt/m.read/%s",
+		url.PathEscape(roomID), url.PathEscape(eventID)),
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	result := ts.sync(alice, since)
+	receipts, ok := ephemeralEvent(result, roomID, "m.receipt")
+	if !ok {
+		t.Fatalf("no m.receipt in the ephemeral stream: %+v", result.Rooms.Join[roomID].Ephemeral)
+	}
+
+	// The shape is event ID -> receipt type -> user ID -> data.
+	byType, _ := receipts.Content[eventID].(map[string]any)
+	byUser, _ := byType["m.read"].(map[string]any)
+	if _, found := byUser[bob.UserID]; !found {
+		t.Errorf("m.receipt does not name %s: %+v", bob.UserID, receipts.Content)
+	}
+}
+
+func TestPrivateReceiptStaysWithItsSender(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	eventID := ts.sendMessage(alice, roomID, "read me privately")
+	aliceSince := ts.sync(alice, "").NextBatch
+	bobSince := ts.sync(bob, "").NextBatch
+
+	expectStatus(t, ts.post(fmt.Sprintf("/_matrix/client/v3/rooms/%s/receipt/m.read.private/%s",
+		url.PathEscape(roomID), url.PathEscape(eventID)),
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	// Bob sees his own.
+	own := ts.sync(bob, bobSince)
+	if _, ok := ephemeralEvent(own, roomID, "m.receipt"); !ok {
+		t.Error("a private receipt must be reported back to its sender")
+	}
+
+	// Alice does not.
+	theirs := ts.sync(alice, aliceSince)
+	if e, ok := ephemeralEvent(theirs, roomID, "m.receipt"); ok {
+		t.Errorf("a private receipt leaked to another member: %+v", e.Content)
+	}
+}
+
+func TestReceiptForAnUnknownEventIsNotFound(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/receipt/m.read/$nope",
+		alice.AccessToken, map[string]any{})
+	expectError(t, rec, http.StatusNotFound, "M_NOT_FOUND")
+}
+
+func TestReceiptOfAnUnknownTypeIsRefused(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	eventID := ts.sendMessage(alice, roomID, "hello")
+
+	rec := ts.post(fmt.Sprintf("/_matrix/client/v3/rooms/%s/receipt/m.invented/%s",
+		url.PathEscape(roomID), url.PathEscape(eventID)), alice.AccessToken, map[string]any{})
+	expectError(t, rec, http.StatusBadRequest, "M_INVALID_PARAM")
+}
+
+func TestReadMarkersSetTheFullyReadAccountData(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	eventID := ts.sendMessage(alice, roomID, "up to here")
+	since := ts.sync(alice, "").NextBatch
+
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/read_markers",
+		alice.AccessToken, map[string]any{
+			"m.fully_read": eventID,
+			"m.read":       eventID,
+		}), http.StatusOK)
+
+	result := ts.sync(alice, since)
+	room := result.Rooms.Join[roomID]
+	if !contains(typesOf(room.AccountData.Events), "m.fully_read") {
+		t.Errorf("room account data = %v, want m.fully_read", typesOf(room.AccountData.Events))
+	}
+	if _, ok := ephemeralEvent(result, roomID, "m.receipt"); !ok {
+		t.Error("the m.read in the same request should have become a receipt")
+	}
+}
+
+func TestReadMarkersNeedsAtLeastOneMarker(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/read_markers",
+		alice.AccessToken, map[string]any{})
+	expectError(t, rec, http.StatusBadRequest, "M_MISSING_PARAM")
+}
+
+func TestSendToDeviceReachesTheDeviceAndIsThenDrained(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+
+	bobSince := ts.sync(bob, "").NextBatch
+
+	expectStatus(t, ts.put("/_matrix/client/v3/sendToDevice/m.room_key/txn1",
+		alice.AccessToken, map[string]any{
+			"messages": map[string]any{
+				bob.UserID: map[string]any{
+					bob.DeviceID: map[string]any{"session_id": "abc"},
+				},
+			},
+		}), http.StatusOK)
+
+	delivered := ts.sync(bob, bobSince)
+	if len(delivered.ToDevice.Events) != 1 {
+		t.Fatalf("to_device = %+v, want one message", delivered.ToDevice.Events)
+	}
+	msg := delivered.ToDevice.Events[0]
+	if msg.Type != "m.room_key" || msg.Sender != alice.UserID {
+		t.Errorf("to_device event = %+v, want an m.room_key from %s", msg, alice.UserID)
+	}
+	if msg.Content["session_id"] != "abc" {
+		t.Errorf("content = %+v, want the body as it was sent", msg.Content)
+	}
+
+	// Syncing past the batch acknowledges it; the next sync must not
+	// deliver it again.
+	again := ts.sync(bob, delivered.NextBatch)
+	if len(again.ToDevice.Events) != 0 {
+		t.Errorf("to_device = %+v, want nothing after the batch was acknowledged", again.ToDevice.Events)
+	}
+
+	// Alice's own devices were not addressed and must see nothing.
+	aliceResult := ts.sync(alice, "")
+	if len(aliceResult.ToDevice.Events) != 0 {
+		t.Errorf("the sender received their own to-device message: %+v", aliceResult.ToDevice.Events)
+	}
+}
+
+func TestSendToDeviceWildcardAddressesEveryDevice(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	bobSecond := ts.login(bob, "SECONDDEVICE")
+
+	firstSince := ts.sync(bob, "").NextBatch
+	secondSince := ts.sync(bobSecond, "").NextBatch
+
+	expectStatus(t, ts.put("/_matrix/client/v3/sendToDevice/m.room_key/txn1",
+		alice.AccessToken, map[string]any{
+			"messages": map[string]any{
+				bob.UserID: map[string]any{"*": map[string]any{"session_id": "shared"}},
+			},
+		}), http.StatusOK)
+
+	for _, session := range []struct {
+		name  string
+		acc   account
+		since string
+	}{
+		{"the first device", bob, firstSince},
+		{"the second device", bobSecond, secondSince},
+	} {
+		result := ts.sync(session.acc, session.since)
+		if len(result.ToDevice.Events) != 1 {
+			t.Errorf("%s got %+v, want one message", session.name, result.ToDevice.Events)
+		}
+	}
+}
+
+func TestSendToDeviceToARemoteUserIsDroppedNotRefused(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+
+	// Federation is off. Failing the request would stop the local
+	// deliveries in the same batch from happening at all.
+	expectStatus(t, ts.put("/_matrix/client/v3/sendToDevice/m.room_key/txn1",
+		alice.AccessToken, map[string]any{
+			"messages": map[string]any{
+				"@carol:elsewhere.example": map[string]any{"ANY": map[string]any{}},
+			},
+		}), http.StatusOK)
+}
+
+func TestPresenceRoundTrips(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+
+	expectStatus(t, ts.put("/_matrix/client/v3/presence/"+url.PathEscape(alice.UserID)+"/status",
+		alice.AccessToken, map[string]any{
+			"presence":   "unavailable",
+			"status_msg": "in a meeting",
+		}), http.StatusOK)
+
+	rec := ts.get("/_matrix/client/v3/presence/"+url.PathEscape(alice.UserID)+"/status",
+		alice.AccessToken)
+	expectStatus(t, rec, http.StatusOK)
+
+	body := jsonBody(t, rec)
+	if body["presence"] != "unavailable" {
+		t.Errorf("presence = %v, want unavailable", body["presence"])
+	}
+	if body["status_msg"] != "in a meeting" {
+		t.Errorf("status_msg = %v, want \"in a meeting\"", body["status_msg"])
+	}
+}
+
+func TestPresenceOfAUserWhoNeverSetItIsOffline(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+
+	rec := ts.get("/_matrix/client/v3/presence/"+url.PathEscape(bob.UserID)+"/status",
+		alice.AccessToken)
+	expectStatus(t, rec, http.StatusOK)
+
+	if got := jsonBody(t, rec)["presence"]; got != "offline" {
+		t.Errorf("presence = %v, want offline", got)
+	}
+}
+
+func TestPresenceIsOnlyAboutTheCaller(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+
+	rec := ts.put("/_matrix/client/v3/presence/"+url.PathEscape(bob.UserID)+"/status",
+		alice.AccessToken, map[string]any{"presence": "online"})
+	expectError(t, rec, http.StatusForbidden, "M_FORBIDDEN")
+}
+
+func TestPresenceRejectsAnUnknownState(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+
+	rec := ts.put("/_matrix/client/v3/presence/"+url.PathEscape(alice.UserID)+"/status",
+		alice.AccessToken, map[string]any{"presence": "asleep"})
+	expectError(t, rec, http.StatusBadRequest, "M_INVALID_PARAM")
+}
+
+func TestPresenceReachesSomebodySharingARoom(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	since := ts.sync(alice, "").NextBatch
+
+	expectStatus(t, ts.put("/_matrix/client/v3/presence/"+url.PathEscape(bob.UserID)+"/status",
+		bob.AccessToken, map[string]any{"presence": "unavailable", "status_msg": "away"}),
+		http.StatusOK)
+
+	result := ts.sync(alice, since)
+	var found bool
+	for _, e := range result.Presence.Events {
+		if e.Sender == bob.UserID && e.Content["presence"] == "unavailable" {
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("presence = %+v, want %s reported as unavailable", result.Presence.Events, bob.UserID)
+	}
+}

server/internal/clientapi/login_test.go

@@ -479,32 +479,14 @@ func TestCapabilities(t *testing.T) {
 	}
 }
 
-// TestSyncPlaceholder documents what the Phase 1 /sync does: it answers a
-// valid token with an empty, well-formed response, and refuses a revoked
-// one. Phase 4 replaces it with the real syncapi.
-func TestSyncPlaceholder(t *testing.T) {
-	ts := newTestServer(t, nil)
-	acc := ts.register("alice", "superuser")
-
-	rec := ts.get("/_matrix/client/v3/sync", acc.AccessToken)
-	expectStatus(t, rec, http.StatusOK)
-
-	body := jsonBody(t, rec)
-	if next, _ := body["next_batch"].(string); next == "" {
-		t.Error("/sync must return a next_batch token")
-	}
-	rooms, _ := body["rooms"].(map[string]any)
-	for _, key := range []string{"join", "invite", "leave"} {
-		if _, present := rooms[key]; !present {
-			t.Errorf("rooms is missing %s: %s", key, rec.Body.String())
-		}
-	}
-}
-
 func TestUnknownClientEndpoint(t *testing.T) {
 	ts := newTestServer(t, nil)
 
-	rec := ts.get("/_matrix/client/v3/rooms/!a:b/messages", "")
+	// A real endpoint this build does not serve: third-party protocols
+	// belong to the application service API of PLAN.md phase 9. The spec's
+	// answer for an endpoint a server does not implement is
+	// M_UNRECOGNIZED, and the router's catch-all is what gives it.
+	rec := ts.get("/_matrix/client/v3/thirdparty/protocols", "")
 	expectError(t, rec, http.StatusNotFound, "M_UNRECOGNIZED")
 }
 

server/internal/clientapi/messages_test.go

@@ -0,0 +1,344 @@
+package clientapi_test
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+	"testing"
+)
+
+// /messages and /context: a room's history, which is the other half of the
+// read model. The thing worth pinning down is that paging terminates and
+// that it loses nothing — a client that pages from the bottom of a room to
+// the top must see every event exactly once.
+
+type messagesResult struct {
+	Start string      `json:"start"`
+	End   string      `json:"end"`
+	Chunk []syncEvent `json:"chunk"`
+	State []syncEvent `json:"state"`
+}
+
+// messages performs one /messages call.
+func (ts *testServer) messages(acc account, roomID string, query url.Values) messagesResult {
+	ts.t.Helper()
+
+	path := "/_matrix/client/v3/rooms/" + url.PathEscape(roomID) + "/messages"
+	if encoded := query.Encode(); encoded != "" {
+		path += "?" + encoded
+	}
+	rec := ts.get(path, acc.AccessToken)
+	expectStatus(ts.t, rec, http.StatusOK)
+	return decode[messagesResult](ts.t, rec)
+}
+
+func TestMessagesPagesBackwardsThroughTheWholeRoom(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	const messages = 7
+	for i := range messages {
+		ts.sendMessage(alice, roomID, fmt.Sprintf("line %d", i))
+	}
+
+	// Page backwards in twos until the server stops handing out a token.
+	var (
+		seen  []string
+		from  string
+		pages int
+	)
+	for {
+		query := url.Values{"dir": {"b"}, "limit": {"2"}}
+		if from != "" {
+			query.Set("from", from)
+		}
+		page := ts.messages(alice, roomID, query)
+		seen = append(seen, bodiesOf(page.Chunk)...)
+
+		pages++
+		if pages > 20 {
+			t.Fatal("paging backwards did not terminate")
+		}
+		if page.End == "" {
+			break
+		}
+		if len(page.Chunk) == 0 {
+			t.Fatalf("page %d was empty but carried an end token %q", pages, page.End)
+		}
+		from = page.End
+	}
+
+	// Every message exactly once. The order is newest first within a page
+	// and the pages walk backwards, so the whole set is what matters here;
+	// the ordering inside a page is covered below.
+	if len(seen) != messages {
+		t.Fatalf("saw %d messages over %d pages, want %d: %v", len(seen), pages, messages, seen)
+	}
+	counts := map[string]int{}
+	for _, body := range seen {
+		counts[body]++
+	}
+	for i := range messages {
+		body := fmt.Sprintf("line %d", i)
+		if counts[body] != 1 {
+			t.Errorf("%q appeared %d times, want once", body, counts[body])
+		}
+	}
+}
+
+func TestMessagesBackwardsStartsAtTheNewestEvent(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "older")
+	ts.sendMessage(alice, roomID, "newest")
+
+	page := ts.messages(alice, roomID, url.Values{"dir": {"b"}, "limit": {"1"}})
+	got := bodiesOf(page.Chunk)
+	if len(got) != 1 || got[0] != "newest" {
+		t.Errorf("chunk = %v, want [newest]: a backwards page with no ?from= starts at the end of the room", got)
+	}
+}
+
+func TestMessagesForwardsFromTheStartOfTheRoom(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "first")
+	ts.sendMessage(alice, roomID, "second")
+
+	page := ts.messages(alice, roomID, url.Values{"dir": {"f"}, "limit": {"100"}})
+	got := typesOf(page.Chunk)
+	if len(got) == 0 || got[0] != "m.room.create" {
+		t.Fatalf("chunk starts with %v, want the create event first", got)
+	}
+	bodies := bodiesOf(page.Chunk)
+	if len(bodies) != 2 || bodies[0] != "first" || bodies[1] != "second" {
+		t.Errorf("bodies = %v, want [first second] in order", bodies)
+	}
+}
+
+func TestMessagesClosesTheGapASyncReported(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	for i := range 5 {
+		ts.sendMessage(alice, roomID, fmt.Sprintf("line %d", i))
+	}
+
+	// A sync with a tight limit leaves a gap and hands back the token to
+	// close it with. That token has to be a valid ?from= here: it is the
+	// whole reason the two endpoints share a token format.
+	result := ts.syncWith(alice, url.Values{
+		"filter": {`{"room":{"timeline":{"limit":2}}}`},
+	})
+	room := result.Rooms.Join[roomID]
+	if !room.Timeline.Limited || room.Timeline.PrevBatch == "" {
+		t.Fatalf("expected a limited timeline with a prev_batch, got %+v", room.Timeline)
+	}
+
+	page := ts.messages(alice, roomID, url.Values{
+		"dir":   {"b"},
+		"from":  {room.Timeline.PrevBatch},
+		"limit": {"10"},
+	})
+
+	// Nothing the sync already showed may appear again.
+	synced := map[string]bool{}
+	for _, e := range room.Timeline.Events {
+		synced[e.EventID] = true
+	}
+	for _, e := range page.Chunk {
+		if synced[e.EventID] {
+			t.Errorf("%s was in both the sync and the page that was meant to precede it", e.EventID)
+		}
+	}
+	if len(page.Chunk) == 0 {
+		t.Error("the page closing the gap was empty")
+	}
+}
+
+func TestMessagesAppliesAFilter(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "a message")
+
+	page := ts.messages(alice, roomID, url.Values{
+		"dir":    {"b"},
+		"limit":  {"50"},
+		"filter": {`{"types":["m.room.message"]}`},
+	})
+	for _, e := range page.Chunk {
+		if e.Type != "m.room.message" {
+			t.Errorf("chunk contains %s, want only m.room.message", e.Type)
+		}
+	}
+	if len(page.Chunk) != 1 {
+		t.Errorf("chunk has %d events, want the one message", len(page.Chunk))
+	}
+}
+
+func TestMessagesRejectsABadDirection(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.get("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/messages?dir=sideways",
+		alice.AccessToken)
+	expectError(t, rec, http.StatusBadRequest, "M_INVALID_PARAM")
+}
+
+func TestMessagesRefusesANonMember(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	// A private room: nothing about it is world-readable.
+	roomID := ts.createRoom(alice, map[string]any{"preset": "private_chat"})
+
+	rec := ts.get("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/messages?dir=b",
+		bob.AccessToken)
+	expectError(t, rec, http.StatusForbidden, "M_FORBIDDEN")
+}
+
+type contextResult struct {
+	Start        string      `json:"start"`
+	End          string      `json:"end"`
+	EventsBefore []syncEvent `json:"events_before"`
+	Event        syncEvent   `json:"event"`
+	EventsAfter  []syncEvent `json:"events_after"`
+	State        []syncEvent `json:"state"`
+}
+
+func TestContextReturnsTheEventAndItsSurroundings(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	ts.sendMessage(alice, roomID, "before 1")
+	ts.sendMessage(alice, roomID, "before 2")
+	target := ts.sendMessage(alice, roomID, "the target")
+	ts.sendMessage(alice, roomID, "after 1")
+	ts.sendMessage(alice, roomID, "after 2")
+
+	rec := ts.get(fmt.Sprintf("/_matrix/client/v3/rooms/%s/context/%s?limit=5",
+		url.PathEscape(roomID), url.PathEscape(target)), alice.AccessToken)
+	expectStatus(t, rec, http.StatusOK)
+	result := decode[contextResult](t, rec)
+
+	if result.Event.EventID != target {
+		t.Errorf("event = %s, want %s", result.Event.EventID, target)
+	}
+	// events_before is newest first, so the message immediately before
+	// the target comes first.
+	before := bodiesOf(result.EventsBefore)
+	if len(before) == 0 || before[0] != "before 2" {
+		t.Errorf("events_before = %v, want it to start with \"before 2\"", before)
+	}
+	after := bodiesOf(result.EventsAfter)
+	if len(after) == 0 || after[0] != "after 1" {
+		t.Errorf("events_after = %v, want it to start with \"after 1\"", after)
+	}
+	if result.Start == "" || result.End == "" {
+		t.Error("a context must carry both paging tokens")
+	}
+	if len(result.State) == 0 {
+		t.Error("a context must carry the state needed to render its chunk")
+	}
+}
+
+func TestContextStateIsLazyLoadedWhenAsked(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	target := ts.sendMessage(alice, roomID, "only one speaker")
+
+	rec := ts.get(fmt.Sprintf("/_matrix/client/v3/rooms/%s/context/%s?filter=%s",
+		url.PathEscape(roomID), url.PathEscape(target),
+		url.QueryEscape(`{"lazy_load_members":true}`)), alice.AccessToken)
+	expectStatus(t, rec, http.StatusOK)
+	result := decode[contextResult](t, rec)
+
+	for _, e := range result.State {
+		if e.Type != "m.room.member" {
+			t.Errorf("lazy-loaded state contains %s, want only member events", e.Type)
+		}
+	}
+	if len(result.State) != 1 {
+		t.Errorf("state has %d events, want just the one sender's member event", len(result.State))
+	}
+}
+
+func TestContextOfAnUnknownEventIsNotFound(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.get("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/context/$nope",
+		alice.AccessToken)
+	expectError(t, rec, http.StatusNotFound, "M_NOT_FOUND")
+}
+
+func TestMessagesBackwardsChunkIsNewestFirst(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "first")
+	ts.sendMessage(alice, roomID, "second")
+	ts.sendMessage(alice, roomID, "third")
+
+	// The spec says a backwards page is in reverse chronological order,
+	// because that is the order a client scrolling up consumes it in.
+	page := ts.messages(alice, roomID, url.Values{
+		"dir":    {"b"},
+		"limit":  {"50"},
+		"filter": {`{"types":["m.room.message"]}`},
+	})
+	got := bodiesOf(page.Chunk)
+	want := []string{"third", "second", "first"}
+	if len(got) != len(want) {
+		t.Fatalf("chunk = %v, want %v", got, want)
+	}
+	for i := range got {
+		if got[i] != want[i] {
+			t.Fatalf("chunk = %v, want %v", got, want)
+		}
+	}
+
+	// Forwards is the other way round.
+	forwards := ts.messages(alice, roomID, url.Values{
+		"dir":    {"f"},
+		"limit":  {"50"},
+		"filter": {`{"types":["m.room.message"]}`},
+	})
+	if got := bodiesOf(forwards.Chunk); len(got) == 0 || got[0] != "first" {
+		t.Errorf("forwards chunk = %v, want it to start with \"first\"", got)
+	}
+}
+
+func TestMessagesStartNamesWhereThePageActuallyBegan(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "only")
+
+	// ?from= is optional. Without it, `start` has to name the end of the
+	// room the page began at: a client that paged backwards and then used
+	// `start` to page forwards again would otherwise be handed the whole
+	// room a second time.
+	page := ts.messages(alice, roomID, url.Values{"dir": {"b"}, "limit": {"50"}})
+	if page.Start == "" || page.Start == "s0" {
+		t.Fatalf("start = %q, want the position the page began at", page.Start)
+	}
+
+	forwards := ts.messages(alice, roomID, url.Values{
+		"dir":   {"f"},
+		"from":  {page.Start},
+		"limit": {"50"},
+	})
+	if len(forwards.Chunk) != 0 {
+		t.Errorf("paging forwards from the end returned %d events, want none",
+			len(forwards.Chunk))
+	}
+}

server/internal/clientapi/rooms.go

@@ -313,12 +313,10 @@ type membersResponse struct {
 
 // handleGetMembers implements GET /_matrix/client/v3/rooms/{roomID}/members.
 //
-// The ?at= parameter asks for the membership as of a point in the event
-// stream, and is accepted and ignored. That is not a shortcut: there is one
-// stream position in this build, because /sync has no stream to be at until
-// Phase 4, so the membership as of any token a client can hold *is* the
-// current membership. Phase 4 replaces this endpoint's filtering along with
-// /sync and has to honour it then — see docs/rooms.md.
+// The ?at= parameter is a sync token, and names the point in the event
+// stream to report the membership as of. A client rendering an older part
+// of a room's timeline needs it: the names and avatars to show against
+// those messages are the ones their senders had then, not now.
 func (c *ClientAPI) handleGetMembers(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
 	roomID, err := pathRoomID(r)
 	if err != nil {
@@ -333,11 +331,19 @@ func (c *ClientAPI) handleGetMembers(w http.ResponseWriter, r *http.Request, aut
 		}
 	}
 
+	// ?at= only has a meaning once there is a stream, so it is read here
+	// rather than in the roomserver: the token format is the syncapi's.
+	at, err := ParseSyncPosition(query.Get("at"))
+	if err != nil {
+		return err
+	}
+
 	members, err := c.rooms.Members(r.Context(), roomserver.MembersRequest{
 		RoomID:        roomID,
 		User:          auth.UserID,
 		Membership:    membership,
 		NotMembership: notMembership,
+		At:            at,
 	})
 	if err != nil {
 		return roomError(err)

server/internal/clientapi/rooms_test.go

@@ -252,9 +252,9 @@ func TestMembershipEndpoints(t *testing.T) {
 		bob.AccessToken)
 	expectError(t, rec, http.StatusBadRequest, httputil.ErrCodeBadJSON)
 
-	// ?at= is accepted and ignored: there is one stream position in this
-	// build, so the membership as of any token a client can hold is the
-	// current membership. Phase 4 has to honour it properly.
+	// ?at= names a stream position to report the membership as of; the
+	// zero token is the start of the room, which every member postdates.
+	// TestMembersAtAnEarlierPosition covers what it actually returns.
 	rec = ts.get("/_matrix/client/v3/rooms/"+escaped+"/members?at=s0_0", bob.AccessToken)
 	expectStatus(t, rec, http.StatusOK)
 

server/internal/clientapi/sync.go

@@ -1,74 +1,264 @@
 package clientapi
 
 import (
+	"encoding/json"
+	"errors"
 	"net/http"
+	"strconv"
+	"time"
 
 	"github.com/dariuszjaniak/warp/server/internal/httputil"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
 	"github.com/dariuszjaniak/warp/server/internal/userapi"
 )
 
-// syncResponse is the subset of the /sync body this build can fill in.
-type syncResponse struct {
-	NextBatch   string          `json:"next_batch"`
-	Rooms       syncRooms       `json:"rooms"`
-	Presence    syncEventList   `json:"presence"`
-	AccountData syncEventList   `json:"account_data"`
-	ToDevice    syncEventList   `json:"to_device"`
-	DeviceLists syncDeviceLists `json:"device_lists"`
-}
-
-type syncRooms struct {
-	Join   map[string]any `json:"join"`
-	Invite map[string]any `json:"invite"`
-	Leave  map[string]any `json:"leave"`
-	Knock  map[string]any `json:"knock"`
-}
-
-type syncEventList struct {
-	Events []any `json:"events"`
-}
-
-type syncDeviceLists struct {
-	Changed []string `json:"changed"`
-	Left    []string `json:"left"`
-}
-
-// syncPlaceholderToken is the only stream position this build has.
+// syncError maps a syncapi error onto the Matrix error the spec answers it
+// with.
 //
-// It is a real token shape rather than an empty string so that a client
-// storing it and sending it back as ?since= gets the same answer.
-const syncPlaceholderToken = "s0_0"
+// It defers to roomError for everything that came out of the roomserver,
+// which is most of it: a sync read is mostly room reads, and the two have
+// to answer "unknown room" the same way whichever endpoint asked.
+func syncError(err error) error {
+	var matrixErr *httputil.MatrixError
+	switch {
+	case err == nil:
+		return nil
+	case errors.As(err, &matrixErr):
+		return err
 
-// handleSync implements GET /_matrix/client/v3/sync as far as this build
-// can.
+	case errors.Is(err, syncapi.ErrNoSuchFilter):
+		return httputil.NotFound("Unknown filter")
+	case errors.Is(err, syncapi.ErrInvalidFilter),
+		errors.Is(err, syncapi.ErrInvalidParam):
+		return httputil.InvalidParam("%s", err.Error())
+	case errors.Is(err, syncapi.ErrInvalidToken):
+		return httputil.InvalidParam("%s", err.Error())
+	case errors.Is(err, syncapi.ErrPresenceDisabled):
+		return httputil.Forbidden("Presence is disabled on this server")
+
+	default:
+		return roomError(err)
+	}
+}
+
+// handleSync implements GET /_matrix/client/v3/sync.
 //
-// It is a placeholder, and since Phase 3 it is an incomplete answer rather
-// than a complete one: there are rooms now, and this reports none of them.
-// Filling it in means the event stream — stream tokens, incremental deltas,
-// long polling, filters and lazy-loaded members — which is the syncapi of
-// PLAN.md phase 4, and which replaces this wholesale.
-//
-// It stays because of what it does buy: a valid token syncs and a revoked
-// one does not, which is the first thing every client and the test suite
-// check, and which cannot be observed at all without an endpoint here. A
-// client can meanwhile see its rooms through /joined_rooms and read them
-// through /rooms/{roomID}/state. See docs/rooms.md.
-func (c *ClientAPI) handleSync(w http.ResponseWriter, _ *http.Request, _ userapi.Authenticated) error {
-	// The `timeout` parameter is ignored rather than honoured: blocking for
-	// 30 seconds to return a response this build cannot change would hold a
-	// connection open for nothing.
-	httputil.WriteJSON(w, http.StatusOK, syncResponse{
-		NextBatch: syncPlaceholderToken,
-		Rooms: syncRooms{
-			Join:   map[string]any{},
-			Invite: map[string]any{},
-			Leave:  map[string]any{},
-			Knock:  map[string]any{},
-		},
-		Presence:    syncEventList{Events: []any{}},
-		AccountData: syncEventList{Events: []any{}},
-		ToDevice:    syncEventList{Events: []any{}},
-		DeviceLists: syncDeviceLists{Changed: []string{}, Left: []string{}},
+// It is the one endpoint that is expected to block. The handler holds the
+// request open for up to the configured long-poll timeout, and the syncapi
+// returns as soon as anything the client cares about happens — so the
+// timeout decides how often an idle client re-establishes its request, not
+// how quickly it hears about a message.
+func (c *ClientAPI) handleSync(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	query := r.URL.Query()
+
+	timeout, err := durationParam(query.Get("timeout"))
+	if err != nil {
+		return err
+	}
+	fullState, err := boolParam(query.Get("full_state"))
+	if err != nil {
+		return err
+	}
+
+	resp, err := c.sync.Sync(r.Context(), syncapi.SyncRequest{
+		User:        auth.UserID,
+		Localpart:   auth.Localpart,
+		Device:      auth.DeviceID,
+		Since:       query.Get("since"),
+		Filter:      query.Get("filter"),
+		Timeout:     timeout,
+		FullState:   fullState,
+		SetPresence: query.Get("set_presence"),
 	})
+	if err != nil {
+		// A client that gives up on a long poll cancels the request, which
+		// arrives here as a context error. Writing a response to a
+		// connection nobody is reading would be the only thing a handler
+		// could do about it, so the request is simply dropped.
+		if ctxErr := r.Context().Err(); ctxErr != nil && errors.Is(err, ctxErr) {
+			return nil //nolint:nilerr // the client is gone; there is nobody to answer
+		}
+		return syncError(err)
+	}
+
+	// A sync response is never cached: it is a position in a stream, and a
+	// cached one would hand a client the same batch twice.
+	httputil.NoCache(w)
+	httputil.WriteJSON(w, http.StatusOK, resp)
 	return nil
 }
+
+// handleMessages implements GET /_matrix/client/v3/rooms/{roomID}/messages.
+func (c *ClientAPI) handleMessages(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	roomID, err := pathRoomID(r)
+	if err != nil {
+		return err
+	}
+	query := r.URL.Query()
+
+	direction := storage.DirectionBackward
+	switch query.Get("dir") {
+	case "f":
+		direction = storage.DirectionForward
+	case "b", "":
+		// The spec makes `dir` required; a client that leaves it out means
+		// "scroll up", which is what every one of them is doing.
+	default:
+		return httputil.InvalidParam("dir must be \"b\" or \"f\"")
+	}
+
+	limit, err := intParam(query.Get("limit"))
+	if err != nil {
+		return err
+	}
+
+	resp, err := c.sync.Messages(r.Context(), syncapi.MessagesRequest{
+		RoomID:    roomID,
+		User:      auth.UserID,
+		Localpart: auth.Localpart,
+		From:      query.Get("from"),
+		To:        query.Get("to"),
+		Direction: direction,
+		Limit:     limit,
+		Filter:    query.Get("filter"),
+	})
+	if err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, resp)
+	return nil
+}
+
+// handleContext implements
+// GET /_matrix/client/v3/rooms/{roomID}/context/{eventID}.
+func (c *ClientAPI) handleContext(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	roomID, err := pathRoomID(r)
+	if err != nil {
+		return err
+	}
+	eventID := r.PathValue("eventID")
+	if eventID == "" {
+		return httputil.MissingParam("An event ID is required")
+	}
+
+	limit, err := intParam(r.URL.Query().Get("limit"))
+	if err != nil {
+		return err
+	}
+
+	resp, err := c.sync.Context(r.Context(), syncapi.ContextRequest{
+		RoomID:    roomID,
+		EventID:   eventID,
+		User:      auth.UserID,
+		Localpart: auth.Localpart,
+		Limit:     limit,
+		Filter:    r.URL.Query().Get("filter"),
+	})
+	if err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, resp)
+	return nil
+}
+
+// filterResponse is what POST /user/{userID}/filter returns.
+type filterResponse struct {
+	FilterID string `json:"filter_id"`
+}
+
+// handlePostFilter implements POST /_matrix/client/v3/user/{userID}/filter.
+func (c *ClientAPI) handlePostFilter(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	localpart, err := c.requireOwnAccountData(auth, r.PathValue("userID"))
+	if err != nil {
+		return err
+	}
+
+	// The filter is stored and returned byte for byte, so it is read as
+	// raw JSON rather than decoded and re-encoded: a client that compares
+	// what it gets back with what it sent must see the same thing.
+	var body json.RawMessage
+	if err := httputil.DecodeJSON(w, r, &body); err != nil {
+		return err
+	}
+
+	filterID, err := c.sync.SaveFilter(r.Context(), localpart, body)
+	if err != nil {
+		return syncError(err)
+	}
+	httputil.WriteJSON(w, http.StatusOK, filterResponse{FilterID: filterID})
+	return nil
+}
+
+// handleGetFilter implements
+// GET /_matrix/client/v3/user/{userID}/filter/{filterID}.
+func (c *ClientAPI) handleGetFilter(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error {
+	localpart, err := c.requireOwnAccountData(auth, r.PathValue("userID"))
+	if err != nil {
+		return err
+	}
+	filterID := r.PathValue("filterID")
+	if filterID == "" {
+		return httputil.MissingParam("A filter ID is required")
+	}
+
+	stored, err := c.sync.StoredFilter(r.Context(), localpart, filterID)
+	if err != nil {
+		return syncError(err)
+	}
+	httputil.WriteRawJSON(w, http.StatusOK, stored)
+	return nil
+}
+
+// ParseSyncPosition reads a sync token from a query parameter and returns
+// the event-stream position in it, or 0 for an absent one.
+//
+// It lives here rather than in the roomserver because the token format is
+// the syncapi's, and /rooms/{id}/members?at= is the one room endpoint that
+// takes one.
+func ParseSyncPosition(raw string) (int64, error) {
+	if raw == "" {
+		return 0, nil
+	}
+	token, err := syncapi.ParseToken(raw)
+	if err != nil {
+		return 0, httputil.InvalidParam("%v", err)
+	}
+	return token.PDU, nil
+}
+
+// durationParam reads a millisecond duration from a query parameter.
+func durationParam(raw string) (time.Duration, error) {
+	if raw == "" {
+		return 0, nil
+	}
+	ms, err := strconv.ParseInt(raw, 10, 64)
+	if err != nil || ms < 0 {
+		return 0, httputil.InvalidParam("timeout must be a non-negative number of milliseconds")
+	}
+	return time.Duration(ms) * time.Millisecond, nil
+}
+
+// intParam reads a non-negative integer from a query parameter.
+func intParam(raw string) (int, error) {
+	if raw == "" {
+		return 0, nil
+	}
+	value, err := strconv.Atoi(raw)
+	if err != nil || value < 0 {
+		return 0, httputil.InvalidParam("limit must be a non-negative integer")
+	}
+	return value, nil
+}
+
+// boolParam reads a boolean from a query parameter.
+func boolParam(raw string) (bool, error) {
+	if raw == "" {
+		return false, nil
+	}
+	value, err := strconv.ParseBool(raw)
+	if err != nil {
+		return false, httputil.InvalidParam("%q is not a boolean", raw)
+	}
+	return value, nil
+}

server/internal/clientapi/sync_test.go

@@ -0,0 +1,664 @@
+package clientapi_test
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+	"testing"
+	"time"
+)
+
+// The /sync tests. They go through HTTP rather than calling the syncapi
+// directly, because what is worth pinning down is the response a client
+// sees: the shape of the body, which section a room lands in, and whether a
+// token handed back can be handed in again.
+
+// syncResult is the handful of things these tests read out of a response.
+// It is a typed struct rather than a map because a sync response is nested
+// four deep, and `body["rooms"].(map[string]any)["join"]` repeated thirty
+// times is unreadable.
+type syncResult struct {
+	NextBatch string `json:"next_batch"`
+	Rooms     struct {
+		Join map[string]struct {
+			Timeline struct {
+				Events    []syncEvent `json:"events"`
+				Limited   bool        `json:"limited"`
+				PrevBatch string      `json:"prev_batch"`
+			} `json:"timeline"`
+			State struct {
+				Events []syncEvent `json:"events"`
+			} `json:"state"`
+			Ephemeral struct {
+				Events []syncEvent `json:"events"`
+			} `json:"ephemeral"`
+			AccountData struct {
+				Events []syncEvent `json:"events"`
+			} `json:"account_data"`
+			Summary *struct {
+				Heroes             []string `json:"m.heroes"`
+				JoinedMemberCount  int      `json:"m.joined_member_count"`
+				InvitedMemberCount int      `json:"m.invited_member_count"`
+			} `json:"summary"`
+		} `json:"join"`
+		Invite map[string]struct {
+			InviteState struct {
+				Events []syncEvent `json:"events"`
+			} `json:"invite_state"`
+		} `json:"invite"`
+		Leave map[string]struct {
+			Timeline struct {
+				Events []syncEvent `json:"events"`
+			} `json:"timeline"`
+			State struct {
+				Events []syncEvent `json:"events"`
+			} `json:"state"`
+		} `json:"leave"`
+	} `json:"rooms"`
+	Presence struct {
+		Events []syncEvent `json:"events"`
+	} `json:"presence"`
+	AccountData struct {
+		Events []syncEvent `json:"events"`
+	} `json:"account_data"`
+	ToDevice struct {
+		Events []syncEvent `json:"events"`
+	} `json:"to_device"`
+	DeviceLists struct {
+		Changed []string `json:"changed"`
+		Left    []string `json:"left"`
+	} `json:"device_lists"`
+}
+
+type syncEvent struct {
+	EventID  string         `json:"event_id"`
+	RoomID   string         `json:"room_id"`
+	Type     string         `json:"type"`
+	StateKey *string        `json:"state_key"`
+	Sender   string         `json:"sender"`
+	Content  map[string]any `json:"content"`
+}
+
+// sync performs one /sync, with no timeout: every test but the long-poll
+// ones wants the answer as it stands.
+func (ts *testServer) sync(acc account, since string) syncResult {
+	ts.t.Helper()
+	return ts.syncWith(acc, url.Values{"since": {since}})
+}
+
+// syncWith performs a /sync with arbitrary query parameters.
+func (ts *testServer) syncWith(acc account, query url.Values) syncResult {
+	ts.t.Helper()
+
+	path := "/_matrix/client/v3/sync"
+	if encoded := query.Encode(); encoded != "" {
+		path += "?" + encoded
+	}
+	rec := ts.get(path, acc.AccessToken)
+	expectStatus(ts.t, rec, http.StatusOK)
+
+	result := decode[syncResult](ts.t, rec)
+	if result.NextBatch == "" {
+		ts.t.Fatalf("/sync returned no next_batch: %s", rec.Body.String())
+	}
+	return result
+}
+
+// sendMessage sends an m.room.message and returns its event ID.
+func (ts *testServer) sendMessage(acc account, roomID, body string) string {
+	ts.t.Helper()
+
+	path := fmt.Sprintf("/_matrix/client/v3/rooms/%s/send/m.room.message/%s",
+		url.PathEscape(roomID), url.PathEscape(fmt.Sprintf("txn-%s-%d", body, time.Now().UnixNano())))
+	rec := ts.put(path, acc.AccessToken, map[string]any{"msgtype": "m.text", "body": body})
+	expectStatus(ts.t, rec, http.StatusOK)
+
+	eventID, _ := jsonBody(ts.t, rec)["event_id"].(string)
+	if eventID == "" {
+		ts.t.Fatalf("send returned no event ID: %s", rec.Body.String())
+	}
+	return eventID
+}
+
+// bodiesOf projects timeline events onto their message bodies, which is
+// what most of these tests actually assert on.
+func bodiesOf(events []syncEvent) []string {
+	out := make([]string, 0, len(events))
+	for _, e := range events {
+		if body, ok := e.Content["body"].(string); ok {
+			out = append(out, body)
+		}
+	}
+	return out
+}
+
+// typesOf projects events onto their types.
+func typesOf(events []syncEvent) []string {
+	out := make([]string, 0, len(events))
+	for _, e := range events {
+		out = append(out, e.Type)
+	}
+	return out
+}
+
+func contains(haystack []string, needle string) bool {
+	for _, s := range haystack {
+		if s == needle {
+			return true
+		}
+	}
+	return false
+}
+
+func TestSyncInitialReportsTheRoomAndItsHistory(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, map[string]any{"name": "Rome"})
+	ts.sendMessage(alice, roomID, "hello")
+
+	result := ts.sync(alice, "")
+
+	room, ok := result.Rooms.Join[roomID]
+	if !ok {
+		t.Fatalf("the room is not in the join section: %+v", result.Rooms.Join)
+	}
+	if got := bodiesOf(room.Timeline.Events); !contains(got, "hello") {
+		t.Errorf("timeline bodies = %v, want it to contain \"hello\"", got)
+	}
+
+	// The room was created and filled in fewer events than the default
+	// timeline limit, so the whole of it is in the timeline and there is
+	// nothing before it to report as state.
+	if room.Timeline.Limited {
+		t.Error("a room shorter than the timeline limit must not be reported as limited")
+	}
+
+	// A sync's events carry no room_id: they are already nested under the
+	// room they belong to.
+	for _, e := range room.Timeline.Events {
+		if e.RoomID != "" {
+			t.Errorf("%s carries a room_id in a sync response", e.Type)
+			break
+		}
+	}
+
+	// device_lists is empty until Phase 7 and must still be present: a
+	// client reads a missing one as a malformed response.
+	if result.DeviceLists.Changed == nil || result.DeviceLists.Left == nil {
+		t.Error("device_lists must carry both lists even when empty")
+	}
+}
+
+func TestSyncIncrementalReturnsOnlyWhatIsNew(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "first")
+
+	first := ts.sync(alice, "")
+	ts.sendMessage(alice, roomID, "second")
+	second := ts.sync(alice, first.NextBatch)
+
+	room, ok := second.Rooms.Join[roomID]
+	if !ok {
+		t.Fatalf("the room is not in the join section: %+v", second.Rooms.Join)
+	}
+	got := bodiesOf(room.Timeline.Events)
+	if len(got) != 1 || got[0] != "second" {
+		t.Errorf("timeline bodies = %v, want exactly [second]", got)
+	}
+
+	// An incremental sync with no gap reports no state: every state change
+	// in the window is a timeline event the client already has.
+	if len(room.State.Events) != 0 {
+		t.Errorf("state = %v, want none on a gapless incremental sync", typesOf(room.State.Events))
+	}
+}
+
+func TestSyncIncrementalIsEmptyWhenNothingHappened(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	first := ts.sync(alice, "")
+	second := ts.sync(alice, first.NextBatch)
+
+	if len(second.Rooms.Join) != 0 {
+		t.Errorf("join = %v, want nothing when nothing happened", second.Rooms.Join)
+	}
+	_ = roomID
+}
+
+func TestSyncTokenRoundTripsThroughTheClient(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	first := ts.sync(alice, "")
+	// Syncing twice from the same token must give the same answer: a token
+	// is a position, not a cursor the server advances for itself.
+	a := ts.sync(alice, first.NextBatch)
+	ts.sendMessage(alice, roomID, "once")
+	b := ts.sync(alice, first.NextBatch)
+	c := ts.sync(alice, first.NextBatch)
+
+	if len(a.Rooms.Join) != 0 {
+		t.Error("the first re-sync should have been empty")
+	}
+	if got := bodiesOf(b.Rooms.Join[roomID].Timeline.Events); len(got) != 1 {
+		t.Errorf("second re-sync timeline = %v, want one event", got)
+	}
+	if got := bodiesOf(c.Rooms.Join[roomID].Timeline.Events); len(got) != 1 {
+		t.Errorf("third re-sync timeline = %v, want the same one event", got)
+	}
+}
+
+func TestSyncRejectsAnUnreadableToken(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+
+	for _, since := range []string{"nonsense", "s-1", "sx", "s1_2_3_4_5_6"} {
+		rec := ts.get("/_matrix/client/v3/sync?since="+url.QueryEscape(since), alice.AccessToken)
+		expectError(t, rec, http.StatusBadRequest, "M_INVALID_PARAM")
+	}
+}
+
+func TestSyncAcceptsATokenFromAnOlderServer(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	ts.sendMessage(alice, roomID, "hello")
+
+	// A token with only the event position in it, as a client stored
+	// before the other streams existed. It must be read as "seen nothing"
+	// of those streams rather than refused.
+	result := ts.syncWith(alice, url.Values{"since": {"s1"}})
+	if len(result.Rooms.Join) == 0 {
+		t.Error("a short token must still sync")
+	}
+}
+
+func TestSyncLongPollWakesOnAnEvent(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	since := ts.sync(alice, "").NextBatch
+
+	done := make(chan syncResult, 1)
+	go func() {
+		done <- ts.syncWith(alice, url.Values{
+			"since":   {since},
+			"timeout": {"10000"},
+		})
+	}()
+
+	// The sync has to be blocked before the event is sent, or it would
+	// find the event without ever having waited and the test would pass
+	// without testing anything.
+	waitForListener(t, ts)
+	ts.sendMessage(alice, roomID, "woken")
+
+	select {
+	case result := <-done:
+		got := bodiesOf(result.Rooms.Join[roomID].Timeline.Events)
+		if !contains(got, "woken") {
+			t.Errorf("timeline = %v, want it to contain \"woken\"", got)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("a blocked /sync was not woken by an event in its room")
+	}
+}
+
+func TestSyncLongPollReturnsEmptyOnTimeout(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	ts.createRoom(alice, nil)
+	since := ts.sync(alice, "").NextBatch
+
+	start := time.Now()
+	result := ts.syncWith(alice, url.Values{"since": {since}, "timeout": {"150"}})
+	elapsed := time.Since(start)
+
+	if len(result.Rooms.Join) != 0 {
+		t.Errorf("join = %v, want nothing", result.Rooms.Join)
+	}
+	if elapsed < 100*time.Millisecond {
+		t.Errorf("a 150ms long poll returned after %v; it did not wait", elapsed)
+	}
+}
+
+// waitForListener blocks until a /sync has registered with the notifier, so
+// that a test can be sure the sync it started is actually waiting.
+func waitForListener(t *testing.T, ts *testServer) {
+	t.Helper()
+
+	deadline := time.Now().Add(5 * time.Second)
+	for time.Now().Before(deadline) {
+		if ts.syncAPI.Notifier().Listeners() > 0 {
+			return
+		}
+		time.Sleep(time.Millisecond)
+	}
+	t.Fatal("no /sync ever blocked on the notifier")
+}
+
+func TestSyncReportsAnInviteAsStrippedState(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"name": "Carthage"})
+
+	rec := ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/invite",
+		alice.AccessToken, map[string]any{"user_id": bob.UserID})
+	expectStatus(t, rec, http.StatusOK)
+
+	result := ts.sync(bob, "")
+	invite, ok := result.Rooms.Invite[roomID]
+	if !ok {
+		t.Fatalf("the room is not in the invite section: %+v", result.Rooms.Invite)
+	}
+	if !contains(typesOf(invite.InviteState.Events), "m.room.name") {
+		t.Errorf("invite_state = %v, want it to name the room", typesOf(invite.InviteState.Events))
+	}
+	if !contains(typesOf(invite.InviteState.Events), "m.room.member") {
+		t.Error("invite_state must carry the member events that make the invite legible")
+	}
+	// An invitee gets no timeline at all: they are not a member.
+	if _, joined := result.Rooms.Join[roomID]; joined {
+		t.Error("an invited room must not be in the join section")
+	}
+}
+
+func TestSyncReportsALeaveAndThenStops(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+	ts.sendMessage(alice, roomID, "while bob is here")
+
+	since := ts.sync(bob, "").NextBatch
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/leave",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	afterLeave := ts.sync(bob, since)
+	if _, ok := afterLeave.Rooms.Leave[roomID]; !ok {
+		t.Fatalf("the leave was not reported: %+v", afterLeave.Rooms)
+	}
+
+	// Everything after the leave is somebody else's business.
+	ts.sendMessage(alice, roomID, "after bob left")
+	quiet := ts.sync(bob, afterLeave.NextBatch)
+	for _, section := range []any{quiet.Rooms.Join[roomID], quiet.Rooms.Leave[roomID]} {
+		_ = section
+	}
+	if left, ok := quiet.Rooms.Leave[roomID]; ok {
+		if got := bodiesOf(left.Timeline.Events); contains(got, "after bob left") {
+			t.Errorf("a departed member was sent %v", got)
+		}
+	}
+}
+
+func TestSyncSummaryNamesARoomWithNoName(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{
+		"preset": "trusted_private_chat",
+		"invite": []string{bob.UserID},
+	})
+
+	result := ts.sync(alice, "")
+	room := result.Rooms.Join[roomID]
+	if room.Summary == nil {
+		t.Fatal("a room with no name must carry a summary to name it by")
+	}
+	if !contains(room.Summary.Heroes, bob.UserID) {
+		t.Errorf("heroes = %v, want it to name %s", room.Summary.Heroes, bob.UserID)
+	}
+	if room.Summary.JoinedMemberCount != 1 {
+		t.Errorf("joined_member_count = %d, want 1", room.Summary.JoinedMemberCount)
+	}
+	if room.Summary.InvitedMemberCount != 1 {
+		t.Errorf("invited_member_count = %d, want 1", room.Summary.InvitedMemberCount)
+	}
+}
+
+func TestSyncLimitedTimelineCarriesTheStateBeforeIt(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	since := ts.sync(alice, "").NextBatch
+
+	// A state change and then more messages than the timeline limit, so
+	// the state change falls into the gap and has to be reported
+	// separately.
+	expectStatus(t, ts.put("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/state/m.room.topic",
+		alice.AccessToken, map[string]any{"topic": "in the gap"}), http.StatusOK)
+	for i := range 5 {
+		ts.sendMessage(alice, roomID, fmt.Sprintf("message %d", i))
+	}
+
+	result := ts.syncWith(alice, url.Values{
+		"since":  {since},
+		"filter": {`{"room":{"timeline":{"limit":2}}}`},
+	})
+	room := result.Rooms.Join[roomID]
+	if !room.Timeline.Limited {
+		t.Fatal("a timeline cut short by the limit must say so")
+	}
+	if room.Timeline.PrevBatch == "" {
+		t.Error("a limited timeline must carry a prev_batch to close the gap with")
+	}
+	if !contains(typesOf(room.State.Events), "m.room.topic") {
+		t.Errorf("state = %v, want the topic change that fell into the gap",
+			typesOf(room.State.Events))
+	}
+}
+
+func TestSyncFullStateReturnsEverythingAgain(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, map[string]any{"name": "Thebes"})
+
+	since := ts.sync(alice, "").NextBatch
+	result := ts.syncWith(alice, url.Values{
+		"since":      {since},
+		"full_state": {"true"},
+		"filter":     {`{"room":{"timeline":{"limit":1}}}`},
+	})
+
+	room, ok := result.Rooms.Join[roomID]
+	if !ok {
+		t.Fatalf("full_state must report every room: %+v", result.Rooms.Join)
+	}
+	if !contains(typesOf(room.State.Events), "m.room.create") {
+		t.Errorf("state = %v, want the whole state including m.room.create",
+			typesOf(room.State.Events))
+	}
+}
+
+func TestSyncLazyLoadingLeavesOutUninvolvedMembers(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	// Only Alice speaks, so only Alice's member event is needed to render
+	// the timeline.
+	for i := range 4 {
+		ts.sendMessage(alice, roomID, fmt.Sprintf("line %d", i))
+	}
+
+	result := ts.syncWith(alice, url.Values{
+		"filter": {`{"room":{"timeline":{"limit":2,"lazy_load_members":true}}}`},
+	})
+	room := result.Rooms.Join[roomID]
+
+	members := map[string]bool{}
+	for _, e := range room.State.Events {
+		if e.Type == "m.room.member" && e.StateKey != nil {
+			members[*e.StateKey] = true
+		}
+	}
+	if !members[alice.UserID] {
+		t.Error("lazy loading must still send the member event of a timeline sender")
+	}
+	if members[bob.UserID] {
+		t.Error("lazy loading must leave out a member who does not appear in the timeline")
+	}
+}
+
+func TestSyncAppliesATypeFilter(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+	since := ts.sync(alice, "").NextBatch
+
+	ts.sendMessage(alice, roomID, "a message")
+	expectStatus(t, ts.put("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/state/m.room.topic",
+		alice.AccessToken, map[string]any{"topic": "a topic"}), http.StatusOK)
+
+	result := ts.syncWith(alice, url.Values{
+		"since":  {since},
+		"filter": {`{"room":{"timeline":{"not_types":["m.room.message"]}}}`},
+	})
+	got := typesOf(result.Rooms.Join[roomID].Timeline.Events)
+	if contains(got, "m.room.message") {
+		t.Errorf("timeline = %v, want the message excluded", got)
+	}
+	if !contains(got, "m.room.topic") {
+		t.Errorf("timeline = %v, want the topic kept", got)
+	}
+}
+
+func TestSyncUsesAnUploadedFilter(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.post("/_matrix/client/v3/user/"+url.PathEscape(alice.UserID)+"/filter",
+		alice.AccessToken, map[string]any{
+			"room": map[string]any{
+				"timeline": map[string]any{"types": []string{"m.room.message"}},
+			},
+		})
+	expectStatus(t, rec, http.StatusOK)
+	filterID, _ := jsonBody(t, rec)["filter_id"].(string)
+	if filterID == "" {
+		t.Fatalf("no filter ID was returned: %s", rec.Body.String())
+	}
+
+	// It comes back exactly as it was uploaded.
+	rec = ts.get("/_matrix/client/v3/user/"+url.PathEscape(alice.UserID)+"/filter/"+filterID,
+		alice.AccessToken)
+	expectStatus(t, rec, http.StatusOK)
+
+	ts.sendMessage(alice, roomID, "kept")
+	result := ts.syncWith(alice, url.Values{"filter": {filterID}})
+
+	got := typesOf(result.Rooms.Join[roomID].Timeline.Events)
+	if !contains(got, "m.room.message") {
+		t.Errorf("timeline = %v, want the message", got)
+	}
+	if contains(got, "m.room.create") {
+		t.Errorf("timeline = %v, want only messages", got)
+	}
+}
+
+func TestGetUnknownFilterIsNotFound(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+
+	rec := ts.get("/_matrix/client/v3/user/"+url.PathEscape(alice.UserID)+"/filter/nope",
+		alice.AccessToken)
+	expectError(t, rec, http.StatusNotFound, "M_NOT_FOUND")
+}
+
+func TestFilterBelongsToItsOwner(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+
+	rec := ts.post("/_matrix/client/v3/user/"+url.PathEscape(bob.UserID)+"/filter",
+		alice.AccessToken, map[string]any{})
+	expectError(t, rec, http.StatusForbidden, "M_FORBIDDEN")
+}
+
+func TestSyncReportsAccountData(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	since := ts.sync(alice, "").NextBatch
+
+	expectStatus(t, ts.put("/_matrix/client/v3/user/"+url.PathEscape(alice.UserID)+
+		"/account_data/m.direct", alice.AccessToken, map[string]any{"@bob:warp.example": []string{}}),
+		http.StatusOK)
+
+	result := ts.sync(alice, since)
+	if !contains(typesOf(result.AccountData.Events), "m.direct") {
+		t.Errorf("account_data = %v, want m.direct", typesOf(result.AccountData.Events))
+	}
+	// Account data is not an event: it has a type and a content and
+	// nothing else.
+	for _, e := range result.AccountData.Events {
+		if e.EventID != "" || e.Sender != "" {
+			t.Errorf("account data was reported as an event: %+v", e)
+		}
+	}
+}
+
+func TestMembersAtAnEarlierPosition(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	bob := ts.register("bob", "hunter2-hunter2-hunter2")
+	roomID := ts.createRoom(alice, map[string]any{"preset": "public_chat"})
+
+	// A token from before Bob joined.
+	before := ts.sync(alice, "").NextBatch
+	expectStatus(t, ts.post("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join",
+		bob.AccessToken, map[string]any{}), http.StatusOK)
+
+	membersAt := func(at string) []string {
+		t.Helper()
+		path := "/_matrix/client/v3/rooms/" + url.PathEscape(roomID) + "/members"
+		if at != "" {
+			path += "?at=" + url.QueryEscape(at)
+		}
+		rec := ts.get(path, alice.AccessToken)
+		expectStatus(t, rec, http.StatusOK)
+
+		body := decode[struct {
+			Chunk []syncEvent `json:"chunk"`
+		}](t, rec)
+
+		out := make([]string, 0, len(body.Chunk))
+		for _, e := range body.Chunk {
+			if e.StateKey != nil {
+				out = append(out, *e.StateKey)
+			}
+		}
+		return out
+	}
+
+	if got := membersAt(""); !contains(got, bob.UserID) {
+		t.Errorf("current members = %v, want it to include %s", got, bob.UserID)
+	}
+	if got := membersAt(before); contains(got, bob.UserID) {
+		t.Errorf("members at an earlier position = %v, want it to leave out %s", got, bob.UserID)
+	}
+}
+
+func TestMembersRejectsABadAtToken(t *testing.T) {
+	ts := newTestServer(t, nil)
+	alice := ts.register("alice", "correct-horse-battery-staple")
+	roomID := ts.createRoom(alice, nil)
+
+	rec := ts.get("/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/members?at=nonsense",
+		alice.AccessToken)
+	expectError(t, rec, http.StatusBadRequest, "M_INVALID_PARAM")
+}

server/internal/config/config.go

@@ -22,6 +22,7 @@ type Config struct {
 	Listen       Listen       `yaml:"listen"`
 	Database     Database     `yaml:"database"`
 	ClientAPI    ClientAPI    `yaml:"client_api"`
+	Sync         Sync         `yaml:"sync"`
 	Federation   Federation   `yaml:"federation"`
 	Registration Registration `yaml:"registration"`
 	Logging      Logging      `yaml:"logging"`
@@ -143,6 +144,13 @@ type RateLimit struct {
 	// into a room that everybody in it then sees, so it is worth a tighter
 	// bucket than a read.
 	Join RateLimitBucket `yaml:"join"`
+
+	// Sync covers /sync and the chatty writes behind it: typing,
+	// receipts, to-device messages and presence. It is the loosest bucket
+	// of the lot on purpose — a client is in /sync all the time, and a
+	// long poll that is refused leaves it reconnecting in a loop, which is
+	// the one case where rate limiting makes the load worse.
+	Sync RateLimitBucket `yaml:"sync"`
 }
 
 // RateLimitBucket is one token bucket: Burst requests may arrive back to
@@ -152,6 +160,27 @@ type RateLimitBucket struct {
 	Interval time.Duration `yaml:"interval"`
 }
 
+// Sync configures the read model: /sync and the streams it carries.
+type Sync struct {
+	// Presence turns the presence stream on. It is off by default
+	// (PLAN.md open question 3): presence is the noisiest thing in Matrix
+	// — every user's state change goes to everybody they share a room
+	// with — and a private homeserver gains little from it. With it off
+	// the presence endpoints answer M_FORBIDDEN and the presence section
+	// of a sync is empty.
+	Presence bool `yaml:"presence" env:"WARP_SYNC_PRESENCE"`
+
+	// TypingTimeout is how long a typing notification lasts without being
+	// renewed, and the cap on what a client may ask for.
+	TypingTimeout time.Duration `yaml:"typing_timeout" env:"WARP_SYNC_TYPING_TIMEOUT"`
+
+	// LongPollTimeout caps a /sync's ?timeout=, whatever the client asks
+	// for. Holding a connection longer buys nothing — a sync is woken the
+	// moment anything happens — and a client asking for ten minutes would
+	// hold a connection for ten minutes.
+	LongPollTimeout time.Duration `yaml:"long_poll_timeout" env:"WARP_SYNC_LONG_POLL_TIMEOUT"`
+}
+
 // Federation controls the server-to-server surface. See PLAN.md section 3: it
 // is a first-class mode, not an afterthought.
 type Federation struct {
@@ -253,8 +282,16 @@ func Default() *Config {
 				// and a client that hits either limit is a script.
 				RoomCreate: RateLimitBucket{Burst: 10, Interval: time.Second},
 				Join:       RateLimitBucket{Burst: 10, Interval: time.Second},
+				// Loose enough that a client in a long poll with typing
+				// and receipts on top never notices it.
+				Sync: RateLimitBucket{Burst: 100, Interval: 20 * time.Millisecond},
 			},
 		},
+		Sync: Sync{
+			Presence:        false,
+			TypingTimeout:   30 * time.Second,
+			LongPollTimeout: 30 * time.Second,
+		},
 		Federation: Federation{
 			Enabled:            false,
 			AllowRuntimeToggle: true,
@@ -359,6 +396,7 @@ func (c *Config) Validate() error {
 			{"register", c.ClientAPI.RateLimit.Register},
 			{"room_create", c.ClientAPI.RateLimit.RoomCreate},
 			{"join", c.ClientAPI.RateLimit.Join},
+			{"sync", c.ClientAPI.RateLimit.Sync},
 		} {
 			if b.bucket.Burst < 1 {
 				bad("client_api.rate_limit.%s.burst: must be at least 1", b.name)
@@ -368,6 +406,13 @@ func (c *Config) Validate() error {
 			}
 		}
 	}
+	if c.Sync.TypingTimeout <= 0 {
+		bad("sync.typing_timeout: must be positive")
+	}
+	if c.Sync.LongPollTimeout <= 0 {
+		bad("sync.long_poll_timeout: must be positive")
+	}
+
 	for i, token := range c.Registration.Tokens {
 		if strings.TrimSpace(token) == "" {
 			bad("registration.tokens[%d]: must not be empty", i)

server/internal/config/config_test.go

@@ -160,4 +160,15 @@ func TestExampleConfigIsValid(t *testing.T) {
 	if cfg.Database.Driver != "sqlite" {
 		t.Errorf("example driver = %q, want sqlite", cfg.Database.Driver)
 	}
+	if cfg.Sync.Presence {
+		t.Error("the shipped example must keep presence off (PLAN.md open question 3)")
+	}
+	// A zero here would mean the example omitted the section and the
+	// validation below is passing on the defaults rather than on the file.
+	if cfg.Sync.TypingTimeout == 0 || cfg.Sync.LongPollTimeout == 0 {
+		t.Error("the shipped example must set the sync timeouts")
+	}
+	if cfg.ClientAPI.RateLimit.Sync.Burst == 0 {
+		t.Error("the shipped example must set the sync rate-limit bucket")
+	}
 }

server/internal/roomserver/directory.go

@@ -173,11 +173,7 @@ func (r *API) ResolveAlias(ctx context.Context, alias string) (ResolvedAlias, er
 
 // RoomAliases lists the local aliases of a room, for a user who may see it.
 func (r *API) RoomAliases(ctx context.Context, roomID, userID string) ([]string, error) {
-	room, rules, err := r.roomAndRules(ctx, roomID)
-	if err != nil {
-		return nil, err
-	}
-	if err := r.requireVisibility(ctx, room, rules, userID); err != nil {
+	if _, err := r.requireReadable(ctx, roomID, userID); err != nil {
 		return nil, err
 	}
 	aliases, err := r.store.AliasesForRoom(ctx, roomID)

server/internal/roomserver/query.go

@@ -34,8 +34,16 @@ func (r *API) currentState(ctx context.Context, room storage.Room, rules roomver
 // is `unsigned`, which is per-server annotation: the age of the event, the
 // state it replaced, and the redaction that stripped it.
 type ClientEvent struct {
-	EventID        string          `json:"event_id"`
-	RoomID         string          `json:"room_id"`
+	EventID string `json:"event_id"`
+
+	// RoomID is omitted when empty, because a /sync response carries its
+	// events without one: they are already nested under the room they
+	// belong to, and repeating it on every event of a client with a
+	// thousand rooms is a measurable part of every response. Every other
+	// endpoint sets it, and the spec has a separate name for the shape
+	// without it — ClientEventWithoutRoomID — rather than two event types.
+	RoomID string `json:"room_id,omitempty"`
+
 	Type           string          `json:"type"`
 	StateKey       *string         `json:"state_key,omitempty"`
 	Sender         string          `json:"sender"`
@@ -98,11 +106,12 @@ func (r *API) unsigned(stored storage.StoredEvent, e *event.Event) json.RawMessa
 
 // Event returns one event of a room, if the user may see it.
 func (r *API) Event(ctx context.Context, roomID, userID, eventID string) (ClientEvent, error) {
-	room, rules, err := r.roomAndRules(ctx, roomID)
+	_, rules, err := r.roomAndRules(ctx, roomID)
 	if err != nil {
 		return ClientEvent{}, err
 	}
-	if err := r.requireVisibility(ctx, room, rules, userID); err != nil {
+	visible, err := r.requireReadable(ctx, roomID, userID)
+	if err != nil {
 		return ClientEvent{}, err
 	}
 
@@ -110,6 +119,12 @@ func (r *API) Event(ctx context.Context, roomID, userID, eventID string) (Client
 	if err != nil {
 		return ClientEvent{}, err
 	}
+	// An event outside the user's window is reported as unknown rather
+	// than as forbidden: whether a particular event ID exists in a room is
+	// itself something they are not entitled to know.
+	if !visible.Contains(stored.StreamOrdering) {
+		return ClientEvent{}, fmt.Errorf("event %q: %w", eventID, ErrNoSuchEvent)
+	}
 	e, err := parseEvent(stored, rules)
 	if err != nil {
 		return ClientEvent{}, err
@@ -117,29 +132,18 @@ func (r *API) Event(ctx context.Context, roomID, userID, eventID string) (Client
 	return r.clientEvent(stored, e, roomID), nil
 }
 
-// RoomState returns a room's current state in the client shape.
+// RoomState returns a room's state in the client shape: the current state
+// for a member, and the state as it was when they left for a user who has.
 func (r *API) RoomState(ctx context.Context, roomID, userID string) ([]ClientEvent, error) {
-	room, rules, err := r.roomAndRules(ctx, roomID)
+	_, rules, err := r.roomAndRules(ctx, roomID)
 	if err != nil {
 		return nil, err
 	}
-	if err := r.requireVisibility(ctx, room, rules, userID); err != nil {
-		return nil, err
-	}
-
-	stored, err := r.store.CurrentStateEvents(ctx, roomID)
+	visible, err := r.requireReadable(ctx, roomID, userID)
 	if err != nil {
 		return nil, err
 	}
-	out := make([]ClientEvent, 0, len(stored))
-	for _, s := range stored {
-		e, err := parseEvent(s, rules)
-		if err != nil {
-			return nil, err
-		}
-		out = append(out, r.clientEvent(s, e, roomID))
-	}
-	return out, nil
+	return r.stateAt(ctx, roomID, rules, visible.To)
 }
 
 // StateEventContent returns the content of one piece of a room's current
@@ -148,23 +152,11 @@ func (r *API) RoomState(ctx context.Context, roomID, userID string) ([]ClientEve
 // The endpoint returns the content alone, not the event: a quirk of the
 // Client-Server API that predates the rest of it, and one clients depend on.
 func (r *API) StateEventContent(ctx context.Context, roomID, userID, eventType, stateKey string) (json.RawMessage, error) {
-	room, rules, err := r.roomAndRules(ctx, roomID)
+	e, err := r.StateEvent(ctx, roomID, userID, eventType, stateKey)
 	if err != nil {
 		return nil, err
 	}
-	if err := r.requireVisibility(ctx, room, rules, userID); err != nil {
-		return nil, err
-	}
-
-	stored, err := r.store.CurrentStateEvent(ctx, roomID, eventType, stateKey)
-	if err != nil {
-		return nil, err
-	}
-	e, err := parseEvent(stored, rules)
-	if err != nil {
-		return nil, err
-	}
-	return e.Content(), nil
+	return e.Content, nil
 }
 
 // StateEvent returns one piece of a room's current state as a whole event.
@@ -172,14 +164,33 @@ func (r *API) StateEventContent(ctx context.Context, roomID, userID, eventType,
 // It is what ?format=event asks for on the state endpoint, and the only way
 // a client can see who set a piece of state and when.
 func (r *API) StateEvent(ctx context.Context, roomID, userID, eventType, stateKey string) (ClientEvent, error) {
-	room, rules, err := r.roomAndRules(ctx, roomID)
+	_, rules, err := r.roomAndRules(ctx, roomID)
 	if err != nil {
 		return ClientEvent{}, err
 	}
-	if err := r.requireVisibility(ctx, room, rules, userID); err != nil {
+	visible, err := r.requireReadable(ctx, roomID, userID)
+	if err != nil {
 		return ClientEvent{}, err
 	}
 
+	// A member reads the one row they asked for. A user who has left has to
+	// be served from the state as of their departure, which means reading
+	// the snapshot and picking the tuple out of it — the room has moved on
+	// since, and the current row is not theirs to see.
+	if visible.To != 0 {
+		state, err := r.stateAt(ctx, roomID, rules, visible.To)
+		if err != nil {
+			return ClientEvent{}, err
+		}
+		for _, e := range state {
+			if e.Type == eventType && e.StateKey != nil && *e.StateKey == stateKey {
+				return e, nil
+			}
+		}
+		return ClientEvent{}, fmt.Errorf("state %s/%s of %q: %w",
+			eventType, stateKey, roomID, ErrNoSuchEvent)
+	}
+
 	stored, err := r.store.CurrentStateEvent(ctx, roomID, eventType, stateKey)
 	if err != nil {
 		return ClientEvent{}, err
@@ -201,44 +212,88 @@ type MembersRequest struct {
 	// Membership and NotMembership filter the result; either may be empty.
 	Membership    string
 	NotMembership string
+
+	// At is the stream position to report the membership as of, or 0 for
+	// the current one.
+	//
+	// It is what ?at= asks for, and what a client that is rendering an
+	// older part of a room's timeline needs: the names and avatars to show
+	// against those messages are the ones their senders had then, not now.
+	At int64
 }
 
 // Members returns a room's m.room.member events.
 func (r *API) Members(ctx context.Context, req MembersRequest) ([]ClientEvent, error) {
-	room, rules, err := r.roomAndRules(ctx, req.RoomID)
+	_, rules, err := r.roomAndRules(ctx, req.RoomID)
 	if err != nil {
 		return nil, err
 	}
-	if err := r.requireVisibility(ctx, room, rules, req.User); err != nil {
-		return nil, err
-	}
-
-	var filter []string
-	if req.Membership != "" {
-		filter = append(filter, req.Membership)
-	}
-	stored, err := r.store.Members(ctx, req.RoomID, filter)
+	visible, err := r.requireReadable(ctx, req.RoomID, req.User)
 	if err != nil {
 		return nil, err
 	}
 
-	out := make([]ClientEvent, 0, len(stored))
-	for _, s := range stored {
-		e, err := parseEvent(s, rules)
+	// The position to report as of: the one asked for, held inside what
+	// the user may see. A user who has left is pinned to their departure
+	// whatever they asked for — the room has moved on since, and the
+	// current membership is not theirs to read.
+	at := req.At
+	if visible.To != 0 && (at == 0 || at > visible.To) {
+		at = visible.To
+	}
+	if at != 0 && at <= visible.From {
+		return nil, fmt.Errorf("%w: %s may not see %s as it was at %d",
+			ErrForbidden, req.User, req.RoomID, req.At)
+	}
+
+	var members []ClientEvent
+	if at != 0 {
+		state, err := r.stateAt(ctx, req.RoomID, rules, at)
 		if err != nil {
 			return nil, err
 		}
-		if req.NotMembership != "" {
-			membership, err := e.Membership()
-			if err == nil && membership == req.NotMembership {
-				continue
-			}
+		members = filterClientEvents(state, storage.TimelineFilter{Types: []string{event.TypeMember}})
+	} else {
+		var filter []string
+		if req.Membership != "" {
+			filter = append(filter, req.Membership)
 		}
-		out = append(out, r.clientEvent(s, e, req.RoomID))
+		stored, err := r.store.Members(ctx, req.RoomID, filter)
+		if err != nil {
+			return nil, err
+		}
+		if members, err = r.clientEvents(stored, rules, req.RoomID); err != nil {
+			return nil, err
+		}
+	}
+
+	out := make([]ClientEvent, 0, len(members))
+	for _, e := range members {
+		membership := membershipOf(e)
+		if req.Membership != "" && membership != req.Membership {
+			continue
+		}
+		if req.NotMembership != "" && membership == req.NotMembership {
+			continue
+		}
+		out = append(out, e)
 	}
 	return out, nil
 }
 
+// membershipOf reads content.membership out of a projected member event.
+//
+// A member event that reached a room always has one — the authorisation
+// rules refuse one that does not — so an empty answer means the event is
+// not a member event at all.
+func membershipOf(e ClientEvent) string {
+	var content struct {
+		Membership string `json:"membership"`
+	}
+	_ = json.Unmarshal(e.Content, &content)
+	return content.Membership
+}
+
 // RoomMember is one joined member, as /joined_members reports them.
 type RoomMember struct {
 	DisplayName string `json:"display_name,omitempty"`
@@ -333,47 +388,6 @@ func (r *API) requireJoined(ctx context.Context, roomID, userID string) error {
 	return nil
 }
 
-// requireVisibility refuses a caller who may not see a room's state.
-//
-// This is a room-level test, not the per-event history visibility the spec
-// defines. A member sees the room; so does anyone at all when the room's
-// history is world_readable; a user who has left sees the state as it was
-// when they left, which is their own membership event's snapshot and which
-// this returns the current state for instead.
-//
-// That last case is the gap, and it is deliberate. Serving a departed
-// member the state as of their leave event means answering every state read
-// from a historical snapshot and filtering each event by the visibility in
-// force when it was sent — which is the same machinery /messages and
-// /context need, and so belongs with them in Phase 4 rather than being
-// built twice. Until then a user who has left a room is refused, which errs
-// towards showing them too little.
-func (r *API) requireVisibility(ctx context.Context, room storage.Room, rules roomversion.Rules, userID string) error {
-	entry, err := r.store.Membership(ctx, room.ID, userID)
-	if err != nil {
-		return err
-	}
-	if entry.Membership == event.MembershipJoin {
-		return nil
-	}
-
-	summary, err := r.store.RoomSummary(ctx, room.ID)
-	if err != nil {
-		return err
-	}
-	if summary.HistoryVisibility == HistoryVisibilityWorldReadable {
-		return nil
-	}
-
-	// An invited user can see enough of the room to decide, which the spec
-	// delivers as stripped state on the invite rather than as a state read.
-	if entry.Membership == event.MembershipInvite {
-		return fmt.Errorf("%w: %s has been invited to %s but has not joined it",
-			ErrForbidden, userID, room.ID)
-	}
-	return fmt.Errorf("%w: %s is not in %s", ErrForbidden, userID, room.ID)
-}
-
 // CanSendStateEvent reports whether a user may send a state event of a type
 // into a room.
 //

server/internal/roomserver/roomserver.go

@@ -98,10 +98,14 @@ type Notification struct {
 // they were persisted.
 //
 // It is the emit stage of the input pipeline and the seam the sync API
-// (PLAN.md phase 4) plugs into: a client long-polling /sync has to be woken
-// the moment an event lands, and the input pipeline is the only place that
-// knows one has. Phase 3 ships without a notifier, so this is nil and the
-// stage is skipped.
+// plugs into: a client long-polling /sync has to be woken the moment an
+// event lands, and the input pipeline is the only place that knows one
+// has. *syncapi.Notifier satisfies it; nil skips the stage, which is what
+// a build with no sync API does.
+//
+// An implementation must not block. It is called with the room's write
+// lock still held, so anything slow here serialises the room's writers
+// behind the slowest reader of it.
 type Notifier interface {
 	EventPersisted(ctx context.Context, n Notification)
 }

server/internal/roomserver/roomserver_test.go

@@ -39,6 +39,10 @@ type fixture struct {
 	api   *roomserver.API
 	store storage.RoomStore
 	db    storage.Database
+
+	// sends counts the messages the fill helper has sent, so that each
+	// one gets its own transaction ID; see fill in stream_test.go.
+	sends int
 }
 
 // profiles is a stand-in for the userapi, so that these tests do not need

server/internal/roomserver/roomsync.go

@@ -0,0 +1,326 @@
+package roomserver
+
+import (
+	"context"
+	"strings"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/matrix/roomversion"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// RoomSyncRequest asks what one user sees of one room between two
+// positions.
+//
+// It is the read model's largest operation and the one /sync is built on.
+// It is a single call rather than a timeline read plus two state reads
+// because the three answers have to agree with each other: which state the
+// response reports depends on where the timeline turned out to start, and
+// working that out in the caller would mean the caller deciding what the
+// timeline was — which is the question it asked.
+type RoomSyncRequest struct {
+	RoomID string
+
+	// User is the full user ID syncing.
+	User string
+
+	// Since is the position the user has already seen, exclusive. Zero is
+	// an initial sync.
+	Since int64
+
+	// To is the top of this sync, inclusive. It is fixed by the caller for
+	// every room in one response, so that a token names one moment rather
+	// than a different one per room.
+	To int64
+
+	// TimelineLimit caps the events in the timeline.
+	TimelineLimit int
+
+	// TimelineFilter and StateFilter narrow the two halves of the
+	// response. The state filter is applied in this package rather than in
+	// SQL: a state read is a set of tuples, not a window of a stream.
+	TimelineFilter storage.TimelineFilter
+	StateFilter    storage.TimelineFilter
+
+	// FullState asks for the room's whole state even on an incremental
+	// sync, which is what ?full_state=true means.
+	FullState bool
+
+	// LazyLoadMembers leaves the member events out of the state except for
+	// the senders who appear in the timeline.
+	LazyLoadMembers bool
+}
+
+// RoomSync is what one user sees of one room between two positions.
+type RoomSync struct {
+	// Membership is the user's current membership in the room.
+	Membership string
+
+	Timeline Timeline
+
+	// State is the room's state as of the start of the timeline on an
+	// initial sync, and what changed between Since and the start of the
+	// timeline on an incremental one. For an incremental sync with no gap
+	// it is empty: every state change in the window is in the timeline
+	// itself.
+	State []ClientEvent
+
+	// Summary is the room's member counts and heroes.
+	Summary RoomSyncSummary
+
+	// Visible is the span of the stream the user may read, which the
+	// caller needs in order to bound the ephemeral streams the same way.
+	Visible VisibleRange
+}
+
+// RoomSync reads one room for one user.
+func (r *API) RoomSync(ctx context.Context, req RoomSyncRequest) (RoomSync, error) {
+	room, rules, err := r.roomAndRules(ctx, req.RoomID)
+	if err != nil {
+		return RoomSync{}, err
+	}
+	summary, err := r.store.RoomSummary(ctx, room.ID)
+	if err != nil {
+		return RoomSync{}, err
+	}
+	visible, err := r.visibleRange(ctx, room.ID, rules, req.User, summary.HistoryVisibility)
+	if err != nil {
+		return RoomSync{}, err
+	}
+	if err := visible.refusal(room.ID, req.User); err != nil {
+		return RoomSync{}, err
+	}
+
+	out := RoomSync{Membership: visible.Membership, Visible: visible}
+
+	// The timeline is read backwards even though it is reported forwards.
+	// A client wants the *newest* events of the window it has missed — the
+	// bottom of the room, which is what it is looking at — and a forward
+	// read with a limit would hand it the oldest instead and then call the
+	// rest a gap.
+	out.Timeline, err = r.timeline(ctx, rules, visible, TimelineRequest{
+		RoomID:    req.RoomID,
+		User:      req.User,
+		Low:       req.Since,
+		High:      req.To,
+		Direction: storage.DirectionBackward,
+		Limit:     req.TimelineLimit,
+		Filter:    req.TimelineFilter,
+	})
+	if err != nil {
+		return RoomSync{}, err
+	}
+
+	if out.State, err = r.syncState(ctx, rules, visible, req, out.Timeline); err != nil {
+		return RoomSync{}, err
+	}
+	if out.Summary, err = r.RoomSummaryFor(ctx, req.RoomID, req.User); err != nil {
+		return RoomSync{}, err
+	}
+	return out, nil
+}
+
+// syncState decides which state a room's half of a sync response reports.
+//
+// The rule the spec sets is that a client which applies `state` and then
+// `timeline` ends up with the room's state at the end of the timeline. That
+// gives three cases:
+//
+//   - An initial sync, or ?full_state=true: the whole state as of the start
+//     of the timeline.
+//   - An incremental sync whose timeline reaches back to `since`: nothing.
+//     Every state change in between is a timeline event the client is
+//     already being given.
+//   - An incremental sync whose timeline was cut short — a gap: what
+//     changed between `since` and where the timeline starts. Those changes
+//     are not in the timeline and the client would otherwise never see
+//     them.
+func (r *API) syncState(ctx context.Context, rules roomversion.Rules, visible VisibleRange, req RoomSyncRequest, timeline Timeline) ([]ClientEvent, error) {
+	statePos, haveState := statePosition(timeline, req.To, visible)
+
+	switch {
+	case !haveState:
+		// The timeline reaches back to the room's create event, so there
+		// is nothing before it to report.
+		return nil, nil
+
+	case req.FullState || req.Since == 0:
+		state, err := r.stateAt(ctx, req.RoomID, rules, statePos)
+		if err != nil {
+			return nil, err
+		}
+		return r.shapeState(ctx, rules, req, timeline, state)
+
+	case !timeline.Limited:
+		return nil, nil
+
+	default:
+		before, err := r.stateAt(ctx, req.RoomID, rules, req.Since)
+		if err != nil {
+			return nil, err
+		}
+		after, err := r.stateAt(ctx, req.RoomID, rules, statePos)
+		if err != nil {
+			return nil, err
+		}
+		return r.shapeState(ctx, rules, req, timeline, stateDelta(before, after))
+	}
+}
+
+// statePosition is the position a room's state is reported as of, and
+// whether there is any state to report at all.
+//
+// The state belongs at the start of the timeline, so that applying it and
+// then the timeline lands on the room's current state. A room with no
+// timeline events in this window is reported as of the top of the sync
+// instead: the state may still have been read for the first time.
+func statePosition(timeline Timeline, to int64, visible VisibleRange) (int64, bool) {
+	if len(timeline.Events) > 0 {
+		return timeline.Start, timeline.Start > 0
+	}
+	if visible.To != 0 && (to == 0 || to > visible.To) {
+		to = visible.To
+	}
+	return to, to > 0
+}
+
+// shapeState applies the state filter and lazy loading to a state set.
+func (r *API) shapeState(ctx context.Context, rules roomversion.Rules, req RoomSyncRequest, timeline Timeline, state []ClientEvent) ([]ClientEvent, error) {
+	state = filterClientEvents(state, req.StateFilter)
+	if !req.LazyLoadMembers {
+		return state, nil
+	}
+
+	// Lazy loading: drop the member events and put back only the ones the
+	// client needs to render the timeline it is being given.
+	//
+	// The spec also lets a server leave out members it has already sent
+	// this device, which would need per-device bookkeeping of what each
+	// one has seen. Warp sends them every time instead: it is the
+	// behaviour `include_redundant_members: true` asks for, so no client
+	// can be surprised by it, and it costs one state read rather than a
+	// table.
+	kept := make([]ClientEvent, 0, len(state))
+	for _, e := range state {
+		if e.Type != event.TypeMember {
+			kept = append(kept, e)
+		}
+	}
+
+	senders := timelineSenders(timeline)
+	members, err := r.memberEvents(ctx, req.RoomID, rules, senders)
+	if err != nil {
+		return nil, err
+	}
+	return append(kept, filterClientEvents(members, req.StateFilter)...), nil
+}
+
+// timelineSenders lists the distinct senders of a timeline, in the order
+// they first appear.
+func timelineSenders(timeline Timeline) []string {
+	seen := make(map[string]struct{}, len(timeline.Events))
+	out := make([]string, 0, len(timeline.Events))
+	for _, e := range timeline.Events {
+		if _, dup := seen[e.Sender]; dup {
+			continue
+		}
+		seen[e.Sender] = struct{}{}
+		out = append(out, e.Sender)
+	}
+	return out
+}
+
+// stateKey identifies one piece of state.
+type stateKeyTuple struct {
+	eventType string
+	stateKey  string
+}
+
+// tupleOf is the state tuple of a client event, and whether it has one.
+func tupleOf(e ClientEvent) (stateKeyTuple, bool) {
+	if e.StateKey == nil {
+		return stateKeyTuple{}, false
+	}
+	return stateKeyTuple{eventType: e.Type, stateKey: *e.StateKey}, true
+}
+
+// stateDelta is what changed between two state sets: every tuple of `after`
+// whose event is not the one `before` had.
+//
+// A tuple that `before` had and `after` does not is left out. State is only
+// ever replaced, never removed — there is no way to unset a piece of room
+// state — so such a tuple can only come from state resolution dropping an
+// event, and there is nothing a client could be sent that would express it.
+func stateDelta(before, after []ClientEvent) []ClientEvent {
+	had := make(map[stateKeyTuple]string, len(before))
+	for _, e := range before {
+		if tuple, ok := tupleOf(e); ok {
+			had[tuple] = e.EventID
+		}
+	}
+
+	out := make([]ClientEvent, 0, len(after))
+	for _, e := range after {
+		tuple, ok := tupleOf(e)
+		if !ok {
+			continue
+		}
+		if had[tuple] == e.EventID {
+			continue
+		}
+		out = append(out, e)
+	}
+	return out
+}
+
+// filterClientEvents applies the part of an event filter that is a
+// comparison on the event itself.
+//
+// It is the same filter the store applies to a timeline read, applied in Go
+// because a state read is a set of tuples rather than a window of a stream:
+// there is no index to push it down to.
+func filterClientEvents(events []ClientEvent, filter storage.TimelineFilter) []ClientEvent {
+	if filter.IsEmpty() {
+		return events
+	}
+	out := make([]ClientEvent, 0, len(events))
+	for _, e := range events {
+		if matchesFilter(e, filter) {
+			out = append(out, e)
+		}
+	}
+	return out
+}
+
+// matchesFilter reports whether one event passes a filter. Exclusions win
+// over inclusions, which is what the spec requires.
+func matchesFilter(e ClientEvent, filter storage.TimelineFilter) bool {
+	switch {
+	case matchesAny(e.Type, filter.NotTypes):
+		return false
+	case matchesAny(e.Sender, filter.NotSenders):
+		return false
+	case len(filter.Types) > 0 && !matchesAny(e.Type, filter.Types):
+		return false
+	case len(filter.Senders) > 0 && !matchesAny(e.Sender, filter.Senders):
+		return false
+	}
+	return true
+}
+
+// matchesAny reports whether value matches one of the filter patterns, each
+// of which may end in '*' as the spec's wildcard.
+func matchesAny(value string, patterns []string) bool {
+	for _, pattern := range patterns {
+		if prefix, isWildcard := strings.CutSuffix(pattern, "*"); isWildcard {
+			if strings.HasPrefix(value, prefix) {
+				return true
+			}
+			continue
+		}
+		if value == pattern {
+			return true
+		}
+	}
+	return false
+}

server/internal/roomserver/stream.go

@@ -0,0 +1,481 @@
+package roomserver
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"slices"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/matrix/roomversion"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// The read model: the event stream as /sync, /messages and /context read
+// it.
+//
+// It is here rather than in the syncapi because reading an event means
+// parsing it under its room's version, and that is the roomserver's
+// business — the syncapi would otherwise need its own copy of the room
+// version registry, the event parser and the client projection, and a
+// second copy of those would be a second place for them to disagree. What
+// the syncapi gets instead is windows of ClientEvents with their positions,
+// and it assembles the response shape around them.
+
+// MembershipAt is a user's current membership in a room, with the position
+// of the event that set it.
+type MembershipAt struct {
+	RoomID     string
+	Membership string
+	EventID    string
+
+	// Position is the stream position of the membership event.
+	Position int64
+}
+
+// TimelineRequest is one window of a room's event stream.
+//
+// The bounds are the store's: Low exclusive, High inclusive, both
+// directions, and zero meaning unbounded at that end. Direction decides
+// only which end the limit trims and the order the store reads in — the
+// events always come back oldest first.
+type TimelineRequest struct {
+	RoomID string
+
+	// User is the full user ID the window is read on behalf of. The result
+	// is clamped to what their membership and the room's history
+	// visibility allow.
+	User string
+
+	Low  int64
+	High int64
+
+	Direction storage.TimelineDirection
+
+	// Limit caps how many events come back.
+	Limit int
+
+	// Filter narrows which events are read. Applying it here rather than
+	// after the limit is what keeps a filtered page from coming back short
+	// and looking like the end of the room.
+	Filter storage.TimelineFilter
+}
+
+// Timeline is a window of a room's event stream.
+type Timeline struct {
+	// Events are in stream order, oldest first, whichever direction was
+	// asked for: a client renders a timeline forwards even when it paged
+	// backwards to get it.
+	Events []ClientEvent
+
+	// Start and End are the positions bounding the window: Start below the
+	// first event and End at the last. They are the two tokens a client
+	// needs to ask for the next page in either direction, and either can
+	// be handed back as a bound without re-reading the event it names.
+	Start int64
+	End   int64
+
+	// Limited reports that the limit cut the window short, so there is
+	// more of it beyond the end the limit trimmed.
+	Limited bool
+}
+
+// StreamTop returns the top of this server's event stream.
+func (r *API) StreamTop(ctx context.Context) (int64, error) {
+	return r.store.MaxStreamOrdering(ctx)
+}
+
+// UserMemberships returns the rooms a user has one of the given memberships
+// in, with the position of each membership event.
+func (r *API) UserMemberships(ctx context.Context, userID string, memberships []string) ([]MembershipAt, error) {
+	stored, err := r.store.MembershipsForUser(ctx, userID, memberships)
+	if err != nil {
+		return nil, err
+	}
+	out := make([]MembershipAt, 0, len(stored))
+	for _, m := range stored {
+		out = append(out, MembershipAt{
+			RoomID:     m.RoomID,
+			Membership: m.Membership,
+			EventID:    m.EventID,
+			Position:   m.StreamOrdering,
+		})
+	}
+	return out, nil
+}
+
+// RoomsWithEvents returns the subset of rooms that have an event after from
+// and at or before to.
+func (r *API) RoomsWithEvents(ctx context.Context, roomIDs []string, from, to int64) ([]string, error) {
+	return r.store.RoomsWithEvents(ctx, roomIDs, from, to)
+}
+
+// Timeline reads one window of a room's event stream, as far as the user is
+// allowed to see it.
+func (r *API) Timeline(ctx context.Context, req TimelineRequest) (Timeline, error) {
+	room, rules, err := r.roomAndRules(ctx, req.RoomID)
+	if err != nil {
+		return Timeline{}, err
+	}
+	visible, err := r.requireReadable(ctx, room.ID, req.User)
+	if err != nil {
+		return Timeline{}, err
+	}
+	return r.timeline(ctx, rules, visible, req)
+}
+
+// timeline is Timeline with the room's rules and the user's visible range
+// already in hand, for the sync loop that reads many rooms at once.
+func (r *API) timeline(ctx context.Context, rules roomversion.Rules, visible VisibleRange, req TimelineRequest) (Timeline, error) {
+	// An empty window is reported at the bound the caller came in on, so
+	// that the tokens it hands back are still usable.
+	empty := Timeline{Start: req.Low, End: req.Low}
+	if req.Direction == storage.DirectionBackward && req.High > 0 {
+		empty = Timeline{Start: req.High, End: req.High}
+	}
+	if req.Limit <= 0 {
+		return empty, nil
+	}
+
+	low, high, any := visible.Clamp(req.Low, req.High)
+	if !any {
+		return empty, nil
+	}
+
+	// One more than asked for, so that "there is more" can be told from
+	// "that was all" without a second query.
+	stored, err := r.store.Timeline(ctx, storage.TimelineRequest{
+		RoomID:    req.RoomID,
+		Low:       low,
+		High:      high,
+		Direction: req.Direction,
+		Limit:     req.Limit + 1,
+		Filter:    req.Filter,
+	})
+	if err != nil {
+		return Timeline{}, err
+	}
+
+	out := empty
+	if len(stored) > req.Limit {
+		out.Limited = true
+		stored = stored[:req.Limit]
+	}
+	if len(stored) == 0 {
+		return out, nil
+	}
+
+	// The store returned them in the direction asked for; a client always
+	// renders a timeline oldest first.
+	if req.Direction == storage.DirectionBackward {
+		slices.Reverse(stored)
+	}
+
+	out.Events = make([]ClientEvent, 0, len(stored))
+	for _, s := range stored {
+		e, err := parseEvent(s, rules)
+		if err != nil {
+			return Timeline{}, err
+		}
+		out.Events = append(out.Events, r.clientEvent(s, e, req.RoomID))
+	}
+	// Start sits below the oldest event, so that handing it back as a High
+	// bound returns the page before this one; End is at the newest, so that
+	// handing it back as a Low bound returns the page after.
+	out.Start = stored[0].StreamOrdering - 1
+	out.End = stored[len(stored)-1].StreamOrdering
+	return out, nil
+}
+
+// EventPosition returns an event's position in the stream, for a user who
+// may see it.
+//
+// /context needs it: the timeline reads are bounded by position rather than
+// by event ID, so the event a context is centred on has to be located
+// before either side of it can be read.
+func (r *API) EventPosition(ctx context.Context, roomID, userID, eventID string) (int64, error) {
+	visible, err := r.requireReadable(ctx, roomID, userID)
+	if err != nil {
+		return 0, err
+	}
+	stored, err := r.store.Event(ctx, roomID, eventID)
+	if err != nil {
+		return 0, err
+	}
+	if !visible.Contains(stored.StreamOrdering) {
+		return 0, fmt.Errorf("event %q: %w", eventID, ErrNoSuchEvent)
+	}
+	return stored.StreamOrdering, nil
+}
+
+// StateAt returns a room's state as of a stream position, in the client
+// shape.
+//
+// It is what a sync with a gap in it and a leaver's state read both need:
+// the state current state cannot give, because the room has moved on since.
+func (r *API) StateAt(ctx context.Context, roomID, userID string, pos int64) ([]ClientEvent, error) {
+	room, rules, err := r.roomAndRules(ctx, roomID)
+	if err != nil {
+		return nil, err
+	}
+	visible, err := r.requireReadable(ctx, room.ID, userID)
+	if err != nil {
+		return nil, err
+	}
+	if visible.To != 0 && (pos == 0 || pos > visible.To) {
+		pos = visible.To
+	}
+	return r.stateAt(ctx, roomID, rules, pos)
+}
+
+// stateAt is StateAt with the checks already done.
+//
+// A position of zero is the room's current state, which is both cheaper to
+// read and what every caller that is up to date wants.
+func (r *API) stateAt(ctx context.Context, roomID string, rules roomversion.Rules, pos int64) ([]ClientEvent, error) {
+	var (
+		entries []storage.StateEntry
+		err     error
+	)
+	if pos == 0 {
+		stored, err := r.store.CurrentStateEvents(ctx, roomID)
+		if err != nil {
+			return nil, err
+		}
+		return r.clientEvents(stored, rules, roomID)
+	}
+	if entries, err = r.store.StateAtStream(ctx, roomID, pos); err != nil {
+		return nil, err
+	}
+	return r.clientEventsByID(ctx, roomID, rules, stateEntryIDs(entries))
+}
+
+// StateBeforeEvent returns the state of a room as of an event's
+// prev_events: the state /context reports alongside it.
+func (r *API) StateBeforeEvent(ctx context.Context, roomID, userID, eventID string) ([]ClientEvent, error) {
+	room, rules, err := r.roomAndRules(ctx, roomID)
+	if err != nil {
+		return nil, err
+	}
+	visible, err := r.requireReadable(ctx, room.ID, userID)
+	if err != nil {
+		return nil, err
+	}
+
+	stored, err := r.store.Event(ctx, roomID, eventID)
+	if err != nil {
+		return nil, err
+	}
+	if !visible.Contains(stored.StreamOrdering) {
+		return nil, fmt.Errorf("%w: %s may not see %s", ErrForbidden, userID, eventID)
+	}
+
+	entries, err := r.store.StateBefore(ctx, roomID, eventID)
+	if err != nil {
+		return nil, err
+	}
+	state, err := r.clientEventsByID(ctx, roomID, rules, stateEntryIDs(entries))
+	if err != nil {
+		return nil, err
+	}
+	// The event itself, when it is a state event, is part of the state
+	// after it but not before it. /context wants the state the event was
+	// sent against, so it is left out — except that a client rendering the
+	// event needs the sender's member event, which is in the snapshot
+	// already.
+	return state, nil
+}
+
+// MemberEvents returns the m.room.member events of specific users in a
+// room, in the client shape.
+//
+// Lazy-loading is what this is for: a client that asked for it is sent the
+// members who appear in the timeline it was given, rather than all of them.
+func (r *API) MemberEvents(ctx context.Context, roomID string, userIDs []string) ([]ClientEvent, error) {
+	if len(userIDs) == 0 {
+		return nil, nil
+	}
+	_, rules, err := r.roomAndRules(ctx, roomID)
+	if err != nil {
+		return nil, err
+	}
+	return r.memberEvents(ctx, roomID, rules, userIDs)
+}
+
+// memberEvents is MemberEvents with the rules in hand.
+func (r *API) memberEvents(ctx context.Context, roomID string, rules roomversion.Rules, userIDs []string) ([]ClientEvent, error) {
+	out := make([]ClientEvent, 0, len(userIDs))
+	for _, userID := range userIDs {
+		stored, err := r.store.CurrentStateEvent(ctx, roomID, event.TypeMember, userID)
+		switch {
+		case isNotFound(err):
+			// A member who appears in the timeline but is not in the
+			// room's current state: their membership was reset by state
+			// resolution, or the event is old enough to predate it. Not an
+			// error, and nothing to send.
+			continue
+		case err != nil:
+			return nil, err
+		}
+		e, err := parseEvent(stored, rules)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, r.clientEvent(stored, e, roomID))
+	}
+	return out, nil
+}
+
+// StrippedEvent is a piece of room state as it travels with an invite:
+// enough for a client to show the user what they are being asked to join,
+// and nothing that would let them read the room.
+//
+// It has no event ID and no timestamp on purpose — the spec's "stripped
+// state" is explicitly not an event, and a client must not put it in a
+// timeline.
+type StrippedEvent struct {
+	Type     string          `json:"type"`
+	StateKey string          `json:"state_key"`
+	Sender   string          `json:"sender"`
+	Content  json.RawMessage `json:"content"`
+}
+
+// strippedStateTypes is what the spec says an invite carries: the state a
+// client needs to render the room in an invite list.
+var strippedStateTypes = []string{
+	event.TypeCreate,
+	event.TypeJoinRules,
+	TypeCanonicalAlias,
+	TypeName,
+	TypeAvatar,
+	TypeTopic,
+	TypeEncryption,
+	event.TypeHistoryVisibility,
+}
+
+// StrippedState returns the stripped state of a room, plus the member
+// events of the users named.
+//
+// The member events are what makes an invite legible: a client showing
+// "Alice invited you" needs Alice's display name, and the invitee's own
+// member event carries the invite's reason.
+func (r *API) StrippedState(ctx context.Context, roomID string, members []string) ([]StrippedEvent, error) {
+	_, rules, err := r.roomAndRules(ctx, roomID)
+	if err != nil {
+		return nil, err
+	}
+	return r.strippedState(ctx, roomID, rules, members)
+}
+
+// strippedState is StrippedState with the rules in hand.
+func (r *API) strippedState(ctx context.Context, roomID string, rules roomversion.Rules, members []string) ([]StrippedEvent, error) {
+	var out []StrippedEvent
+
+	appendState := func(eventType, stateKey string) error {
+		stored, err := r.store.CurrentStateEvent(ctx, roomID, eventType, stateKey)
+		switch {
+		case isNotFound(err):
+			return nil
+		case err != nil:
+			return err
+		}
+		e, err := parseEvent(stored, rules)
+		if err != nil {
+			return err
+		}
+		key, _ := e.StateKey()
+		out = append(out, StrippedEvent{
+			Type:     e.Type(),
+			StateKey: key,
+			Sender:   e.Sender(),
+			Content:  e.Content(),
+		})
+		return nil
+	}
+
+	for _, eventType := range strippedStateTypes {
+		if err := appendState(eventType, ""); err != nil {
+			return nil, err
+		}
+	}
+	for _, member := range members {
+		if err := appendState(event.TypeMember, member); err != nil {
+			return nil, err
+		}
+	}
+	return out, nil
+}
+
+// RoomSyncSummary is the `summary` block of a room in a sync response.
+//
+// It exists for rooms with no name: a client showing a direct message has
+// to call it something, and the heroes are the members it names it after.
+type RoomSyncSummary struct {
+	// Heroes are up to five members of the room other than the user
+	// syncing, which is what a client falls back to for a room with no
+	// name and no canonical alias.
+	Heroes []string `json:"m.heroes,omitempty"`
+
+	JoinedMemberCount  int `json:"m.joined_member_count"`
+	InvitedMemberCount int `json:"m.invited_member_count"`
+}
+
+// maxHeroes is how many members a summary names. The spec says five.
+const maxHeroes = 5
+
+// RoomSummaryFor builds the summary block of a room for one user.
+func (r *API) RoomSummaryFor(ctx context.Context, roomID, userID string) (RoomSyncSummary, error) {
+	entries, err := r.store.MembershipEntries(ctx, roomID,
+		[]string{event.MembershipJoin, event.MembershipInvite})
+	if err != nil {
+		return RoomSyncSummary{}, err
+	}
+
+	var summary RoomSyncSummary
+	for _, entry := range entries {
+		switch entry.Membership {
+		case event.MembershipJoin:
+			summary.JoinedMemberCount++
+		case event.MembershipInvite:
+			summary.InvitedMemberCount++
+		}
+		if entry.UserID != userID && len(summary.Heroes) < maxHeroes {
+			summary.Heroes = append(summary.Heroes, entry.UserID)
+		}
+	}
+	return summary, nil
+}
+
+// clientEvents projects stored events into the client shape.
+func (r *API) clientEvents(stored []storage.StoredEvent, rules roomversion.Rules, roomID string) ([]ClientEvent, error) {
+	out := make([]ClientEvent, 0, len(stored))
+	for _, s := range stored {
+		e, err := parseEvent(s, rules)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, r.clientEvent(s, e, roomID))
+	}
+	return out, nil
+}
+
+// clientEventsByID reads events by ID and projects them.
+func (r *API) clientEventsByID(ctx context.Context, roomID string, rules roomversion.Rules, eventIDs []string) ([]ClientEvent, error) {
+	if len(eventIDs) == 0 {
+		return nil, nil
+	}
+	stored, err := r.store.EventsByID(ctx, roomID, eventIDs)
+	if err != nil {
+		return nil, err
+	}
+	return r.clientEvents(stored, rules, roomID)
+}
+
+// stateEntryIDs projects state entries onto the event IDs behind them.
+func stateEntryIDs(entries []storage.StateEntry) []string {
+	out := make([]string, 0, len(entries))
+	for _, entry := range entries {
+		out = append(out, entry.EventID)
+	}
+	return out
+}

server/internal/roomserver/stream_test.go

@@ -0,0 +1,421 @@
+package roomserver_test
+
+import (
+	"encoding/json"
+	"errors"
+	"strconv"
+	"testing"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// The read model: timeline windows, the state at a position, and the
+// visible range that bounds both. The visible range is the interesting part
+// — it is where history visibility is decided, and the one place in the
+// component where getting it wrong shows somebody a room they left.
+
+// fill sends n messages into a room and returns their event IDs in order.
+// The bodies are "line 0", "line 1" and so on, counting from zero within
+// each call.
+//
+// Every send gets its own transaction ID, counted across the whole
+// fixture. Reusing one would make the second send an idempotent retry of
+// the first, and a test would then be asserting about a room with half the
+// events it thought it had.
+func (f *fixture) fill(roomID, sender string, n int) []string {
+	f.t.Helper()
+	out := make([]string, 0, n)
+	for i := range n {
+		f.sends++
+		eventID, err := f.sendMessage(roomID, sender,
+			"line "+strconv.Itoa(i), "fill-"+strconv.Itoa(f.sends))
+		if err != nil {
+			f.t.Fatalf("sendMessage: %v", err)
+		}
+		out = append(out, eventID)
+	}
+	return out
+}
+
+// publicRoom creates a room anybody may join.
+func (f *fixture) publicRoom() string {
+	f.t.Helper()
+	return f.createRoom(roomserver.CreateRoomRequest{Preset: "public_chat"}).RoomID
+}
+
+func (f *fixture) join(roomID, userID string) {
+	f.t.Helper()
+	if _, err := f.api.JoinRoom(f.t.Context(), roomserver.JoinRequest{
+		RoomID: roomID, User: userID,
+	}); err != nil {
+		f.t.Fatalf("JoinRoom(%s): %v", userID, err)
+	}
+}
+
+func (f *fixture) leave(roomID, userID string) {
+	f.t.Helper()
+	if err := f.api.LeaveRoom(f.t.Context(), roomID, userID, ""); err != nil {
+		f.t.Fatalf("LeaveRoom(%s): %v", userID, err)
+	}
+}
+
+// bodies projects client events onto their message bodies.
+func bodies(events []roomserver.ClientEvent) []string {
+	out := make([]string, 0, len(events))
+	for _, e := range events {
+		var content struct {
+			Body string `json:"body"`
+		}
+		if err := json.Unmarshal(e.Content, &content); err == nil && content.Body != "" {
+			out = append(out, content.Body)
+		}
+	}
+	return out
+}
+
+func TestTimelineIsOldestFirstWhicheverWayItWasRead(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.fill(roomID, alice, 4)
+
+	backwards, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID:    roomID,
+		User:      alice,
+		Direction: storage.DirectionBackward,
+		Limit:     2,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+
+	// Reading backwards keeps the newest two, and hands them back oldest
+	// first: a client renders a timeline forwards whichever way it paged.
+	got := bodies(backwards.Events)
+	if len(got) != 2 || got[0] != "line 2" || got[1] != "line 3" {
+		t.Errorf("backwards timeline = %v, want [line 2 line 3]", got)
+	}
+	if !backwards.Limited {
+		t.Error("a window the limit cut short must say so")
+	}
+	if backwards.Start >= backwards.End {
+		t.Errorf("Start %d must sit below End %d", backwards.Start, backwards.End)
+	}
+
+	// Start is below the oldest event, so handing it back as a High bound
+	// returns the page before this one, with nothing repeated.
+	earlier, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID:    roomID,
+		User:      alice,
+		High:      backwards.Start,
+		Direction: storage.DirectionBackward,
+		Limit:     2,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+	if got := bodies(earlier.Events); len(got) != 2 || got[1] != "line 1" {
+		t.Errorf("the page before = %v, want it to end at \"line 1\"", got)
+	}
+}
+
+func TestTimelineEndIsUsableAsTheNextLowBound(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.fill(roomID, alice, 3)
+
+	first, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: alice, Direction: storage.DirectionForward, Limit: 2,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+	next, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: alice, Low: first.End,
+		Direction: storage.DirectionForward, Limit: 2,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+
+	// Nothing may appear in both pages: End is a position already seen.
+	seen := map[string]bool{}
+	for _, e := range first.Events {
+		seen[e.EventID] = true
+	}
+	for _, e := range next.Events {
+		if seen[e.EventID] {
+			t.Errorf("%s appeared in two consecutive pages", e.EventID)
+		}
+	}
+}
+
+func TestVisibleRangeOfAMember(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+
+	visible, err := f.api.VisibleRange(t.Context(), roomID, alice)
+	if err != nil {
+		t.Fatalf("VisibleRange: %v", err)
+	}
+	if !visible.Allowed {
+		t.Fatal("a joined member must be allowed to read the room")
+	}
+	// The default preset is shared history, so a member sees all of it and
+	// is not bounded above.
+	if visible.From != 0 || visible.To != 0 {
+		t.Errorf("range = %+v, want it unbounded for shared history", visible)
+	}
+	if visible.Membership != event.MembershipJoin {
+		t.Errorf("membership = %q, want join", visible.Membership)
+	}
+}
+
+func TestVisibleRangeOfAStranger(t *testing.T) {
+	f := newFixture(t, nil)
+	// private_chat is invite-only with shared history: a stranger sees
+	// nothing of it.
+	roomID := f.createRoom(roomserver.CreateRoomRequest{Preset: "private_chat"}).RoomID
+
+	visible, err := f.api.VisibleRange(t.Context(), roomID, bob)
+	if err != nil {
+		t.Fatalf("VisibleRange: %v", err)
+	}
+	if visible.Allowed {
+		t.Error("a stranger must not be allowed to read a private room")
+	}
+
+	if _, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: bob, Direction: storage.DirectionBackward, Limit: 10,
+	}); !errors.Is(err, roomserver.ErrForbidden) {
+		t.Errorf("Timeline for a stranger = %v, want ErrForbidden", err)
+	}
+}
+
+func TestVisibleRangeOfADepartedMemberEndsAtTheirLeave(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.join(roomID, bob)
+	f.fill(roomID, alice, 2)
+	f.leave(roomID, bob)
+	f.fill(roomID, alice, 2)
+
+	visible, err := f.api.VisibleRange(t.Context(), roomID, bob)
+	if err != nil {
+		t.Fatalf("VisibleRange: %v", err)
+	}
+	if !visible.Allowed {
+		t.Fatal("a departed member keeps what they saw while they were in the room")
+	}
+	if visible.To == 0 {
+		t.Fatal("a departed member must be bounded above at their own leave")
+	}
+
+	timeline, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: bob, Direction: storage.DirectionBackward, Limit: 50,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+	// They see the two messages sent while they were in the room and
+	// neither of the two sent afterwards. Each fill counts its bodies from
+	// zero, so what is asserted is the count: four messages exist in the
+	// room and Bob may read two of them.
+	if got := bodies(timeline.Events); len(got) != 2 {
+		t.Errorf("a departed member's timeline = %v, want the two messages from before they left", got)
+	}
+}
+
+func TestVisibleRangeWithJoinedHistoryStartsAtTheJoin(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.createRoom(roomserver.CreateRoomRequest{
+		Preset: "public_chat",
+		InitialState: []roomserver.StateEvent{{
+			Type:    event.TypeHistoryVisibility,
+			Content: []byte(`{"history_visibility":"joined"}`),
+		}},
+	}).RoomID
+
+	f.fill(roomID, alice, 2)
+	f.join(roomID, bob)
+	f.fill(roomID, alice, 2)
+
+	timeline, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: bob, Direction: storage.DirectionBackward, Limit: 50,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+
+	// Only the messages sent after Bob joined. The two sent before carry
+	// the same bodies — each fill counts from zero — so what is asserted
+	// is the count: four messages exist and Bob may read two of them.
+	got := bodies(timeline.Events)
+	want := []string{"line 0", "line 1"}
+	if len(got) != len(want) {
+		t.Fatalf("timeline = %v, want %v", got, want)
+	}
+	for i := range got {
+		if got[i] != want[i] {
+			t.Fatalf("timeline = %v, want %v", got, want)
+		}
+	}
+	// Bob's own member event is the first thing he sees, which is how a
+	// client renders "you joined".
+	var sawOwnJoin bool
+	for _, e := range timeline.Events {
+		if e.Type == event.TypeMember && e.StateKey != nil && *e.StateKey == bob {
+			sawOwnJoin = true
+		}
+	}
+	if !sawOwnJoin {
+		t.Error("a user with joined history visibility must see their own join")
+	}
+}
+
+func TestVisibleRangeOfAWorldReadableRoom(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.createRoom(roomserver.CreateRoomRequest{
+		Preset: "public_chat",
+		InitialState: []roomserver.StateEvent{{
+			Type:    event.TypeHistoryVisibility,
+			Content: []byte(`{"history_visibility":"world_readable"}`),
+		}},
+	}).RoomID
+	f.fill(roomID, alice, 2)
+
+	visible, err := f.api.VisibleRange(t.Context(), roomID, bob)
+	if err != nil {
+		t.Fatalf("VisibleRange: %v", err)
+	}
+	if !visible.Allowed {
+		t.Fatal("a world-readable room is readable by anybody")
+	}
+
+	timeline, err := f.api.Timeline(t.Context(), roomserver.TimelineRequest{
+		RoomID: roomID, User: bob, Direction: storage.DirectionBackward, Limit: 50,
+	})
+	if err != nil {
+		t.Fatalf("Timeline: %v", err)
+	}
+	if got := bodies(timeline.Events); len(got) != 2 {
+		t.Errorf("a stranger's timeline of a world-readable room = %v, want both messages", got)
+	}
+}
+
+func TestRoomStateOfADepartedMemberIsAsOfTheirLeave(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.join(roomID, bob)
+
+	if _, err := f.api.SendEvent(t.Context(), roomserver.SendEventRequest{
+		RoomID: roomID, Sender: alice, Type: roomserver.TypeTopic, StateKey: stringPtr(""),
+		Content: []byte(`{"topic":"while bob was here"}`),
+	}); err != nil {
+		t.Fatalf("setting the topic: %v", err)
+	}
+	f.leave(roomID, bob)
+	if _, err := f.api.SendEvent(t.Context(), roomserver.SendEventRequest{
+		RoomID: roomID, Sender: alice, Type: roomserver.TypeTopic, StateKey: stringPtr(""),
+		Content: []byte(`{"topic":"after bob left"}`),
+	}); err != nil {
+		t.Fatalf("setting the topic again: %v", err)
+	}
+
+	// Phase 3 refused a departed member outright and said so in a comment;
+	// they are now served the room as it was when they left.
+	topic, err := f.api.StateEventContent(t.Context(), roomID, bob, roomserver.TypeTopic, "")
+	if err != nil {
+		t.Fatalf("StateEventContent for a departed member: %v", err)
+	}
+	var content struct {
+		Topic string `json:"topic"`
+	}
+	if err := json.Unmarshal(topic, &content); err != nil {
+		t.Fatalf("unmarshal the topic: %v", err)
+	}
+	if content.Topic != "while bob was here" {
+		t.Errorf("topic = %q, want the one in force when they left", content.Topic)
+	}
+}
+
+func TestEventOutsideTheVisibleRangeIsUnknown(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.join(roomID, bob)
+	f.leave(roomID, bob)
+
+	later := f.fill(roomID, alice, 1)[0]
+
+	// Not "forbidden": whether a particular event ID exists in a room is
+	// itself something a departed member is not entitled to know.
+	if _, err := f.api.Event(t.Context(), roomID, bob, later); !errors.Is(err, roomserver.ErrNoSuchEvent) {
+		t.Errorf("Event after a leave = %v, want ErrNoSuchEvent", err)
+	}
+}
+
+func TestRoomSummaryCountsAndHeroes(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.publicRoom()
+	f.join(roomID, bob)
+	if err := f.api.InviteUser(t.Context(), roomserver.InviteRequest{
+		RoomID: roomID, Sender: alice, Target: carol,
+	}); err != nil {
+		t.Fatalf("InviteUser: %v", err)
+	}
+
+	summary, err := f.api.RoomSummaryFor(t.Context(), roomID, alice)
+	if err != nil {
+		t.Fatalf("RoomSummaryFor: %v", err)
+	}
+	if summary.JoinedMemberCount != 2 {
+		t.Errorf("joined = %d, want 2", summary.JoinedMemberCount)
+	}
+	if summary.InvitedMemberCount != 1 {
+		t.Errorf("invited = %d, want 1", summary.InvitedMemberCount)
+	}
+	// The heroes are everybody but the user asking: a client names a room
+	// after the other people in it.
+	for _, hero := range summary.Heroes {
+		if hero == alice {
+			t.Errorf("heroes = %v, want it to leave out the user asking", summary.Heroes)
+		}
+	}
+	if len(summary.Heroes) != 2 {
+		t.Errorf("heroes = %v, want both the other members", summary.Heroes)
+	}
+}
+
+func TestStrippedStateIsEnoughToRenderAnInvite(t *testing.T) {
+	f := newFixture(t, nil)
+	roomID := f.createRoom(roomserver.CreateRoomRequest{
+		Preset: "private_chat",
+		Name:   "The Study",
+	}).RoomID
+	if err := f.api.InviteUser(t.Context(), roomserver.InviteRequest{
+		RoomID: roomID, Sender: alice, Target: bob,
+	}); err != nil {
+		t.Fatalf("InviteUser: %v", err)
+	}
+
+	stripped, err := f.api.StrippedState(t.Context(), roomID, []string{bob, alice})
+	if err != nil {
+		t.Fatalf("StrippedState: %v", err)
+	}
+
+	types := map[string]bool{}
+	for _, e := range stripped {
+		types[e.Type] = true
+		// Stripped state is explicitly not an event, so it carries no ID
+		// and no timestamp: a client must not put it in a timeline.
+		if e.Sender == "" {
+			t.Errorf("stripped %s has no sender", e.Type)
+		}
+	}
+	for _, want := range []string{event.TypeCreate, event.TypeJoinRules, roomserver.TypeName, event.TypeMember} {
+		if !types[want] {
+			t.Errorf("stripped state is missing %s (have %v)", want, types)
+		}
+	}
+}

server/internal/roomserver/visibility.go

@@ -0,0 +1,234 @@
+package roomserver
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/matrix/roomversion"
+)
+
+// VisibleRange is the span of a room's event stream a user may read.
+//
+// It is a range rather than a yes or no because membership is not a yes or
+// no either. A user who has left a room keeps what they saw while they were
+// in it and sees nothing after their leave; a user who joined a room whose
+// history visibility is `joined` sees nothing from before their join. Both
+// are the same question — where does this user's window start and end — and
+// answering it once per room is what lets a timeline read be a single
+// bounded query rather than a filter applied to every event in it.
+type VisibleRange struct {
+	// Allowed is false when the user may see nothing of the room at all.
+	Allowed bool
+
+	// From is the first position the user may see, exclusive. Zero means
+	// the start of the room.
+	From int64
+
+	// To is the last position the user may see, inclusive. Zero means the
+	// top of the stream, and goes on meaning that as the stream grows.
+	To int64
+
+	// Membership is the user's current membership, empty for a user who was
+	// never in the room.
+	Membership string
+
+	// HistoryVisibility is the room's current history visibility, which the
+	// bounds above were derived from.
+	HistoryVisibility string
+}
+
+// Contains reports whether a position is inside the range.
+func (v VisibleRange) Contains(pos int64) bool {
+	switch {
+	case !v.Allowed:
+		return false
+	case pos <= v.From:
+		return false
+	case v.To != 0 && pos > v.To:
+		return false
+	}
+	return true
+}
+
+// Clamp narrows a caller's window to what the range allows, reporting
+// whether anything is left of it.
+//
+// The bounds are exclusive at the bottom and inclusive at the top, the same
+// as a storage.TimelineRequest, so the result can be handed straight to
+// one.
+func (v VisibleRange) Clamp(from, to int64) (int64, int64, bool) {
+	if !v.Allowed {
+		return 0, 0, false
+	}
+	from = max(from, v.From)
+	if v.To != 0 && (to == 0 || to > v.To) {
+		to = v.To
+	}
+	if to != 0 && from >= to {
+		return 0, 0, false
+	}
+	return from, to, true
+}
+
+// VisibleRange works out what a user may read of a room.
+//
+// The rules are the spec's history visibility:
+//
+//   - world_readable: everything, to anybody, member or not.
+//   - shared: everything, to anybody who is or has been a member.
+//   - invited: from the user's invite onwards.
+//   - joined: from the user's join onwards.
+//
+// A user who has left or was banned is bounded at their own membership
+// event in every case, which is what stops a kicked user's client from
+// going on streaming the room. An invitee is told enough to decide — the
+// stripped state that travels with the invite — and no timeline.
+//
+// The approximation here is the room's *current* history visibility. The
+// spec judges each event by the visibility in force when it was sent, so a
+// room that was once world_readable and is now `joined` hides history here
+// that the spec would show, and the reverse shows history it would hide.
+// Doing it exactly means resolving the history visibility at every event in
+// the window — a state read per event — and belongs with the state storage
+// work of PLAN.md open question 1. Until then this follows the setting the
+// room's members last agreed on, which is the one they would expect to
+// apply.
+func (r *API) VisibleRange(ctx context.Context, roomID, userID string) (VisibleRange, error) {
+	room, rules, err := r.roomAndRules(ctx, roomID)
+	if err != nil {
+		return VisibleRange{}, err
+	}
+	summary, err := r.store.RoomSummary(ctx, room.ID)
+	if err != nil {
+		return VisibleRange{}, err
+	}
+	return r.visibleRange(ctx, roomID, rules, userID, summary.HistoryVisibility)
+}
+
+// visibleRange is VisibleRange with the room's history visibility already
+// in hand, for the callers — a sync over many rooms — that have it.
+func (r *API) visibleRange(ctx context.Context, roomID string, rules roomversion.Rules, userID, historyVisibility string) (VisibleRange, error) {
+	out := VisibleRange{HistoryVisibility: historyVisibility}
+
+	changes, err := r.membershipHistory(ctx, roomID, rules, userID)
+	if err != nil {
+		return VisibleRange{}, err
+	}
+	if len(changes) > 0 {
+		out.Membership = changes[len(changes)-1].membership
+	}
+
+	worldReadable := historyVisibility == HistoryVisibilityWorldReadable
+	switch out.Membership {
+	case event.MembershipJoin:
+		out.Allowed = true
+	case event.MembershipLeave, event.MembershipBan:
+		out.Allowed = true
+		// Bounded at their own membership event: what they saw stays
+		// theirs, what happened afterwards does not.
+		out.To = changes[len(changes)-1].position
+	default:
+		// Invited, knocking, or never a member. Only a world-readable room
+		// has anything to show them.
+		out.Allowed = worldReadable
+	}
+
+	// World-readable removes the floor — they could read the room as a
+	// stranger, so there is nothing to gain by hiding it from them as a
+	// member — but not the ceiling: a leaver's *membership* view still ends
+	// where they left, and the stranger's view is what /messages on a
+	// world-readable room gives them anyway.
+	if !worldReadable && out.Allowed {
+		switch historyVisibility {
+		case HistoryVisibilityInvited, HistoryVisibilityJoined:
+			out.From = spanStart(changes, historyVisibility)
+		}
+	}
+	return out, nil
+}
+
+// membershipChange is one of a user's membership events in a room.
+type membershipChange struct {
+	membership string
+	position   int64
+}
+
+// membershipHistory reads a user's membership events in a room, oldest
+// first.
+func (r *API) membershipHistory(ctx context.Context, roomID string, rules roomversion.Rules, userID string) ([]membershipChange, error) {
+	stored, err := r.store.UserMembershipEvents(ctx, roomID, userID)
+	if err != nil {
+		return nil, err
+	}
+	out := make([]membershipChange, 0, len(stored))
+	for _, s := range stored {
+		e, err := parseEvent(s, rules)
+		if err != nil {
+			return nil, err
+		}
+		membership, err := e.Membership()
+		if err != nil {
+			// A member event with no membership could not have been
+			// accepted into the room, so this is a corrupted row rather
+			// than something a client sent. Skipping it would widen the
+			// window silently.
+			return nil, fmt.Errorf("member event %s of %s has no membership: %w", s.EventID, roomID, err)
+		}
+		out = append(out, membershipChange{membership: membership, position: s.StreamOrdering})
+	}
+	return out, nil
+}
+
+// spanStart is the floor of the user's current membership span, exclusive.
+//
+// For `joined` visibility that is the join that began the span they are in
+// — or, for a user who has left, the one that began the span they left
+// from. For `invited` it is the invite that preceded that join, because an
+// invitee is entitled to the room from the moment they were asked.
+//
+// A user who joined, left and rejoined therefore sees their latest span and
+// not the earlier ones, which is what the spec's per-event rule works out
+// to for everything after their last join.
+func spanStart(changes []membershipChange, historyVisibility string) int64 {
+	// Walk back to the membership event that opened the current span.
+	start := len(changes) - 1
+	for start > 0 && changes[start].membership != event.MembershipJoin {
+		start--
+	}
+	for start > 0 && changes[start-1].membership == event.MembershipJoin {
+		// Consecutive joins are profile changes, not rejoins; the span
+		// began at the first of them.
+		start--
+	}
+	if historyVisibility == HistoryVisibilityInvited &&
+		start > 0 && changes[start-1].membership == event.MembershipInvite {
+		start--
+	}
+	// Exclusive, so one below the event itself: the membership event is the
+	// first thing the user sees, which is how a client renders "you joined".
+	return changes[start].position - 1
+}
+
+// requireReadable refuses a caller who may see nothing of a room, and
+// returns the range for one who may.
+func (r *API) requireReadable(ctx context.Context, roomID, userID string) (VisibleRange, error) {
+	visible, err := r.VisibleRange(ctx, roomID, userID)
+	if err != nil {
+		return VisibleRange{}, err
+	}
+	return visible, visible.refusal(roomID, userID)
+}
+
+// refusal turns a range that allows nothing into the error a client is
+// given for it.
+func (v VisibleRange) refusal(roomID, userID string) error {
+	if v.Allowed {
+		return nil
+	}
+	if v.Membership == event.MembershipInvite {
+		return fmt.Errorf("%w: %s has been invited to %s but has not joined it",
+			ErrForbidden, userID, roomID)
+	}
+	return fmt.Errorf("%w: %s is not in %s", ErrForbidden, userID, roomID)
+}

server/internal/setup/phase1_test.go

@@ -74,11 +74,13 @@ func TestClientAPIIsWiredUp(t *testing.T) {
 		t.Fatalf("registration returned no access token: %s", rec.Body.String())
 	}
 
+	// /sync is not in this list: it belongs to the syncapi, which needs
+	// the roomserver, which needs a signing key this server was started
+	// without. TestSyncIsWiredUp covers it.
 	for _, path := range []string{
 		"/_matrix/client/v3/account/whoami",
 		"/_matrix/client/v3/capabilities",
 		"/_matrix/client/v3/devices",
-		"/_matrix/client/v3/sync",
 	} {
 		if rec := getWithToken(t, srv, path, token); rec.Code != http.StatusOK {
 			t.Errorf("GET %s = %d, want 200 (body: %s)", path, rec.Code, rec.Body.String())

server/internal/setup/phase4_test.go

@@ -0,0 +1,149 @@
+package setup_test
+
+import (
+	"bytes"
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"path/filepath"
+	"testing"
+
+	"github.com/dariuszjaniak/warp/server/internal/config"
+	"github.com/dariuszjaniak/warp/server/internal/matrix/signing"
+	"github.com/dariuszjaniak/warp/server/internal/setup"
+)
+
+// newSigningServer is newTestServer with a real signing key, which is what
+// the roomserver and therefore the syncapi need: a server started without
+// one has no rooms and so no stream to read.
+func newSigningServer(t *testing.T, tweak func(*config.Config)) *setup.Server {
+	t.Helper()
+
+	key, err := signing.Generate("test")
+	if err != nil {
+		t.Fatalf("signing.Generate: %v", err)
+	}
+	keyPath := filepath.Join(t.TempDir(), "signing.key")
+	if err := key.Save(keyPath); err != nil {
+		t.Fatalf("saving the signing key: %v", err)
+	}
+
+	cfg := config.Default()
+	cfg.Global.ServerName = "warp.example"
+	cfg.Global.PrivateKeyPath = keyPath
+	cfg.Database.ConnectionString = "file:" + t.Name() + "?mode=memory&cache=shared"
+	cfg.Registration.Enabled = true
+	// The limiter keys on the client address, and every httptest request
+	// comes from the same one.
+	cfg.ClientAPI.RateLimit.Enabled = false
+	if tweak != nil {
+		tweak(cfg)
+	}
+	if err := cfg.Validate(); err != nil {
+		t.Fatalf("test config is invalid: %v", err)
+	}
+
+	srv, err := setup.New(t.Context(), cfg, setup.Options{Logger: slog.New(slog.DiscardHandler)})
+	if err != nil {
+		t.Fatalf("setup.New: %v", err)
+	}
+	t.Cleanup(func() { _ = srv.Close() })
+	return srv
+}
+
+// registerThrough walks the registration flow and returns the access token.
+func registerThrough(t *testing.T, srv *setup.Server, localpart string) string {
+	t.Helper()
+
+	body := map[string]any{"username": localpart, "password": "sUp3rs3kr1t"}
+	rec := post(t, srv, "/_matrix/client/v3/register", body, "")
+	if rec.Code != http.StatusUnauthorized {
+		t.Fatalf("register status = %d, want 401 (body: %s)", rec.Code, rec.Body.String())
+	}
+	body["auth"] = map[string]any{
+		"type":    "m.login.dummy",
+		"session": decode[map[string]any](t, rec)["session"],
+	}
+
+	rec = post(t, srv, "/_matrix/client/v3/register", body, "")
+	if rec.Code != http.StatusOK {
+		t.Fatalf("register status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
+	}
+	token, _ := decode[map[string]any](t, rec)["access_token"].(string)
+	if token == "" {
+		t.Fatalf("registration returned no access token: %s", rec.Body.String())
+	}
+	return token
+}
+
+// put sends a JSON body with PUT through the fully wrapped handler.
+func put(t *testing.T, srv *setup.Server, path string, body any, token string) *httptest.ResponseRecorder {
+	t.Helper()
+
+	raw, err := json.Marshal(body)
+	if err != nil {
+		t.Fatal(err)
+	}
+	req := httptest.NewRequest(http.MethodPut, path, bytes.NewReader(raw))
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Authorization", "Bearer "+token)
+	rec := httptest.NewRecorder()
+	srv.Handler().ServeHTTP(rec, req)
+	return rec
+}
+
+// TestSyncIsWiredUp is the Phase 4 counterpart of TestClientAPIIsWiredUp:
+// it goes through the whole stack rather than a hand-built router, to catch
+// an endpoint that was written but never registered.
+func TestSyncIsWiredUp(t *testing.T) {
+	srv := newSigningServer(t, func(c *config.Config) { c.Sync.Presence = true })
+	token := registerThrough(t, srv, "alice")
+
+	if srv.Sync() == nil {
+		t.Fatal("a server with a signing key must have a syncapi")
+	}
+
+	rec := post(t, srv, "/_matrix/client/v3/createRoom", map[string]any{}, token)
+	if rec.Code != http.StatusOK {
+		t.Fatalf("createRoom status = %d (body: %s)", rec.Code, rec.Body.String())
+	}
+	roomID, _ := decode[map[string]any](t, rec)["room_id"].(string)
+	if roomID == "" {
+		t.Fatalf("createRoom returned no room ID: %s", rec.Body.String())
+	}
+
+	for _, path := range []string{
+		"/_matrix/client/v3/sync",
+		"/_matrix/client/v3/rooms/" + roomID + "/messages?dir=b",
+		"/_matrix/client/v3/presence/@alice:warp.example/status",
+	} {
+		if rec := getWithToken(t, srv, path, token); rec.Code != http.StatusOK {
+			t.Errorf("GET %s = %d, want 200 (body: %s)", path, rec.Code, rec.Body.String())
+		}
+	}
+}
+
+// TestPresenceIsOffByDefault pins down PLAN.md open question 3: presence is
+// the noisiest thing in Matrix and a new install does not get it.
+func TestPresenceIsOffByDefault(t *testing.T) {
+	if config.Default().Sync.Presence {
+		t.Error("presence must be off in a default configuration")
+	}
+
+	srv := newSigningServer(t, nil)
+	token := registerThrough(t, srv, "alice")
+
+	// The endpoint is registered and refuses, rather than being absent: a
+	// client that gets a 404 concludes the server is too old to have
+	// presence and stops asking, where a refusal says it is switched off.
+	rec := put(t, srv, "/_matrix/client/v3/presence/@alice:warp.example/status",
+		map[string]any{"presence": "online"}, token)
+	if rec.Code != http.StatusForbidden {
+		t.Errorf("PUT /presence with presence off = %d, want 403 (body: %s)",
+			rec.Code, rec.Body.String())
+	}
+	if code, _ := decode[map[string]any](t, rec)["errcode"].(string); code != "M_FORBIDDEN" {
+		t.Errorf("errcode = %q, want M_FORBIDDEN", code)
+	}
+}

server/internal/setup/routes.go

@@ -12,10 +12,12 @@ import (
 //
 // Keep it honest: a version belongs here once the endpoints a client
 // actually uses from it work. Accounts, sessions, devices, profiles,
-// account data and — with Phase 3 — rooms all do. What is still missing is
-// the event stream: /sync returns an empty placeholder and there is no
-// /messages, so a client can create a room and send into it but cannot read
-// its history. The list grows when Phase 4 lands.
+// account data, rooms and — with Phase 4 — the event stream all do, which
+// is enough for a client to be used. What is still missing is end-to-end
+// encryption (Phase 7), media (Phase 6), push (Phase 8) and federation
+// (Phase 5); a client discovers each of those from the endpoint it needs
+// rather than from this list, which is why their absence does not make a
+// version listed here a lie.
 var SpecVersions = []string{"v1.13", "v1.14"}
 
 // UnstableFeatures advertises MSC support. Simplified Sliding Sync

server/internal/setup/server.go

@@ -25,6 +25,7 @@ import (
 	"github.com/dariuszjaniak/warp/server/internal/settings"
 	"github.com/dariuszjaniak/warp/server/internal/storage"
 	_ "github.com/dariuszjaniak/warp/server/internal/storage/drivers" // link in the storage drivers
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
 	"github.com/dariuszjaniak/warp/server/internal/userapi"
 )
 
@@ -41,6 +42,7 @@ type Server struct {
 
 	users     *userapi.API
 	rooms     *roomserver.API
+	sync      *syncapi.API
 	clientAPI *clientapi.ClientAPI
 	adminAPI  *adminapi.AdminAPI
 
@@ -149,21 +151,48 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error)
 	// and the Phase 1 tests start without a key, and then start without
 	// rooms — the C-S room endpoints answer M_UNRECOGNIZED rather than
 	// pretending to work.
+	//
+	// The syncapi comes with it, because a sync is almost entirely room
+	// reads: a server with no rooms has no stream to read, and /sync
+	// answering M_UNRECOGNIZED says so where an empty success would not.
+	//
+	// The notifier is built first and handed to both: it is the
+	// roomserver's emit stage and the syncapi's wake-up, which is exactly
+	// the seam Phase 3 left for this.
 	if s.signingKey != nil {
+		notifier := syncapi.NewNotifier()
+
 		s.rooms, err = roomserver.New(roomserver.Options{
 			ServerName:        cfg.Global.ServerName,
 			Store:             s.db.Rooms(),
 			Profiles:          s.users,
 			SigningKey:        s.signingKey,
 			FederationEnabled: settingsService.FederationEnabled,
+			Notifier:          notifier,
 			Logger:            log,
 		})
 		if err != nil {
 			_ = s.closeDB()
 			return nil, err
 		}
+
+		s.sync, err = syncapi.New(syncapi.Options{
+			ServerName:      cfg.Global.ServerName,
+			Rooms:           s.rooms,
+			Accounts:        s.users,
+			Store:           s.db.Sync(),
+			Notifier:        notifier,
+			PresenceEnabled: cfg.Sync.Presence,
+			TypingTimeout:   cfg.Sync.TypingTimeout,
+			LongPollTimeout: cfg.Sync.LongPollTimeout,
+			Logger:          log,
+		})
+		if err != nil {
+			_ = s.closeDB()
+			return nil, err
+		}
 	} else {
-		log.Warn("started without a signing key: rooms are unavailable")
+		log.Warn("started without a signing key: rooms and /sync are unavailable")
 	}
 
 	s.clientAPI, err = clientapi.New(clientapi.Options{
@@ -171,6 +200,7 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error)
 		ClientBaseURL:      cfg.ClientBaseURL(),
 		Users:              s.users,
 		Rooms:              s.rooms,
+		Sync:               s.sync,
 		Settings:           settingsService,
 		Federation:         s.federation,
 		Registration:       cfg.Registration,
@@ -252,6 +282,10 @@ func (s *Server) Users() *userapi.API { return s.users }
 // without a signing key.
 func (s *Server) Rooms() *roomserver.API { return s.rooms }
 
+// Sync exposes the syncapi component, or nil when the server started
+// without a signing key.
+func (s *Server) Sync() *syncapi.API { return s.sync }
+
 // SigningKey returns the server signing key, or nil when it was skipped.
 func (s *Server) SigningKey() *signing.KeyPair { return s.signingKey }
 
@@ -286,6 +320,7 @@ func (s *Server) Run(ctx context.Context) error {
 		"database", s.db.Driver(),
 		"federation", s.settings.FederationEnabled(),
 		"registration", s.settings.RegistrationEnabled(),
+		"presence", s.cfg.Sync.Presence,
 	)
 
 	errCh := make(chan error, 1)
@@ -321,6 +356,11 @@ func (s *Server) Run(ctx context.Context) error {
 
 // Close releases the resources New acquired.
 func (s *Server) Close() error {
+	if s.sync != nil {
+		// The typing tracker holds a timer per typing user; a server
+		// shutting down should not wait for them.
+		s.sync.Close()
+	}
 	return s.closeDB()
 }
 

server/internal/storage/pocketbase/pocketbase.go

@@ -57,6 +57,9 @@ func (*Database) Users() storage.UserStore { return notImplementedUsers{} }
 // Rooms implements storage.Database.
 func (*Database) Rooms() storage.RoomStore { return notImplementedRooms{} }
 
+// Sync implements storage.Database.
+func (*Database) Sync() storage.SyncStore { return notImplementedSync{} }
+
 // Close implements storage.Database.
 func (*Database) Close() error { return nil }
 
@@ -227,6 +230,30 @@ func (notImplementedRooms) RoomsForUser(context.Context, string, []string) ([]st
 	return nil, storage.ErrNotImplemented
 }
 
+func (notImplementedRooms) MembershipsForUser(context.Context, string, []string) ([]storage.SyncMembership, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedRooms) UserMembershipEvents(context.Context, string, string) ([]storage.StoredEvent, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedRooms) MaxStreamOrdering(context.Context) (int64, error) {
+	return 0, storage.ErrNotImplemented
+}
+
+func (notImplementedRooms) Timeline(context.Context, storage.TimelineRequest) ([]storage.StoredEvent, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedRooms) RoomsWithEvents(context.Context, []string, int64, int64) ([]string, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedRooms) StateAtStream(context.Context, string, int64) ([]storage.StateEntry, error) {
+	return nil, storage.ErrNotImplemented
+}
+
 func (notImplementedRooms) TransactionEvent(context.Context, storage.TransactionRef) (string, error) {
 	return "", storage.ErrNotImplemented
 }
@@ -271,9 +298,71 @@ func (notImplementedRooms) RoomStats(context.Context) (storage.RoomStats, error)
 	return storage.RoomStats{}, storage.ErrNotImplemented
 }
 
+// notImplementedSync satisfies storage.SyncStore the same way. A sync that
+// returned an empty success would be worse than an error: a client would
+// take it for "nothing has happened" and stop asking.
+type notImplementedSync struct{}
+
+func (notImplementedSync) StreamPositions(context.Context) (storage.StreamPositions, error) {
+	return storage.StreamPositions{}, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) SetReceipt(context.Context, storage.Receipt) (int64, bool, error) {
+	return 0, false, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) Receipts(context.Context, []string, int64, int64) ([]storage.Receipt, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) ReceiptsForUser(context.Context, string, []string) ([]storage.Receipt, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) SendToDevice(context.Context, []storage.NewToDeviceMessage) (int64, error) {
+	return 0, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) ToDeviceMessages(context.Context, string, string, int64, int64, int) ([]storage.ToDeviceMessage, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) DeleteToDeviceMessages(context.Context, string, string, int64) error {
+	return storage.ErrNotImplemented
+}
+
+func (notImplementedSync) SetPresence(context.Context, storage.PresenceState) (int64, bool, error) {
+	return 0, false, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) Presence(context.Context, string) (storage.PresenceState, error) {
+	return storage.PresenceState{}, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) PresenceForUsers(context.Context, []string, int64, int64) ([]storage.PresenceState, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) AccountDataChanges(context.Context, string, int64, int64) ([]storage.AccountDataEntry, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) AccountDataAt(context.Context, string, int64) ([]storage.AccountDataEntry, error) {
+	return nil, storage.ErrNotImplemented
+}
+
+func (notImplementedSync) SaveFilter(context.Context, string, []byte, int64) (string, error) {
+	return "", storage.ErrNotImplemented
+}
+
+func (notImplementedSync) Filter(context.Context, string, string) ([]byte, error) {
+	return nil, storage.ErrNotImplemented
+}
+
 // Compile-time proof that the stub keeps up with the interfaces.
 var (
 	_ storage.Database  = (*Database)(nil)
 	_ storage.UserStore = notImplementedUsers{}
 	_ storage.RoomStore = notImplementedRooms{}
+	_ storage.SyncStore = notImplementedSync{}
 )

server/internal/storage/postgres/migrations/0004_syncapi.sql

@@ -0,0 +1,100 @@
+-- The sync read model: the streams /sync reports that are not room events,
+-- and the filters a client uploads.
+--
+-- A room event's position in the event stream is its stream_ordering in
+-- roomserver_events, and nothing here duplicates it. What is here is
+-- everything else a sync response carries, each with a stream of its own:
+-- read receipts, to-device messages, presence and account data all change
+-- without any event being sent, and a client that has seen receipt 7 but
+-- only event 4 has to be able to say so in one token.
+--
+-- Typing notifications are deliberately absent. They expire after seconds,
+-- they are worthless after a restart, and writing them to disk would mean a
+-- row per keystroke-burst per room; they live in memory in the syncapi.
+
+-- One counter row per stream, incremented inside the writing transaction.
+-- Same reasoning as roomserver_sequences: an UPDATE takes a row lock in
+-- both engines and hands out gapless positions, which a sequence does not.
+CREATE TABLE IF NOT EXISTS sync_sequences (
+    name  TEXT   NOT NULL PRIMARY KEY,
+    value BIGINT NOT NULL
+);
+
+INSERT INTO sync_sequences (name, value) VALUES ('receipt', 0)      ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('to_device', 0)    ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('account_data', 0) ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('presence', 0)     ON CONFLICT (name) DO NOTHING;
+
+-- Account data gets a stream position so that an incremental sync can ask
+-- for the blobs that changed rather than re-sending every one of them. The
+-- column goes on the existing table rather than into a parallel one: there
+-- is exactly one current value per (user, room, type), and a second table
+-- would be a second thing to keep in step with it.
+ALTER TABLE user_account_data ADD COLUMN stream_pos BIGINT NOT NULL DEFAULT 0;
+
+CREATE INDEX IF NOT EXISTS user_account_data_stream_idx
+    ON user_account_data (localpart, stream_pos);
+
+-- Read receipts. Only the latest receipt of each (user, type, thread) in a
+-- room is kept: a receipt supersedes the one before it, and /sync reports
+-- the current one rather than a history.
+CREATE TABLE IF NOT EXISTS sync_receipts (
+    room_id      TEXT   NOT NULL,
+    user_id      TEXT   NOT NULL,
+    -- m.read or m.read.private.
+    receipt_type TEXT   NOT NULL,
+    -- The thread the receipt is for, or '' for the room's main timeline.
+    -- Threads are Phase 10; the column is here because the key would
+    -- otherwise have to change when they arrive, and a receipt table is
+    -- not something to migrate twice.
+    thread_id    TEXT   NOT NULL DEFAULT '',
+    event_id     TEXT   NOT NULL,
+    -- When the receipt was sent, which clients show as "read at".
+    ts_ms        BIGINT NOT NULL,
+    stream_pos   BIGINT NOT NULL,
+    PRIMARY KEY (room_id, user_id, receipt_type, thread_id)
+);
+
+CREATE INDEX IF NOT EXISTS sync_receipts_stream_idx
+    ON sync_receipts (stream_pos);
+
+CREATE INDEX IF NOT EXISTS sync_receipts_room_stream_idx
+    ON sync_receipts (room_id, stream_pos);
+
+-- To-device messages: a queue per device, drained by the device
+-- acknowledging a later sync token rather than by reading it.
+CREATE TABLE IF NOT EXISTS sync_to_device (
+    stream_pos BIGINT NOT NULL PRIMARY KEY,
+    localpart  TEXT   NOT NULL,
+    device_id  TEXT   NOT NULL,
+    sender     TEXT   NOT NULL,
+    event_type TEXT   NOT NULL,
+    content    TEXT   NOT NULL,
+    sent_at_ms BIGINT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS sync_to_device_device_idx
+    ON sync_to_device (localpart, device_id, stream_pos);
+
+-- Presence. One row per user, the current state and nothing historical.
+CREATE TABLE IF NOT EXISTS sync_presence (
+    user_id        TEXT   NOT NULL PRIMARY KEY,
+    presence       TEXT   NOT NULL,
+    status_msg     TEXT   NOT NULL DEFAULT '',
+    last_active_ms BIGINT NOT NULL,
+    stream_pos     BIGINT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS sync_presence_stream_idx
+    ON sync_presence (stream_pos);
+
+-- Uploaded filters. The JSON is stored as the client sent it: a filter is
+-- returned verbatim by GET /user/{userID}/filter/{filterID}, and clients
+-- compare what they get back with what they uploaded.
+CREATE TABLE IF NOT EXISTS sync_filters (
+    localpart     TEXT   NOT NULL,
+    filter_id     TEXT   NOT NULL,
+    filter_json   TEXT   NOT NULL,
+    created_at_ms BIGINT NOT NULL,
+    PRIMARY KEY (localpart, filter_id)
+);

server/internal/storage/rooms.go

@@ -310,6 +310,45 @@ type RoomStore interface {
 	// in, ordered by room ID.
 	RoomsForUser(ctx context.Context, userID string, memberships []string) ([]string, error)
 
+	// MembershipsForUser is RoomsForUser with the membership and the
+	// position of the event that set it: one query for everything a sync
+	// needs to know about where a user stands.
+	MembershipsForUser(ctx context.Context, userID string, memberships []string) ([]SyncMembership, error)
+
+	// UserMembershipEvents returns every m.room.member event of one user in
+	// one room, oldest first.
+	//
+	// It is how far back a user's own view of a room reaches: a user who
+	// joined, left and rejoined has several spans, and only their own
+	// membership events say where each one began. There is one row per
+	// membership change of one user in one room, so this stays small
+	// however busy the room is.
+	UserMembershipEvents(ctx context.Context, roomID, userID string) ([]StoredEvent, error)
+
+	// MaxStreamOrdering returns the top of this server's event stream, or 0
+	// for a server that has persisted no events.
+	MaxStreamOrdering(ctx context.Context) (int64, error)
+
+	// Timeline reads one window of a room's event stream. See
+	// TimelineRequest: the bounds are exclusive and the limit is required.
+	Timeline(ctx context.Context, req TimelineRequest) ([]StoredEvent, error)
+
+	// RoomsWithEvents returns the subset of roomIDs that have an event
+	// after from and at or before to.
+	//
+	// It is what keeps an incremental sync down to one query for the common
+	// case of nothing having happened: a user in two hundred rooms would
+	// otherwise need two hundred timeline reads to find that out.
+	RoomsWithEvents(ctx context.Context, roomIDs []string, from, to int64) ([]string, error)
+
+	// StateAtStream returns the room's state as of a stream position: the
+	// state after the last event at or before pos.
+	//
+	// It answers the two questions current state cannot. A user who has
+	// left sees the room as it was when they left, and a sync with a gap in
+	// it has to say which state changed inside that gap.
+	StateAtStream(ctx context.Context, roomID string, pos int64) ([]StateEntry, error)
+
 	// TransactionEvent returns the event a client transaction ID already
 	// produced, or ErrNoSuchEvent when this is the first time it is seen.
 	TransactionEvent(ctx context.Context, ref TransactionRef) (string, error)

server/internal/storage/sqlite/migrations/0004_syncapi.sql

@@ -0,0 +1,100 @@
+-- The sync read model: the streams /sync reports that are not room events,
+-- and the filters a client uploads.
+--
+-- A room event's position in the event stream is its stream_ordering in
+-- roomserver_events, and nothing here duplicates it. What is here is
+-- everything else a sync response carries, each with a stream of its own:
+-- read receipts, to-device messages, presence and account data all change
+-- without any event being sent, and a client that has seen receipt 7 but
+-- only event 4 has to be able to say so in one token.
+--
+-- Typing notifications are deliberately absent. They expire after seconds,
+-- they are worthless after a restart, and writing them to disk would mean a
+-- row per keystroke-burst per room; they live in memory in the syncapi.
+
+-- One counter row per stream, incremented inside the writing transaction.
+-- Same reasoning as roomserver_sequences: an UPDATE takes a row lock in
+-- both engines and hands out gapless positions, which a sequence does not.
+CREATE TABLE IF NOT EXISTS sync_sequences (
+    name  TEXT   NOT NULL PRIMARY KEY,
+    value BIGINT NOT NULL
+);
+
+INSERT INTO sync_sequences (name, value) VALUES ('receipt', 0)      ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('to_device', 0)    ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('account_data', 0) ON CONFLICT (name) DO NOTHING;
+INSERT INTO sync_sequences (name, value) VALUES ('presence', 0)     ON CONFLICT (name) DO NOTHING;
+
+-- Account data gets a stream position so that an incremental sync can ask
+-- for the blobs that changed rather than re-sending every one of them. The
+-- column goes on the existing table rather than into a parallel one: there
+-- is exactly one current value per (user, room, type), and a second table
+-- would be a second thing to keep in step with it.
+ALTER TABLE user_account_data ADD COLUMN stream_pos BIGINT NOT NULL DEFAULT 0;
+
+CREATE INDEX IF NOT EXISTS user_account_data_stream_idx
+    ON user_account_data (localpart, stream_pos);
+
+-- Read receipts. Only the latest receipt of each (user, type, thread) in a
+-- room is kept: a receipt supersedes the one before it, and /sync reports
+-- the current one rather than a history.
+CREATE TABLE IF NOT EXISTS sync_receipts (
+    room_id      TEXT   NOT NULL,
+    user_id      TEXT   NOT NULL,
+    -- m.read or m.read.private.
+    receipt_type TEXT   NOT NULL,
+    -- The thread the receipt is for, or '' for the room's main timeline.
+    -- Threads are Phase 10; the column is here because the key would
+    -- otherwise have to change when they arrive, and a receipt table is
+    -- not something to migrate twice.
+    thread_id    TEXT   NOT NULL DEFAULT '',
+    event_id     TEXT   NOT NULL,
+    -- When the receipt was sent, which clients show as "read at".
+    ts_ms        BIGINT NOT NULL,
+    stream_pos   BIGINT NOT NULL,
+    PRIMARY KEY (room_id, user_id, receipt_type, thread_id)
+);
+
+CREATE INDEX IF NOT EXISTS sync_receipts_stream_idx
+    ON sync_receipts (stream_pos);
+
+CREATE INDEX IF NOT EXISTS sync_receipts_room_stream_idx
+    ON sync_receipts (room_id, stream_pos);
+
+-- To-device messages: a queue per device, drained by the device
+-- acknowledging a later sync token rather than by reading it.
+CREATE TABLE IF NOT EXISTS sync_to_device (
+    stream_pos BIGINT NOT NULL PRIMARY KEY,
+    localpart  TEXT   NOT NULL,
+    device_id  TEXT   NOT NULL,
+    sender     TEXT   NOT NULL,
+    event_type TEXT   NOT NULL,
+    content    TEXT   NOT NULL,
+    sent_at_ms BIGINT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS sync_to_device_device_idx
+    ON sync_to_device (localpart, device_id, stream_pos);
+
+-- Presence. One row per user, the current state and nothing historical.
+CREATE TABLE IF NOT EXISTS sync_presence (
+    user_id        TEXT   NOT NULL PRIMARY KEY,
+    presence       TEXT   NOT NULL,
+    status_msg     TEXT   NOT NULL DEFAULT '',
+    last_active_ms BIGINT NOT NULL,
+    stream_pos     BIGINT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS sync_presence_stream_idx
+    ON sync_presence (stream_pos);
+
+-- Uploaded filters. The JSON is stored as the client sent it: a filter is
+-- returned verbatim by GET /user/{userID}/filter/{filterID}, and clients
+-- compare what they get back with what they uploaded.
+CREATE TABLE IF NOT EXISTS sync_filters (
+    localpart     TEXT   NOT NULL,
+    filter_id     TEXT   NOT NULL,
+    filter_json   TEXT   NOT NULL,
+    created_at_ms BIGINT NOT NULL,
+    PRIMARY KEY (localpart, filter_id)
+);

server/internal/storage/sqlstore/rooms.go

@@ -167,27 +167,10 @@ func (s *Rooms) Write(ctx context.Context, w storage.RoomWrite) (int64, error) {
 	return ordering, nil
 }
 
-const (
-	bumpSequenceSQL   = `UPDATE roomserver_sequences SET value = value + 1 WHERE name = ?`
-	selectSequenceSQL = `SELECT value FROM roomserver_sequences WHERE name = ?`
-)
-
-// nextSequence hands out the next value of a counter.
-//
-// The UPDATE takes a row lock that both engines hold until the transaction
-// ends, so two concurrent writers queue rather than collide, and a value
-// handed out by a transaction that then rolls back is never used — which is
-// what keeps the stream gapless. A database sequence would be faster and
-// would not have that property.
+// nextSequence hands out the next value of one of this component's
+// counters; see sequences.go for why they are rows rather than sequences.
 func (s *Rooms) nextSequence(ctx context.Context, tx *sql.Tx, name string) (int64, error) {
-	if _, err := tx.ExecContext(ctx, s.q(bumpSequenceSQL), name); err != nil {
-		return 0, fmt.Errorf("advance the %s counter: %w", name, err)
-	}
-	var value int64
-	if err := tx.QueryRowContext(ctx, s.q(selectSequenceSQL), name).Scan(&value); err != nil {
-		return 0, fmt.Errorf("read the %s counter: %w", name, err)
-	}
-	return value, nil
+	return nextSequence(ctx, tx, s.q, roomSequences, name)
 }
 
 const (

server/internal/storage/sqlstore/sequences.go

@@ -0,0 +1,63 @@
+package sqlstore
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+)
+
+// The sequence tables. Each component owns its own, so that a component
+// added later does not need a migration to a table another one is already
+// writing.
+const (
+	roomSequences = "roomserver_sequences"
+	syncSequences = "sync_sequences"
+)
+
+// The counters in sync_sequences.
+const (
+	seqReceipt     = "receipt"
+	seqToDevice    = "to_device"
+	seqAccountData = "account_data"
+	seqPresence    = "presence"
+)
+
+// execQuerier is the part of *sql.Tx and *sql.DB that the sequence helpers
+// need, so that a counter can be read outside a transaction and bumped
+// inside one.
+type execQuerier interface {
+	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
+	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
+}
+
+// nextSequence hands out the next value of a counter.
+//
+// The UPDATE takes a row lock that both engines hold until the transaction
+// ends, so two concurrent writers queue rather than collide, and a value
+// handed out by a transaction that then rolls back is never used — which is
+// what keeps a stream gapless. A database sequence would be faster and
+// would not have that property: PostgreSQL's are explicitly not
+// transactional, so a rolled-back write would leave a hole that every
+// reader of the stream would have to be taught to skip.
+//
+// It must be called inside a transaction. Called on a bare *sql.DB the two
+// statements are two transactions, and two callers can then be handed the
+// same value.
+func nextSequence(ctx context.Context, db execQuerier, q func(string) string, table, name string) (int64, error) {
+	bump := fmt.Sprintf(`UPDATE %s SET value = value + 1 WHERE name = ?`, table)
+	if _, err := db.ExecContext(ctx, q(bump), name); err != nil {
+		return 0, fmt.Errorf("advance the %s counter: %w", name, err)
+	}
+	return currentSequence(ctx, db, q, table, name)
+}
+
+// currentSequence reads a counter without advancing it: the top of a stream
+// as a sync token has to name it.
+func currentSequence(ctx context.Context, db execQuerier, q func(string) string, table, name string) (int64, error) {
+	query := fmt.Sprintf(`SELECT value FROM %s WHERE name = ?`, table)
+	var value int64
+	if err := db.QueryRowContext(ctx, q(query), name).Scan(&value); err != nil {
+		return 0, fmt.Errorf("read the %s counter: %w", name, err)
+	}
+	return value, nil
+}

server/internal/storage/sqlstore/sqlstore.go

@@ -58,6 +58,7 @@ type Store struct {
 	settings *Settings
 	users    *Users
 	rooms    *Rooms
+	sync     *Sync
 }
 
 // New builds a Store. It does not touch the database.
@@ -75,6 +76,7 @@ func New(opts Options) (*Store, error) {
 		settings: NewSettings(opts.DB, opts.Settings),
 		users:    NewUsers(opts.DB, opts.Rebind, opts.IsUniqueViolation),
 		rooms:    NewRooms(opts.DB, opts.Rebind, opts.IsUniqueViolation),
+		sync:     NewSync(opts.DB, opts.Rebind, opts.IsUniqueViolation),
 	}, nil
 }
 
@@ -99,6 +101,9 @@ func (s *Store) Users() storage.UserStore { return s.users }
 // Rooms implements storage.Database.
 func (s *Store) Rooms() storage.RoomStore { return s.rooms }
 
+// Sync implements storage.Database.
+func (s *Store) Sync() storage.SyncStore { return s.sync }
+
 // Migrate implements storage.Database.
 func (s *Store) Migrate(ctx context.Context) error {
 	ms, err := s.migrations()

server/internal/storage/sqlstore/sync.go

@@ -0,0 +1,585 @@
+package sqlstore
+
+import (
+	"context"
+	"crypto/rand"
+	"database/sql"
+	"encoding/base64"
+	"errors"
+	"fmt"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// Sync is the shared SQL implementation of storage.SyncStore: the streams a
+// sync response carries that are not room events.
+//
+// Like the other component stores it is written once with `?` placeholders
+// and rebound per driver. Every stream here follows the same shape — a
+// counter row handed out inside the writing transaction, and a stream_pos
+// column on the row it stamps — so that a reader can ask any of them "what
+// changed after position N" with one index lookup.
+type Sync struct {
+	db       *sql.DB
+	rebind   func(string) string
+	isUnique func(error) bool
+}
+
+// NewSync builds a sync store.
+func NewSync(db *sql.DB, rebind func(string) string, isUnique func(error) bool) *Sync {
+	if rebind == nil {
+		rebind = func(query string) string { return query }
+	}
+	if isUnique == nil {
+		isUnique = func(error) bool { return false }
+	}
+	return &Sync{db: db, rebind: rebind, isUnique: isUnique}
+}
+
+// maxToDeviceBatch bounds how many to-device messages one sync carries. A
+// device that has been offline for a week can have a long queue, and the
+// rest of it is still there on the next sync.
+const maxToDeviceBatch = 100
+
+// q rebinds a statement for this driver.
+func (s *Sync) q(query string) string { return s.rebind(query) }
+
+// tx runs fn in a transaction, rolling back on any error.
+func (s *Sync) tx(ctx context.Context, fn func(*sql.Tx) error) error {
+	tx, err := s.db.BeginTx(ctx, nil)
+	if err != nil {
+		return fmt.Errorf("begin transaction: %w", err)
+	}
+	defer func() { _ = tx.Rollback() }()
+
+	if err := fn(tx); err != nil {
+		return err
+	}
+	if err := tx.Commit(); err != nil {
+		return fmt.Errorf("commit transaction: %w", err)
+	}
+	return nil
+}
+
+const selectStreamPositionsSQL = `
+	SELECT name, value FROM sync_sequences`
+
+// StreamPositions implements storage.SyncStore.
+//
+// The PDU position comes from the events table and the rest from the
+// counter rows, and both are read here so that a caller gets one consistent
+// set. The counters are read rather than the rows they stamp because a
+// stream whose latest row was deleted — a drained to-device queue — must
+// not appear to have gone backwards.
+func (s *Sync) StreamPositions(ctx context.Context) (storage.StreamPositions, error) {
+	var out storage.StreamPositions
+
+	err := s.db.QueryRowContext(ctx, s.q(selectMaxStreamOrderingSQL)).Scan(&out.PDU)
+	if err != nil {
+		return storage.StreamPositions{}, fmt.Errorf("read the top of the event stream: %w", err)
+	}
+
+	rows, err := s.db.QueryContext(ctx, s.q(selectStreamPositionsSQL))
+	if err != nil {
+		return storage.StreamPositions{}, fmt.Errorf("read the stream positions: %w", err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	for rows.Next() {
+		var (
+			name  string
+			value int64
+		)
+		if err := rows.Scan(&name, &value); err != nil {
+			return storage.StreamPositions{}, fmt.Errorf("read the stream positions: %w", err)
+		}
+		switch name {
+		case seqReceipt:
+			out.Receipt = value
+		case seqToDevice:
+			out.ToDevice = value
+		case seqAccountData:
+			out.AccountData = value
+		case seqPresence:
+			out.Presence = value
+		}
+	}
+	if err := rows.Err(); err != nil {
+		return storage.StreamPositions{}, fmt.Errorf("read the stream positions: %w", err)
+	}
+	return out, nil
+}
+
+const (
+	selectReceiptPositionSQL = `
+		SELECT e.stream_ordering FROM roomserver_events e
+		WHERE e.room_id = ? AND e.event_id = ?`
+
+	selectCurrentReceiptSQL = `
+		SELECT r.event_id, e.stream_ordering, r.stream_pos
+		FROM sync_receipts r
+		LEFT JOIN roomserver_events e ON e.event_id = r.event_id
+		WHERE r.room_id = ? AND r.user_id = ? AND r.receipt_type = ? AND r.thread_id = ?`
+
+	upsertReceiptSQL = `
+		INSERT INTO sync_receipts
+			(room_id, user_id, receipt_type, thread_id, event_id, ts_ms, stream_pos)
+		VALUES (?, ?, ?, ?, ?, ?, ?)
+		ON CONFLICT (room_id, user_id, receipt_type, thread_id) DO UPDATE SET
+			event_id = excluded.event_id,
+			ts_ms = excluded.ts_ms,
+			stream_pos = excluded.stream_pos`
+)
+
+// SetReceipt implements storage.SyncStore.
+//
+// A receipt that would move backwards is dropped. Two clients of one user
+// read a room at different speeds, and letting the slower one win would
+// make the unread marker jump backwards on the other — so the comparison is
+// on the position of the event being acknowledged, not on when the receipt
+// arrived.
+func (s *Sync) SetReceipt(ctx context.Context, r storage.Receipt) (int64, bool, error) {
+	var (
+		pos   int64
+		moved bool
+	)
+	err := s.tx(ctx, func(tx *sql.Tx) error {
+		var target int64
+		err := tx.QueryRowContext(ctx, s.q(selectReceiptPositionSQL), r.RoomID, r.EventID).Scan(&target)
+		switch {
+		case errors.Is(err, sql.ErrNoRows):
+			return fmt.Errorf("event %q: %w", r.EventID, storage.ErrNoSuchEvent)
+		case err != nil:
+			return fmt.Errorf("locate the event a receipt names: %w", err)
+		}
+
+		var (
+			haveEventID string
+			havePos     sql.NullInt64
+			haveStream  int64
+		)
+		err = tx.QueryRowContext(ctx, s.q(selectCurrentReceiptSQL),
+			r.RoomID, r.UserID, r.ReceiptType, r.ThreadID).Scan(&haveEventID, &havePos, &haveStream)
+		switch {
+		case err == nil:
+			// A receipt whose event has since been purged has no position
+			// to compare against, and the new one supersedes it.
+			if havePos.Valid && havePos.Int64 >= target {
+				pos, moved = haveStream, false
+				return nil
+			}
+		case !errors.Is(err, sql.ErrNoRows):
+			return fmt.Errorf("read the current receipt: %w", err)
+		}
+
+		if pos, err = nextSequence(ctx, tx, s.q, syncSequences, seqReceipt); err != nil {
+			return err
+		}
+		if _, err := tx.ExecContext(ctx, s.q(upsertReceiptSQL),
+			r.RoomID, r.UserID, r.ReceiptType, r.ThreadID, r.EventID, r.TimestampMS, pos); err != nil {
+			return fmt.Errorf("write the receipt: %w", err)
+		}
+		moved = true
+		return nil
+	})
+	if err != nil {
+		return 0, false, fmt.Errorf("set receipt in %q: %w", r.RoomID, err)
+	}
+	return pos, moved, nil
+}
+
+const receiptColumns = `
+	room_id, user_id, receipt_type, thread_id, event_id, ts_ms, stream_pos`
+
+func scanReceipts(rows *sql.Rows) ([]storage.Receipt, error) {
+	defer func() { _ = rows.Close() }()
+
+	var out []storage.Receipt
+	for rows.Next() {
+		var r storage.Receipt
+		if err := rows.Scan(&r.RoomID, &r.UserID, &r.ReceiptType, &r.ThreadID,
+			&r.EventID, &r.TimestampMS, &r.StreamPos); err != nil {
+			return nil, err
+		}
+		out = append(out, r)
+	}
+	return out, rows.Err()
+}
+
+// Receipts implements storage.SyncStore.
+func (s *Sync) Receipts(ctx context.Context, roomIDs []string, from, to int64) ([]storage.Receipt, error) {
+	if len(roomIDs) == 0 {
+		return nil, nil
+	}
+
+	query := `SELECT ` + receiptColumns + ` FROM sync_receipts
+		WHERE room_id IN (` + placeholders(len(roomIDs)) + `) AND stream_pos > ?`
+	args := make([]any, 0, len(roomIDs)+2)
+	for _, id := range roomIDs {
+		args = append(args, id)
+	}
+	args = append(args, from)
+	if to > 0 {
+		query += ` AND stream_pos <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY stream_pos`
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read receipts: %w", err)
+	}
+	receipts, err := scanReceipts(rows)
+	if err != nil {
+		return nil, fmt.Errorf("read receipts: %w", err)
+	}
+	return receipts, nil
+}
+
+// ReceiptsForUser implements storage.SyncStore.
+func (s *Sync) ReceiptsForUser(ctx context.Context, userID string, roomIDs []string) ([]storage.Receipt, error) {
+	if len(roomIDs) == 0 {
+		return nil, nil
+	}
+
+	query := `SELECT ` + receiptColumns + ` FROM sync_receipts
+		WHERE user_id = ? AND room_id IN (` + placeholders(len(roomIDs)) + `)
+		ORDER BY stream_pos`
+	args := make([]any, 0, len(roomIDs)+1)
+	args = append(args, userID)
+	for _, id := range roomIDs {
+		args = append(args, id)
+	}
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read the receipts of %q: %w", userID, err)
+	}
+	receipts, err := scanReceipts(rows)
+	if err != nil {
+		return nil, fmt.Errorf("read the receipts of %q: %w", userID, err)
+	}
+	return receipts, nil
+}
+
+const insertToDeviceSQL = `
+	INSERT INTO sync_to_device
+		(stream_pos, localpart, device_id, sender, event_type, content, sent_at_ms)
+	VALUES (?, ?, ?, ?, ?, ?, ?)`
+
+// SendToDevice implements storage.SyncStore.
+//
+// The whole batch shares one transaction but takes a position each: a
+// message to two of a user's devices is two independent deliveries, and one
+// device acknowledging its copy must not drop the other's.
+func (s *Sync) SendToDevice(ctx context.Context, msgs []storage.NewToDeviceMessage) (int64, error) {
+	var top int64
+	err := s.tx(ctx, func(tx *sql.Tx) error {
+		if len(msgs) == 0 {
+			var err error
+			top, err = currentSequence(ctx, tx, s.q, syncSequences, seqToDevice)
+			return err
+		}
+		for _, m := range msgs {
+			pos, err := nextSequence(ctx, tx, s.q, syncSequences, seqToDevice)
+			if err != nil {
+				return err
+			}
+			if _, err := tx.ExecContext(ctx, s.q(insertToDeviceSQL),
+				pos, m.Localpart, m.DeviceID, m.Sender, m.Type, string(m.Content), m.SentAtMS); err != nil {
+				return fmt.Errorf("queue a to-device message for %s/%s: %w", m.Localpart, m.DeviceID, err)
+			}
+			top = pos
+		}
+		return nil
+	})
+	if err != nil {
+		return 0, fmt.Errorf("send to-device messages: %w", err)
+	}
+	return top, nil
+}
+
+const selectToDeviceSQL = `
+	SELECT stream_pos, localpart, device_id, sender, event_type, content, sent_at_ms
+	FROM sync_to_device
+	WHERE localpart = ? AND device_id = ? AND stream_pos > ?`
+
+// ToDeviceMessages implements storage.SyncStore.
+func (s *Sync) ToDeviceMessages(ctx context.Context, localpart, deviceID string, from, to int64, limit int) ([]storage.ToDeviceMessage, error) {
+	if limit <= 0 || limit > maxToDeviceBatch {
+		limit = maxToDeviceBatch
+	}
+
+	query := selectToDeviceSQL
+	args := []any{localpart, deviceID, from}
+	if to > 0 {
+		query += ` AND stream_pos <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY stream_pos LIMIT ?`
+	args = append(args, limit)
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read to-device messages for %s/%s: %w", localpart, deviceID, err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	var out []storage.ToDeviceMessage
+	for rows.Next() {
+		var (
+			m       storage.ToDeviceMessage
+			content string
+		)
+		if err := rows.Scan(&m.StreamPos, &m.Localpart, &m.DeviceID,
+			&m.Sender, &m.Type, &content, &m.SentAtMS); err != nil {
+			return nil, fmt.Errorf("read to-device messages for %s/%s: %w", localpart, deviceID, err)
+		}
+		m.Content = []byte(content)
+		out = append(out, m)
+	}
+	return out, rows.Err()
+}
+
+const deleteToDeviceSQL = `
+	DELETE FROM sync_to_device
+	WHERE localpart = ? AND device_id = ? AND stream_pos <= ?`
+
+// DeleteToDeviceMessages implements storage.SyncStore.
+func (s *Sync) DeleteToDeviceMessages(ctx context.Context, localpart, deviceID string, pos int64) error {
+	if pos <= 0 {
+		return nil
+	}
+	if _, err := s.db.ExecContext(ctx, s.q(deleteToDeviceSQL), localpart, deviceID, pos); err != nil {
+		return fmt.Errorf("delete to-device messages for %s/%s: %w", localpart, deviceID, err)
+	}
+	return nil
+}
+
+const (
+	upsertPresenceSQL = `
+		INSERT INTO sync_presence (user_id, presence, status_msg, last_active_ms, stream_pos)
+		VALUES (?, ?, ?, ?, ?)
+		ON CONFLICT (user_id) DO UPDATE SET
+			presence = excluded.presence,
+			status_msg = excluded.status_msg,
+			last_active_ms = excluded.last_active_ms,
+			stream_pos = excluded.stream_pos`
+
+	touchPresenceSQL = `
+		UPDATE sync_presence SET last_active_ms = ? WHERE user_id = ?`
+)
+
+// SetPresence implements storage.SyncStore.
+//
+// A write that changes nothing a peer would notice refreshes the
+// last-active timestamp in place and leaves the stream where it was. Every
+// /sync is such a write — a client says it is still online by syncing — and
+// moving the stream each time would wake every peer's long poll, which
+// would sync, which would move it again.
+func (s *Sync) SetPresence(ctx context.Context, p storage.PresenceState) (int64, bool, error) {
+	var (
+		pos   int64
+		moved bool
+	)
+	err := s.tx(ctx, func(tx *sql.Tx) error {
+		var existing storage.PresenceState
+		err := tx.QueryRowContext(ctx, s.q(selectPresenceSQL), p.UserID).Scan(
+			&existing.UserID, &existing.Presence, &existing.StatusMsg,
+			&existing.LastActiveMS, &existing.StreamPos)
+		switch {
+		case err == nil:
+			if existing.Presence == p.Presence && existing.StatusMsg == p.StatusMsg {
+				pos = existing.StreamPos
+				if _, err := tx.ExecContext(ctx, s.q(touchPresenceSQL),
+					p.LastActiveMS, p.UserID); err != nil {
+					return fmt.Errorf("refresh the last-active timestamp: %w", err)
+				}
+				return nil
+			}
+		case !errors.Is(err, sql.ErrNoRows):
+			return fmt.Errorf("read the current presence: %w", err)
+		}
+
+		if pos, err = nextSequence(ctx, tx, s.q, syncSequences, seqPresence); err != nil {
+			return err
+		}
+		if _, err := tx.ExecContext(ctx, s.q(upsertPresenceSQL),
+			p.UserID, p.Presence, p.StatusMsg, p.LastActiveMS, pos); err != nil {
+			return fmt.Errorf("write presence: %w", err)
+		}
+		moved = true
+		return nil
+	})
+	if err != nil {
+		return 0, false, fmt.Errorf("set the presence of %q: %w", p.UserID, err)
+	}
+	return pos, moved, nil
+}
+
+const presenceColumns = `user_id, presence, status_msg, last_active_ms, stream_pos`
+
+const selectPresenceSQL = `
+	SELECT ` + presenceColumns + ` FROM sync_presence WHERE user_id = ?`
+
+// Presence implements storage.SyncStore.
+func (s *Sync) Presence(ctx context.Context, userID string) (storage.PresenceState, error) {
+	var p storage.PresenceState
+	err := s.db.QueryRowContext(ctx, s.q(selectPresenceSQL), userID).
+		Scan(&p.UserID, &p.Presence, &p.StatusMsg, &p.LastActiveMS, &p.StreamPos)
+	switch {
+	case errors.Is(err, sql.ErrNoRows):
+		return storage.PresenceState{}, fmt.Errorf("presence of %q: %w", userID, storage.ErrNoSuchUser)
+	case err != nil:
+		return storage.PresenceState{}, fmt.Errorf("read the presence of %q: %w", userID, err)
+	}
+	return p, nil
+}
+
+// PresenceForUsers implements storage.SyncStore.
+func (s *Sync) PresenceForUsers(ctx context.Context, userIDs []string, from, to int64) ([]storage.PresenceState, error) {
+	if len(userIDs) == 0 {
+		return nil, nil
+	}
+
+	query := `SELECT ` + presenceColumns + ` FROM sync_presence
+		WHERE user_id IN (` + placeholders(len(userIDs)) + `) AND stream_pos > ?`
+	args := make([]any, 0, len(userIDs)+2)
+	for _, id := range userIDs {
+		args = append(args, id)
+	}
+	args = append(args, from)
+	if to > 0 {
+		query += ` AND stream_pos <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY stream_pos`
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read presence: %w", err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	var out []storage.PresenceState
+	for rows.Next() {
+		var p storage.PresenceState
+		if err := rows.Scan(&p.UserID, &p.Presence, &p.StatusMsg, &p.LastActiveMS, &p.StreamPos); err != nil {
+			return nil, fmt.Errorf("read presence: %w", err)
+		}
+		out = append(out, p)
+	}
+	return out, rows.Err()
+}
+
+const selectAccountDataChangesSQL = `
+	SELECT room_id, data_type, stream_pos FROM user_account_data
+	WHERE localpart = ? AND stream_pos > ?`
+
+// AccountDataChanges implements storage.SyncStore.
+func (s *Sync) AccountDataChanges(ctx context.Context, localpart string, from, to int64) ([]storage.AccountDataEntry, error) {
+	query := selectAccountDataChangesSQL
+	args := []any{localpart, from}
+	if to > 0 {
+		query += ` AND stream_pos <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY stream_pos`
+
+	return s.accountData(ctx, localpart, query, args)
+}
+
+const selectAllAccountDataSQL = `
+	SELECT room_id, data_type, stream_pos FROM user_account_data
+	WHERE localpart = ?`
+
+// AccountDataAt implements storage.SyncStore.
+func (s *Sync) AccountDataAt(ctx context.Context, localpart string, to int64) ([]storage.AccountDataEntry, error) {
+	query := selectAllAccountDataSQL
+	args := []any{localpart}
+	if to > 0 {
+		query += ` AND stream_pos <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY room_id, data_type`
+
+	return s.accountData(ctx, localpart, query, args)
+}
+
+func (s *Sync) accountData(ctx context.Context, localpart, query string, args []any) ([]storage.AccountDataEntry, error) {
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read the account data of %q: %w", localpart, err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	var out []storage.AccountDataEntry
+	for rows.Next() {
+		var e storage.AccountDataEntry
+		if err := rows.Scan(&e.RoomID, &e.DataType, &e.StreamPos); err != nil {
+			return nil, fmt.Errorf("read the account data of %q: %w", localpart, err)
+		}
+		out = append(out, e)
+	}
+	return out, rows.Err()
+}
+
+const insertFilterSQL = `
+	INSERT INTO sync_filters (localpart, filter_id, filter_json, created_at_ms)
+	VALUES (?, ?, ?, ?)`
+
+// SaveFilter implements storage.SyncStore.
+//
+// The ID is random rather than a counter. The spec says only that it is
+// opaque, and a counter would be a per-user one: two users would both have
+// a filter "1", so a filter ID leaking into a URL or a log would say which
+// user it belonged to only by accident of context.
+func (s *Sync) SaveFilter(ctx context.Context, localpart string, filter []byte, createdAtMS int64) (string, error) {
+	for attempt := range 3 {
+		filterID, err := newFilterID()
+		if err != nil {
+			return "", err
+		}
+		_, err = s.db.ExecContext(ctx, s.q(insertFilterSQL),
+			localpart, filterID, string(filter), createdAtMS)
+		switch {
+		case err == nil:
+			return filterID, nil
+		case s.isUnique(err) && attempt < 2:
+			// 96 bits of randomness collided, or the caller's own ID did.
+			// Either way the answer is another ID.
+			continue
+		default:
+			return "", fmt.Errorf("save a filter for %q: %w", localpart, err)
+		}
+	}
+	return "", fmt.Errorf("save a filter for %q: could not find an unused filter ID", localpart)
+}
+
+// newFilterID mints a filter ID: 12 random bytes, URL-safe base64.
+func newFilterID() (string, error) {
+	var raw [12]byte
+	if _, err := rand.Read(raw[:]); err != nil {
+		return "", fmt.Errorf("generate a filter ID: %w", err)
+	}
+	return base64.RawURLEncoding.EncodeToString(raw[:]), nil
+}
+
+const selectFilterSQL = `
+	SELECT filter_json FROM sync_filters WHERE localpart = ? AND filter_id = ?`
+
+// Filter implements storage.SyncStore.
+func (s *Sync) Filter(ctx context.Context, localpart, filterID string) ([]byte, error) {
+	var body string
+	err := s.db.QueryRowContext(ctx, s.q(selectFilterSQL), localpart, filterID).Scan(&body)
+	switch {
+	case errors.Is(err, sql.ErrNoRows):
+		return nil, fmt.Errorf("filter %q: %w", filterID, storage.ErrNoSuchFilter)
+	case err != nil:
+		return nil, fmt.Errorf("read filter %q: %w", filterID, err)
+	}
+	return []byte(body), nil
+}
+
+// Compile-time proof that the shared implementation satisfies the interface.
+var _ storage.SyncStore = (*Sync)(nil)

server/internal/storage/sqlstore/timeline.go

@@ -0,0 +1,331 @@
+package sqlstore
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"strings"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// The stream reads of the room store: the queries the sync API and
+// /messages are built on.
+//
+// They live apart from the rest of the room store because they are the only
+// queries in it that are about *order* rather than about identity. Every
+// other read answers "what is the state of this room" or "what is this
+// event"; these answer "what happened between here and there", which is a
+// different access pattern and the one the (room_id, stream_ordering) index
+// exists for.
+
+const selectMaxStreamOrderingSQL = `
+	SELECT COALESCE(MAX(stream_ordering), 0) FROM roomserver_events`
+
+// MaxStreamOrdering implements storage.RoomStore.
+//
+// It reads the events table rather than the counter in
+// roomserver_sequences. The counter is advanced inside the writing
+// transaction and is therefore already at the new value while that
+// transaction is still open; a sync token taken from it would name a
+// position whose event no reader can see yet, and the event would be
+// skipped for good once the transaction committed.
+func (s *Rooms) MaxStreamOrdering(ctx context.Context) (int64, error) {
+	var top int64
+	if err := s.db.QueryRowContext(ctx, s.q(selectMaxStreamOrderingSQL)).Scan(&top); err != nil {
+		return 0, fmt.Errorf("read the top of the event stream: %w", err)
+	}
+	return top, nil
+}
+
+// MembershipsForUser implements storage.RoomStore.
+func (s *Rooms) MembershipsForUser(ctx context.Context, userID string, memberships []string) ([]storage.SyncMembership, error) {
+	query := `
+		SELECT cs.room_id, cs.membership, cs.event_id, e.stream_ordering
+		FROM roomserver_current_state cs
+		JOIN roomserver_events e ON e.event_id = cs.event_id
+		WHERE cs.event_type = 'm.room.member' AND cs.state_key = ?`
+	args := []any{userID}
+	if len(memberships) > 0 {
+		query += ` AND cs.membership IN (` + placeholders(len(memberships)) + `)`
+		for _, m := range memberships {
+			args = append(args, m)
+		}
+	}
+	query += ` ORDER BY cs.room_id`
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read the memberships of %q: %w", userID, err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	var out []storage.SyncMembership
+	for rows.Next() {
+		var m storage.SyncMembership
+		if err := rows.Scan(&m.RoomID, &m.Membership, &m.EventID, &m.StreamOrdering); err != nil {
+			return nil, fmt.Errorf("read the memberships of %q: %w", userID, err)
+		}
+		out = append(out, m)
+	}
+	return out, rows.Err()
+}
+
+var selectUserMembershipEventsSQL = `
+	SELECT ` + eventColumns + `
+	FROM roomserver_events e
+	WHERE e.room_id = ? AND e.event_type = 'm.room.member' AND e.state_key = ?
+	ORDER BY e.stream_ordering`
+
+// UserMembershipEvents implements storage.RoomStore.
+func (s *Rooms) UserMembershipEvents(ctx context.Context, roomID, userID string) ([]storage.StoredEvent, error) {
+	rows, err := s.db.QueryContext(ctx, s.q(selectUserMembershipEventsSQL), roomID, userID)
+	if err != nil {
+		return nil, fmt.Errorf("read the membership events of %q in %q: %w", userID, roomID, err)
+	}
+	events, err := scanEvents(rows)
+	if err != nil {
+		return nil, fmt.Errorf("read the membership events of %q in %q: %w", userID, roomID, err)
+	}
+	return events, nil
+}
+
+// Timeline implements storage.RoomStore.
+func (s *Rooms) Timeline(ctx context.Context, req storage.TimelineRequest) ([]storage.StoredEvent, error) {
+	if req.Limit <= 0 {
+		return nil, nil
+	}
+
+	conditions := []string{`e.room_id = ?`}
+	args := []any{req.RoomID}
+	if req.Low > 0 {
+		conditions = append(conditions, `e.stream_ordering > ?`)
+		args = append(args, req.Low)
+	}
+	if req.High > 0 {
+		conditions = append(conditions, `e.stream_ordering <= ?`)
+		args = append(args, req.High)
+	}
+
+	// The order decides which end the LIMIT trims: reading backwards keeps
+	// the newest of the window, which is what a timeline is.
+	order := `ASC`
+	if req.Direction == storage.DirectionBackward {
+		order = `DESC`
+	}
+
+	if clause, filterArgs := filterConditions(req.Filter); clause != "" {
+		conditions = append(conditions, clause)
+		args = append(args, filterArgs...)
+	}
+	args = append(args, req.Limit)
+
+	query := `SELECT ` + eventColumns + `
+		FROM roomserver_events e
+		WHERE ` + strings.Join(conditions, " AND ") + `
+		ORDER BY e.stream_ordering ` + order + `
+		LIMIT ?`
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("read the timeline of %q: %w", req.RoomID, err)
+	}
+	events, err := scanEvents(rows)
+	if err != nil {
+		return nil, fmt.Errorf("read the timeline of %q: %w", req.RoomID, err)
+	}
+	return events, nil
+}
+
+// filterConditions renders the part of an event filter the store applies in
+// SQL, as a WHERE fragment and its arguments.
+//
+// The exclusions are ANDed in after the inclusions, which is what makes
+// not_types win over types for an event matching both — the behaviour the
+// spec requires.
+func filterConditions(f storage.TimelineFilter) (string, []any) {
+	var (
+		clauses []string
+		args    []any
+	)
+	add := func(column string, patterns []string, negate bool) {
+		if len(patterns) == 0 {
+			return
+		}
+		clause, clauseArgs := patternMatch(column, patterns)
+		if negate {
+			clause = `NOT (` + clause + `)`
+		}
+		clauses = append(clauses, clause)
+		args = append(args, clauseArgs...)
+	}
+	add(`e.event_type`, f.Types, false)
+	add(`e.event_type`, f.NotTypes, true)
+	add(`e.sender`, f.Senders, false)
+	add(`e.sender`, f.NotSenders, true)
+
+	if len(clauses) == 0 {
+		return "", nil
+	}
+	return strings.Join(clauses, " AND "), args
+}
+
+// patternMatch renders "this column matches any of these patterns".
+//
+// A filter pattern may end in '*', which the spec defines as matching any
+// suffix — "m.room.*" is how a client asks for every room event type. Exact
+// patterns go into one IN list so that the common case is a single indexed
+// lookup; a wildcard becomes a LIKE with an explicit escape character, so
+// that a '%' or '_' inside the pattern is matched literally rather than
+// being read as a LIKE wildcard of its own.
+func patternMatch(column string, patterns []string) (string, []any) {
+	var (
+		exact    []string
+		wildcard []string
+		args     []any
+	)
+	for _, p := range patterns {
+		if prefix, isWildcard := strings.CutSuffix(p, "*"); isWildcard {
+			wildcard = append(wildcard, prefix)
+			continue
+		}
+		exact = append(exact, p)
+	}
+
+	var parts []string
+	if len(exact) > 0 {
+		parts = append(parts, column+` IN (`+placeholders(len(exact))+`)`)
+		for _, p := range exact {
+			args = append(args, p)
+		}
+	}
+	for _, prefix := range wildcard {
+		parts = append(parts, column+` LIKE ? ESCAPE '\'`)
+		args = append(args, escapeLike(prefix)+`%`)
+	}
+	return strings.Join(parts, " OR "), args
+}
+
+// escapeLike makes a literal string safe to embed in a LIKE pattern that
+// declares '\' as its escape character.
+func escapeLike(s string) string {
+	var b strings.Builder
+	b.Grow(len(s))
+	for _, r := range s {
+		if r == '\\' || r == '%' || r == '_' {
+			b.WriteRune('\\')
+		}
+		b.WriteRune(r)
+	}
+	return b.String()
+}
+
+// RoomsWithEvents implements storage.RoomStore.
+func (s *Rooms) RoomsWithEvents(ctx context.Context, roomIDs []string, from, to int64) ([]string, error) {
+	if len(roomIDs) == 0 {
+		return nil, nil
+	}
+
+	query := `
+		SELECT DISTINCT room_id FROM roomserver_events
+		WHERE room_id IN (` + placeholders(len(roomIDs)) + `) AND stream_ordering > ?`
+	args := make([]any, 0, len(roomIDs)+2)
+	for _, id := range roomIDs {
+		args = append(args, id)
+	}
+	args = append(args, from)
+	if to > 0 {
+		query += ` AND stream_ordering <= ?`
+		args = append(args, to)
+	}
+	query += ` ORDER BY room_id`
+
+	rows, err := s.db.QueryContext(ctx, s.q(query), args...)
+	if err != nil {
+		return nil, fmt.Errorf("find the rooms with events after %d: %w", from, err)
+	}
+	defer func() { _ = rows.Close() }()
+
+	var out []string
+	for rows.Next() {
+		var roomID string
+		if err := rows.Scan(&roomID); err != nil {
+			return nil, fmt.Errorf("find the rooms with events after %d: %w", from, err)
+		}
+		out = append(out, roomID)
+	}
+	return out, rows.Err()
+}
+
+const selectEventAtStreamSQL = `
+	SELECT e.event_id, e.event_type, e.state_key, e.state_before_id
+	FROM roomserver_events e
+	WHERE e.room_id = ? AND e.stream_ordering <= ?
+	ORDER BY e.stream_ordering DESC
+	LIMIT 1`
+
+const selectSnapshotEntriesSQL = `
+	SELECT se.event_type, se.state_key, se.event_id, ''
+	FROM roomserver_state_entries se
+	WHERE se.snapshot_id = ?
+	ORDER BY se.event_type, se.state_key`
+
+// StateAtStream implements storage.RoomStore.
+//
+// The state after a position is the snapshot recorded against the last
+// event at or before it, plus that event itself when it is a state event:
+// the snapshot is the state the event was authorised against, which is the
+// state *before* it.
+func (s *Rooms) StateAtStream(ctx context.Context, roomID string, pos int64) ([]storage.StateEntry, error) {
+	if pos <= 0 {
+		return nil, nil
+	}
+
+	var (
+		eventID    string
+		eventType  string
+		stateKey   sql.NullString
+		snapshotID int64
+	)
+	err := s.db.QueryRowContext(ctx, s.q(selectEventAtStreamSQL), roomID, pos).
+		Scan(&eventID, &eventType, &stateKey, &snapshotID)
+	switch {
+	case errors.Is(err, sql.ErrNoRows):
+		// Nothing had happened in the room by then, which for a position
+		// before its create event is the honest answer.
+		return nil, nil
+	case err != nil:
+		return nil, fmt.Errorf("read the state of %q at %d: %w", roomID, pos, err)
+	}
+
+	var entries []storage.StateEntry
+	if snapshotID != 0 {
+		rows, qErr := s.db.QueryContext(ctx, s.q(selectSnapshotEntriesSQL), snapshotID)
+		if qErr != nil {
+			return nil, fmt.Errorf("read the state of %q at %d: %w", roomID, pos, qErr)
+		}
+		if entries, err = scanStateEntries(rows); err != nil {
+			return nil, fmt.Errorf("read the state of %q at %d: %w", roomID, pos, err)
+		}
+	}
+	if stateKey.Valid {
+		entries = upsertStateEntry(entries, storage.StateEntry{
+			Type:     eventType,
+			StateKey: stateKey.String,
+			EventID:  eventID,
+		})
+	}
+	return entries, nil
+}
+
+// upsertStateEntry replaces the matching tuple in entries, or appends it.
+func upsertStateEntry(entries []storage.StateEntry, entry storage.StateEntry) []storage.StateEntry {
+	for i, existing := range entries {
+		if existing.Type == entry.Type && existing.StateKey == entry.StateKey {
+			entries[i] = entry
+			return entries
+		}
+	}
+	return append(entries, entry)
+}

server/internal/storage/sqlstore/users.go

@@ -610,10 +610,10 @@ const (
 		WHERE localpart = ? AND room_id = ? AND data_type = ?`
 
 	upsertAccountDataSQL = `
-		INSERT INTO user_account_data (localpart, room_id, data_type, content)
-		VALUES (?, ?, ?, ?)
+		INSERT INTO user_account_data (localpart, room_id, data_type, content, stream_pos)
+		VALUES (?, ?, ?, ?, ?)
 		ON CONFLICT (localpart, room_id, data_type)
-		DO UPDATE SET content = excluded.content`
+		DO UPDATE SET content = excluded.content, stream_pos = excluded.stream_pos`
 )
 
 // AccountData implements storage.UserStore.
@@ -632,9 +632,26 @@ func (u *Users) AccountData(ctx context.Context, localpart, roomID, dataType str
 // SetAccountData implements storage.UserStore. The content is stored as text
 // rather than a JSON column type: the two engines disagree on what JSON
 // normalisation does, and account data has to come back out byte for byte.
+//
+// The write also stamps the row with a position in the account data stream,
+// taken inside the same transaction as the write itself. Account data is
+// one of the streams /sync reports, and a position assigned separately
+// would either be visible before the content it describes or after it — in
+// the second case a client would be told the stream had moved and then find
+// the old value.
 func (u *Users) SetAccountData(ctx context.Context, localpart, roomID, dataType string, content []byte) error {
-	if _, err := u.db.ExecContext(ctx, u.q(upsertAccountDataSQL),
-		localpart, roomID, dataType, string(content)); err != nil {
+	err := u.tx(ctx, func(tx *sql.Tx) error {
+		pos, err := nextSequence(ctx, tx, u.q, syncSequences, seqAccountData)
+		if err != nil {
+			return err
+		}
+		if _, err := tx.ExecContext(ctx, u.q(upsertAccountDataSQL),
+			localpart, roomID, dataType, string(content), pos); err != nil {
+			return err
+		}
+		return nil
+	})
+	if err != nil {
 		return fmt.Errorf("set account data %q: %w", dataType, err)
 	}
 	return nil

server/internal/storage/storage.go

@@ -67,6 +67,10 @@ type Database interface {
 	// Rooms is the roomserver store: rooms, events, room state, aliases.
 	Rooms() RoomStore
 
+	// Sync is the syncapi store: receipts, to-device messages, presence and
+	// uploaded filters.
+	Sync() SyncStore
+
 	// Close releases the connection pool.
 	Close() error
 }

server/internal/storage/sync.go

@@ -0,0 +1,296 @@
+package storage
+
+import (
+	"context"
+	"errors"
+)
+
+// Errors the sync store returns for cases a caller has to tell apart.
+var (
+	// ErrNoSuchFilter is returned for a filter ID the user has not
+	// uploaded.
+	ErrNoSuchFilter = errors.New("no such filter")
+)
+
+// StreamPositions is the top of every stream a sync token points into, read
+// in one call.
+//
+// A sync response is assembled from several independent streams, and the
+// token it returns has to name a position in each. Reading them one at a
+// time would let a stream move between two of the reads, and the client
+// would then be handed a token that skips whatever landed in the gap.
+type StreamPositions struct {
+	// PDU is the room event stream: the highest stream ordering in
+	// roomserver_events.
+	PDU int64
+
+	Receipt     int64
+	ToDevice    int64
+	AccountData int64
+	Presence    int64
+}
+
+// TimelineDirection is which way a timeline read walks the stream.
+type TimelineDirection string
+
+// The two directions, spelled as the Client-Server API spells them in
+// /messages' `dir` parameter.
+const (
+	// DirectionForward reads oldest first, which is the order /sync hands
+	// events to a client.
+	DirectionForward TimelineDirection = "f"
+
+	// DirectionBackward reads newest first, which is how /messages pages
+	// back through a room's history.
+	DirectionBackward TimelineDirection = "b"
+)
+
+// TimelineRequest is one window of a room's event stream.
+//
+// The window is named by its two ends rather than by a direction and a
+// starting point, because the two are independent: which end the limit
+// trims, and which order the rows come back in, is Direction's business,
+// and nothing else about the request changes with it. Spelling the bounds
+// as "from" and "to" instead would mean each of them being the low bound in
+// one direction and the high bound in the other, which is exactly the
+// off-by-one every pagination bug is made of.
+//
+// Low is exclusive and High inclusive, in both directions. That is what
+// lets a token a client was handed be passed straight back without
+// re-reading the event it names: a position is somewhere a client has got
+// to, not an event it still wants.
+type TimelineRequest struct {
+	RoomID string
+
+	// Low is an exclusive lower bound. Zero means the start of the room.
+	Low int64
+
+	// High is an inclusive upper bound. Zero means the top of the stream,
+	// and goes on meaning that as the stream grows.
+	High int64
+
+	// Direction decides which end of the window the limit trims — the
+	// bottom when reading backwards, the top when reading forwards — and
+	// the order the rows come back in.
+	Direction TimelineDirection
+
+	// Limit caps the rows returned. It is required: an unbounded timeline
+	// read is a way to ask the server to serialise a whole room.
+	Limit int
+
+	// Filter narrows which events come back. The store applies it in SQL
+	// because a filter applied after the limit would return fewer events
+	// than the client asked for and then look like the end of the room.
+	Filter TimelineFilter
+}
+
+// TimelineFilter is the part of a Client-Server API event filter that the
+// store can apply itself.
+//
+// Only the fields that are a plain comparison on an indexed or stored
+// column are here. The rest of a filter — `contains_url`, lazy-loading,
+// `include_redundant_members` — is about what an event means or about how a
+// response is assembled, and is applied by the syncapi.
+type TimelineFilter struct {
+	// Types, when non-empty, admits only these event types. A trailing '*'
+	// is a wildcard, as the spec defines for filter patterns.
+	Types []string
+
+	// NotTypes excludes types, and wins over Types for an event matching
+	// both, which is what the spec requires.
+	NotTypes []string
+
+	// Senders, when non-empty, admits only these senders.
+	Senders []string
+
+	// NotSenders excludes senders, and likewise wins.
+	NotSenders []string
+}
+
+// IsEmpty reports whether the filter admits everything.
+func (f TimelineFilter) IsEmpty() bool {
+	return len(f.Types) == 0 && len(f.NotTypes) == 0 &&
+		len(f.Senders) == 0 && len(f.NotSenders) == 0
+}
+
+// SyncMembership is a user's current membership in one room, with the
+// position of the event that set it.
+//
+// The position is what bounds what a user who has left may see: their view
+// of the room ends at their own leave event, and a room they were kicked
+// out of must not keep streaming to them.
+type SyncMembership struct {
+	RoomID     string
+	Membership string
+	EventID    string
+
+	// StreamOrdering is the position of the membership event.
+	StreamOrdering int64
+}
+
+// Receipt is one read receipt.
+type Receipt struct {
+	RoomID      string
+	UserID      string
+	ReceiptType string
+
+	// ThreadID is the thread the receipt is for, or empty for the room's
+	// main timeline.
+	ThreadID string
+
+	EventID string
+
+	// TimestampMS is when the receipt was sent, which is what a client
+	// shows as "read at".
+	TimestampMS int64
+
+	// StreamPos is the receipt's position in the receipt stream.
+	StreamPos int64
+}
+
+// ToDeviceMessage is one message addressed to a single device.
+type ToDeviceMessage struct {
+	// StreamPos is the message's position in the to-device stream.
+	StreamPos int64
+
+	Localpart string
+	DeviceID  string
+	Sender    string
+	Type      string
+
+	// Content is the message body as the sender wrote it.
+	Content []byte
+
+	SentAtMS int64
+}
+
+// NewToDeviceMessage is one message to enqueue.
+type NewToDeviceMessage struct {
+	Localpart string
+
+	// DeviceID is the device to deliver to. The caller has already expanded
+	// a wildcard addressee into one of these per device: which devices a
+	// user has is the userapi's business, not the store's.
+	DeviceID string
+
+	Sender   string
+	Type     string
+	Content  []byte
+	SentAtMS int64
+}
+
+// PresenceState is a user's presence as the store holds it.
+type PresenceState struct {
+	UserID string
+
+	// Presence is "online", "offline" or "unavailable".
+	Presence string
+
+	StatusMsg string
+
+	// LastActiveMS is when the user was last seen doing something.
+	LastActiveMS int64
+
+	// StreamPos is the position of the last change to this row.
+	StreamPos int64
+}
+
+// AccountDataEntry locates one account data blob that has changed. The
+// content is not carried: a sync response needs to know which blobs to
+// re-read, and reading them is a separate, cacheable call.
+type AccountDataEntry struct {
+	// RoomID is empty for global account data.
+	RoomID   string
+	DataType string
+
+	StreamPos int64
+}
+
+// StoredFilter is an uploaded filter.
+type StoredFilter struct {
+	Localpart string
+	FilterID  string
+
+	// JSON is the filter exactly as the client uploaded it.
+	JSON []byte
+
+	CreatedAtMS int64
+}
+
+// SyncStore persists the streams a sync response carries that are not room
+// events, plus the filters clients upload: the storage half of the syncapi
+// component.
+//
+// Typing notifications are not here on purpose. They expire in seconds and
+// are worthless across a restart, so the syncapi keeps them in memory.
+type SyncStore interface {
+	// StreamPositions reads the top of every stream in one call.
+	StreamPositions(ctx context.Context) (StreamPositions, error)
+
+	// SetReceipt records a read receipt, replacing any earlier one from the
+	// same user for the same type and thread, and returns its stream
+	// position.
+	//
+	// A receipt that would move backwards is not written and reports the
+	// current position with moved false: receipts only ever advance, and a
+	// client syncing on two devices would otherwise flap.
+	SetReceipt(ctx context.Context, r Receipt) (pos int64, moved bool, err error)
+
+	// Receipts returns the current receipts of the given rooms whose
+	// position is after from and at or before to. A from of 0 returns every
+	// current receipt, which is what an initial sync needs.
+	Receipts(ctx context.Context, roomIDs []string, from, to int64) ([]Receipt, error)
+
+	// ReceiptsForUser returns one user's own current receipts in the given
+	// rooms, whatever their position. A client needs its own read markers
+	// on an initial sync even when it set them long ago.
+	ReceiptsForUser(ctx context.Context, userID string, roomIDs []string) ([]Receipt, error)
+
+	// SendToDevice enqueues messages for delivery and returns the highest
+	// position it wrote. An empty batch is not an error and returns the
+	// current position.
+	SendToDevice(ctx context.Context, msgs []NewToDeviceMessage) (int64, error)
+
+	// ToDeviceMessages returns the messages queued for one device whose
+	// position is after from and at or before to, oldest first.
+	ToDeviceMessages(ctx context.Context, localpart, deviceID string, from, to int64, limit int) ([]ToDeviceMessage, error)
+
+	// DeleteToDeviceMessages removes the messages for a device at or before
+	// pos. A device acknowledges a batch by syncing past it, so this is
+	// what the next sync does with the batch before it.
+	DeleteToDeviceMessages(ctx context.Context, localpart, deviceID string, pos int64) error
+
+	// SetPresence records a user's presence and returns its stream
+	// position, reporting whether anything a peer would care about
+	// changed.
+	//
+	// A write that only refreshes LastActiveMS does not move the stream.
+	// A client says it is still online by syncing, so every sync is one of
+	// these — and a stream that moved each time would wake every peer's
+	// long poll, which would sync, which would move it again.
+	SetPresence(ctx context.Context, p PresenceState) (pos int64, moved bool, err error)
+
+	// Presence reads one user's presence, returning ErrNoSuchUser when
+	// nothing has ever been recorded for them.
+	Presence(ctx context.Context, userID string) (PresenceState, error)
+
+	// PresenceForUsers returns the presence of the given users whose
+	// position is after from and at or before to. A from of 0 returns every
+	// current state.
+	PresenceForUsers(ctx context.Context, userIDs []string, from, to int64) ([]PresenceState, error)
+
+	// AccountDataChanges lists the account data of a user that changed
+	// after from and at or before to.
+	AccountDataChanges(ctx context.Context, localpart string, from, to int64) ([]AccountDataEntry, error)
+
+	// AccountDataAt lists every account data blob a user has, as an initial
+	// sync needs.
+	AccountDataAt(ctx context.Context, localpart string, to int64) ([]AccountDataEntry, error)
+
+	// SaveFilter stores a filter under a new ID and returns it.
+	SaveFilter(ctx context.Context, localpart string, filter []byte, createdAtMS int64) (string, error)
+
+	// Filter reads an uploaded filter, returning ErrNoSuchFilter when the
+	// ID is unknown.
+	Filter(ctx context.Context, localpart, filterID string) ([]byte, error)
+}

server/internal/storage/sync_test.go

@@ -0,0 +1,698 @@
+package storage_test
+
+import (
+	"database/sql"
+	"errors"
+	"os"
+	"testing"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// The sync store and the stream reads of the room store, against every
+// driver that can run without external services.
+
+// migratedSyncDB opens a driver, migrates it and empties both the
+// roomserver and the sync tables.
+func migratedSyncDB(t *testing.T, driver string) storage.Database {
+	t.Helper()
+	db := migratedRoomDB(t, driver)
+	resetSyncTables(t, driver)
+	return db
+}
+
+// resetSyncTables empties the sync tables and rewinds their counters, for
+// the same reason resetRoomTables exists: the PostgreSQL tests share one
+// database.
+func resetSyncTables(t *testing.T, driver string) {
+	t.Helper()
+	if driver != storage.DriverPostgres {
+		return
+	}
+
+	db, err := sql.Open("pgx", os.Getenv("WARP_TEST_POSTGRES_DSN"))
+	if err != nil {
+		t.Fatalf("open postgres to reset the sync tables: %v", err)
+	}
+	defer func() { _ = db.Close() }()
+
+	if _, err := db.ExecContext(t.Context(), `TRUNCATE
+		sync_receipts, sync_to_device, sync_presence, sync_filters`); err != nil {
+		t.Fatalf("reset the sync tables: %v", err)
+	}
+	if _, err := db.ExecContext(t.Context(),
+		`UPDATE sync_sequences SET value = 0`); err != nil {
+		t.Fatalf("rewind the sync counters: %v", err)
+	}
+}
+
+// writeMessages appends n plausible message events to a room and returns
+// their IDs in order. The room store never parses an event, so the JSON
+// only has to be JSON.
+func writeMessages(t *testing.T, rooms storage.RoomStore, roomID string, ids ...string) {
+	t.Helper()
+	for _, id := range ids {
+		if _, err := rooms.Write(t.Context(), storage.RoomWrite{
+			Event:       newTestEvent(roomID, id, "m.room.message", nil),
+			Extremities: []string{id},
+		}); err != nil {
+			t.Fatalf("Write %s: %v", id, err)
+		}
+	}
+}
+
+func TestTimelineBoundsAndDirection(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+			writeMessages(t, rooms, testRoomID, "$a", "$b", "$c", "$d", "$e")
+
+			top, err := rooms.MaxStreamOrdering(ctx)
+			if err != nil {
+				t.Fatalf("MaxStreamOrdering: %v", err)
+			}
+			if top != 5 {
+				t.Fatalf("MaxStreamOrdering = %d, want 5", top)
+			}
+
+			for _, tc := range []struct {
+				name      string
+				req       storage.TimelineRequest
+				wantOrder []string
+			}{
+				{
+					// Low is exclusive, High inclusive, in both
+					// directions: that is what lets a token be handed back
+					// without re-reading the event it names.
+					name: "forwards from the start",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, Direction: storage.DirectionForward, Limit: 10,
+					},
+					wantOrder: []string{"$a", "$b", "$c", "$d", "$e"},
+				},
+				{
+					name: "forwards past a position",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, Low: 2,
+						Direction: storage.DirectionForward, Limit: 10,
+					},
+					wantOrder: []string{"$c", "$d", "$e"},
+				},
+				{
+					name: "forwards up to a position",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, High: 3,
+						Direction: storage.DirectionForward, Limit: 10,
+					},
+					wantOrder: []string{"$a", "$b", "$c"},
+				},
+				{
+					// The limit trims the far end, so a backwards read
+					// keeps the newest: that is what a timeline is.
+					name: "backwards keeps the newest",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, Direction: storage.DirectionBackward, Limit: 2,
+					},
+					wantOrder: []string{"$e", "$d"},
+				},
+				{
+					name: "forwards keeps the oldest",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, Direction: storage.DirectionForward, Limit: 2,
+					},
+					wantOrder: []string{"$a", "$b"},
+				},
+				{
+					name: "a window with both bounds",
+					req: storage.TimelineRequest{
+						RoomID: testRoomID, Low: 1, High: 4,
+						Direction: storage.DirectionBackward, Limit: 10,
+					},
+					wantOrder: []string{"$d", "$c", "$b"},
+				},
+			} {
+				t.Run(tc.name, func(t *testing.T) {
+					events, err := rooms.Timeline(ctx, tc.req)
+					if err != nil {
+						t.Fatalf("Timeline: %v", err)
+					}
+					got := make([]string, 0, len(events))
+					for _, e := range events {
+						got = append(got, e.EventID)
+					}
+					if len(got) != len(tc.wantOrder) {
+						t.Fatalf("Timeline returned %v, want %v", got, tc.wantOrder)
+					}
+					for i := range got {
+						if got[i] != tc.wantOrder[i] {
+							t.Fatalf("Timeline returned %v, want %v", got, tc.wantOrder)
+						}
+					}
+				})
+			}
+		})
+	}
+}
+
+func TestTimelineFilterIsAppliedInSQL(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+
+			// Two types and two senders.
+			for _, e := range []storage.StoredEvent{
+				newTestEvent(testRoomID, "$msg1", "m.room.message", nil),
+				newTestEvent(testRoomID, "$topic", "m.room.topic", stateKey("")),
+				func() storage.StoredEvent {
+					e := newTestEvent(testRoomID, "$msg2", "m.room.message", nil)
+					e.Sender = "@bob:warp.example"
+					return e
+				}(),
+			} {
+				if _, err := rooms.Write(ctx, storage.RoomWrite{
+					Event: e, Extremities: []string{e.EventID},
+				}); err != nil {
+					t.Fatalf("Write %s: %v", e.EventID, err)
+				}
+			}
+
+			for _, tc := range []struct {
+				name   string
+				filter storage.TimelineFilter
+				want   []string
+			}{
+				{"types", storage.TimelineFilter{Types: []string{"m.room.message"}},
+					[]string{"$msg1", "$msg2"}},
+				{"not_types", storage.TimelineFilter{NotTypes: []string{"m.room.message"}},
+					[]string{"$topic"}},
+				{"senders", storage.TimelineFilter{Senders: []string{"@bob:warp.example"}},
+					[]string{"$msg2"}},
+				{"not_senders", storage.TimelineFilter{NotSenders: []string{"@bob:warp.example"}},
+					[]string{"$msg1", "$topic"}},
+				{
+					// A trailing '*' is the spec's wildcard.
+					"a wildcard type", storage.TimelineFilter{Types: []string{"m.room.m*"}},
+					[]string{"$msg1", "$msg2"},
+				},
+				{
+					// not_types wins over types for an event matching
+					// both, which the spec requires.
+					"exclusions win",
+					storage.TimelineFilter{
+						Types:    []string{"m.room.message"},
+						NotTypes: []string{"m.room.message"},
+					},
+					nil,
+				},
+			} {
+				t.Run(tc.name, func(t *testing.T) {
+					events, err := rooms.Timeline(ctx, storage.TimelineRequest{
+						RoomID:    testRoomID,
+						Direction: storage.DirectionForward,
+						Limit:     10,
+						Filter:    tc.filter,
+					})
+					if err != nil {
+						t.Fatalf("Timeline: %v", err)
+					}
+					got := make([]string, 0, len(events))
+					for _, e := range events {
+						got = append(got, e.EventID)
+					}
+					if len(got) != len(tc.want) {
+						t.Fatalf("Timeline returned %v, want %v", got, tc.want)
+					}
+					for i := range got {
+						if got[i] != tc.want[i] {
+							t.Fatalf("Timeline returned %v, want %v", got, tc.want)
+						}
+					}
+				})
+			}
+		})
+	}
+}
+
+func TestTimelineWildcardDoesNotTreatPatternCharactersAsWildcards(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+			writeMessages(t, rooms, testRoomID, "$msg")
+
+			// A filter pattern is matched with LIKE, so a '%' or '_' in it
+			// has to be escaped or it would match anything.
+			events, err := rooms.Timeline(ctx, storage.TimelineRequest{
+				RoomID:    testRoomID,
+				Direction: storage.DirectionForward,
+				Limit:     10,
+				Filter:    storage.TimelineFilter{Types: []string{"m.room.m%*"}},
+			})
+			if err != nil {
+				t.Fatalf("Timeline: %v", err)
+			}
+			if len(events) != 0 {
+				t.Errorf("a '%%' in a filter pattern matched %d events; it must be literal", len(events))
+			}
+		})
+	}
+}
+
+func TestRoomsWithEventsNarrowsToWhatChanged(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+
+			const other = "!other:warp.example"
+			for _, id := range []string{testRoomID, other} {
+				if err := rooms.CreateRoom(ctx, newTestRoom(id)); err != nil {
+					t.Fatalf("CreateRoom %s: %v", id, err)
+				}
+			}
+			writeMessages(t, rooms, testRoomID, "$a")
+			writeMessages(t, rooms, other, "$b")
+			writeMessages(t, rooms, testRoomID, "$c")
+
+			// Only the first room has an event after position 2.
+			changed, err := rooms.RoomsWithEvents(ctx, []string{testRoomID, other}, 2, 0)
+			if err != nil {
+				t.Fatalf("RoomsWithEvents: %v", err)
+			}
+			if len(changed) != 1 || changed[0] != testRoomID {
+				t.Errorf("RoomsWithEvents = %v, want [%s]", changed, testRoomID)
+			}
+
+			// Nothing happened after the top of the stream.
+			quiet, err := rooms.RoomsWithEvents(ctx, []string{testRoomID, other}, 3, 0)
+			if err != nil {
+				t.Fatalf("RoomsWithEvents: %v", err)
+			}
+			if len(quiet) != 0 {
+				t.Errorf("RoomsWithEvents past the top = %v, want nothing", quiet)
+			}
+		})
+	}
+}
+
+func TestStateAtStreamIsTheStateAfterAPosition(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+
+			create := []storage.StateEntry{{Type: "m.room.create", StateKey: "", EventID: "$create"}}
+			if _, err := rooms.Write(ctx, storage.RoomWrite{
+				Event:       newTestEvent(testRoomID, "$create", "m.room.create", stateKey("")),
+				StateDelta:  create,
+				Extremities: []string{"$create"},
+			}); err != nil {
+				t.Fatalf("Write the create event: %v", err)
+			}
+
+			// A topic, authorised against the state the create event left.
+			if _, err := rooms.Write(ctx, storage.RoomWrite{
+				Event:       newTestEvent(testRoomID, "$topic", "m.room.topic", stateKey("")),
+				StateBefore: create,
+				StateDelta:  []storage.StateEntry{{Type: "m.room.topic", StateKey: "", EventID: "$topic"}},
+				Extremities: []string{"$topic"},
+			}); err != nil {
+				t.Fatalf("Write the topic: %v", err)
+			}
+
+			// A message after it, whose state-before includes the topic.
+			if _, err := rooms.Write(ctx, storage.RoomWrite{
+				Event: newTestEvent(testRoomID, "$msg", "m.room.message", nil),
+				StateBefore: append(create,
+					storage.StateEntry{Type: "m.room.topic", StateKey: "", EventID: "$topic"}),
+				Extremities: []string{"$msg"},
+			}); err != nil {
+				t.Fatalf("Write the message: %v", err)
+			}
+
+			for _, tc := range []struct {
+				pos   int64
+				types []string
+			}{
+				// Before anything happened.
+				{0, nil},
+				// After the create event: the state is the create event,
+				// which is the snapshot *before* it plus itself.
+				{1, []string{"m.room.create"}},
+				// After the topic.
+				{2, []string{"m.room.create", "m.room.topic"}},
+				// After a message, which changes no state.
+				{3, []string{"m.room.create", "m.room.topic"}},
+			} {
+				entries, err := rooms.StateAtStream(ctx, testRoomID, tc.pos)
+				if err != nil {
+					t.Fatalf("StateAtStream(%d): %v", tc.pos, err)
+				}
+				if len(entries) != len(tc.types) {
+					t.Errorf("StateAtStream(%d) returned %d entries, want %d (%v)",
+						tc.pos, len(entries), len(tc.types), tc.types)
+					continue
+				}
+				have := map[string]bool{}
+				for _, e := range entries {
+					have[e.Type] = true
+				}
+				for _, want := range tc.types {
+					if !have[want] {
+						t.Errorf("StateAtStream(%d) is missing %s", tc.pos, want)
+					}
+				}
+			}
+		})
+	}
+}
+
+func TestReceiptsOnlyEverAdvance(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			db := migratedSyncDB(t, driver)
+			rooms, sync := db.Rooms(), db.Sync()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+			writeMessages(t, rooms, testRoomID, "$first", "$second")
+
+			receipt := storage.Receipt{
+				RoomID:      testRoomID,
+				UserID:      "@alice:warp.example",
+				ReceiptType: "m.read",
+				EventID:     "$second",
+				TimestampMS: 1_700_000_000_000,
+			}
+			pos, moved, err := sync.SetReceipt(ctx, receipt)
+			if err != nil {
+				t.Fatalf("SetReceipt: %v", err)
+			}
+			if !moved || pos == 0 {
+				t.Fatalf("SetReceipt = (%d, %v), want a position and moved", pos, moved)
+			}
+
+			// Backwards: two of a user's clients read a room at different
+			// speeds, and letting the slower one win would make the unread
+			// marker jump backwards on the other.
+			backwards := receipt
+			backwards.EventID = "$first"
+			back, moved, err := sync.SetReceipt(ctx, backwards)
+			if err != nil {
+				t.Fatalf("SetReceipt backwards: %v", err)
+			}
+			if moved {
+				t.Error("a receipt that would move backwards was written")
+			}
+			if back != pos {
+				t.Errorf("the position moved from %d to %d on a rejected receipt", pos, back)
+			}
+
+			current, err := sync.Receipts(ctx, []string{testRoomID}, 0, 0)
+			if err != nil {
+				t.Fatalf("Receipts: %v", err)
+			}
+			if len(current) != 1 || current[0].EventID != "$second" {
+				t.Errorf("Receipts = %+v, want the one receipt at $second", current)
+			}
+		})
+	}
+}
+
+func TestReceiptForAnUnknownEventIsRefused(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			db := migratedSyncDB(t, driver)
+			if err := db.Rooms().CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+
+			_, _, err := db.Sync().SetReceipt(ctx, storage.Receipt{
+				RoomID:      testRoomID,
+				UserID:      "@alice:warp.example",
+				ReceiptType: "m.read",
+				EventID:     "$nope",
+			})
+			if !errors.Is(err, storage.ErrNoSuchEvent) {
+				t.Errorf("SetReceipt for an unknown event = %v, want ErrNoSuchEvent", err)
+			}
+		})
+	}
+}
+
+func TestToDeviceQueueIsPerDeviceAndDrainable(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			sync := migratedSyncDB(t, driver).Sync()
+
+			top, err := sync.SendToDevice(ctx, []storage.NewToDeviceMessage{
+				{Localpart: "alice", DeviceID: "ONE", Sender: "@bob:warp.example",
+					Type: "m.room_key", Content: []byte(`{"n":1}`)},
+				{Localpart: "alice", DeviceID: "TWO", Sender: "@bob:warp.example",
+					Type: "m.room_key", Content: []byte(`{"n":2}`)},
+				{Localpart: "alice", DeviceID: "ONE", Sender: "@bob:warp.example",
+					Type: "m.room_key", Content: []byte(`{"n":3}`)},
+			})
+			if err != nil {
+				t.Fatalf("SendToDevice: %v", err)
+			}
+			if top != 3 {
+				t.Errorf("SendToDevice returned %d, want 3: every message takes a position", top)
+			}
+
+			one, err := sync.ToDeviceMessages(ctx, "alice", "ONE", 0, 0, 0)
+			if err != nil {
+				t.Fatalf("ToDeviceMessages: %v", err)
+			}
+			if len(one) != 2 {
+				t.Fatalf("device ONE has %d messages, want 2", len(one))
+			}
+
+			// Draining one device's queue leaves the other's alone: a
+			// message to two devices is two independent deliveries.
+			if err := sync.DeleteToDeviceMessages(ctx, "alice", "ONE", one[len(one)-1].StreamPos); err != nil {
+				t.Fatalf("DeleteToDeviceMessages: %v", err)
+			}
+			if remaining, err := sync.ToDeviceMessages(ctx, "alice", "ONE", 0, 0, 0); err != nil {
+				t.Fatalf("ToDeviceMessages: %v", err)
+			} else if len(remaining) != 0 {
+				t.Errorf("device ONE still has %d messages after draining", len(remaining))
+			}
+			if two, err := sync.ToDeviceMessages(ctx, "alice", "TWO", 0, 0, 0); err != nil {
+				t.Fatalf("ToDeviceMessages: %v", err)
+			} else if len(two) != 1 {
+				t.Errorf("device TWO has %d messages, want its own 1", len(two))
+			}
+		})
+	}
+}
+
+func TestPresenceStreamMovesOnlyOnARealChange(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			sync := migratedSyncDB(t, driver).Sync()
+
+			state := storage.PresenceState{
+				UserID: "@alice:warp.example", Presence: "online", LastActiveMS: 1_000,
+			}
+			first, moved, err := sync.SetPresence(ctx, state)
+			if err != nil {
+				t.Fatalf("SetPresence: %v", err)
+			}
+			if !moved {
+				t.Error("the first presence write must move the stream")
+			}
+
+			// Every /sync is a write that only refreshes the timestamp. A
+			// stream that moved each time would wake every peer's long
+			// poll, which would sync, which would move it again.
+			state.LastActiveMS = 2_000
+			again, moved, err := sync.SetPresence(ctx, state)
+			if err != nil {
+				t.Fatalf("SetPresence: %v", err)
+			}
+			if moved {
+				t.Error("a write that refreshes only the timestamp must not move the stream")
+			}
+			if again != first {
+				t.Errorf("the position moved from %d to %d without a change", first, again)
+			}
+
+			// The timestamp did update, even though the stream did not.
+			read, err := sync.Presence(ctx, state.UserID)
+			if err != nil {
+				t.Fatalf("Presence: %v", err)
+			}
+			if read.LastActiveMS != 2_000 {
+				t.Errorf("last_active_ms = %d, want it refreshed to 2000", read.LastActiveMS)
+			}
+
+			state.Presence = "unavailable"
+			third, moved, err := sync.SetPresence(ctx, state)
+			if err != nil {
+				t.Fatalf("SetPresence: %v", err)
+			}
+			if !moved || third <= first {
+				t.Errorf("a real change gave (%d, %v), want a higher position and moved", third, moved)
+			}
+		})
+	}
+}
+
+func TestPresenceOfAnUnknownUser(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			sync := migratedSyncDB(t, driver).Sync()
+			_, err := sync.Presence(t.Context(), "@nobody:warp.example")
+			if !errors.Is(err, storage.ErrNoSuchUser) {
+				t.Errorf("Presence of an unknown user = %v, want ErrNoSuchUser", err)
+			}
+		})
+	}
+}
+
+func TestFiltersAreStoredVerbatimAndScopedToTheirOwner(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			sync := migratedSyncDB(t, driver).Sync()
+
+			// Spacing and key order are preserved: a client compares what
+			// it gets back with what it sent.
+			raw := []byte(`{"room":{"timeline":{"limit":5}},"event_fields":["type"]}`)
+			filterID, err := sync.SaveFilter(ctx, "alice", raw, 1_700_000_000_000)
+			if err != nil {
+				t.Fatalf("SaveFilter: %v", err)
+			}
+			if filterID == "" {
+				t.Fatal("SaveFilter returned no ID")
+			}
+
+			read, err := sync.Filter(ctx, "alice", filterID)
+			if err != nil {
+				t.Fatalf("Filter: %v", err)
+			}
+			if string(read) != string(raw) {
+				t.Errorf("Filter = %s, want it byte for byte: %s", read, raw)
+			}
+
+			// A filter belongs to the user who uploaded it.
+			if _, err := sync.Filter(ctx, "bob", filterID); !errors.Is(err, storage.ErrNoSuchFilter) {
+				t.Errorf("another user's filter = %v, want ErrNoSuchFilter", err)
+			}
+			if _, err := sync.Filter(ctx, "alice", "nope"); !errors.Is(err, storage.ErrNoSuchFilter) {
+				t.Errorf("an unknown filter = %v, want ErrNoSuchFilter", err)
+			}
+		})
+	}
+}
+
+func TestStreamPositionsReadsEveryStream(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			db := migratedSyncDB(t, driver)
+			rooms, sync := db.Rooms(), db.Sync()
+
+			positions, err := sync.StreamPositions(ctx)
+			if err != nil {
+				t.Fatalf("StreamPositions: %v", err)
+			}
+			if positions != (storage.StreamPositions{}) {
+				t.Errorf("a fresh database is at %+v, want every stream at zero", positions)
+			}
+
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+			writeMessages(t, rooms, testRoomID, "$a")
+			if _, _, err := sync.SetReceipt(ctx, storage.Receipt{
+				RoomID: testRoomID, UserID: "@alice:warp.example",
+				ReceiptType: "m.read", EventID: "$a",
+			}); err != nil {
+				t.Fatalf("SetReceipt: %v", err)
+			}
+			if _, err := sync.SendToDevice(ctx, []storage.NewToDeviceMessage{
+				{Localpart: "alice", DeviceID: "ONE", Type: "m.room_key", Content: []byte(`{}`)},
+			}); err != nil {
+				t.Fatalf("SendToDevice: %v", err)
+			}
+			if _, _, err := sync.SetPresence(ctx, storage.PresenceState{
+				UserID: "@alice:warp.example", Presence: "online",
+			}); err != nil {
+				t.Fatalf("SetPresence: %v", err)
+			}
+
+			positions, err = sync.StreamPositions(ctx)
+			if err != nil {
+				t.Fatalf("StreamPositions: %v", err)
+			}
+			if positions.PDU != 1 || positions.Receipt != 1 ||
+				positions.ToDevice != 1 || positions.Presence != 1 {
+				t.Errorf("StreamPositions = %+v, want every written stream at 1", positions)
+			}
+		})
+	}
+}
+
+func TestUserMembershipEventsAreInStreamOrder(t *testing.T) {
+	for _, driver := range testDrivers {
+		t.Run(driver, func(t *testing.T) {
+			ctx := t.Context()
+			rooms := migratedSyncDB(t, driver).Rooms()
+			if err := rooms.CreateRoom(ctx, newTestRoom(testRoomID)); err != nil {
+				t.Fatalf("CreateRoom: %v", err)
+			}
+
+			const alice = "@alice:warp.example"
+			for _, id := range []string{"$join", "$leave", "$rejoin"} {
+				e := newTestEvent(testRoomID, id, "m.room.member", stateKey(alice))
+				if _, err := rooms.Write(ctx, storage.RoomWrite{
+					Event:       e,
+					Extremities: []string{id},
+				}); err != nil {
+					t.Fatalf("Write %s: %v", id, err)
+				}
+			}
+			// Somebody else's membership must not be in the answer.
+			other := newTestEvent(testRoomID, "$bob", "m.room.member", stateKey("@bob:warp.example"))
+			if _, err := rooms.Write(ctx, storage.RoomWrite{
+				Event: other, Extremities: []string{"$bob"},
+			}); err != nil {
+				t.Fatalf("Write $bob: %v", err)
+			}
+
+			events, err := rooms.UserMembershipEvents(ctx, testRoomID, alice)
+			if err != nil {
+				t.Fatalf("UserMembershipEvents: %v", err)
+			}
+			want := []string{"$join", "$leave", "$rejoin"}
+			if len(events) != len(want) {
+				t.Fatalf("UserMembershipEvents returned %d events, want %d", len(events), len(want))
+			}
+			for i, e := range events {
+				if e.EventID != want[i] {
+					t.Errorf("event %d = %s, want %s", i, e.EventID, want[i])
+				}
+			}
+		})
+	}
+}

server/internal/syncapi/context.go

@@ -0,0 +1,178 @@
+package syncapi
+
+import (
+	"context"
+
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// GET /rooms/{roomID}/context/{eventID}: one event and what surrounds it.
+//
+// It is what a permalink opens, and what jumping to a search result or a
+// reply does. The answer is the event, a few events either side of it, the
+// tokens to go on paging in both directions, and the state of the room as
+// of the event — because a client that jumps into the middle of a room's
+// history has none of it.
+
+// ContextRequest is one context read.
+type ContextRequest struct {
+	RoomID  string
+	EventID string
+
+	User      string
+	Localpart string
+
+	// Limit is how many events to return in total, including the event
+	// itself. The spec's default is 10, and the events either side of the
+	// target get half each.
+	Limit int
+
+	// Filter is the ?filter= parameter, a RoomEventFilter as for
+	// /messages.
+	Filter string
+}
+
+// ContextResponse is the event and its surroundings.
+type ContextResponse struct {
+	// Start and End are the tokens to page backwards and forwards from.
+	Start string `json:"start,omitempty"`
+	End   string `json:"end,omitempty"`
+
+	// EventsBefore and EventsAfter are oldest-first and oldest-first
+	// respectively — the spec says EventsBefore is in reverse
+	// chronological order, which is what a client scrolling up consumes.
+	EventsBefore []roomserver.ClientEvent `json:"events_before"`
+	Event        roomserver.ClientEvent   `json:"event"`
+	EventsAfter  []roomserver.ClientEvent `json:"events_after"`
+
+	// State is the room's state as of the event, which is what a client
+	// needs to render the names and avatars of the people in the chunk.
+	State []roomserver.ClientEvent `json:"state"`
+}
+
+// DefaultContextLimit is the spec's default for /context.
+const DefaultContextLimit = 10
+
+// Context answers one context read.
+func (s *API) Context(ctx context.Context, req ContextRequest) (ContextResponse, error) {
+	filter, err := s.messageFilter(ctx, req.Localpart, req.Filter)
+	if err != nil {
+		return ContextResponse{}, err
+	}
+
+	limit := req.Limit
+	if limit <= 0 {
+		limit = DefaultContextLimit
+	}
+	limit = min(limit, MaxTimelineLimit)
+
+	// The target event is read first, and through the roomserver, so that
+	// the visibility check happens before anything else: an event the user
+	// may not see has to come back as unknown rather than as a context
+	// with an empty middle.
+	target, err := s.rooms.Event(ctx, req.RoomID, req.User, req.EventID)
+	if err != nil {
+		return ContextResponse{}, err
+	}
+
+	// The target's own position. The roomserver's timeline reads are
+	// bounded by position rather than by event ID, so it has to be found
+	// before either side can be read — which is one extra read, and the
+	// reason /context costs more than /messages.
+	position, err := s.rooms.EventPosition(ctx, req.RoomID, req.User, req.EventID)
+	if err != nil {
+		return ContextResponse{}, err
+	}
+
+	// Half the budget each side, with the odd one going backwards: a
+	// client opening a permalink scrolls up from it more often than down.
+	after := limit / 2
+	before := limit - after - 1
+	if before < 0 {
+		before = 0
+	}
+
+	out := ContextResponse{
+		Event:        target,
+		EventsBefore: []roomserver.ClientEvent{},
+		EventsAfter:  []roomserver.ClientEvent{},
+		State:        []roomserver.ClientEvent{},
+	}
+
+	if before > 0 {
+		earlier, err := s.rooms.Timeline(ctx, roomserver.TimelineRequest{
+			RoomID: req.RoomID,
+			User:   req.User,
+			// Strictly before the target: High is inclusive, and the
+			// target itself is reported separately.
+			High:      position - 1,
+			Direction: storage.DirectionBackward,
+			Limit:     before,
+			Filter:    filter.TimelineFilter(),
+		})
+		if err != nil {
+			return ContextResponse{}, err
+		}
+		// The spec wants this half newest first: it is what a client
+		// prepends as the user scrolls up.
+		out.EventsBefore = reversed(earlier.Events)
+		out.Start = pduToken(earlier.Start).String()
+	}
+	if out.Start == "" {
+		out.Start = pduToken(position - 1).String()
+	}
+
+	if after > 0 {
+		later, err := s.rooms.Timeline(ctx, roomserver.TimelineRequest{
+			RoomID: req.RoomID,
+			User:   req.User,
+			// Strictly after the target: Low is exclusive.
+			Low:       position,
+			Direction: storage.DirectionForward,
+			Limit:     after,
+			Filter:    filter.TimelineFilter(),
+		})
+		if err != nil {
+			return ContextResponse{}, err
+		}
+		out.EventsAfter = later.Events
+		out.End = pduToken(later.End).String()
+	}
+	if out.End == "" {
+		out.End = pduToken(position).String()
+	}
+
+	state, err := s.contextState(ctx, req, filter, out)
+	if err != nil {
+		return ContextResponse{}, err
+	}
+	out.State = state
+	return out, nil
+}
+
+// contextState is the state a context read reports.
+//
+// With lazy loading it is the member events of everybody in the chunk,
+// which is all a client needs to render it. Without, it is the room's whole
+// state as of the event — which for a large room is most of the response,
+// and is why every client asks for lazy loading.
+func (s *API) contextState(ctx context.Context, req ContextRequest, filter Filter, resp ContextResponse) ([]roomserver.ClientEvent, error) {
+	if filter.LazyLoadMembers() {
+		chunk := make([]roomserver.ClientEvent, 0, len(resp.EventsBefore)+len(resp.EventsAfter)+1)
+		chunk = append(chunk, resp.EventsBefore...)
+		chunk = append(chunk, resp.Event)
+		chunk = append(chunk, resp.EventsAfter...)
+		return s.rooms.MemberEvents(ctx, req.RoomID, sendersOf(chunk))
+	}
+	return s.rooms.StateBeforeEvent(ctx, req.RoomID, req.User, req.EventID)
+}
+
+// reversed returns a copy of events in the opposite order.
+func reversed(events []roomserver.ClientEvent) []roomserver.ClientEvent {
+	out := make([]roomserver.ClientEvent, len(events))
+	for i, e := range events {
+		out[len(events)-1-i] = e
+	}
+	return out
+}

server/internal/syncapi/errors.go

@@ -0,0 +1,48 @@
+package syncapi
+
+import (
+	"fmt"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/id"
+)
+
+// The component's errors are plain wrapped sentinels rather than HTTP
+// errors, the same as the roomserver's: this package is called from the
+// Client-Server handlers and, later, from the application service API, and
+// a component that returned HTTP statuses would be deciding something that
+// is not its to decide. The handler maps them; see clientapi.syncError.
+
+// errForbiddenf is a refusal the caller is told the reason for.
+//
+// The reason is passed through to the client on purpose, as it is in the
+// roomserver: "M_FORBIDDEN" on its own sends a user to their operator and
+// the operator to the logs, and nothing in these messages is anything the
+// caller could not work out from the room's state.
+func errForbiddenf(format string, args ...any) error {
+	return fmt.Errorf("%w: %s", ErrForbidden, fmt.Sprintf(format, args...))
+}
+
+// errInvalidParamf is a request parameter this server will not accept.
+func errInvalidParamf(format string, args ...any) error {
+	return fmt.Errorf("%w: %s", ErrInvalidParam, fmt.Sprintf(format, args...))
+}
+
+// errInvalidTokenf is a ?since= or ?from= that could not be read.
+func errInvalidTokenf(format string, args ...any) error {
+	return fmt.Errorf("%w: %s", ErrInvalidToken, fmt.Sprintf(format, args...))
+}
+
+// localpartOf returns the localpart of a user ID of this server.
+func (s *API) localpartOf(userID string) (string, bool) {
+	localpart, serverName, err := id.SplitUserID(userID)
+	if err != nil || serverName != s.serverName {
+		return "", false
+	}
+	return localpart, true
+}
+
+// isLocal reports whether a user ID belongs to this server.
+func (s *API) isLocal(userID string) bool {
+	_, ok := s.localpartOf(userID)
+	return ok
+}

server/internal/syncapi/filter.go

@@ -0,0 +1,374 @@
+package syncapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// Filters: what a client asks to be left out of a sync.
+//
+// A filter is uploaded once and referred to by ID on every later sync,
+// which is the point of it — the filter a client wants is long and it does
+// not change, so sending it on every request would be the largest part of
+// every request. The ?filter= parameter also accepts an inline filter as
+// JSON, which is what a client that does not want to upload one uses.
+
+// Errors a filter can fail with.
+var (
+	// ErrNoSuchFilter is an unknown filter ID.
+	ErrNoSuchFilter = storage.ErrNoSuchFilter
+
+	// ErrInvalidFilter is a filter that is not usable: not an object, or
+	// a limit that is not a number.
+	ErrInvalidFilter = errors.New("invalid filter")
+)
+
+// Filter is the whole of a Client-Server API filter.
+//
+// The fields Warp does not act on are still parsed, so that a client
+// uploading a complete filter gets it back unchanged and is not told its
+// filter was rejected because of a field this build ignores.
+type Filter struct {
+	// EventFields names the properties to keep on each event. Warp does
+	// not trim events: see trimToFields.
+	EventFields []string `json:"event_fields,omitempty"`
+
+	// EventFormat is "client" or "federation". Only "client" is offered —
+	// the federation format is a server's view of an event and the
+	// Client-Server API has no business handing it out.
+	EventFormat string `json:"event_format,omitempty"`
+
+	Presence    *EventFilter `json:"presence,omitempty"`
+	AccountData *EventFilter `json:"account_data,omitempty"`
+	Room        *RoomFilter  `json:"room,omitempty"`
+}
+
+// EventFilter narrows a stream of events that are not in a room.
+type EventFilter struct {
+	Limit      *int     `json:"limit,omitempty"`
+	Types      []string `json:"types,omitempty"`
+	NotTypes   []string `json:"not_types,omitempty"`
+	Senders    []string `json:"senders,omitempty"`
+	NotSenders []string `json:"not_senders,omitempty"`
+}
+
+// RoomFilter narrows the room half of a sync.
+type RoomFilter struct {
+	Rooms    []string `json:"rooms,omitempty"`
+	NotRooms []string `json:"not_rooms,omitempty"`
+
+	// IncludeLeave asks for the rooms the user has left. Without it a sync
+	// reports only the leaves that happened inside the window, which is
+	// what a client needs to notice one.
+	IncludeLeave bool `json:"include_leave,omitempty"`
+
+	Ephemeral   *RoomEventFilter `json:"ephemeral,omitempty"`
+	AccountData *RoomEventFilter `json:"account_data,omitempty"`
+	Timeline    *RoomEventFilter `json:"timeline,omitempty"`
+	State       *StateFilter     `json:"state,omitempty"`
+}
+
+// RoomEventFilter narrows a stream of room events.
+type RoomEventFilter struct {
+	Limit      *int     `json:"limit,omitempty"`
+	Types      []string `json:"types,omitempty"`
+	NotTypes   []string `json:"not_types,omitempty"`
+	Senders    []string `json:"senders,omitempty"`
+	NotSenders []string `json:"not_senders,omitempty"`
+	Rooms      []string `json:"rooms,omitempty"`
+	NotRooms   []string `json:"not_rooms,omitempty"`
+
+	// ContainsURL selects events whose content has a `url` property, which
+	// is how a client builds a room's media gallery. Warp does not act on
+	// it: see the note in Validate.
+	ContainsURL *bool `json:"contains_url,omitempty"`
+
+	// LazyLoadMembers leaves out the member events of everybody who does
+	// not appear in the timeline being returned. It is what makes a room
+	// with ten thousand members openable.
+	LazyLoadMembers bool `json:"lazy_load_members,omitempty"`
+
+	// IncludeRedundantMembers asks for those member events even when the
+	// client has already been sent them. Warp always behaves as though
+	// this were set; see roomserver.shapeState.
+	IncludeRedundantMembers bool `json:"include_redundant_members,omitempty"`
+
+	// UnreadThreadNotifications is a threads feature; threads are
+	// PLAN.md phase 10.
+	UnreadThreadNotifications bool `json:"unread_thread_notifications,omitempty"`
+}
+
+// StateFilter is a RoomEventFilter applied to a room's state. The spec
+// gives it the same shape, and the only reason it is a separate type is
+// that the state half of a response has no limit worth applying: a partial
+// state would be a lie about the room rather than a shorter answer.
+type StateFilter = RoomEventFilter
+
+// DefaultTimelineLimit is how many timeline events a room's half of a sync
+// carries when the client does not ask for a number. The spec's default is
+// 10; 20 is a screenful on a phone and still small.
+const DefaultTimelineLimit = 20
+
+// MaxTimelineLimit caps what a client may ask for in one room, on /sync and
+// on /messages alike.
+const MaxTimelineLimit = 1000
+
+// MaxFilterSize caps an uploaded filter. A filter is a list of types and
+// senders; anything larger is not a filter.
+const MaxFilterSize = 64 << 10
+
+// ParseFilter reads a filter from JSON and validates it.
+func ParseFilter(raw []byte) (Filter, error) {
+	if len(raw) > MaxFilterSize {
+		return Filter{}, fmt.Errorf("%w: a filter may be at most %d bytes", ErrInvalidFilter, MaxFilterSize)
+	}
+	var f Filter
+	if err := json.Unmarshal(raw, &f); err != nil {
+		return Filter{}, fmt.Errorf("%w: %v", ErrInvalidFilter, err)
+	}
+	if err := f.Validate(); err != nil {
+		return Filter{}, err
+	}
+	return f, nil
+}
+
+// unmarshalStrict reads JSON into v, refusing a body larger than a filter
+// may be. The size check is here as well as in ParseFilter because an
+// inline filter arrives in a query parameter, where nothing else bounds it.
+func unmarshalStrict(raw []byte, v any) error {
+	if len(raw) > MaxFilterSize {
+		return fmt.Errorf("a filter may be at most %d bytes", MaxFilterSize)
+	}
+	return json.Unmarshal(raw, v)
+}
+
+// Validate checks the fields this server acts on.
+//
+// A field Warp ignores is not an error. The spec's filters are a superset
+// of what any server acts on, and a client that uploads `contains_url`
+// would otherwise be unable to sync at all — whereas ignoring it returns
+// more events than asked for, which every client copes with because a
+// server is allowed to return a superset.
+func (f Filter) Validate() error {
+	if f.EventFormat != "" && f.EventFormat != "client" {
+		return fmt.Errorf("%w: event_format %q is not supported; only \"client\" is",
+			ErrInvalidFilter, f.EventFormat)
+	}
+	for _, check := range []struct {
+		name  string
+		limit *int
+	}{
+		{"presence.limit", limitOf(f.Presence)},
+		{"account_data.limit", limitOf(f.AccountData)},
+	} {
+		if check.limit != nil && *check.limit < 0 {
+			return fmt.Errorf("%w: %s must not be negative", ErrInvalidFilter, check.name)
+		}
+	}
+	if f.Room == nil {
+		return nil
+	}
+	for _, check := range []struct {
+		name   string
+		filter *RoomEventFilter
+	}{
+		{"room.timeline.limit", f.Room.Timeline},
+		{"room.ephemeral.limit", f.Room.Ephemeral},
+		{"room.account_data.limit", f.Room.AccountData},
+	} {
+		if check.filter != nil && check.filter.Limit != nil && *check.filter.Limit < 0 {
+			return fmt.Errorf("%w: %s must not be negative", ErrInvalidFilter, check.name)
+		}
+	}
+	return nil
+}
+
+func limitOf(f *EventFilter) *int {
+	if f == nil {
+		return nil
+	}
+	return f.Limit
+}
+
+// TimelineLimit is how many timeline events per room this filter asks for.
+func (f Filter) TimelineLimit() int {
+	if f.Room == nil || f.Room.Timeline == nil || f.Room.Timeline.Limit == nil {
+		return DefaultTimelineLimit
+	}
+	return clampLimit(*f.Room.Timeline.Limit)
+}
+
+// clampLimit holds a client's limit inside what this server will serve.
+func clampLimit(limit int) int {
+	switch {
+	case limit <= 0:
+		// Zero is a legitimate request for no timeline at all — a client
+		// that only wants state — and is honoured rather than defaulted.
+		return 0
+	case limit > MaxTimelineLimit:
+		return MaxTimelineLimit
+	}
+	return limit
+}
+
+// LazyLoadMembers reports whether the filter asks for lazy-loaded members.
+func (f Filter) LazyLoadMembers() bool {
+	return f.Room != nil && f.Room.Timeline != nil && f.Room.Timeline.LazyLoadMembers
+}
+
+// IncludeLeave reports whether the rooms the user has left should be
+// reported in full rather than only when the leave is new.
+func (f Filter) IncludeLeave() bool {
+	return f.Room != nil && f.Room.IncludeLeave
+}
+
+// TimelineFilter projects the timeline filter onto what the store applies.
+func (f Filter) TimelineFilter() storage.TimelineFilter {
+	if f.Room == nil {
+		return storage.TimelineFilter{}
+	}
+	return roomEventFilter(f.Room.Timeline)
+}
+
+// StateFilter projects the state filter onto what the roomserver applies.
+func (f Filter) StateFilter() storage.TimelineFilter {
+	if f.Room == nil {
+		return storage.TimelineFilter{}
+	}
+	return roomEventFilter(f.Room.State)
+}
+
+// EphemeralFilter projects the ephemeral filter: the typing and receipt
+// events of a room.
+func (f Filter) EphemeralFilter() storage.TimelineFilter {
+	if f.Room == nil {
+		return storage.TimelineFilter{}
+	}
+	return roomEventFilter(f.Room.Ephemeral)
+}
+
+// RoomAccountDataFilter projects the per-room account data filter.
+func (f Filter) RoomAccountDataFilter() storage.TimelineFilter {
+	if f.Room == nil {
+		return storage.TimelineFilter{}
+	}
+	return roomEventFilter(f.Room.AccountData)
+}
+
+// AccountDataFilter projects the global account data filter.
+func (f Filter) AccountDataFilter() storage.TimelineFilter {
+	return eventFilter(f.AccountData)
+}
+
+// PresenceFilter projects the presence filter.
+func (f Filter) PresenceFilter() storage.TimelineFilter {
+	return eventFilter(f.Presence)
+}
+
+func roomEventFilter(f *RoomEventFilter) storage.TimelineFilter {
+	if f == nil {
+		return storage.TimelineFilter{}
+	}
+	return storage.TimelineFilter{
+		Types:      f.Types,
+		NotTypes:   f.NotTypes,
+		Senders:    f.Senders,
+		NotSenders: f.NotSenders,
+	}
+}
+
+func eventFilter(f *EventFilter) storage.TimelineFilter {
+	if f == nil {
+		return storage.TimelineFilter{}
+	}
+	return storage.TimelineFilter{
+		Types:      f.Types,
+		NotTypes:   f.NotTypes,
+		Senders:    f.Senders,
+		NotSenders: f.NotSenders,
+	}
+}
+
+// AllowsRoom reports whether a room passes the filter's room lists.
+//
+// The lists are at two levels — room.rooms and, per section,
+// room.timeline.rooms — and both apply, with the exclusions winning, which
+// is what the spec requires.
+func (f Filter) AllowsRoom(roomID string) bool {
+	if f.Room == nil {
+		return true
+	}
+	if !allowsRoom(roomID, f.Room.Rooms, f.Room.NotRooms) {
+		return false
+	}
+	if f.Room.Timeline != nil {
+		return allowsRoom(roomID, f.Room.Timeline.Rooms, f.Room.Timeline.NotRooms)
+	}
+	return true
+}
+
+func allowsRoom(roomID string, include, exclude []string) bool {
+	for _, pattern := range exclude {
+		if matches(roomID, pattern) {
+			return false
+		}
+	}
+	if len(include) == 0 {
+		return true
+	}
+	for _, pattern := range include {
+		if matches(roomID, pattern) {
+			return true
+		}
+	}
+	return false
+}
+
+// matches applies one filter pattern, in which a trailing '*' is the
+// spec's wildcard.
+func matches(value, pattern string) bool {
+	if prefix, isWildcard := strings.CutSuffix(pattern, "*"); isWildcard {
+		return strings.HasPrefix(value, prefix)
+	}
+	return value == pattern
+}
+
+// SaveFilter stores a filter for a user and returns its ID.
+func (s *API) SaveFilter(ctx context.Context, localpart string, raw []byte) (string, error) {
+	if _, err := ParseFilter(raw); err != nil {
+		return "", err
+	}
+	return s.store.SaveFilter(ctx, localpart, raw, s.nowMS())
+}
+
+// StoredFilter returns an uploaded filter exactly as it was uploaded.
+//
+// Byte-for-byte is deliberate: a client that compares what it gets back
+// with what it sent would see a difference that is not there if the filter
+// were re-encoded through a Go struct, because the fields this build does
+// not act on would come back in a different order or not at all.
+func (s *API) StoredFilter(ctx context.Context, localpart, filterID string) ([]byte, error) {
+	return s.store.Filter(ctx, localpart, filterID)
+}
+
+// filterFor resolves the ?filter= parameter: a filter ID, an inline filter
+// as JSON, or nothing.
+func (s *API) filterFor(ctx context.Context, localpart, raw string) (Filter, error) {
+	switch {
+	case raw == "":
+		return Filter{}, nil
+	case strings.HasPrefix(raw, "{"):
+		return ParseFilter([]byte(raw))
+	}
+
+	stored, err := s.store.Filter(ctx, localpart, raw)
+	if err != nil {
+		return Filter{}, err
+	}
+	return ParseFilter(stored)
+}

server/internal/syncapi/filter_test.go

@@ -0,0 +1,144 @@
+package syncapi_test
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
+)
+
+func TestParseFilterReadsTheFieldsThatMatter(t *testing.T) {
+	filter, err := syncapi.ParseFilter([]byte(`{
+		"room": {
+			"timeline": {"limit": 5, "types": ["m.room.message"], "lazy_load_members": true},
+			"include_leave": true
+		}
+	}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+
+	if got := filter.TimelineLimit(); got != 5 {
+		t.Errorf("TimelineLimit() = %d, want 5", got)
+	}
+	if !filter.LazyLoadMembers() {
+		t.Error("LazyLoadMembers() = false, want true")
+	}
+	if !filter.IncludeLeave() {
+		t.Error("IncludeLeave() = false, want true")
+	}
+	if types := filter.TimelineFilter().Types; len(types) != 1 || types[0] != "m.room.message" {
+		t.Errorf("timeline types = %v, want [m.room.message]", types)
+	}
+}
+
+func TestFilterDefaultsAreTheSpecsOwn(t *testing.T) {
+	var filter syncapi.Filter
+
+	if got := filter.TimelineLimit(); got != syncapi.DefaultTimelineLimit {
+		t.Errorf("TimelineLimit() = %d, want the default %d", got, syncapi.DefaultTimelineLimit)
+	}
+	if filter.LazyLoadMembers() {
+		t.Error("lazy loading must be off unless asked for")
+	}
+	if filter.IncludeLeave() {
+		t.Error("left rooms must be left out unless asked for")
+	}
+	if !filter.AllowsRoom("!anything:warp.example") {
+		t.Error("an empty filter must allow every room")
+	}
+}
+
+func TestFilterClampsAnAbsurdLimit(t *testing.T) {
+	filter, err := syncapi.ParseFilter([]byte(`{"room":{"timeline":{"limit":1000000}}}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+	if got := filter.TimelineLimit(); got != syncapi.MaxTimelineLimit {
+		t.Errorf("TimelineLimit() = %d, want it capped at %d", got, syncapi.MaxTimelineLimit)
+	}
+}
+
+func TestFilterHonoursAZeroLimit(t *testing.T) {
+	// Zero is a legitimate request for no timeline at all — a client that
+	// only wants state — and must not be read as "unset".
+	filter, err := syncapi.ParseFilter([]byte(`{"room":{"timeline":{"limit":0}}}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+	if got := filter.TimelineLimit(); got != 0 {
+		t.Errorf("TimelineLimit() = %d, want 0", got)
+	}
+}
+
+func TestFilterRoomLists(t *testing.T) {
+	filter, err := syncapi.ParseFilter([]byte(`{
+		"room": {"rooms": ["!kept:warp.example"], "not_rooms": ["!dropped:warp.example"]}
+	}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+
+	for _, tc := range []struct {
+		roomID string
+		want   bool
+	}{
+		{"!kept:warp.example", true},
+		{"!dropped:warp.example", false},
+		{"!unmentioned:warp.example", false},
+	} {
+		if got := filter.AllowsRoom(tc.roomID); got != tc.want {
+			t.Errorf("AllowsRoom(%q) = %v, want %v", tc.roomID, got, tc.want)
+		}
+	}
+}
+
+func TestFilterExclusionsWinOverInclusions(t *testing.T) {
+	// The spec is explicit: a value matching both lists is excluded.
+	filter, err := syncapi.ParseFilter([]byte(`{
+		"room": {"rooms": ["!both:warp.example"], "not_rooms": ["!both:warp.example"]}
+	}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+	if filter.AllowsRoom("!both:warp.example") {
+		t.Error("a room in both lists must be excluded")
+	}
+}
+
+func TestFilterRejectsAnUnsupportedEventFormat(t *testing.T) {
+	// The federation format is a server's view of an event, and the
+	// Client-Server API has no business handing it out.
+	_, err := syncapi.ParseFilter([]byte(`{"event_format":"federation"}`))
+	if !errors.Is(err, syncapi.ErrInvalidFilter) {
+		t.Errorf("error = %v, want ErrInvalidFilter", err)
+	}
+}
+
+func TestFilterRejectsNonsense(t *testing.T) {
+	for _, raw := range []string{
+		`not json`,
+		`[]`,
+		`{"room":{"timeline":{"limit":-1}}}`,
+	} {
+		if _, err := syncapi.ParseFilter([]byte(raw)); !errors.Is(err, syncapi.ErrInvalidFilter) {
+			t.Errorf("ParseFilter(%q) error = %v, want ErrInvalidFilter", raw, err)
+		}
+	}
+}
+
+func TestFilterIgnoresFieldsThisServerDoesNotActOn(t *testing.T) {
+	// A server is allowed to return a superset of what a filter asked
+	// for, and a client that uploads `contains_url` must still be able to
+	// sync. Refusing the filter would be the one outcome that breaks it.
+	filter, err := syncapi.ParseFilter([]byte(`{
+		"event_fields": ["type", "content"],
+		"room": {"timeline": {"contains_url": true, "unread_thread_notifications": true}}
+	}`))
+	if err != nil {
+		t.Fatalf("ParseFilter: %v", err)
+	}
+	if got := filter.TimelineLimit(); got != syncapi.DefaultTimelineLimit {
+		t.Errorf("TimelineLimit() = %d, want the default", got)
+	}
+}

server/internal/syncapi/messages.go

@@ -0,0 +1,210 @@
+package syncapi
+
+import (
+	"context"
+
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// GET /rooms/{roomID}/messages: a room's history, a page at a time.
+//
+// It is the other half of /sync. A sync hands a client the live stream and,
+// when a client has been away, a `limited` timeline with a gap in it;
+// /messages is how the client closes that gap, and how it scrolls back
+// through a room it has just opened. The tokens are the same ones — a sync's
+// prev_batch is a valid ?from= here, and an ?end= from here is a valid
+// ?since= for /sync — which is what makes the two compose.
+
+// MessagesRequest is one page of a room's history.
+type MessagesRequest struct {
+	RoomID string
+
+	// User is the full user ID asking, and Localpart is their localpart,
+	// for resolving a filter ID.
+	User      string
+	Localpart string
+
+	// From is where to page from. Empty means the end of the room for a
+	// backwards page and the start of it for a forwards one, which is what
+	// a client opening a room sends.
+	From string
+
+	// To bounds the far end of the page. Empty is unbounded.
+	To string
+
+	// Direction is "b" to page backwards, which is what scrolling up does,
+	// or "f" to page forwards. The spec requires it.
+	Direction storage.TimelineDirection
+
+	// Limit caps the events returned.
+	Limit int
+
+	// Filter is the ?filter= parameter: a filter ID or an inline
+	// RoomEventFilter as JSON.
+	Filter string
+}
+
+// MessagesResponse is one page.
+type MessagesResponse struct {
+	// Start is the token this page began at, which is the ?from= the
+	// client sent back to it.
+	Start string `json:"start"`
+
+	// End is the token for the next page. It is left out when the page
+	// reached the end of what the client may see, which is how a client
+	// knows to stop scrolling.
+	End string `json:"end,omitempty"`
+
+	Chunk []roomserver.ClientEvent `json:"chunk"`
+
+	// State is the state of the events in the chunk: the member events of
+	// their senders, when the filter asked for lazy loading. A client
+	// needs them to render a name against a message from somebody who is
+	// no longer in the room.
+	State []roomserver.ClientEvent `json:"state,omitempty"`
+}
+
+// Messages answers one page of a room's history.
+func (s *API) Messages(ctx context.Context, req MessagesRequest) (MessagesResponse, error) {
+	from, err := ParseToken(req.From)
+	if err != nil {
+		return MessagesResponse{}, errInvalidTokenf("from: %v", err)
+	}
+	to, err := ParseToken(req.To)
+	if err != nil {
+		return MessagesResponse{}, errInvalidTokenf("to: %v", err)
+	}
+
+	filter, err := s.messageFilter(ctx, req.Localpart, req.Filter)
+	if err != nil {
+		return MessagesResponse{}, err
+	}
+
+	direction := req.Direction
+	if direction != storage.DirectionForward {
+		// Backwards is the default: the spec makes `dir` required, but
+		// every client that leaves it out means "scroll up".
+		direction = storage.DirectionBackward
+	}
+
+	limit := req.Limit
+	if limit <= 0 {
+		limit = DefaultTimelineLimit
+	}
+	limit = min(limit, MaxTimelineLimit)
+
+	// Which end of the window the client's two tokens are depends on which
+	// way it is paging: scrolling up, `from` is the ceiling and `to` the
+	// floor, and scrolling down it is the other way round. An unset bound
+	// is zero, which the read already takes as "unbounded at that end" —
+	// so paging backwards from the end of the room needs nothing special.
+	low, high := from.PDU, to.PDU
+	if direction == storage.DirectionBackward {
+		low, high = to.PDU, from.PDU
+	}
+
+	timeline, err := s.rooms.Timeline(ctx, roomserver.TimelineRequest{
+		RoomID:    req.RoomID,
+		User:      req.User,
+		Low:       low,
+		High:      high,
+		Direction: direction,
+		Limit:     limit,
+		Filter:    filter.TimelineFilter(),
+	})
+	if err != nil {
+		return MessagesResponse{}, err
+	}
+
+	// The chunk is in the direction asked for: the spec says a backwards
+	// page is in reverse chronological order, because that is the order a
+	// client scrolling up consumes it in. The read returns events oldest
+	// first — which is what /sync wants — so a backwards page is turned
+	// round here.
+	chunk := timeline.Events
+	if direction == storage.DirectionBackward {
+		chunk = reversed(chunk)
+	}
+	if chunk == nil {
+		chunk = []roomserver.ClientEvent{}
+	}
+
+	out := MessagesResponse{
+		Start: pduToken(from.PDU).String(),
+		Chunk: chunk,
+	}
+	// `from` is optional, and without it `start` has to name the end of
+	// the room the page actually began at rather than position zero: a
+	// client that paged backwards from the end and then used `start` to
+	// page forwards again would otherwise be handed the whole room.
+	if req.From == "" {
+		if direction == storage.DirectionBackward {
+			out.Start = pduToken(timeline.End).String()
+		} else {
+			out.Start = pduToken(timeline.Start).String()
+		}
+	}
+
+	// `end` is left out at the end of the room, which is how a client
+	// knows to stop asking. Reading backwards the far end is Start, the
+	// position below the oldest event; reading forwards it is End.
+	//
+	// The read fetches one more row than the limit, so `Limited` is the
+	// whole answer to "is there more": a page that came back exactly full
+	// and happened to be the last one still correctly has no `end`.
+	if timeline.Limited {
+		if direction == storage.DirectionBackward {
+			out.End = pduToken(timeline.Start).String()
+		} else {
+			out.End = pduToken(timeline.End).String()
+		}
+	}
+
+	if filter.LazyLoadMembers() {
+		state, err := s.rooms.MemberEvents(ctx, req.RoomID, sendersOf(chunk))
+		if err != nil {
+			return MessagesResponse{}, err
+		}
+		out.State = state
+	}
+	return out, nil
+}
+
+// messageFilter resolves the ?filter= of /messages.
+//
+// The parameter is a RoomEventFilter here rather than the whole filter that
+// /sync takes — the spec defines it that way, because a filter about
+// presence or account data would mean nothing to a room's history. A filter
+// ID, though, names a full filter, so both shapes have to be accepted: an
+// inline one is wrapped into the room's timeline slot, and a stored one is
+// used as it stands.
+func (s *API) messageFilter(ctx context.Context, localpart, raw string) (Filter, error) {
+	if raw == "" {
+		return Filter{}, nil
+	}
+	if raw[0] != '{' {
+		return s.filterFor(ctx, localpart, raw)
+	}
+
+	var roomEvents RoomEventFilter
+	if err := unmarshalStrict([]byte(raw), &roomEvents); err != nil {
+		return Filter{}, errInvalidParamf("filter: %v", err)
+	}
+	return Filter{Room: &RoomFilter{Timeline: &roomEvents}}, nil
+}
+
+// sendersOf lists the distinct senders of a chunk, in the order they first
+// appear.
+func sendersOf(events []roomserver.ClientEvent) []string {
+	seen := make(map[string]struct{}, len(events))
+	out := make([]string, 0, len(events))
+	for _, e := range events {
+		if _, dup := seen[e.Sender]; dup {
+			continue
+		}
+		seen[e.Sender] = struct{}{}
+		out = append(out, e.Sender)
+	}
+	return out
+}

server/internal/syncapi/notifier.go

@@ -0,0 +1,207 @@
+package syncapi
+
+import (
+	"context"
+	"sync"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+)
+
+// Notifier wakes the long-polling syncs that something concerns.
+//
+// It is the other half of the roomserver's emit stage: the input pipeline
+// is the only thing that knows an event has landed, and a client blocked in
+// /sync for thirty seconds has to hear about it in milliseconds rather than
+// when its timeout expires. Polling the database instead would mean every
+// connected client querying every second, which is the cost this avoids.
+//
+// Waking is by room and by user rather than globally. A sync registers the
+// rooms it just read plus its own user ID; an event wakes the listeners of
+// its room, and a membership event also wakes the listener of the user it
+// names — which is how an invite reaches a client that is not in the room
+// yet and so is not listening for it. An event in a room nobody is polling
+// wakes nobody.
+//
+// A wake-up is only ever a hint. The listener re-reads the streams from the
+// database and works out for itself what changed, so a spurious wake-up
+// costs one query and a missed one costs at most the sync timeout. That is
+// what makes it safe for this to hold no state about what each client has
+// seen.
+type Notifier struct {
+	mu    sync.Mutex
+	rooms map[string]map[*Listener]struct{}
+	users map[string]map[*Listener]struct{}
+}
+
+// NewNotifier returns a notifier with no listeners.
+func NewNotifier() *Notifier {
+	return &Notifier{
+		rooms: map[string]map[*Listener]struct{}{},
+		users: map[string]map[*Listener]struct{}{},
+	}
+}
+
+// Listener is one blocked sync's wake-up channel.
+type Listener struct {
+	notifier *Notifier
+	wake     chan struct{}
+
+	userID string
+	rooms  []string
+}
+
+// Listen registers interest in a user and a set of rooms.
+//
+// The caller must Close the listener. Registering before reading the
+// streams is what closes the race: an event that lands between the
+// registration and the read leaves the channel already signalled, so the
+// following Wait returns at once and the sync reads again rather than
+// blocking on a change it just missed.
+func (n *Notifier) Listen(userID string, roomIDs []string) *Listener {
+	l := &Listener{
+		notifier: n,
+		// One slot, so a wake-up is never lost and never queued: the
+		// listener re-reads everything anyway, so two wake-ups and one are
+		// the same instruction.
+		wake:   make(chan struct{}, 1),
+		userID: userID,
+		rooms:  roomIDs,
+	}
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	if userID != "" {
+		add(n.users, userID, l)
+	}
+	for _, roomID := range roomIDs {
+		add(n.rooms, roomID, l)
+	}
+	return l
+}
+
+// Wait returns the channel a wake-up arrives on.
+func (l *Listener) Wait() <-chan struct{} { return l.wake }
+
+// Close deregisters the listener. It is safe to call more than once.
+func (l *Listener) Close() {
+	if l == nil || l.notifier == nil {
+		return
+	}
+	n := l.notifier
+	l.notifier = nil
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	if l.userID != "" {
+		remove(n.users, l.userID, l)
+	}
+	for _, roomID := range l.rooms {
+		remove(n.rooms, roomID, l)
+	}
+}
+
+// EventPersisted implements roomserver.Notifier: it is called once per
+// event the input pipeline has committed, in the order they were committed.
+//
+// It must not block. The call happens with the room's write lock still
+// held, so anything slow here would serialise the room's writers behind the
+// slowest reader of it — which is why waking is a non-blocking send onto a
+// one-slot channel and nothing else.
+func (n *Notifier) EventPersisted(_ context.Context, notification roomserver.Notification) {
+	n.NotifyRoom(notification.RoomID)
+
+	// A membership event also concerns the user it names, who for an
+	// invite is not in the room and so is not listening for it.
+	e := notification.Event
+	if e == nil || e.Type() != event.TypeMember {
+		return
+	}
+	if stateKey, ok := e.StateKey(); ok && stateKey != "" {
+		n.NotifyUser(stateKey)
+	}
+}
+
+// NotifyRoom wakes the listeners of one room. It is what the ephemeral
+// streams — typing and receipts — use, since those are about a room but are
+// not events.
+func (n *Notifier) NotifyRoom(roomID string) {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	wakeAll(n.rooms[roomID])
+}
+
+// NotifyUser wakes the listeners of one user: their to-device messages,
+// their account data and the invites addressed to them.
+func (n *Notifier) NotifyUser(userID string) {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	wakeAll(n.users[userID])
+}
+
+// NotifyUsers wakes several users at once, under one lock.
+func (n *Notifier) NotifyUsers(userIDs []string) {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	for _, userID := range userIDs {
+		wakeAll(n.users[userID])
+	}
+}
+
+// Listeners reports how many syncs are currently blocked, for metrics and
+// for the tests that would otherwise have to guess.
+func (n *Notifier) Listeners() int {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+
+	seen := map[*Listener]struct{}{}
+	for _, set := range n.rooms {
+		for l := range set {
+			seen[l] = struct{}{}
+		}
+	}
+	for _, set := range n.users {
+		for l := range set {
+			seen[l] = struct{}{}
+		}
+	}
+	return len(seen)
+}
+
+// wakeAll signals every listener in a set without blocking on any of them.
+func wakeAll(set map[*Listener]struct{}) {
+	for l := range set {
+		select {
+		case l.wake <- struct{}{}:
+		default:
+			// Already signalled and not yet read: the listener will see
+			// this change when it acts on the signal it has.
+		}
+	}
+}
+
+func add(index map[string]map[*Listener]struct{}, key string, l *Listener) {
+	set, ok := index[key]
+	if !ok {
+		set = map[*Listener]struct{}{}
+		index[key] = set
+	}
+	set[l] = struct{}{}
+}
+
+func remove(index map[string]map[*Listener]struct{}, key string, l *Listener) {
+	set, ok := index[key]
+	if !ok {
+		return
+	}
+	delete(set, l)
+	if len(set) == 0 {
+		// An empty set left behind would make the index grow with every
+		// room any client has ever polled.
+		delete(index, key)
+	}
+}
+
+// Compile-time proof that the notifier is what the roomserver's emit stage
+// expects.
+var _ roomserver.Notifier = (*Notifier)(nil)

server/internal/syncapi/notifier_test.go

@@ -0,0 +1,150 @@
+package syncapi_test
+
+import (
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
+)
+
+// The notifier is the one piece of this component that several goroutines
+// touch at once, so it is tested on its own rather than only through a
+// blocked /sync.
+
+// woken reports whether a listener has been signalled, without blocking.
+func woken(l *syncapi.Listener) bool {
+	select {
+	case <-l.Wait():
+		return true
+	default:
+		return false
+	}
+}
+
+func TestNotifierWakesOnlyWhoIsConcerned(t *testing.T) {
+	n := syncapi.NewNotifier()
+
+	inRoom := n.Listen("@alice:warp.example", []string{"!a:warp.example"})
+	elsewhere := n.Listen("@bob:warp.example", []string{"!b:warp.example"})
+	t.Cleanup(inRoom.Close)
+	t.Cleanup(elsewhere.Close)
+
+	n.NotifyRoom("!a:warp.example")
+
+	if !woken(inRoom) {
+		t.Error("a listener on the room was not woken")
+	}
+	if woken(elsewhere) {
+		t.Error("a listener on another room was woken")
+	}
+}
+
+func TestNotifierWakesByUser(t *testing.T) {
+	n := syncapi.NewNotifier()
+
+	// A user who is in no rooms at all still has to be reachable: an
+	// invite, a to-device message and account data all arrive without any
+	// room they are listening to.
+	listener := n.Listen("@alice:warp.example", nil)
+	t.Cleanup(listener.Close)
+
+	n.NotifyUser("@alice:warp.example")
+	if !woken(listener) {
+		t.Error("a listener was not woken by a notification addressed to its user")
+	}
+}
+
+func TestNotifierCoalescesWakeups(t *testing.T) {
+	n := syncapi.NewNotifier()
+	listener := n.Listen("@alice:warp.example", []string{"!a:warp.example"})
+	t.Cleanup(listener.Close)
+
+	// A listener re-reads everything when it wakes, so two wake-ups and
+	// one are the same instruction. What matters is that the second does
+	// not block the notifier.
+	for range 100 {
+		n.NotifyRoom("!a:warp.example")
+	}
+
+	if !woken(listener) {
+		t.Fatal("the listener was not woken at all")
+	}
+	if woken(listener) {
+		t.Error("wake-ups queued up instead of coalescing")
+	}
+}
+
+func TestNotifierForgetsAClosedListener(t *testing.T) {
+	n := syncapi.NewNotifier()
+
+	listener := n.Listen("@alice:warp.example", []string{"!a:warp.example"})
+	if n.Listeners() != 1 {
+		t.Fatalf("Listeners() = %d, want 1", n.Listeners())
+	}
+
+	listener.Close()
+	if n.Listeners() != 0 {
+		t.Errorf("Listeners() = %d after Close, want 0", n.Listeners())
+	}
+	// Closing twice must not panic or double-remove: a sync closes its
+	// listener on every path out, and some of those paths overlap.
+	listener.Close()
+
+	// Nothing is left behind to be woken, and notifying an unknown room is
+	// not an error.
+	n.NotifyRoom("!a:warp.example")
+	n.NotifyUser("@alice:warp.example")
+}
+
+func TestNotifierUnderConcurrentUse(t *testing.T) {
+	n := syncapi.NewNotifier()
+
+	// Listeners coming and going while notifications arrive is the normal
+	// state of a server: every /sync registers, reads and deregisters.
+	// Run under `go test -race` this is the test that would catch a lock
+	// held in the wrong place.
+	var wg sync.WaitGroup
+	stop := make(chan struct{})
+
+	for i := range 8 {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			for {
+				select {
+				case <-stop:
+					return
+				default:
+				}
+				l := n.Listen("@alice:warp.example", []string{"!a:warp.example"})
+				_ = woken(l)
+				l.Close()
+			}
+		}(i)
+	}
+	for range 4 {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for {
+				select {
+				case <-stop:
+					return
+				default:
+				}
+				n.NotifyRoom("!a:warp.example")
+				n.NotifyUser("@alice:warp.example")
+				n.NotifyUsers([]string{"@alice:warp.example", "@bob:warp.example"})
+			}
+		}()
+	}
+
+	time.Sleep(50 * time.Millisecond)
+	close(stop)
+	wg.Wait()
+
+	if n.Listeners() != 0 {
+		t.Errorf("Listeners() = %d after everything closed, want 0", n.Listeners())
+	}
+}

server/internal/syncapi/presence.go

@@ -0,0 +1,251 @@
+package syncapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"maps"
+	"slices"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// Presence: who is online, and the status message they set.
+//
+// It is off by default (PLAN.md open question 3). Presence is the single
+// noisiest thing in Matrix — every user's state change is sent to everybody
+// who shares a room with them, which over federation is quadratic in the
+// worst case, and Synapse deployments routinely turn it off for exactly
+// that reason. A private homeserver with a dozen users has no such problem,
+// so it is a switch rather than a decision.
+//
+// With it off, the endpoints answer M_FORBIDDEN and the presence section of
+// a sync response is empty. They are not unregistered: a client that gets
+// a 404 from /presence concludes the server is too old to have it and
+// stops asking, whereas a refusal says this server has it switched off.
+
+// The presence states the spec defines.
+const (
+	PresenceOnline      = "online"
+	PresenceOffline     = "offline"
+	PresenceUnavailable = "unavailable"
+)
+
+// maxStatusMsgLength caps a status message. The spec sets no limit; this is
+// long enough for a sentence and short enough that it cannot be used as
+// storage.
+const maxStatusMsgLength = 512
+
+// PresenceRequest is PUT /presence/{userID}/status.
+type PresenceRequest struct {
+	// User is the full user ID whose presence is being set, which the
+	// handler has already checked is the caller.
+	User string
+
+	Presence string
+
+	// StatusMsg is the free-text status. An explicitly empty string
+	// clears it, which is why this is a pointer: a request that does not
+	// mention it leaves it alone.
+	StatusMsg *string
+}
+
+// SetPresence records a user's presence.
+func (s *API) SetPresence(ctx context.Context, req PresenceRequest) error {
+	if !s.presenceEnabled {
+		return ErrPresenceDisabled
+	}
+	switch req.Presence {
+	case PresenceOnline, PresenceOffline, PresenceUnavailable:
+	default:
+		return errInvalidParamf("%q is not a presence state", req.Presence)
+	}
+	if req.StatusMsg != nil && len(*req.StatusMsg) > maxStatusMsgLength {
+		return errInvalidParamf("a status message may be at most %d bytes", maxStatusMsgLength)
+	}
+
+	state := storage.PresenceState{
+		UserID:       req.User,
+		Presence:     req.Presence,
+		LastActiveMS: s.nowMS(),
+	}
+	// Keep the status message a request did not mention.
+	if req.StatusMsg != nil {
+		state.StatusMsg = *req.StatusMsg
+	} else {
+		switch existing, err := s.store.Presence(ctx, req.User); {
+		case err == nil:
+			state.StatusMsg = existing.StatusMsg
+		case !errors.Is(err, storage.ErrNoSuchUser):
+			return err
+		}
+	}
+
+	_, moved, err := s.store.SetPresence(ctx, state)
+	if err != nil {
+		return err
+	}
+	// Nothing a peer would notice changed, so there is nobody to wake.
+	if !moved {
+		return nil
+	}
+	return s.notifyPresence(ctx, req.User)
+}
+
+// Presence reads a user's presence, which GET /presence/{userID}/status
+// answers.
+//
+// A user nothing has ever been recorded for is offline rather than unknown.
+// That is what the spec says to report and it is also the truth: a user who
+// has never told the server they were online is not online.
+func (s *API) Presence(ctx context.Context, userID string) (storage.PresenceState, error) {
+	if !s.presenceEnabled {
+		return storage.PresenceState{}, ErrPresenceDisabled
+	}
+	if !s.isLocal(userID) {
+		// Remote presence needs federation (Phase 5). Reporting offline
+		// would be a guess; saying so is not.
+		return storage.PresenceState{}, errForbiddenf(
+			"%s belongs to another server and federation is required to ask it", userID)
+	}
+
+	switch state, err := s.store.Presence(ctx, userID); {
+	case err == nil:
+		return state, nil
+	case errors.Is(err, storage.ErrNoSuchUser):
+		return storage.PresenceState{UserID: userID, Presence: PresenceOffline}, nil
+	default:
+		return storage.PresenceState{}, err
+	}
+}
+
+// notifyPresence wakes the users who share a room with this one.
+//
+// Presence is published to everybody a user shares a room with, which is
+// the set this walks: their rooms, and the members of each. It is the one
+// genuinely fan-out notification in the component, and the reason presence
+// is a switch.
+func (s *API) notifyPresence(ctx context.Context, userID string) error {
+	rooms, err := s.rooms.JoinedRooms(ctx, userID)
+	if err != nil {
+		return err
+	}
+
+	peers := map[string]struct{}{}
+	for _, roomID := range rooms {
+		members, err := s.rooms.JoinedMembers(ctx, roomID, userID)
+		if err != nil {
+			return err
+		}
+		for member := range members {
+			peers[member] = struct{}{}
+		}
+	}
+	delete(peers, userID)
+
+	s.notifier.NotifyUsers(slices.Sorted(maps.Keys(peers)))
+	// Their own other devices want it too.
+	s.notifier.NotifyUser(userID)
+	return nil
+}
+
+// presenceEvents reads the presence of everybody the user shares a room
+// with, between two positions.
+func (s *API) presenceEvents(ctx context.Context, userID string, rooms []string, from, to int64, filter storage.TimelineFilter) ([]presenceEvent, error) {
+	if !s.presenceEnabled || len(rooms) == 0 {
+		return nil, nil
+	}
+
+	peers := map[string]struct{}{userID: {}}
+	for _, roomID := range rooms {
+		entries, err := s.rooms.JoinedMembers(ctx, roomID, userID)
+		if err != nil {
+			// A room the user is no longer joined to is not an error here:
+			// their presence section simply does not include its members.
+			if errors.Is(err, ErrForbidden) {
+				continue
+			}
+			return nil, err
+		}
+		for member := range entries {
+			peers[member] = struct{}{}
+		}
+	}
+
+	states, err := s.store.PresenceForUsers(ctx, slices.Sorted(maps.Keys(peers)), from, to)
+	if err != nil {
+		return nil, err
+	}
+
+	out := make([]presenceEvent, 0, len(states))
+	for _, state := range states {
+		e, err := newPresenceEvent(state, s.nowMS())
+		if err != nil {
+			return nil, err
+		}
+		if !matchesEventFilter(e.Type, e.Sender, filter) {
+			continue
+		}
+		out = append(out, e)
+	}
+	return out, nil
+}
+
+// presenceEvent is the m.presence EDU of one user.
+type presenceEvent struct {
+	Type    string          `json:"type"`
+	Sender  string          `json:"sender"`
+	Content json.RawMessage `json:"content"`
+}
+
+// newPresenceEvent builds the EDU for one user's state.
+//
+// `last_active_ago` is computed on the way out rather than stored, for the
+// same reason `age` is on an event: a value written down at write time
+// would be wrong by however long the row sat in the database.
+func newPresenceEvent(state storage.PresenceState, nowMS int64) (presenceEvent, error) {
+	content := map[string]any{
+		"presence":         state.Presence,
+		"currently_active": state.Presence == PresenceOnline,
+	}
+	if state.StatusMsg != "" {
+		content["status_msg"] = state.StatusMsg
+	}
+	if state.LastActiveMS > 0 {
+		content["last_active_ago"] = max(nowMS-state.LastActiveMS, 0)
+	}
+
+	body, err := json.Marshal(content)
+	if err != nil {
+		return presenceEvent{}, fmt.Errorf("marshalling the presence of %s: %w", state.UserID, err)
+	}
+	return presenceEvent{Type: "m.presence", Sender: state.UserID, Content: body}, nil
+}
+
+// matchesEventFilter is the filter check for the streams whose events are
+// not room events: presence and global account data. Their filters have the
+// same types and senders fields, and exclusions win the same way.
+func matchesEventFilter(eventType, sender string, filter storage.TimelineFilter) bool {
+	switch {
+	case anyMatch(eventType, filter.NotTypes):
+		return false
+	case sender != "" && anyMatch(sender, filter.NotSenders):
+		return false
+	case len(filter.Types) > 0 && !anyMatch(eventType, filter.Types):
+		return false
+	case len(filter.Senders) > 0 && sender != "" && !anyMatch(sender, filter.Senders):
+		return false
+	}
+	return true
+}
+
+// anyMatch applies a list of filter patterns.
+func anyMatch(value string, patterns []string) bool {
+	for _, pattern := range patterns {
+		if matches(value, pattern) {
+			return true
+		}
+	}
+	return false
+}

server/internal/syncapi/receipts.go

@@ -0,0 +1,267 @@
+package syncapi
+
+import (
+	"cmp"
+	"context"
+	"encoding/json"
+	"fmt"
+	"slices"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// The receipt types the Client-Server API defines.
+const (
+	// ReceiptRead is public: everybody in the room sees it.
+	ReceiptRead = "m.read"
+
+	// ReceiptReadPrivate is the user's own, and is reported only back to
+	// them. It exists so that a user can mark a room read without telling
+	// the room they read it.
+	ReceiptReadPrivate = "m.read.private"
+
+	// ReceiptFullyRead is not a receipt at all: it is the read marker, and
+	// is stored as the room account data type m.fully_read. It is named
+	// here because POST /read_markers takes all three in one body.
+	ReceiptFullyRead = "m.fully_read"
+)
+
+// requireJoined refuses a caller who is not in the room.
+//
+// Every write in this file is a claim about a room — "I am typing in it",
+// "I have read this far" — rather than data belonging to the caller, so
+// being in the room is the condition for all of them.
+func (s *API) requireJoined(ctx context.Context, roomID, userID string) error {
+	membership, err := s.rooms.Membership(ctx, roomID, userID)
+	if err != nil {
+		return err
+	}
+	if membership != event.MembershipJoin {
+		return errForbiddenf("%s is not joined to %s", userID, roomID)
+	}
+	return nil
+}
+
+// ReceiptRequest is POST /rooms/{roomID}/receipt/{receiptType}/{eventID}.
+type ReceiptRequest struct {
+	RoomID string
+
+	// User is the full user ID sending the receipt.
+	User string
+
+	ReceiptType string
+	EventID     string
+
+	// ThreadID is the thread the receipt is for, or empty for the room's
+	// main timeline. Threads are PLAN.md phase 10; the value is stored so
+	// that a client that sends one gets it back, rather than having its
+	// threaded receipt silently collapse onto the main timeline's.
+	ThreadID string
+}
+
+// SetReceipt records a read receipt.
+func (s *API) SetReceipt(ctx context.Context, req ReceiptRequest) error {
+	switch req.ReceiptType {
+	case ReceiptRead, ReceiptReadPrivate, ReceiptFullyRead:
+	default:
+		return errInvalidParamf("%q is not a receipt type this server accepts", req.ReceiptType)
+	}
+
+	if err := s.requireJoined(ctx, req.RoomID, req.User); err != nil {
+		return err
+	}
+
+	// The spec routes m.fully_read through this endpoint too, and it is
+	// account data rather than a receipt — but it is still a claim about a
+	// room the caller has to be in, which is why the check above comes
+	// first.
+	if req.ReceiptType == ReceiptFullyRead {
+		return s.setFullyRead(ctx, req.RoomID, req.User, req.EventID)
+	}
+
+	_, moved, err := s.store.SetReceipt(ctx, storage.Receipt{
+		RoomID:      req.RoomID,
+		UserID:      req.User,
+		ReceiptType: req.ReceiptType,
+		ThreadID:    req.ThreadID,
+		EventID:     req.EventID,
+		TimestampMS: s.nowMS(),
+	})
+	if err != nil {
+		return err
+	}
+
+	// Nothing moved means the user had already read further, and there is
+	// nothing for anybody to be woken about.
+	if !moved {
+		return nil
+	}
+	// A private receipt concerns only its sender; a public one is part of
+	// the room's ephemeral stream and everybody in it wants it.
+	if req.ReceiptType == ReceiptReadPrivate {
+		s.notifier.NotifyUser(req.User)
+		return nil
+	}
+	s.notifier.NotifyRoom(req.RoomID)
+	return nil
+}
+
+// ReadMarkersRequest is POST /rooms/{roomID}/read_markers.
+type ReadMarkersRequest struct {
+	RoomID string
+	User   string
+
+	// FullyRead is the event up to which the user has read, stored as the
+	// room account data m.fully_read. It is where a client draws the
+	// "unread from here" line, which is a different thing from a receipt:
+	// the line stays where the user left it even as they scroll past.
+	FullyRead string
+
+	// Read and ReadPrivate are ordinary receipts, sent in the same request
+	// because a client moving the marker almost always moves both.
+	Read        string
+	ReadPrivate string
+}
+
+// SetReadMarkers implements the read markers endpoint: up to three
+// independent writes from one request.
+//
+// They are not atomic, and cannot usefully be: the marker is account data
+// and the receipts are the receipt stream, which are two streams with two
+// positions. A failure partway through leaves the earlier writes in place,
+// which is the same state a client would reach by sending them separately
+// and losing its connection — and which the next call corrects.
+func (s *API) SetReadMarkers(ctx context.Context, req ReadMarkersRequest) error {
+	if err := s.requireJoined(ctx, req.RoomID, req.User); err != nil {
+		return err
+	}
+
+	if req.FullyRead != "" {
+		if err := s.setFullyRead(ctx, req.RoomID, req.User, req.FullyRead); err != nil {
+			return err
+		}
+	}
+	for _, receipt := range []struct {
+		receiptType string
+		eventID     string
+	}{
+		{ReceiptRead, req.Read},
+		{ReceiptReadPrivate, req.ReadPrivate},
+	} {
+		if receipt.eventID == "" {
+			continue
+		}
+		if err := s.SetReceipt(ctx, ReceiptRequest{
+			RoomID:      req.RoomID,
+			User:        req.User,
+			ReceiptType: receipt.receiptType,
+			EventID:     receipt.eventID,
+		}); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// setFullyRead writes the read marker as room account data.
+func (s *API) setFullyRead(ctx context.Context, roomID, userID, eventID string) error {
+	localpart, ok := s.localpartOf(userID)
+	if !ok {
+		return errForbiddenf("%s is not a user of this server", userID)
+	}
+	content, err := json.Marshal(map[string]string{"event_id": eventID})
+	if err != nil {
+		return fmt.Errorf("marshalling the read marker: %w", err)
+	}
+	if err := s.accounts.SetAccountData(ctx, localpart, roomID, ReceiptFullyRead, content); err != nil {
+		return err
+	}
+	s.notifier.NotifyUser(userID)
+	return nil
+}
+
+// receiptsEvent builds the m.receipt EDU of one room.
+//
+// The shape is the spec's, and it is an awkward one: a map of event ID to
+// receipt type to user ID to the receipt's data. It is nested that way
+// because a client's job is to answer "who has read up to this event",
+// which this shape makes a lookup and a flat list would make a scan.
+func receiptsEvent(receipts []storage.Receipt) *EphemeralEvent {
+	if len(receipts) == 0 {
+		return nil
+	}
+
+	type receiptData struct {
+		TS       int64  `json:"ts"`
+		ThreadID string `json:"thread_id,omitempty"`
+	}
+	content := map[string]map[string]map[string]receiptData{}
+	for _, r := range receipts {
+		byType, ok := content[r.EventID]
+		if !ok {
+			byType = map[string]map[string]receiptData{}
+			content[r.EventID] = byType
+		}
+		byUser, ok := byType[r.ReceiptType]
+		if !ok {
+			byUser = map[string]receiptData{}
+			byType[r.ReceiptType] = byUser
+		}
+		byUser[r.UserID] = receiptData{TS: r.TimestampMS, ThreadID: r.ThreadID}
+	}
+
+	body, err := json.Marshal(content)
+	if err != nil {
+		// Every value here is a string or an integer, so this cannot
+		// happen; dropping the EDU beats failing the sync if it does.
+		return nil
+	}
+	return &EphemeralEvent{Type: "m.receipt", Content: body}
+}
+
+// roomReceipts reads the receipts a user is entitled to see in a set of
+// rooms, between two positions.
+//
+// Private receipts are filtered here rather than in SQL: the query is the
+// same one the public receipts come from, and a WHERE clause that had to
+// name the asking user would turn one query per sync into two.
+func (s *API) roomReceipts(ctx context.Context, userID string, roomIDs []string, from, to int64) (map[string][]storage.Receipt, error) {
+	if len(roomIDs) == 0 {
+		return nil, nil
+	}
+	receipts, err := s.store.Receipts(ctx, roomIDs, from, to)
+	if err != nil {
+		return nil, err
+	}
+	// An initial sync also wants the user's own receipts however old they
+	// are: a client that has just logged in has to know where it had read
+	// up to, and that receipt was set long before this sync's window.
+	if from == 0 {
+		own, err := s.store.ReceiptsForUser(ctx, userID, roomIDs)
+		if err != nil {
+			return nil, err
+		}
+		receipts = append(receipts, own...)
+	}
+
+	out := map[string][]storage.Receipt{}
+	seen := map[storage.Receipt]struct{}{}
+	for _, r := range receipts {
+		if r.ReceiptType == ReceiptReadPrivate && r.UserID != userID {
+			continue
+		}
+		if _, dup := seen[r]; dup {
+			// The user's own receipts can arrive from both reads above.
+			continue
+		}
+		seen[r] = struct{}{}
+		out[r.RoomID] = append(out[r.RoomID], r)
+	}
+	for roomID := range out {
+		slices.SortFunc(out[roomID], func(a, b storage.Receipt) int {
+			return cmp.Compare(a.StreamPos, b.StreamPos)
+		})
+	}
+	return out, nil
+}

server/internal/syncapi/sync.go

@@ -0,0 +1,759 @@
+package syncapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"slices"
+	"time"
+
+	"github.com/dariuszjaniak/warp/server/internal/matrix/event"
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// GET /sync: the endpoint a client spends its life in.
+//
+// The shape of the work is: fix the top of every stream, so that the whole
+// response describes one moment; work out which rooms the user's view of
+// has changed; read each of them; read the streams that are not about a
+// room; and hand back a token naming where it all got to. If nothing had
+// changed and the client asked to wait, block on the notifier and do it
+// again.
+
+// SyncRequest is one call to /sync.
+type SyncRequest struct {
+	// User is the full user ID syncing; Localpart and Device identify the
+	// session, because a to-device message is addressed to a device rather
+	// than to a user.
+	User      string
+	Localpart string
+	Device    string
+
+	// Since is the ?since= token. Empty is an initial sync.
+	Since string
+
+	// Filter is the ?filter= parameter: a filter ID, an inline filter as
+	// JSON, or empty.
+	Filter string
+
+	// Timeout is how long to wait for something to happen before
+	// answering with nothing. Zero returns immediately.
+	Timeout time.Duration
+
+	// FullState asks for every room's whole state even on an incremental
+	// sync.
+	FullState bool
+
+	// SetPresence is the ?set_presence= parameter: a client says it is
+	// online by syncing, and says it has gone away by syncing with
+	// set_presence=unavailable.
+	SetPresence string
+}
+
+// Response is the body of a /sync.
+type Response struct {
+	NextBatch string `json:"next_batch"`
+
+	Rooms       Rooms           `json:"rooms"`
+	Presence    PresenceList    `json:"presence"`
+	AccountData AccountDataList `json:"account_data"`
+	ToDevice    ToDeviceList    `json:"to_device"`
+
+	// DeviceLists reports whose device list changed, which a client needs
+	// in order to re-fetch their keys. The keyserver is PLAN.md phase 7,
+	// so both lists are always empty; the block is here all the same,
+	// because a client that finds no device_lists treats the response as
+	// malformed rather than as "nothing changed".
+	DeviceLists DeviceLists `json:"device_lists"`
+}
+
+// Rooms is the room half of a sync, split by the user's membership.
+type Rooms struct {
+	Join   map[string]JoinedRoom  `json:"join"`
+	Invite map[string]InvitedRoom `json:"invite"`
+	Leave  map[string]LeftRoom    `json:"leave"`
+	Knock  map[string]KnockedRoom `json:"knock"`
+}
+
+// JoinedRoom is a room the user is in.
+type JoinedRoom struct {
+	// Summary is left out for a room that has a name of its own: it exists
+	// so that a client can name a room that does not.
+	Summary *roomserver.RoomSyncSummary `json:"summary,omitempty"`
+
+	State       EventList          `json:"state"`
+	Timeline    Timeline           `json:"timeline"`
+	Ephemeral   EphemeralEventList `json:"ephemeral"`
+	AccountData AccountDataList    `json:"account_data"`
+}
+
+// InvitedRoom is a room the user has been invited to. It carries no
+// timeline at all: the invitee is not a member, and the stripped state is
+// everything they are entitled to.
+type InvitedRoom struct {
+	InviteState StrippedStateList `json:"invite_state"`
+}
+
+// LeftRoom is a room the user has left, reported so that a client can
+// notice the leave and close the room.
+type LeftRoom struct {
+	State       EventList       `json:"state"`
+	Timeline    Timeline        `json:"timeline"`
+	AccountData AccountDataList `json:"account_data"`
+}
+
+// KnockedRoom is a room the user has knocked on. Knocking needs no stream
+// of its own; the stripped state is what tells the client which room it is
+// waiting on.
+type KnockedRoom struct {
+	KnockState StrippedStateList `json:"knock_state"`
+}
+
+// Timeline is a room's events in this window.
+type Timeline struct {
+	Events []roomserver.ClientEvent `json:"events"`
+
+	// Limited says there is a gap between PrevBatch and what the client
+	// already had: it has to page backwards through /messages to close it.
+	Limited bool `json:"limited"`
+
+	// PrevBatch is the token to page backwards from.
+	PrevBatch string `json:"prev_batch,omitempty"`
+}
+
+// The list wrappers. The Client-Server API wraps every collection of
+// events in an object with an `events` key, and each of these carries a
+// different event shape — which is the whole reason they are separate types
+// rather than one.
+type (
+	// EventList holds room events: a timeline or a piece of room state.
+	EventList struct {
+		Events []roomserver.ClientEvent `json:"events"`
+	}
+
+	// AccountDataList holds account data.
+	AccountDataList struct {
+		Events []AccountDataEvent `json:"events"`
+	}
+
+	// EphemeralEventList holds typing and receipt EDUs, which have a type
+	// and a content and nothing else.
+	EphemeralEventList struct {
+		Events []EphemeralEvent `json:"events"`
+	}
+
+	// PresenceList holds m.presence EDUs, which also have a sender.
+	PresenceList struct {
+		Events []presenceEvent `json:"events"`
+	}
+
+	// ToDeviceList holds to-device messages.
+	ToDeviceList struct {
+		Events []toDeviceEvent `json:"events"`
+	}
+
+	// StrippedStateList holds the stripped state of an invite or a knock.
+	StrippedStateList struct {
+		Events []roomserver.StrippedEvent `json:"events"`
+	}
+)
+
+// EphemeralEvent is an EDU of a room: a typing notification or a set of
+// receipts. It is not an event — no ID, no sender, no place in the room's
+// history — which is why it has a type of its own.
+type EphemeralEvent struct {
+	Type    string          `json:"type"`
+	Content json.RawMessage `json:"content"`
+}
+
+// AccountDataEvent is one account data blob as a sync reports it.
+//
+// It has a type and a content and nothing else — no sender, no ID, no room
+// — because account data is not an event: it is a key-value store the
+// Client-Server API happens to report in an event list, which is one of its
+// older seams. Reporting it as a room event with empty fields would be
+// worse than a type of its own: a client would see an event with no sender
+// and no timestamp and have to decide what that meant.
+type AccountDataEvent struct {
+	Type    string          `json:"type"`
+	Content json.RawMessage `json:"content"`
+}
+
+// DeviceLists is whose end-to-end keys a client should re-fetch.
+type DeviceLists struct {
+	Changed []string `json:"changed"`
+	Left    []string `json:"left"`
+}
+
+// Sync answers one /sync, blocking for up to req.Timeout when there is
+// nothing to report.
+func (s *API) Sync(ctx context.Context, req SyncRequest) (Response, error) {
+	since, err := ParseToken(req.Since)
+	if err != nil {
+		return Response{}, errInvalidTokenf("%v", err)
+	}
+	filter, err := s.filterFor(ctx, req.Localpart, req.Filter)
+	if err != nil {
+		return Response{}, err
+	}
+	if err := s.applySetPresence(ctx, req); err != nil {
+		return Response{}, err
+	}
+
+	// A device acknowledges the to-device batch it was last sent by coming
+	// back with the token that carried it, so this is where the previous
+	// batch is dropped.
+	s.ackToDevice(ctx, req.Localpart, req.Device, since.ToDevice)
+
+	timeout := min(req.Timeout, s.longPollTimeout)
+	deadline := s.now().Add(timeout)
+
+	for {
+		resp, changed, listener, err := s.attempt(ctx, req, filter, since)
+		switch {
+		case err != nil:
+			listener.Close()
+			return Response{}, err
+		case changed || timeout <= 0:
+			listener.Close()
+			return resp, nil
+		}
+
+		wait := time.Until(deadline)
+		if wait <= 0 {
+			listener.Close()
+			return resp, nil
+		}
+
+		woken, err := waitFor(ctx, listener, wait)
+		listener.Close()
+		switch {
+		case err != nil:
+			return Response{}, err
+		case !woken:
+			// Nothing happened in the time the client was willing to wait.
+			// The empty response and its token are the right answer: the
+			// client asks again.
+			return resp, nil
+		}
+	}
+}
+
+// attempt builds one candidate response, with a listener registered before
+// anything was read.
+//
+// Registering first is what closes the race: anything that lands between
+// the registration and the read leaves the channel already signalled, so a
+// wait on it returns at once and the caller reads again rather than
+// blocking on a change it has just missed. The listener is returned rather
+// than closed here because the caller is the one that knows whether it is
+// about to wait on it.
+func (s *API) attempt(ctx context.Context, req SyncRequest, filter Filter, since Token) (Response, bool, *Listener, error) {
+	rooms, err := s.rooms.UserMemberships(ctx, req.User, nil)
+	if err != nil {
+		return Response{}, false, nil, err
+	}
+	listener := s.notifier.Listen(req.User, roomIDsOf(rooms))
+
+	resp, changed, err := s.build(ctx, req, filter, since, rooms)
+	if err != nil {
+		return Response{}, false, listener, err
+	}
+	return resp, changed, listener, nil
+}
+
+// waitFor blocks until the listener is woken, the wait runs out, or the
+// request is cancelled. It reports whether it was woken.
+func waitFor(ctx context.Context, listener *Listener, wait time.Duration) (bool, error) {
+	timer := time.NewTimer(wait)
+	defer timer.Stop()
+
+	select {
+	case <-listener.Wait():
+		return true, nil
+	case <-timer.C:
+		return false, nil
+	case <-ctx.Done():
+		return false, ctx.Err()
+	}
+}
+
+// build assembles one response, reporting whether it found anything — which
+// is what tells Sync whether to block.
+func (s *API) build(ctx context.Context, req SyncRequest, filter Filter, since Token, rooms []roomserver.MembershipAt) (Response, bool, error) {
+	// The top of every stream, read once, so that the whole response
+	// describes one moment and the token it returns names that moment.
+	// Read per section instead, a stream could move between two sections
+	// and the token would skip whatever landed in the gap.
+	positions, err := s.store.StreamPositions(ctx)
+	if err != nil {
+		return Response{}, false, err
+	}
+	to := tokenAt(positions)
+
+	resp := Response{
+		Rooms: Rooms{
+			Join:   map[string]JoinedRoom{},
+			Invite: map[string]InvitedRoom{},
+			Leave:  map[string]LeftRoom{},
+			Knock:  map[string]KnockedRoom{},
+		},
+		Presence:    PresenceList{Events: []presenceEvent{}},
+		AccountData: AccountDataList{Events: []AccountDataEvent{}},
+		ToDevice:    ToDeviceList{Events: []toDeviceEvent{}},
+		// Both lists are empty until Phase 7 and both have to be present.
+		DeviceLists: DeviceLists{Changed: []string{}, Left: []string{}},
+	}
+
+	// The next token. Every stream advances to the top of the window: what
+	// was in it has either been reported or is something this user is not
+	// concerned by, and leaving a position behind would mean re-reading the
+	// same window on every sync forever. The exception is to-device, whose
+	// queue is drained a batch at a time and whose token therefore names
+	// the last message delivered.
+	next := to
+
+	if err := s.buildRooms(ctx, req, filter, since, to, rooms, &resp); err != nil {
+		return Response{}, false, err
+	}
+
+	accountData, err := s.globalAccountData(ctx, req.Localpart, since.AccountData, to.AccountData, filter)
+	if err != nil {
+		return Response{}, false, err
+	}
+	resp.AccountData.Events = accountData
+
+	toDevice, toDevicePos, err := s.toDeviceMessages(ctx, req.Localpart, req.Device, since.ToDevice, to.ToDevice)
+	if err != nil {
+		return Response{}, false, err
+	}
+	resp.ToDevice.Events = toDevice
+	next.ToDevice = toDevicePos
+
+	// A room joined inside this window brings its members' presence with
+	// it: their last presence change is older than the window, so a
+	// window-bounded read would not find them and the client would show
+	// everybody in its new room as offline until they next moved.
+	presenceFrom := since.Presence
+	if joinedInWindow(rooms, since.PDU, to.PDU) {
+		presenceFrom = 0
+	}
+	presence, err := s.presenceEvents(ctx, req.User, joinedRoomIDs(rooms),
+		presenceFrom, to.Presence, filter.PresenceFilter())
+	if err != nil {
+		return Response{}, false, err
+	}
+	resp.Presence.Events = presence
+
+	resp.NextBatch = next.String()
+	return resp, isInteresting(resp, since), nil
+}
+
+// isInteresting reports whether a response is worth returning rather than
+// blocking on.
+//
+// Comparing the tokens would not do. A stream can move without this user
+// being concerned by it — somebody else's receipt in a room they are not in
+// — and a sync that returned on every such move would turn a long poll back
+// into a busy loop. So the test is whether the response has anything in it,
+// with an initial sync as the exception: a client that has just logged in
+// needs an answer whatever is in it.
+func isInteresting(resp Response, since Token) bool {
+	if since.IsInitial() {
+		return true
+	}
+	return len(resp.Rooms.Join) > 0 ||
+		len(resp.Rooms.Invite) > 0 ||
+		len(resp.Rooms.Leave) > 0 ||
+		len(resp.Rooms.Knock) > 0 ||
+		len(resp.AccountData.Events) > 0 ||
+		len(resp.ToDevice.Events) > 0 ||
+		len(resp.Presence.Events) > 0
+}
+
+// buildRooms fills the four room sections.
+func (s *API) buildRooms(ctx context.Context, req SyncRequest, filter Filter, since, to Token, rooms []roomserver.MembershipAt, resp *Response) error {
+	interesting, err := s.changedRooms(ctx, req, filter, since, to, rooms)
+	if err != nil {
+		return err
+	}
+
+	for _, room := range rooms {
+		if _, ok := interesting[room.RoomID]; !ok {
+			continue
+		}
+		if !filter.AllowsRoom(room.RoomID) {
+			continue
+		}
+
+		// A room whose membership moved inside this window is read in full
+		// rather than from the client's position: the ephemeral streams
+		// and the account data of a room just joined are all older than
+		// the window, and a window-bounded read of them would come back
+		// empty.
+		window := roomWindow{
+			since: since,
+			to:    to,
+			full:  since.IsInitial() || req.FullState || room.Position > since.PDU,
+		}
+
+		switch room.Membership {
+		case event.MembershipJoin:
+			joined, err := s.joinedRoom(ctx, req, filter, window, room)
+			if err != nil {
+				return err
+			}
+			resp.Rooms.Join[room.RoomID] = joined
+
+		case event.MembershipInvite:
+			state, err := s.rooms.StrippedState(ctx, room.RoomID, []string{req.User})
+			if err != nil {
+				return err
+			}
+			resp.Rooms.Invite[room.RoomID] = InvitedRoom{
+				InviteState: StrippedStateList{Events: state},
+			}
+
+		case event.MembershipKnock:
+			state, err := s.rooms.StrippedState(ctx, room.RoomID, []string{req.User})
+			if err != nil {
+				return err
+			}
+			resp.Rooms.Knock[room.RoomID] = KnockedRoom{
+				KnockState: StrippedStateList{Events: state},
+			}
+
+		case event.MembershipLeave, event.MembershipBan:
+			left, err := s.leftRoom(ctx, req, filter, window, room)
+			if err != nil {
+				return err
+			}
+			resp.Rooms.Leave[room.RoomID] = left
+		}
+	}
+	return nil
+}
+
+// roomWindow is the span one room is read over.
+type roomWindow struct {
+	since Token
+	to    Token
+
+	// full reads the room's streams from the beginning rather than from
+	// the client's position: an initial sync, ?full_state=true, or a room
+	// the user has only just joined.
+	full bool
+}
+
+// from is the position a stream is read from, which is the start of it for
+// a full read.
+func (w roomWindow) from(pos int64) int64 {
+	if w.full {
+		return 0
+	}
+	return pos
+}
+
+// changedRooms is the set of rooms this sync has to look at.
+//
+// On an initial sync that is all of them. On an incremental one it is the
+// rooms with a new event, plus the rooms whose ephemeral or account data
+// changed — neither of which has an event to be found by — plus the rooms
+// the user's own membership changed in, which is how a leave reaches a
+// client even when it was the last thing to happen in the room.
+func (s *API) changedRooms(ctx context.Context, req SyncRequest, filter Filter, since, to Token, rooms []roomserver.MembershipAt) (map[string]struct{}, error) {
+	out := map[string]struct{}{}
+
+	if since.IsInitial() || req.FullState {
+		for _, room := range rooms {
+			// A room left before this sync began is not reported unless
+			// the filter asks for it: a client logging in does not want
+			// every room it has ever left.
+			if isDeparted(room.Membership) && !filter.IncludeLeave() {
+				continue
+			}
+			out[room.RoomID] = struct{}{}
+		}
+		return out, nil
+	}
+
+	changed, err := s.rooms.RoomsWithEvents(ctx, roomIDsOf(rooms), since.PDU, to.PDU)
+	if err != nil {
+		return nil, err
+	}
+	for _, roomID := range changed {
+		out[roomID] = struct{}{}
+	}
+
+	joined := joinedRoomIDs(rooms)
+	if to.Receipt > since.Receipt {
+		receipts, err := s.store.Receipts(ctx, joined, since.Receipt, to.Receipt)
+		if err != nil {
+			return nil, err
+		}
+		for _, r := range receipts {
+			out[r.RoomID] = struct{}{}
+		}
+	}
+
+	// Typing lives in memory and has no position at all: a notification
+	// that started and stopped between two syncs is simply never reported,
+	// and one that is live is reported every time until it expires.
+	for _, roomID := range s.typing.activeRooms() {
+		if slices.Contains(joined, roomID) {
+			out[roomID] = struct{}{}
+		}
+	}
+
+	if to.AccountData > since.AccountData {
+		entries, err := s.store.AccountDataChanges(ctx, req.Localpart, since.AccountData, to.AccountData)
+		if err != nil {
+			return nil, err
+		}
+		for _, entry := range entries {
+			if entry.RoomID != "" {
+				out[entry.RoomID] = struct{}{}
+			}
+		}
+	}
+
+	for _, room := range rooms {
+		if room.Position > since.PDU && room.Position <= to.PDU {
+			out[room.RoomID] = struct{}{}
+		}
+	}
+	return out, nil
+}
+
+// joinedRoom reads one room the user is in.
+func (s *API) joinedRoom(ctx context.Context, req SyncRequest, filter Filter, window roomWindow, room roomserver.MembershipAt) (JoinedRoom, error) {
+	read, err := s.rooms.RoomSync(ctx, s.roomSyncRequest(req, filter, window, room))
+	if err != nil {
+		return JoinedRoom{}, err
+	}
+
+	out := JoinedRoom{
+		State:    EventList{Events: withoutRoomID(read.State)},
+		Timeline: s.timeline(read.Timeline),
+	}
+	// The summary exists to name a room that has no name of its own, so it
+	// goes out only when there is somebody to name it after.
+	if len(read.Summary.Heroes) > 0 {
+		summary := read.Summary
+		out.Summary = &summary
+	}
+
+	if out.Ephemeral.Events, err = s.roomEphemeral(ctx, req.User, room.RoomID, window, filter); err != nil {
+		return JoinedRoom{}, err
+	}
+	if out.AccountData.Events, err = s.roomAccountData(ctx, req.Localpart, room.RoomID, window, filter); err != nil {
+		return JoinedRoom{}, err
+	}
+	return out, nil
+}
+
+// leftRoom reads a room the user has left: everything up to their leave and
+// nothing after it, which the roomserver's visible range takes care of.
+func (s *API) leftRoom(ctx context.Context, req SyncRequest, filter Filter, window roomWindow, room roomserver.MembershipAt) (LeftRoom, error) {
+	read, err := s.rooms.RoomSync(ctx, s.roomSyncRequest(req, filter, window, room))
+	if err != nil {
+		return LeftRoom{}, err
+	}
+
+	out := LeftRoom{
+		State:    EventList{Events: withoutRoomID(read.State)},
+		Timeline: s.timeline(read.Timeline),
+	}
+	if out.AccountData.Events, err = s.roomAccountData(ctx, req.Localpart, room.RoomID, window, filter); err != nil {
+		return LeftRoom{}, err
+	}
+	return out, nil
+}
+
+// roomSyncRequest is the read the two room sections share.
+func (s *API) roomSyncRequest(req SyncRequest, filter Filter, window roomWindow, room roomserver.MembershipAt) roomserver.RoomSyncRequest {
+	return roomserver.RoomSyncRequest{
+		RoomID:          room.RoomID,
+		User:            req.User,
+		Since:           window.since.PDU,
+		To:              window.to.PDU,
+		TimelineLimit:   filter.TimelineLimit(),
+		TimelineFilter:  filter.TimelineFilter(),
+		StateFilter:     filter.StateFilter(),
+		FullState:       req.FullState,
+		LazyLoadMembers: filter.LazyLoadMembers(),
+	}
+}
+
+// timeline projects a read timeline into the response shape.
+func (s *API) timeline(read roomserver.Timeline) Timeline {
+	out := Timeline{
+		Events:  withoutRoomID(read.Events),
+		Limited: read.Limited,
+	}
+	// prev_batch is what the client pages backwards from, so it names the
+	// position below the first event rather than the window asked for.
+	if len(read.Events) > 0 {
+		out.PrevBatch = pduToken(read.Start).String()
+	}
+	return out
+}
+
+// roomEphemeral builds a room's typing and receipt EDUs.
+func (s *API) roomEphemeral(ctx context.Context, userID, roomID string, window roomWindow, filter Filter) ([]EphemeralEvent, error) {
+	out := []EphemeralEvent{}
+	ephemeralFilter := filter.EphemeralFilter()
+
+	if e := s.typingEvent(roomID); e != nil && matchesEventFilter(e.Type, "", ephemeralFilter) {
+		out = append(out, *e)
+	}
+
+	byRoom, err := s.roomReceipts(ctx, userID, []string{roomID},
+		window.from(window.since.Receipt), window.to.Receipt)
+	if err != nil {
+		return nil, err
+	}
+	if e := receiptsEvent(byRoom[roomID]); e != nil && matchesEventFilter(e.Type, "", ephemeralFilter) {
+		out = append(out, *e)
+	}
+	return out, nil
+}
+
+// globalAccountData reads the account data that is not about a room.
+func (s *API) globalAccountData(ctx context.Context, localpart string, from, to int64, filter Filter) ([]AccountDataEvent, error) {
+	entries, err := s.accountDataEntries(ctx, localpart, from, to)
+	if err != nil {
+		return nil, err
+	}
+	return s.accountDataEvents(ctx, localpart, entries, "", filter.AccountDataFilter())
+}
+
+// roomAccountData reads one room's account data.
+func (s *API) roomAccountData(ctx context.Context, localpart, roomID string, window roomWindow, filter Filter) ([]AccountDataEvent, error) {
+	entries, err := s.accountDataEntries(ctx, localpart,
+		window.from(window.since.AccountData), window.to.AccountData)
+	if err != nil {
+		return nil, err
+	}
+	return s.accountDataEvents(ctx, localpart, entries, roomID, filter.RoomAccountDataFilter())
+}
+
+// accountDataEntries lists which of a user's account data changed in a
+// window, or all of it when the window starts at the beginning.
+func (s *API) accountDataEntries(ctx context.Context, localpart string, from, to int64) ([]storage.AccountDataEntry, error) {
+	if from == 0 {
+		return s.store.AccountDataAt(ctx, localpart, to)
+	}
+	return s.store.AccountDataChanges(ctx, localpart, from, to)
+}
+
+// accountDataEvents turns the entries of one scope — global, or one room —
+// into the events a sync reports.
+func (s *API) accountDataEvents(ctx context.Context, localpart string, entries []storage.AccountDataEntry, roomID string, filter storage.TimelineFilter) ([]AccountDataEvent, error) {
+	out := []AccountDataEvent{}
+	for _, entry := range entries {
+		if entry.RoomID != roomID {
+			continue
+		}
+		if !matchesEventFilter(entry.DataType, "", filter) {
+			continue
+		}
+		content, err := s.accounts.AccountData(ctx, localpart, entry.RoomID, entry.DataType)
+		switch {
+		case errors.Is(err, storage.ErrNoSuchAccountData):
+			// Deleted between the two reads: nothing to report.
+			continue
+		case err != nil:
+			return nil, err
+		}
+		out = append(out, AccountDataEvent{Type: entry.DataType, Content: content})
+	}
+	return out, nil
+}
+
+// applySetPresence acts on ?set_presence=.
+//
+// With presence off the parameter is ignored rather than refused: it is on
+// every sync a client sends, and failing them all would make a server with
+// presence off unusable.
+func (s *API) applySetPresence(ctx context.Context, req SyncRequest) error {
+	if !s.presenceEnabled {
+		return nil
+	}
+	state := req.SetPresence
+	if state == "" {
+		state = PresenceOnline
+	}
+	if state == PresenceOffline {
+		// A client that is syncing is not offline, whatever it says.
+		return nil
+	}
+
+	err := s.SetPresence(ctx, PresenceRequest{User: req.User, Presence: state})
+	if errors.Is(err, ErrInvalidParam) {
+		// An unusable value in a parameter that rides on every request is
+		// not worth failing the sync for.
+		s.log.Debug("ignoring an unusable set_presence", "user_id", req.User, "value", req.SetPresence)
+		return nil
+	}
+	return err
+}
+
+// withoutRoomID strips the room ID from the events of a sync response.
+//
+// The spec's sync carries events without a room_id: it would be the same
+// value on every event of the section they are already nested in, and on a
+// client with a thousand rooms that is a measurable part of every response.
+// /messages, /context and the single-event endpoint do carry it, because
+// there the event is not nested under its room.
+func withoutRoomID(events []roomserver.ClientEvent) []roomserver.ClientEvent {
+	out := make([]roomserver.ClientEvent, 0, len(events))
+	for _, e := range events {
+		e.RoomID = ""
+		out = append(out, e)
+	}
+	return out
+}
+
+// roomIDsOf projects memberships onto room IDs.
+func roomIDsOf(rooms []roomserver.MembershipAt) []string {
+	out := make([]string, 0, len(rooms))
+	for _, room := range rooms {
+		out = append(out, room.RoomID)
+	}
+	return out
+}
+
+// joinedRoomIDs is roomIDsOf for the rooms the user is actually in, which
+// is the set the ephemeral and presence streams are about.
+func joinedRoomIDs(rooms []roomserver.MembershipAt) []string {
+	out := make([]string, 0, len(rooms))
+	for _, room := range rooms {
+		if room.Membership == event.MembershipJoin {
+			out = append(out, room.RoomID)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// joinedInWindow reports whether the user joined a room inside this
+// window.
+func joinedInWindow(rooms []roomserver.MembershipAt, from, to int64) bool {
+	for _, room := range rooms {
+		if room.Membership != event.MembershipJoin {
+			continue
+		}
+		if room.Position > from && (to == 0 || room.Position <= to) {
+			return true
+		}
+	}
+	return false
+}
+
+// isDeparted reports whether a membership means the user is out of the
+// room.
+func isDeparted(membership string) bool {
+	return membership == event.MembershipLeave || membership == event.MembershipBan
+}

server/internal/syncapi/syncapi.go

@@ -0,0 +1,216 @@
+// Package syncapi is the read model: it answers "what has happened that I
+// have not seen", which is the question a Matrix client spends its whole
+// life asking.
+//
+// Everything a client learns after it has logged in arrives through here —
+// /sync for the live stream, /messages for a room's history, /context for
+// one event and what surrounds it. The component writes almost nothing:
+// read receipts, to-device messages and presence are its own, and
+// everything else it serves is the roomserver's, read through the roomserver
+// rather than copied. That is the one real architectural choice in this
+// package, and it is a departure from Dendrite, which gives its syncapi its
+// own tables and feeds them from the output stream. In one process the copy
+// buys nothing and costs the hardest bug in a homeserver: a read model that
+// disagrees with the room.
+//
+// What this package owns instead is everything the roomserver has no
+// opinion about:
+//
+//   - Stream tokens: a position in each of the five streams a response
+//     carries, in one opaque string (token.go).
+//   - The notifier: waking a client blocked in a long poll the moment
+//     something concerns it, rather than when its timeout expires
+//     (notifier.go).
+//   - Filters: what a client asked to be left out (filter.go).
+//   - The ephemeral and device streams: typing, receipts, presence and
+//     to-device messages, none of which are events in a room.
+//
+// Phase 4 of PLAN.md. What it does not do yet is notification counts, which
+// need push rules (Phase 8), and device list tracking, which needs the
+// keyserver (Phase 7); both are named in docs/sync.md as the gaps they are.
+package syncapi
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"time"
+
+	"github.com/dariuszjaniak/warp/server/internal/roomserver"
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// Errors callers have to tell apart. The storage and roomserver sentinels
+// are re-exported rather than translated, so that errors.Is works whichever
+// package a caller reaches for.
+var (
+	ErrNoSuchRoom  = storage.ErrNoSuchRoom
+	ErrNoSuchEvent = storage.ErrNoSuchEvent
+
+	// ErrForbidden is a read the caller is not entitled to: a room they are
+	// not in, or somebody else's filter.
+	ErrForbidden = roomserver.ErrForbidden
+
+	// ErrPresenceDisabled is a presence operation on a server that has
+	// presence switched off.
+	ErrPresenceDisabled = errors.New("presence is disabled on this server")
+
+	// ErrInvalidParam is a request parameter this server will not accept:
+	// a receipt type it does not know, a presence state that is not one.
+	ErrInvalidParam = errors.New("invalid parameter")
+
+	// ErrInvalidToken is a ?since= or ?from= that is not a sync token.
+	ErrInvalidToken = errors.New("invalid sync token")
+)
+
+// Accounts is the part of the userapi the sync API needs: account data to
+// put in a response, and the devices of a user to address a to-device
+// message to. *userapi.API satisfies it as it stands.
+type Accounts interface {
+	AccountData(ctx context.Context, localpart, roomID, dataType string) ([]byte, error)
+	SetAccountData(ctx context.Context, localpart, roomID, dataType string, content []byte) error
+	Devices(ctx context.Context, localpart string) ([]storage.Device, error)
+}
+
+// API is the syncapi component.
+type API struct {
+	serverName string
+	rooms      *roomserver.API
+	accounts   Accounts
+	store      storage.SyncStore
+	notifier   *Notifier
+	typing     *typingTracker
+	log        *slog.Logger
+	now        func() time.Time
+
+	// presenceEnabled is the config switch of PLAN.md open question 3.
+	// With presence off the endpoints answer M_FORBIDDEN and the presence
+	// section of a sync is empty, rather than the endpoints disappearing:
+	// a client that finds no /presence assumes an old server and stops
+	// asking, where a clear refusal tells it this one has it turned off.
+	presenceEnabled bool
+
+	// longPollTimeout caps how long a /sync may block, whatever ?timeout=
+	// the client asked for. A client asking for ten minutes would hold a
+	// connection and a database handle for ten minutes.
+	longPollTimeout time.Duration
+}
+
+// Options configure the component.
+type Options struct {
+	// ServerName is the Matrix server name.
+	ServerName string
+
+	// Rooms is the roomserver, which every room read goes through.
+	Rooms *roomserver.API
+
+	// Accounts is the userapi.
+	Accounts Accounts
+
+	// Store is the sync half of storage.
+	Store storage.SyncStore
+
+	// Notifier wakes blocked syncs. It is the same object the roomserver
+	// was given as its Notifier; nil builds one, which is only useful in a
+	// test where nothing is being written.
+	Notifier *Notifier
+
+	// PresenceEnabled turns the presence stream on.
+	PresenceEnabled bool
+
+	// TypingTimeout is how long a typing notification lasts without being
+	// renewed, whatever timeout the client asked for.
+	TypingTimeout time.Duration
+
+	// LongPollTimeout caps a /sync's ?timeout=.
+	LongPollTimeout time.Duration
+
+	Logger *slog.Logger
+
+	// Now is the clock, so tests can expire a typing notification without
+	// waiting for it.
+	Now func() time.Time
+}
+
+// Defaults for the timeouts.
+const (
+	// DefaultTypingTimeout is the spec's own suggestion, and what every
+	// client sends.
+	DefaultTypingTimeout = 30 * time.Second
+
+	// DefaultLongPollTimeout is the timeout clients ask for. Holding a
+	// connection longer buys nothing: the notifier wakes a sync the moment
+	// anything happens, so the timeout only decides how often an idle
+	// client re-establishes its request.
+	DefaultLongPollTimeout = 30 * time.Second
+
+	// MaxTypingTimeout caps what a client may ask for. A typing
+	// notification that outlived its sender's session would leave a room
+	// permanently claiming somebody was typing.
+	MaxTypingTimeout = 2 * time.Minute
+)
+
+// New builds the component.
+func New(opts Options) (*API, error) {
+	switch {
+	case opts.ServerName == "":
+		return nil, errors.New("syncapi: ServerName must not be empty")
+	case opts.Rooms == nil:
+		return nil, errors.New("syncapi: Rooms must not be nil")
+	case opts.Store == nil:
+		return nil, errors.New("syncapi: Store must not be nil")
+	case opts.Accounts == nil:
+		return nil, errors.New("syncapi: Accounts must not be nil")
+	}
+
+	if opts.Notifier == nil {
+		opts.Notifier = NewNotifier()
+	}
+	if opts.TypingTimeout <= 0 {
+		opts.TypingTimeout = DefaultTypingTimeout
+	}
+	if opts.TypingTimeout > MaxTypingTimeout {
+		opts.TypingTimeout = MaxTypingTimeout
+	}
+	if opts.LongPollTimeout <= 0 {
+		opts.LongPollTimeout = DefaultLongPollTimeout
+	}
+	if opts.Logger == nil {
+		opts.Logger = slog.Default()
+	}
+	if opts.Now == nil {
+		opts.Now = time.Now
+	}
+
+	return &API{
+		serverName:      opts.ServerName,
+		rooms:           opts.Rooms,
+		accounts:        opts.Accounts,
+		store:           opts.Store,
+		notifier:        opts.Notifier,
+		typing:          newTypingTracker(opts.TypingTimeout, opts.Notifier, opts.Now),
+		log:             opts.Logger,
+		now:             opts.Now,
+		presenceEnabled: opts.PresenceEnabled,
+		longPollTimeout: opts.LongPollTimeout,
+	}, nil
+}
+
+// Notifier returns the notifier, which the roomserver is wired to as its
+// emit stage.
+func (s *API) Notifier() *Notifier { return s.notifier }
+
+// PresenceEnabled reports whether the presence stream is on, which
+// /capabilities and the admin API both report.
+func (s *API) PresenceEnabled() bool { return s.presenceEnabled }
+
+// LongPollTimeout is the cap a /sync's ?timeout= is held to.
+func (s *API) LongPollTimeout() time.Duration { return s.longPollTimeout }
+
+// nowMS is the current time in the unit the API uses on the wire.
+func (s *API) nowMS() int64 { return s.now().UnixMilli() }
+
+// Close releases what the component started. The typing tracker holds a
+// timer per typing user, and a server shutting down should not wait for
+// them.
+func (s *API) Close() { s.typing.close() }

server/internal/syncapi/todevice.go

@@ -0,0 +1,176 @@
+package syncapi
+
+import (
+	"context"
+	"encoding/json"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// To-device messaging: a message from one device to another, which the
+// server stores and forwards without reading.
+//
+// It exists for end-to-end encryption — a device sending another one a
+// Megolm session key is a to-device message — and the server deliberately
+// cannot tell what is in one. Phase 7 builds the keyserver on top of this;
+// the transport itself belongs here, because the only way a device ever
+// receives one is in a /sync response.
+
+// wildcardDevice is the device ID that means "every device of this user".
+const wildcardDevice = "*"
+
+// SendToDeviceRequest is PUT /sendToDevice/{eventType}/{txnID}.
+type SendToDeviceRequest struct {
+	// Sender is the full user ID sending, which is the caller.
+	Sender string
+
+	EventType string
+
+	// Messages is user ID -> device ID -> content, as the request body
+	// nests it. A device ID of "*" addresses every device of that user.
+	Messages map[string]map[string]json.RawMessage
+}
+
+// SendToDevice queues messages for delivery.
+//
+// A message to a user of another server is dropped rather than refused.
+// That is deliberate: a client encrypting for a room with remote members
+// sends one message per device in one request, and failing the request
+// because federation is off would stop it from delivering to the local
+// devices as well. The dropped recipients simply never see the key and
+// their client reports an undecryptable message, which is what they would
+// see anyway. Phase 5 routes them.
+func (s *API) SendToDevice(ctx context.Context, req SendToDeviceRequest) error {
+	if req.EventType == "" {
+		return errInvalidParamf("an event type is required")
+	}
+
+	var (
+		queued []storage.NewToDeviceMessage
+		woken  []string
+		sentAt = s.nowMS()
+	)
+	for userID, byDevice := range req.Messages {
+		localpart, ok := s.localpartOf(userID)
+		if !ok {
+			s.log.Debug("dropping a to-device message for a remote user",
+				"sender", req.Sender, "recipient", userID)
+			continue
+		}
+
+		for deviceID, content := range byDevice {
+			if deviceID != wildcardDevice {
+				queued = append(queued, storage.NewToDeviceMessage{
+					Localpart: localpart,
+					DeviceID:  deviceID,
+					Sender:    req.Sender,
+					Type:      req.EventType,
+					Content:   contentOrEmpty(content),
+					SentAtMS:  sentAt,
+				})
+				continue
+			}
+
+			// A wildcard is expanded here rather than stored as one: the
+			// set of a user's devices changes, and a message addressed to
+			// "every device" means the ones that existed when it was sent.
+			// Storing the wildcard would deliver it to a device that was
+			// created afterwards, which for a key share is both useless
+			// and a leak.
+			devices, err := s.accounts.Devices(ctx, localpart)
+			if err != nil {
+				return err
+			}
+			for _, device := range devices {
+				queued = append(queued, storage.NewToDeviceMessage{
+					Localpart: localpart,
+					DeviceID:  device.ID,
+					Sender:    req.Sender,
+					Type:      req.EventType,
+					Content:   contentOrEmpty(content),
+					SentAtMS:  sentAt,
+				})
+			}
+		}
+		woken = append(woken, userID)
+	}
+
+	if len(queued) == 0 {
+		return nil
+	}
+	if _, err := s.store.SendToDevice(ctx, queued); err != nil {
+		return err
+	}
+	s.notifier.NotifyUsers(woken)
+	return nil
+}
+
+// contentOrEmpty keeps a null or absent content out of the store: the
+// to-device content is passed through untouched, and "untouched" has to
+// still be valid JSON on the way out.
+func contentOrEmpty(content json.RawMessage) []byte {
+	if len(content) == 0 {
+		return []byte("{}")
+	}
+	return content
+}
+
+// toDeviceEvent is one message in the shape a sync reports it.
+type toDeviceEvent struct {
+	Sender  string          `json:"sender"`
+	Type    string          `json:"type"`
+	Content json.RawMessage `json:"content"`
+}
+
+// toDeviceMessages reads a device's queue between two positions and returns
+// the events plus the position of the last one.
+//
+// The returned position is what the next token carries, and it is the
+// position of the last message *delivered* rather than the top of the
+// stream: a queue longer than one batch is drained over several syncs, and
+// a token naming the top would skip the rest.
+func (s *API) toDeviceMessages(ctx context.Context, localpart, deviceID string, from, to int64) ([]toDeviceEvent, int64, error) {
+	if deviceID == "" {
+		// An access token always names a device, so this cannot happen
+		// through the API; a guest session later might.
+		return nil, from, nil
+	}
+
+	msgs, err := s.store.ToDeviceMessages(ctx, localpart, deviceID, from, to, 0)
+	if err != nil {
+		return nil, 0, err
+	}
+	if len(msgs) == 0 {
+		return nil, from, nil
+	}
+
+	out := make([]toDeviceEvent, 0, len(msgs))
+	for _, m := range msgs {
+		out = append(out, toDeviceEvent{
+			Sender:  m.Sender,
+			Type:    m.Type,
+			Content: m.Content,
+		})
+	}
+	return out, msgs[len(msgs)-1].StreamPos, nil
+}
+
+// ackToDevice deletes the messages a device has acknowledged by syncing
+// past them.
+//
+// A device acknowledges a batch by sending back the token that batch came
+// with, so the deletion happens on the *next* sync rather than when the
+// batch is sent. A client that crashes between the two gets the batch
+// again, which is the right way round: a lost key share leaves a message
+// permanently undecryptable, and a duplicate one is ignored.
+func (s *API) ackToDevice(ctx context.Context, localpart, deviceID string, through int64) {
+	if deviceID == "" || through <= 0 {
+		return
+	}
+	if err := s.store.DeleteToDeviceMessages(ctx, localpart, deviceID, through); err != nil {
+		// Failing the sync over this would be worse than leaving the rows:
+		// the client has the messages, and the next sync will try again.
+		s.log.Warn("could not drop acknowledged to-device messages",
+			"localpart", localpart, "device_id", deviceID, "through", through, "error", err)
+	}
+}

server/internal/syncapi/token.go

@@ -0,0 +1,132 @@
+package syncapi
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/dariuszjaniak/warp/server/internal/storage"
+)
+
+// Token is a position in every stream a sync response carries.
+//
+// A sync response is assembled from streams that move independently: room
+// events, read receipts, to-device messages, account data and presence. A
+// client that has seen receipt 7 but only event 4 has to be able to say so
+// in the one opaque string the API gives it, so the token names a position
+// in each.
+//
+// The wire form is
+//
+//	s{pdu}_{receipt}_{to_device}_{account_data}_{presence}
+//
+// with trailing zeroes left off, so a client that has only ever seen room
+// events carries "s12" rather than "s12_0_0_0_0". The leading "s" is there
+// to make a token recognisable in a log and to leave room for a second
+// token shape later — Simplified Sliding Sync (PLAN.md phase 10) has its
+// own — without any ambiguity about which is which.
+//
+// The spec says only that the token is opaque, and clients treat it that
+// way. It is still worth being readable: a token is the first thing in
+// every bug report about /sync.
+type Token struct {
+	PDU         int64
+	Receipt     int64
+	ToDevice    int64
+	AccountData int64
+	Presence    int64
+}
+
+// tokenPrefix marks a classic /sync token.
+const tokenPrefix = "s"
+
+// IsInitial reports whether the token is the start of every stream, which
+// is what a sync with no ?since= means.
+func (t Token) IsInitial() bool { return t == Token{} }
+
+// String renders the wire form.
+func (t Token) String() string {
+	parts := []int64{t.PDU, t.Receipt, t.ToDevice, t.AccountData, t.Presence}
+	// Trailing zeroes carry no information, and leaving them off keeps the
+	// common token short.
+	last := 0
+	for i, p := range parts {
+		if p != 0 {
+			last = i
+		}
+	}
+
+	var b strings.Builder
+	b.WriteString(tokenPrefix)
+	for i, p := range parts[:last+1] {
+		if i > 0 {
+			b.WriteByte('_')
+		}
+		b.WriteString(strconv.FormatInt(p, 10))
+	}
+	return b.String()
+}
+
+// ParseToken reads a token a client sent back.
+//
+// An empty string is the initial token: /sync without ?since= and
+// /messages without ?from= both mean "from the beginning", and refusing
+// the empty string would make every caller special-case it.
+//
+// A token with fewer fields than this build writes is accepted and the
+// missing ones are zero. That is what lets a token stored by a client
+// before a stream existed go on working: the stream it did not know about
+// reads as "seen nothing", so the client is sent that stream in full once
+// and is in step from then on.
+func ParseToken(raw string) (Token, error) {
+	if raw == "" {
+		return Token{}, nil
+	}
+	body, ok := strings.CutPrefix(raw, tokenPrefix)
+	if !ok {
+		return Token{}, fmt.Errorf("%q is not a sync token: it must start with %q", raw, tokenPrefix)
+	}
+
+	fields := strings.Split(body, "_")
+	if len(fields) > 5 {
+		return Token{}, fmt.Errorf("%q is not a sync token: it has %d positions, this server writes at most 5",
+			raw, len(fields))
+	}
+
+	var positions [5]int64
+	for i, field := range fields {
+		value, err := strconv.ParseInt(field, 10, 64)
+		if err != nil || value < 0 {
+			return Token{}, fmt.Errorf("%q is not a sync token: position %d is %q", raw, i+1, field)
+		}
+		positions[i] = value
+	}
+	return Token{
+		PDU:         positions[0],
+		Receipt:     positions[1],
+		ToDevice:    positions[2],
+		AccountData: positions[3],
+		Presence:    positions[4],
+	}, nil
+}
+
+// tokenAt is the token naming the top of every stream, which is what a
+// sync response hands back when it has drained all of them.
+func tokenAt(positions storage.StreamPositions) Token {
+	return Token{
+		PDU:         positions.PDU,
+		Receipt:     positions.Receipt,
+		ToDevice:    positions.ToDevice,
+		AccountData: positions.AccountData,
+		Presence:    positions.Presence,
+	}
+}
+
+// pduToken is the token for a position in the room event stream alone.
+//
+// It is what /messages and a room's prev_batch hand out: both are about
+// room events only, and a token from either has to be usable as a ?since=
+// for /sync, where the other streams then read as unseen. That is the
+// conservative direction — the client is sent those streams again rather
+// than having them skipped.
+func pduToken(pos int64) Token { return Token{PDU: pos} }

server/internal/syncapi/token_test.go

@@ -0,0 +1,99 @@
+package syncapi_test
+
+import (
+	"testing"
+
+	"github.com/dariuszjaniak/warp/server/internal/syncapi"
+)
+
+func TestTokenRoundTrips(t *testing.T) {
+	for _, tc := range []struct {
+		name  string
+		token syncapi.Token
+		want  string
+	}{
+		{"the initial token", syncapi.Token{}, "s0"},
+		{"events only", syncapi.Token{PDU: 12}, "s12"},
+		{"events and receipts", syncapi.Token{PDU: 12, Receipt: 3}, "s12_3"},
+		{
+			"every stream",
+			syncapi.Token{PDU: 1, Receipt: 2, ToDevice: 3, AccountData: 4, Presence: 5},
+			"s1_2_3_4_5",
+		},
+		{
+			// A zero in the middle has to be written, because the
+			// positions are read by place.
+			"a gap in the middle",
+			syncapi.Token{PDU: 1, Presence: 5},
+			"s1_0_0_0_5",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := tc.token.String(); got != tc.want {
+				t.Fatalf("String() = %q, want %q", got, tc.want)
+			}
+			parsed, err := syncapi.ParseToken(tc.want)
+			if err != nil {
+				t.Fatalf("ParseToken(%q): %v", tc.want, err)
+			}
+			if parsed != tc.token {
+				t.Errorf("ParseToken(%q) = %+v, want %+v", tc.want, parsed, tc.token)
+			}
+		})
+	}
+}
+
+func TestParseTokenAcceptsAnEmptyString(t *testing.T) {
+	// /sync without ?since= and /messages without ?from= both mean "from
+	// the beginning"; refusing the empty string would make every caller
+	// special-case it.
+	parsed, err := syncapi.ParseToken("")
+	if err != nil {
+		t.Fatalf("ParseToken(\"\"): %v", err)
+	}
+	if !parsed.IsInitial() {
+		t.Errorf("ParseToken(\"\") = %+v, want the initial token", parsed)
+	}
+}
+
+func TestParseTokenAcceptsAShortToken(t *testing.T) {
+	// A token a client stored before a stream existed must go on working:
+	// the stream it does not mention reads as "seen nothing", so the
+	// client is sent that stream in full once and is in step afterwards.
+	parsed, err := syncapi.ParseToken("s7")
+	if err != nil {
+		t.Fatalf("ParseToken(\"s7\"): %v", err)
+	}
+	if parsed.PDU != 7 {
+		t.Errorf("PDU = %d, want 7", parsed.PDU)
+	}
+	if parsed.Receipt != 0 || parsed.ToDevice != 0 || parsed.AccountData != 0 || parsed.Presence != 0 {
+		t.Errorf("the streams a short token omits must read as zero: %+v", parsed)
+	}
+}
+
+func TestParseTokenRejectsNonsense(t *testing.T) {
+	for _, raw := range []string{
+		"nonsense",     // no prefix
+		"t1_2",         // a prefix this build does not write
+		"s",            // a prefix and nothing else
+		"s-1",          // a negative position
+		"sx",           // not a number
+		"s1_",          // a trailing separator
+		"s1_2_3_4_5_6", // more positions than there are streams
+		"s1.5",
+	} {
+		if _, err := syncapi.ParseToken(raw); err == nil {
+			t.Errorf("ParseToken(%q) was accepted", raw)
+		}
+	}
+}
+
+func TestParseTokenIsNotFooledByOverflow(t *testing.T) {
+	// A position larger than an int64 is not a position this server ever
+	// handed out, and silently wrapping it would mean a client being sent
+	// the whole of every room again.
+	if _, err := syncapi.ParseToken("s99999999999999999999"); err == nil {
+		t.Error("a position that does not fit in an int64 was accepted")
+	}
+}

server/internal/syncapi/typing.go

@@ -0,0 +1,220 @@
+package syncapi
+
+import (
+	"context"
+	"encoding/json"
+	"slices"
+	"sync"
+	"time"
+)
+
+// Typing notifications are held in memory and nowhere else.
+//
+// They expire in seconds, they are worthless after a restart — nobody is
+// still typing the message they were typing when the server went down —
+// and they arrive one per keystroke-burst per room. Writing them to disk
+// would be the busiest table in the database, holding the least valuable
+// data in it. Dendrite reaches the same conclusion.
+//
+// The cost of the choice is that a restart clears them, which a client
+// cannot tell from everybody having stopped typing, and that a second
+// process would not see them. The second only matters when Warp stops
+// being one process, which PLAN.md says is not the plan.
+
+// typingTracker holds who is typing in which room, and expires them.
+type typingTracker struct {
+	timeout  time.Duration
+	notifier *Notifier
+	now      func() time.Time
+
+	mu sync.Mutex
+	// rooms is room ID -> user ID -> when the notification expires.
+	rooms map[string]map[string]time.Time
+	// timers hold the expiry wake-ups, so that a notification running out
+	// reaches a blocked sync rather than waiting for one to ask.
+	timers map[typingKey]*time.Timer
+	closed bool
+}
+
+type typingKey struct {
+	roomID string
+	userID string
+}
+
+func newTypingTracker(timeout time.Duration, notifier *Notifier, now func() time.Time) *typingTracker {
+	return &typingTracker{
+		timeout:  timeout,
+		notifier: notifier,
+		now:      now,
+		rooms:    map[string]map[string]time.Time{},
+		timers:   map[typingKey]*time.Timer{},
+	}
+}
+
+// set records that a user is typing in a room for a while, or stops them.
+//
+// The timeout a client asks for is honoured up to the configured cap: a
+// client that asks for an hour would leave the room claiming somebody was
+// typing long after they closed it.
+func (t *typingTracker) set(roomID, userID string, typing bool, timeout time.Duration) {
+	t.mu.Lock()
+	if t.closed {
+		t.mu.Unlock()
+		return
+	}
+
+	key := typingKey{roomID: roomID, userID: userID}
+	if timer, ok := t.timers[key]; ok {
+		timer.Stop()
+		delete(t.timers, key)
+	}
+
+	switch {
+	case !typing:
+		if users, ok := t.rooms[roomID]; ok {
+			delete(users, userID)
+			if len(users) == 0 {
+				delete(t.rooms, roomID)
+			}
+		}
+	default:
+		if timeout <= 0 || timeout > t.timeout {
+			timeout = t.timeout
+		}
+		users, ok := t.rooms[roomID]
+		if !ok {
+			users = map[string]time.Time{}
+			t.rooms[roomID] = users
+		}
+		users[userID] = t.now().Add(timeout)
+		t.timers[key] = time.AfterFunc(timeout, func() { t.expire(key) })
+	}
+	t.mu.Unlock()
+
+	// Outside the lock: the notifier takes its own, and a sync woken by it
+	// will want to read this tracker.
+	t.notifier.NotifyRoom(roomID)
+}
+
+// expire drops a notification whose timer has fired and wakes the room.
+func (t *typingTracker) expire(key typingKey) {
+	t.mu.Lock()
+	delete(t.timers, key)
+	users, ok := t.rooms[key.roomID]
+	if ok {
+		delete(users, key.userID)
+		if len(users) == 0 {
+			delete(t.rooms, key.roomID)
+		}
+	}
+	t.mu.Unlock()
+
+	if ok {
+		t.notifier.NotifyRoom(key.roomID)
+	}
+}
+
+// users lists who is typing in a room, sorted, skipping anyone whose
+// notification has run out.
+//
+// The expiry check is here as well as in the timer because the two answer
+// different failure modes: the timer wakes a blocked client, and this makes
+// sure a notification is never *reported* after its deadline even if the
+// timer has not run yet.
+func (t *typingTracker) users(roomID string) []string {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+
+	now := t.now()
+	out := make([]string, 0, len(t.rooms[roomID]))
+	for userID, expires := range t.rooms[roomID] {
+		if expires.After(now) {
+			out = append(out, userID)
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// rooms with anybody typing, which is how a sync knows a room has an
+// ephemeral change without asking per room.
+func (t *typingTracker) activeRooms() []string {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+
+	now := t.now()
+	out := make([]string, 0, len(t.rooms))
+	for roomID, users := range t.rooms {
+		for _, expires := range users {
+			if expires.After(now) {
+				out = append(out, roomID)
+				break
+			}
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// close stops every timer. A server shutting down should not be held up by
+// a typing notification.
+func (t *typingTracker) close() {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+
+	t.closed = true
+	for key, timer := range t.timers {
+		timer.Stop()
+		delete(t.timers, key)
+	}
+	t.rooms = map[string]map[string]time.Time{}
+}
+
+// TypingRequest is PUT /rooms/{roomID}/typing/{userID}.
+type TypingRequest struct {
+	RoomID string
+
+	// User is the full user ID said to be typing, which the handler has
+	// already checked is the caller.
+	User string
+
+	Typing bool
+
+	// Timeout is how long the notification should last. Zero takes the
+	// server's own.
+	Timeout time.Duration
+}
+
+// SetTyping records a typing notification.
+//
+// Being in the room is required: a typing notification is published to
+// everybody in it, so it is a write to the room even though it is not an
+// event in it.
+func (s *API) SetTyping(ctx context.Context, req TypingRequest) error {
+	if err := s.requireJoined(ctx, req.RoomID, req.User); err != nil {
+		return err
+	}
+
+	s.typing.set(req.RoomID, req.User, req.Typing, req.Timeout)
+	return nil
+}
+
+// typingEvent is the m.typing EDU of a room, or nil when nobody is typing.
+//
+// The event is the whole list rather than a delta, which is what the spec
+// defines: a client replaces what it had, so a missed update corrects
+// itself on the next one.
+func (s *API) typingEvent(roomID string) *EphemeralEvent {
+	users := s.typing.users(roomID)
+	if len(users) == 0 {
+		return nil
+	}
+	content, err := json.Marshal(map[string][]string{"user_ids": users})
+	if err != nil {
+		// Marshalling a list of strings cannot fail; if it somehow did,
+		// dropping the notification is better than failing the sync.
+		s.log.Error("marshalling a typing notification failed", "room_id", roomID, "error", err)
+		return nil
+	}
+	return &EphemeralEvent{Type: "m.typing", Content: content}
+}