11 Commits

Include renames

Author	SHA1	Message	Date
tajniak81andClaude Opus 5.5	76dfc98a86	Group voice and video calls with MatrixRTC and LiveKit ... Every call in the web client now goes through MatrixRTC with a LiveKit SFU, the model Element Call and Element X use; the legacy 1:1 m.call.* calls are gone. In a room of two the other person is rung and can answer or decline; in a bigger room a banner offers to join the call. Calls show a grid of participant tiles with mute, camera, screen sharing and leave, and media is end-to-end encrypted in encrypted rooms. Server: - MSC4140 delayed events (send/state with a delay, plus the delayed_events list/restart/cancel/send API), stored and scheduled, so a member whose client drops off is taken out of the call. - POST /user/{userId}/openid/request_token, which lk-jwt-service uses to hand out LiveKit tokens. - org.matrix.msc4143.rtc_foci in /.well-known/matrix/client, configured under client_api.rtc. - Room presets let members send call membership state. - Migration 0014 for both SQL dialects. Deploy: LiveKit and lk-jwt-service in compose (calls.yml) and the helm chart. Docs: docs/calls.md, configuration and deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-18 19:33:29 +02:00
tajniak81andClaude Opus 5.5	325431a5be	Voice and video calls: TURN endpoint and web client calls ... Server: GET /voip/turnServer, configured under client_api.turn. A shared secret (coturn use-auth-secret) gives each user expiring TURN REST API credentials; static credentials are the fallback; with nothing configured it answers {} so clients still try a direct connection. Web client: 1:1 calls (Matrix VoIP via matrix-js-sdk) in rooms with one other person. Voice and video buttons in the room header, a ringing card with answer, answer without video and decline, and a call panel that stays up across rooms: mute, camera, screen sharing, expand, hang up, ring tones. Call events show in the timeline and room list. Works around the SDK keeping a failed microphone request forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-18 18:04:02 +02:00
tajniak81andClaude Opus 5.5	487e0f410b	Phase 10: hardening and ops ... Implements PLAN.md Phase 10. Simplified Sliding Sync (MSC4186) is served at the unstable org.matrix.simplified_msc3575 endpoint and advertised in /versions. Lists are sorted by activity and filtered by DM, invite, encryption, room type, space and tag; room subscriptions, required_state with wildcards, $ME and $LAZY, and the to_device, e2ee, account_data, receipts and typing extensions are supported. Per-connection state is kept in memory and an unknown pos answers M_UNKNOWN_POS. Threads and relations: m.relates_to is indexed on roomserver_events (migration 0012), /relations (with recursion) and /threads are served, and every event a timeline read returns carries its m.thread, m.replace and m.reference summaries in unsigned.m.relations. Notifications carry a thread ID (migration 0013): threaded receipts clear only their thread, and sync reports unread_thread_notifications when the filter asks. Spaces: /hierarchy walks a space breadth first, asking other servers about rooms this one is not in through the new federation hierarchy endpoint; room_summary describes one room. /search runs on FTS5 (sqlite, pocketbase) or tsvector (postgres), bounded to what each user may see, with context, profiles, state, grouping and paging. The PocketBase driver is real: the sqlite schema and store on an embedded PocketBase app's database, with media in its filesystem, local or S3 (database.pocketbase.s3). It runs the full storage test suite. Ops: OpenTelemetry tracing over OTLP/HTTP (tracing.*), with server spans per route, component spans, and trace propagation on federation, push and application service calls. New metrics for persisted events, waiting syncs, sliding sync connections and the database pool, with a Grafana dashboard. deploy/ adds a Compose setup (PostgreSQL, Prometheus, Grafana, Jaeger) and a Helm chart; cmd/warp-loadtest loads a server and reports request and delivery latencies. The load tester found a long-standing bug, fixed here: on a server where no account data had been written, every incremental /sync re-sent the push rules and returned at once, and an all-zero token was taken for an initial sync, so long polls became busy loops. The PostgreSQL migrations and search dialect compile but were not run against a live PostgreSQL in this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 20:21:31 +02:00
tajniak81andClaude Opus 5.5	a94794b866	Phase 9: application services and admin completion ... Implements PLAN.md Phase 9. internal/appservice reads registration files (appservice.config_files), matches namespaces the way Synapse does (anchored at the start), and delivers events to each service as transactions read from the roomserver's event stream at a cursor kept in appservice_state. A transaction in flight is recorded, so after a restart it is resent under the same ID; failures back off from 2s to 5m, and a successful ping cuts the backoff short. In the Client-Server API an as_token acts as the service's sender or, with ?user_id=, as any existing user in its namespace. Services register and log in their users with m.login.application_service; exclusive user and alias namespaces are refused to everybody else with M_EXCLUSIVE. Unknown aliases and users in a namespace are asked of the service first. The ping endpoint reports M_URL_NOT_SET, M_BAD_STATUS, M_CONNECTION_FAILED and M_CONNECTION_TIMEOUT. Services' own requests, and their users' when rate_limited is false, skip the rate limiter. Users can change their password (POST /account/password) and deactivate their account (POST /account/deactivate), both behind the password. Deactivation revokes tokens, deletes devices, optionally erases the profile and leaves every room through a new userapi.AccountObserver. The admin API is complete: user detail, the admin flag, deactivation and reactivation, password resets, devices, media quotas (media.max_user_storage_bytes plus per-user overrides), registration tokens with use limits and expiry (registration.require_token), server notices (internal/servernotices, server_notices.*), an editor for the federation allow/deny lists persisted in server_settings, a delivery view of the application services, and an audit log of every change. Migration 0011 adds the registration token, server notice room, media quota, audit log and appservice cursor tables. Policy lists are now returned sorted. The Complement entrypoint loads the registrations Complement mounts in /complement/appservice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 12:58:28 +02:00
tajniak81andClaude Opus 5.5	b507753172	Phase 8: push and notifications ... Implements PLAN.md Phase 8. internal/pushrules is the rule language: the spec's default ruleset, all six condition kinds, glob and word-boundary matching, intentional mentions switching off the legacy body rules, and rule ordering as Synapse has it. internal/pushserver keeps the rules, pushers and notifications and talks to push gateways. Push rules are the m.push_rules account data, written only by the push server; reads keep the user's own rules and the enabled/actions of each default and take the rest from the current defaults, so new default rules reach every user without a migration. PUT account_data/m.push_rules is refused with 405. The push server listens on the roomserver's emit stage and evaluates each event on its own worker for every local member and an invite's target. Notifications and per-room unread counts are stored in migration 0010; counts are a seventh sync token stream, so /sync reports unread_notifications and picks up count changes made after the event or by a receipt from another device. m.read and m.read.private receipts clear the counts; leaving a room clears it. Pushers (http kind only) push through a bounded sender pool with retries, honour event_id_only, drop push keys the gateway rejects, and go away with their device. GET /notifications pages the history. New config section push: enabled, include_content, gateway_timeout, notification_retention. Account data writes now wake the user's blocked syncs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 11:43:36 +02:00
tajniak81andClaude Opus 5.5	92836f8dcd	Phase 6: media ... Implements PLAN.md Phase 6: the media repository. Clients upload files, reserve an mxc:// URI and upload to it later, download and ask for thumbnails, all through the authenticated media endpoints of Matrix 1.11; other servers download this server's media through the federation media endpoints, and media uploaded elsewhere is fetched over federation on first use and kept. internal/mediaapi is the component. Content goes into a BlobStore named by its SHA-256, so the same file uploaded twice is stored once and the hash doubles as the ETag; the metadata that maps an mxc:// URI onto a blob is in media_repository and media_thumbnails. The local-filesystem store writes to a temporary file and renames it into place once hashed, so nothing ever refers to half a file. A storage driver can bring its own store through storage.BlobProvider, which is where PocketBase's filesystem abstraction (local or S3) plugs in with Phase 10. Reserved uploads belong to the user who made them, take one upload, lapse after a day, and are capped per user; a download of one waits for its content, woken in memory the moment it arrives, for as long as timeout_ms allows. Thumbnails come from a fixed set of configured sizes, so a thousand requested sizes cannot mean a thousand resizes. They are made on first request and kept, never enlarge, honour a JPEG's EXIF orientation, and point at the original when it already fits. The pixel count is checked from the header before anything is decoded, at most four resizes run at once, and concurrent requests for the same thumbnail share one. Remote media is fetched through the signed client: the multipart/mixed answer of Matrix 1.11, with a fallback to the legacy download endpoint for servers that answer M_UNRECOGNIZED, and Location redirects followed by a dialer that refuses loopback, private and link-local addresses. Concurrent requests share one fetch, detached from the request that started it. With federation off nothing is fetched, and what was cached before is still served. Downloads carry a sandboxing CSP, nosniff, inline Content-Disposition only for the types the spec lists as safe, and private immutable caching. The unauthenticated legacy download endpoints are off by default behind media.legacy_unauthenticated_downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 21:26:49 +02:00
tajniak81andClaude Opus 5.5	6aaf554336	Phase 5: federation ... Implements PLAN.md Phase 5: the Server-Server API. Warp joins rooms on other servers and lets their users join its rooms, invites across servers with the invitee's server countersigning, rejects remote invites (locally if the inviter is unreachable), and exchanges events, typing, receipts and to-device messages with other homeservers — all behind the runtime switch, which now starts and stops real outbound traffic. The protocol lives in internal/federation with no knowledge of rooms: X-Matrix signing (fuzzed parser), the spec's discovery algorithm keeping the dial address and the TLS name apart, a signed client that refuses every request while the switch is off or the policy denies the destination, and a key ring that caches other servers' keys for at most a week, with optional notaries. The roomserver takes events from elsewhere through the spec's intake: signatures and hashes (a hash mismatch keeps the redacted form), the rules against the event's own auth events, against the state before it — with /get_missing_events for small gaps and /state_ids for wide ones — and against the current state, where a failure is a soft failure. Events held without their state are outliers: a column on roomserver_events, because the current state of a room joined over federation is almost all outliers, and excluded from every timeline query. Forks are resolved at the room's head. Membership across servers is make_join/send_join, make_leave/send_leave and countersigned invites, in both directions; m.room.server_acl is enforced on everything another server sends. History before a remote join is backfilled when /messages pages past it, stored at negative stream positions. internal/federationapi is the running component: the inbound endpoints, and an outbound queue that persists what it owes each destination, sends it in transactions of 50 PDUs and 100 EDUs, backs off from ten seconds to six hours, survives restarts, and drops events persisted while federation is off rather than delivering a backlog in one burst. Its Module is what the switch starts and stops. Also: a TLS listener for port 8448, trusted key servers, key validity and an outbound CA bundle in the config; per-destination health and backoff reset in the admin API; remote profiles, aliases and directories in the Client-Server API; a Complement federation profile. See docs/federation.md, including its known gaps (no partial-state joins, no key rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 17:14:20 +02:00
tajniak81andClaude Opus 5	d187ad0d17	Phase 4: sync ... Implements PLAN.md Phase 4: the read model. /sync, /messages and /context, the filters that narrow them, and the streams a response carries that are not room events — typing, receipts, presence and to-device messages. With this a client can be used against Warp, which is milestone M1. The one architectural choice is that the syncapi has no copy of the rooms: it reads them through the roomserver. Dendrite gives its syncapi its own tables and feeds them from the output stream, which is right for a design where the two can be separate processes, and Dendrite's could be. Warp is one process by decision, and there the copy buys nothing and costs the hardest class of bug a homeserver has — a read model that disagrees with the room, where a client is shown a message that was rejected and the only way to find out is to compare two tables. So the roomserver grew a read model (stream.go, roomsync.go, visibility.go) rather than exporting its store: reading an event means parsing it under its room's version, and a second copy of the version registry, the parser and the client projection would be a second place for them to disagree. syncapi: stream tokens naming a position in each of five independent streams, so a client that has seen receipt 7 but only event 4 can say so in one opaque string. A token with fewer positions than this build writes is accepted and the rest read as zero, which is what lets a token stored before a stream existed go on working. The PDU position comes from MAX(stream_ordering) and not from the counter row the other four use: the counter is advanced inside the writing transaction, so a token taken from it would name a position whose event no reader can see yet, and the event would be skipped for good once that transaction committed. Long polling goes through a notifier that wakes by room and by user rather than globally — an event wakes the listeners of its room, a membership event also wakes the user it names, so an invite reaches a client that is not in the room yet. A wake-up is only ever a hint: the listener re-reads the streams and decides for itself what changed, which is what makes it safe for the notifier to hold no state about what each client has seen. The listener registers before the read, so an event landing in between leaves the channel signalled and the sync reads again rather than blocking on a change it just missed. EventPersisted must not block, because it is called with the room's write lock still held. "Something happened" is not "a token moved". A stream can move without this user being concerned by it, and returning on every such move would turn a long poll back into a busy loop. The same trap caught presence twice over: a client says it is still online by syncing, so ?set_presence= makes every sync a presence write, and a stream that moved each time would wake every peer, which would sync, which would move it again. SetPresence therefore reports whether anything a peer would notice actually changed, and a write that only refreshes the last-active timestamp leaves the stream alone. Timeline windows are named by their two ends and a direction, Low exclusive and High inclusive in both directions, rather than by "from" and "to" as the wire format spells them. Each of those would otherwise be the low bound in one direction and the high bound in the other, which is exactly the off-by-one every pagination bug is made of; the handlers translate once and nothing below them thinks about it again. A sync's timeline is read backwards and reported forwards, because a client wants the newest events of the window it missed — a forward read with a limit hands it the oldest and then calls the rest a gap. visibility: history visibility is now a span rather than a yes or no, which is what Phase 3 deferred here. A user who has left a room is served the room as it was when they left — their state reads, their members and their timeline — instead of being refused, and is bounded above at their own membership event so a kicked client cannot go on streaming. Their span is found from their own member events, so a user who joined, left and rejoined sees their latest span. ?at= on /members is honoured for the same reason. What is left is that the span is computed from the room's current history_visibility where the spec judges each event by the one in force when it was sent; that is a state read per event and belongs with open question 1. Typing is held in memory and nowhere else: notifications expire in seconds, are worthless across a restart, and arrive one per keystroke-burst per room, so persisting them would be the busiest table in the database holding the least valuable data in it. Receipts only ever advance, compared on the position of the event acknowledged rather than on arrival, because two of a user's clients read at different speeds and the slower one must not drag the marker backwards. To-device queues are per device and drained by the next sync rather than by the one that delivered them, so a client that crashes in between gets the batch again — the right way round, since a lost key share leaves a message permanently undecryptable and a duplicate is ignored. A wildcard addressee is expanded when sent, not stored, because "every device" means the ones that existed then. Presence is off by default, which answers PLAN.md open question 3. It is the noisiest thing in Matrix and the one genuinely fan-out notification here; a private homeserver gains little from it. With it off the endpoints refuse rather than disappearing, because a client that gets a 404 from /presence concludes the server is too old to have it and stops asking. storage: one migration per driver adding the receipt, to-device, presence and filter tables with a counter row each, and a stream position on the existing account data table rather than a parallel one — there is exactly one current value per (user, room, type), and a second table would be a second thing to keep in step. Account data is reported in an event list but is not an event, so it gets its own response type: a room event with an empty sender and no timestamp would make a client decide what that meant. Deliberately missing, each named in docs/sync.md: notification counts (they need push rules, Phase 8), device lists (the keyserver, Phase 7), anything remote (Phase 5), threads (Phase 10). device_lists is sent as two empty lists because a client reads a missing one as malformed; the notification counts are left out entirely, because a zero there is a claim and an absent field is not. Tested through HTTP rather than against the builder, because what is worth pinning down is the response a client sees — which section a room lands in, whether a token handed back can be handed in again, whether paging terminates and loses nothing. The storage tests run against both SQL drivers; the notifier has its own tests for the concurrent case. A live server was driven end to end, which is what caught the two bugs the unit tests had missed: a backwards /messages page must be in reverse chronological order, and `start` must name where the page began rather than position zero when ?from= was omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-16 11:55:27 +02:00
tajniak81andClaude Opus 5	c96f88e994	Phase 3: rooms, local only ... Implements PLAN.md Phase 3: the roomserver, the room half of storage, the Client-Server room endpoints and the admin room endpoints. A client can now create a room, invite, join, leave, kick, ban, send messages and state, redact, name a room with an alias and find one in the public directory. Nothing talks to another server; every operation that would need to is refused with a message that says so rather than a bare M_FORBIDDEN. roomserver: one input pipeline every event goes through, whatever sent it — validate, authorise, resolve, persist, emit. The rules are not reimplemented here; internal/matrix is pure library code and this is what connects it to a database and to a room's history. The Client-Server handlers do no authorisation of their own, deliberately: whether Alice may kick Bob is a question about the room's power levels, and the one place that is answered correctly is the authorisation rules inside the pipeline. A mutex per room serialises writers. Two events racing into one room would each read the same forward extremities, both become one, and fork the room for no reason — which would then need state resolution to put back together. State resolution is wired in anyway for the multi-extremity case. It is unreachable from outside in this phase, and the alternative is a pipeline that authorises against one arbitrary branch of a fork without saying so. Room versions 3 to 12 all work and are each tested end to end. Version 12 earns that coverage: its room ID is the create event's own ID, its create event carries no room_id, and the create event is excluded from every other event's auth_events — so the state auth.Check is given is the event's auth_events plus the create event, which that version identifies by room ID rather than citing. createRoom sends the spec's events in the spec's order, each authorised against what the ones before it produced. A failure part-way leaves an incomplete room, which is the lesser evil: the alternative is holding every event of a new room in memory and writing them in one transaction, making the pipeline that authorises them a special case. The one failure a second client can cause at any time, an alias clash, takes the room row back out. Redactions stay two decisions, as the spec keeps them. The rules let almost any redaction into the room; whether it applies to its target is a separate test, run inside the same transaction as the redaction itself so that a client holding an event ID can always read the stripped event. storage: events keyed by event ID rather than by a surrogate, since an event ID appears in prev_events, auth_events, state snapshots and client requests alike. Stream ordering comes from a counter row bumped inside the write transaction, because a PostgreSQL sequence is explicitly not transactional and a rolled-back write would leave a hole in the stream that /sync would have to be taught to skip. State snapshots are stored per event and deduplicated by content hash, so the table grows with state changes rather than with traffic — and that is what makes "the state after a forward extremity" answerable at all. The room's listing columns are denormalised onto its row because the public directory orders every published room by member count and pages the result, which cannot be done by reading each room's state in turn. Federation-disabled behaviour from PLAN.md section 3 is enforced here. Restricted joins are the exception that works: the rule needs a signature from the authorising user's server verified, and while federation is off that server is always this one. adminapi: room list and detail, purge, force-leave, and room and event counters on the dashboard. clientapi gains its own rate-limit buckets for createRoom and for joining, both of which write events rather than read them. Tested against the real event core and a real database rather than fakes: what is under test is whether an event this server signs is one the authorisation rules accept, and a fake of either would only prove that the pipeline calls what the fake expects. The storage tests run against both SQL drivers. docs/rooms.md describes the pipeline, the storage decisions and the known gaps — chief among them that history visibility is room-level rather than per-event, and that /sync still reports no rooms, both of which belong with the event stream in Phase 4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-16 10:34:57 +02:00
tajniak81andClaude Opus 5	6004bb01de	Phase 1: accounts and sessions ... Implements PLAN.md Phase 1. A client can now register, log in, manage its devices and set its profile; an operator can create accounts and flip the runtime switches. There are still no rooms and no real /sync. userapi: accounts, argon2id passwords in PHC format, devices, access and refresh tokens stored as SHA-256 hashes, account data and the admin flag. Its storage interface is storage.UserStore, implemented once in sqlstore with `?` placeholders; the postgres driver supplies a rebind function and unique-violation recognition rather than keeping a second copy of thirty statements. Timestamps in these tables are int64 milliseconds, the unit the Client-Server API uses on the wire. Client-Server API: register (user-interactive auth with m.login.dummy and optional registration tokens), register/available, token validity, login with m.login.password, logout, logout/all, refresh, account/whoami, capabilities, devices/* with password-gated deletion, profile/*, global and per-room account data. GET /sync is a deliberate placeholder: it answers a valid token with an empty but well-formed response and refuses a revoked one. Phase 1 has no rooms, so that is the whole truth, and it is what makes session revocation observable. Phase 4 replaces it with the syncapi. Auth middleware resolves a bearer token (header or the legacy query parameter) to a device and user, answering an expired token with the soft logout the spec asks for. Rate limiting is a token bucket per endpoint class keyed on the client address, returning M_LIMIT_EXCEEDED with retry_after_ms. adminapi v1 under /_warp/admin/v1: admin-only auth via a normal Matrix token, GET/PUT /federation and /registration driving internal/settings and reporting a config-pinned switch as a 409, GET/POST /users, GET /stats. warp create-account [--admin] creates the first account straight in the database, so a new install with registration off has a way in. Complement is wired up for the federation=off profile: an image meeting its contract, an entrypoint that configures Warp entirely from WARP_* variables, and a runner that builds, clones and executes the target set (TestRegistration, TestLogin, TestDeviceManagement). It has not been run yet -- there is no container runtime on the development machine -- so the three tests' assertions were read from the Complement sources and pinned as httptest tests instead, and CI gained a job that runs the real thing. Also fixes a .gitignore bug from Phase 0: the unanchored `warp` pattern, meant for the built binary, also matched the directory server/cmd/warp and silently excluded the entire command package from the repository. The binary patterns are now anchored to the repository root, and cmd/warp is committed for the first time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-15 18:18:37 +02:00
tajniak81andClaude Opus 5	b4fb60a8ab	Phase 0: bootstrap the Warp homeserver ... Scaffolds the server module and everything Phase 0 of PLAN.md calls for: a binary that boots, serves its discovery endpoints and nothing else yet. - config: one YAML file plus per-field WARP_* environment overrides, unknown fields rejected, validation that reports every problem at once - logging and metrics: slog, a Prometheus registry, per-route HTTP instrumentation labelled by pattern rather than path - httputil: the Matrix error envelope, error-returning handlers, CORS, request logging, panic recovery, and a router over http.ServeMux - endpoints: /_matrix/client/versions, /.well-known/matrix/client, /.well-known/matrix/server (gated), /healthz, /metrics, and M_UNRECOGNIZED for everything else - storage: driver registry, shared database/sql base, checksummed migration runner, sqlite and postgres drivers, pocketbase stub - settings: server_settings table behind a cached service, so the federation and registration switches change without a restart - federation: the gate that makes federation a runtime mode - cmd/warp: serve, migrate, generate-keys, version - Makefile, golangci-lint v2, GitHub Actions CI (Linux and Windows tests, race on Linux, storage tests against PostgreSQL, lint, boot smoke test), Dockerfile, example config and docs Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-15 17:01:13 +02:00