6 Commits

Include renames

Author	SHA1	Message	Date
tajniak81andClaude Opus 5.5	a94794b866	Phase 9: application services and admin completion ... Implements PLAN.md Phase 9. internal/appservice reads registration files (appservice.config_files), matches namespaces the way Synapse does (anchored at the start), and delivers events to each service as transactions read from the roomserver's event stream at a cursor kept in appservice_state. A transaction in flight is recorded, so after a restart it is resent under the same ID; failures back off from 2s to 5m, and a successful ping cuts the backoff short. In the Client-Server API an as_token acts as the service's sender or, with ?user_id=, as any existing user in its namespace. Services register and log in their users with m.login.application_service; exclusive user and alias namespaces are refused to everybody else with M_EXCLUSIVE. Unknown aliases and users in a namespace are asked of the service first. The ping endpoint reports M_URL_NOT_SET, M_BAD_STATUS, M_CONNECTION_FAILED and M_CONNECTION_TIMEOUT. Services' own requests, and their users' when rate_limited is false, skip the rate limiter. Users can change their password (POST /account/password) and deactivate their account (POST /account/deactivate), both behind the password. Deactivation revokes tokens, deletes devices, optionally erases the profile and leaves every room through a new userapi.AccountObserver. The admin API is complete: user detail, the admin flag, deactivation and reactivation, password resets, devices, media quotas (media.max_user_storage_bytes plus per-user overrides), registration tokens with use limits and expiry (registration.require_token), server notices (internal/servernotices, server_notices.*), an editor for the federation allow/deny lists persisted in server_settings, a delivery view of the application services, and an audit log of every change. Migration 0011 adds the registration token, server notice room, media quota, audit log and appservice cursor tables. Policy lists are now returned sorted. The Complement entrypoint loads the registrations Complement mounts in /complement/appservice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 12:58:28 +02:00
tajniak81andClaude Opus 5.5	b507753172	Phase 8: push and notifications ... Implements PLAN.md Phase 8. internal/pushrules is the rule language: the spec's default ruleset, all six condition kinds, glob and word-boundary matching, intentional mentions switching off the legacy body rules, and rule ordering as Synapse has it. internal/pushserver keeps the rules, pushers and notifications and talks to push gateways. Push rules are the m.push_rules account data, written only by the push server; reads keep the user's own rules and the enabled/actions of each default and take the rest from the current defaults, so new default rules reach every user without a migration. PUT account_data/m.push_rules is refused with 405. The push server listens on the roomserver's emit stage and evaluates each event on its own worker for every local member and an invite's target. Notifications and per-room unread counts are stored in migration 0010; counts are a seventh sync token stream, so /sync reports unread_notifications and picks up count changes made after the event or by a receipt from another device. m.read and m.read.private receipts clear the counts; leaving a room clears it. Pushers (http kind only) push through a bounded sender pool with retries, honour event_id_only, drop push keys the gateway rejects, and go away with their device. GET /notifications pages the history. New config section push: enabled, include_content, gateway_timeout, notification_retention. Account data writes now wake the user's blocked syncs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 11:43:36 +02:00
tajniak81andClaude Opus 5.5	fc49f56339	Phase 7: E2EE support ... Implements PLAN.md Phase 7: the server half of end-to-end encryption. Devices upload identity, one-time and fallback keys; other devices query and claim them, across servers too, and every one-time key is handed out exactly once. Cross-signing, signature uploads, device list tracking and server-side key backup complete the set. internal/keyserver is the component. Keys are stored in canonical JSON so that re-uploading the same key is a byte comparison, and signatures uploaded later are kept apart from the key they sign and merged in on read, so a key's own JSON stays what its owner uploaded. Device keys must be for the caller's own device; cross-signing keys must be the uploader's, carry the right usage and, for self- and user-signing keys, be signed by the master key. Signatures are verified against the key as stored. Replacing a master key asks for the password; a first upload or an upload of the keys already stored does not, as the spec allows. A signature of another user's master key is shown only to the user who made it and never federated. Every write that changes what /keys/query answers moves the device list stream in the same transaction, one position per device, and wakes the syncs of everyone sharing a room. The stream is the sixth field of a sync token; older tokens still parse. The syncapi works out device_lists changed and left from the stream and from membership moves in the window, skipping invites and knocks, and reports device_one_time_keys_count and device_unused_fallback_key_types, the latter as [] when empty. GET /keys/changes uses the same logic. Over federation: /user/keys/query, /user/keys/claim and /user/devices/{userId}, and m.device_list_update and m.signing_key_update both ways. Remote keys are not cached; an update about a remote user sharing a room moves them in the stream so local clients re-query. Device display names are not sent to other servers. With federation off, remote users are reported in failures at once. Deleting a device deletes its keys, through a new userapi DeviceObserver. Key backup keeps the better copy of each session by the spec's rule, never reuses a deleted version's number, and answers writes to a stale version with M_WRONG_ROOM_KEYS_VERSION. Storage is migration 0008 for sqlite and postgres, with a PocketBase stub. The Complement target sets gain the key, device list, fallback, cross-signing and backup tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 10:34:08 +02:00
tajniak81andClaude Opus 5.5	6aaf554336	Phase 5: federation ... Implements PLAN.md Phase 5: the Server-Server API. Warp joins rooms on other servers and lets their users join its rooms, invites across servers with the invitee's server countersigning, rejects remote invites (locally if the inviter is unreachable), and exchanges events, typing, receipts and to-device messages with other homeservers — all behind the runtime switch, which now starts and stops real outbound traffic. The protocol lives in internal/federation with no knowledge of rooms: X-Matrix signing (fuzzed parser), the spec's discovery algorithm keeping the dial address and the TLS name apart, a signed client that refuses every request while the switch is off or the policy denies the destination, and a key ring that caches other servers' keys for at most a week, with optional notaries. The roomserver takes events from elsewhere through the spec's intake: signatures and hashes (a hash mismatch keeps the redacted form), the rules against the event's own auth events, against the state before it — with /get_missing_events for small gaps and /state_ids for wide ones — and against the current state, where a failure is a soft failure. Events held without their state are outliers: a column on roomserver_events, because the current state of a room joined over federation is almost all outliers, and excluded from every timeline query. Forks are resolved at the room's head. Membership across servers is make_join/send_join, make_leave/send_leave and countersigned invites, in both directions; m.room.server_acl is enforced on everything another server sends. History before a remote join is backfilled when /messages pages past it, stored at negative stream positions. internal/federationapi is the running component: the inbound endpoints, and an outbound queue that persists what it owes each destination, sends it in transactions of 50 PDUs and 100 EDUs, backs off from ten seconds to six hours, survives restarts, and drops events persisted while federation is off rather than delivering a backlog in one burst. Its Module is what the switch starts and stops. Also: a TLS listener for port 8448, trusted key servers, key validity and an outbound CA bundle in the config; per-destination health and backoff reset in the admin API; remote profiles, aliases and directories in the Client-Server API; a Complement federation profile. See docs/federation.md, including its known gaps (no partial-state joins, no key rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 17:14:20 +02:00
tajniak81andClaude Opus 5	d187ad0d17	Phase 4: sync ... Implements PLAN.md Phase 4: the read model. /sync, /messages and /context, the filters that narrow them, and the streams a response carries that are not room events — typing, receipts, presence and to-device messages. With this a client can be used against Warp, which is milestone M1. The one architectural choice is that the syncapi has no copy of the rooms: it reads them through the roomserver. Dendrite gives its syncapi its own tables and feeds them from the output stream, which is right for a design where the two can be separate processes, and Dendrite's could be. Warp is one process by decision, and there the copy buys nothing and costs the hardest class of bug a homeserver has — a read model that disagrees with the room, where a client is shown a message that was rejected and the only way to find out is to compare two tables. So the roomserver grew a read model (stream.go, roomsync.go, visibility.go) rather than exporting its store: reading an event means parsing it under its room's version, and a second copy of the version registry, the parser and the client projection would be a second place for them to disagree. syncapi: stream tokens naming a position in each of five independent streams, so a client that has seen receipt 7 but only event 4 can say so in one opaque string. A token with fewer positions than this build writes is accepted and the rest read as zero, which is what lets a token stored before a stream existed go on working. The PDU position comes from MAX(stream_ordering) and not from the counter row the other four use: the counter is advanced inside the writing transaction, so a token taken from it would name a position whose event no reader can see yet, and the event would be skipped for good once that transaction committed. Long polling goes through a notifier that wakes by room and by user rather than globally — an event wakes the listeners of its room, a membership event also wakes the user it names, so an invite reaches a client that is not in the room yet. A wake-up is only ever a hint: the listener re-reads the streams and decides for itself what changed, which is what makes it safe for the notifier to hold no state about what each client has seen. The listener registers before the read, so an event landing in between leaves the channel signalled and the sync reads again rather than blocking on a change it just missed. EventPersisted must not block, because it is called with the room's write lock still held. "Something happened" is not "a token moved". A stream can move without this user being concerned by it, and returning on every such move would turn a long poll back into a busy loop. The same trap caught presence twice over: a client says it is still online by syncing, so ?set_presence= makes every sync a presence write, and a stream that moved each time would wake every peer, which would sync, which would move it again. SetPresence therefore reports whether anything a peer would notice actually changed, and a write that only refreshes the last-active timestamp leaves the stream alone. Timeline windows are named by their two ends and a direction, Low exclusive and High inclusive in both directions, rather than by "from" and "to" as the wire format spells them. Each of those would otherwise be the low bound in one direction and the high bound in the other, which is exactly the off-by-one every pagination bug is made of; the handlers translate once and nothing below them thinks about it again. A sync's timeline is read backwards and reported forwards, because a client wants the newest events of the window it missed — a forward read with a limit hands it the oldest and then calls the rest a gap. visibility: history visibility is now a span rather than a yes or no, which is what Phase 3 deferred here. A user who has left a room is served the room as it was when they left — their state reads, their members and their timeline — instead of being refused, and is bounded above at their own membership event so a kicked client cannot go on streaming. Their span is found from their own member events, so a user who joined, left and rejoined sees their latest span. ?at= on /members is honoured for the same reason. What is left is that the span is computed from the room's current history_visibility where the spec judges each event by the one in force when it was sent; that is a state read per event and belongs with open question 1. Typing is held in memory and nowhere else: notifications expire in seconds, are worthless across a restart, and arrive one per keystroke-burst per room, so persisting them would be the busiest table in the database holding the least valuable data in it. Receipts only ever advance, compared on the position of the event acknowledged rather than on arrival, because two of a user's clients read at different speeds and the slower one must not drag the marker backwards. To-device queues are per device and drained by the next sync rather than by the one that delivered them, so a client that crashes in between gets the batch again — the right way round, since a lost key share leaves a message permanently undecryptable and a duplicate is ignored. A wildcard addressee is expanded when sent, not stored, because "every device" means the ones that existed then. Presence is off by default, which answers PLAN.md open question 3. It is the noisiest thing in Matrix and the one genuinely fan-out notification here; a private homeserver gains little from it. With it off the endpoints refuse rather than disappearing, because a client that gets a 404 from /presence concludes the server is too old to have it and stops asking. storage: one migration per driver adding the receipt, to-device, presence and filter tables with a counter row each, and a stream position on the existing account data table rather than a parallel one — there is exactly one current value per (user, room, type), and a second table would be a second thing to keep in step. Account data is reported in an event list but is not an event, so it gets its own response type: a room event with an empty sender and no timestamp would make a client decide what that meant. Deliberately missing, each named in docs/sync.md: notification counts (they need push rules, Phase 8), device lists (the keyserver, Phase 7), anything remote (Phase 5), threads (Phase 10). device_lists is sent as two empty lists because a client reads a missing one as malformed; the notification counts are left out entirely, because a zero there is a claim and an absent field is not. Tested through HTTP rather than against the builder, because what is worth pinning down is the response a client sees — which section a room lands in, whether a token handed back can be handed in again, whether paging terminates and loses nothing. The storage tests run against both SQL drivers; the notifier has its own tests for the concurrent case. A live server was driven end to end, which is what caught the two bugs the unit tests had missed: a backwards /messages page must be in reverse chronological order, and `start` must name where the page began rather than position zero when ?from= was omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-16 11:55:27 +02:00
tajniak81andClaude Opus 5	c96f88e994	Phase 3: rooms, local only ... Implements PLAN.md Phase 3: the roomserver, the room half of storage, the Client-Server room endpoints and the admin room endpoints. A client can now create a room, invite, join, leave, kick, ban, send messages and state, redact, name a room with an alias and find one in the public directory. Nothing talks to another server; every operation that would need to is refused with a message that says so rather than a bare M_FORBIDDEN. roomserver: one input pipeline every event goes through, whatever sent it — validate, authorise, resolve, persist, emit. The rules are not reimplemented here; internal/matrix is pure library code and this is what connects it to a database and to a room's history. The Client-Server handlers do no authorisation of their own, deliberately: whether Alice may kick Bob is a question about the room's power levels, and the one place that is answered correctly is the authorisation rules inside the pipeline. A mutex per room serialises writers. Two events racing into one room would each read the same forward extremities, both become one, and fork the room for no reason — which would then need state resolution to put back together. State resolution is wired in anyway for the multi-extremity case. It is unreachable from outside in this phase, and the alternative is a pipeline that authorises against one arbitrary branch of a fork without saying so. Room versions 3 to 12 all work and are each tested end to end. Version 12 earns that coverage: its room ID is the create event's own ID, its create event carries no room_id, and the create event is excluded from every other event's auth_events — so the state auth.Check is given is the event's auth_events plus the create event, which that version identifies by room ID rather than citing. createRoom sends the spec's events in the spec's order, each authorised against what the ones before it produced. A failure part-way leaves an incomplete room, which is the lesser evil: the alternative is holding every event of a new room in memory and writing them in one transaction, making the pipeline that authorises them a special case. The one failure a second client can cause at any time, an alias clash, takes the room row back out. Redactions stay two decisions, as the spec keeps them. The rules let almost any redaction into the room; whether it applies to its target is a separate test, run inside the same transaction as the redaction itself so that a client holding an event ID can always read the stripped event. storage: events keyed by event ID rather than by a surrogate, since an event ID appears in prev_events, auth_events, state snapshots and client requests alike. Stream ordering comes from a counter row bumped inside the write transaction, because a PostgreSQL sequence is explicitly not transactional and a rolled-back write would leave a hole in the stream that /sync would have to be taught to skip. State snapshots are stored per event and deduplicated by content hash, so the table grows with state changes rather than with traffic — and that is what makes "the state after a forward extremity" answerable at all. The room's listing columns are denormalised onto its row because the public directory orders every published room by member count and pages the result, which cannot be done by reading each room's state in turn. Federation-disabled behaviour from PLAN.md section 3 is enforced here. Restricted joins are the exception that works: the rule needs a signature from the authorising user's server verified, and while federation is off that server is always this one. adminapi: room list and detail, purge, force-leave, and room and event counters on the dashboard. clientapi gains its own rate-limit buckets for createRoom and for joining, both of which write events rather than read them. Tested against the real event core and a real database rather than fakes: what is under test is whether an event this server signs is one the authorisation rules accept, and a fake of either would only prove that the pipeline calls what the fake expects. The storage tests run against both SQL drivers. docs/rooms.md describes the pipeline, the storage decisions and the known gaps — chief among them that history visibility is room-level rather than per-event, and that /sync still reports no rooms, both of which belong with the event stream in Phase 4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-10-16 10:34:57 +02:00