diff --git a/README.md b/README.md index 4beb2f7..e78f56f 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,25 @@ what exists today. ## Status +**Phase 9 — application services and the admin API. Complete.** Bridges +and bots can be registered from their registration files. A service acts +as its own sender or, with `?user_id=`, as any user in its namespace; +registers and logs in its users with `m.login.application_service`; and +has its exclusive namespaces kept from everybody else with `M_EXCLUSIVE`. +Events reach it as transactions read from the event stream at a cursor +kept in the database. A service that was down is caught up from where it +stopped, and a transaction cut off by a restart is resent under the same +ID. Unknown aliases and users in a service's namespace are asked of the +service first, so portals and puppets appear on demand, and the ping +endpoint reports exactly why a service cannot be reached. Users can change +their password and deactivate their account. The admin API is complete: +deactivating and reactivating users, password resets, devices, the admin +flag, media quotas, registration tokens with use limits and expiry, server +notices, an editor for the federation allow and deny lists, a delivery +view of every application service, and an audit log of every change made +through it. Known gaps — no ephemeral events for services, no third-party +lookups — are listed in [docs/appservices.md](docs/appservices.md). + **Phase 8 — push and notifications. Complete.** Every event is checked against the push rules of every local member of its room. Warp has the spec's full default ruleset, all six condition kinds, intentional mentions @@ -50,8 +69,8 @@ federation on first use — with the fallback older servers need, and redirects that cannot be pointed into a private network — and cached, so it is still there with federation switched off. Every download carries the headers that keep an uploaded file from running as a page on the -homeserver's origin. Known gaps — no URL previews, no deletion or quotas -until Phase 9 — are listed in [docs/media.md](docs/media.md). +homeserver's origin. Known gaps — no URL previews, no deletion — are +listed in [docs/media.md](docs/media.md). **Phase 5 — federation. Complete.** Warp talks to other homeservers. It joins rooms on other servers and lets their users join its rooms, invites @@ -108,9 +127,11 @@ Client-Server API: | | | |---|---| -| `POST /register`, `GET /register/available` | sign-up, with `m.login.dummy` and optional registration tokens | -| `GET`, `POST /login` · `POST /logout`, `/logout/all` · `POST /refresh` | sessions, with `m.login.password` and refresh tokens | +| `POST /register`, `GET /register/available` | sign-up, with `m.login.dummy` and optional registration tokens; application services register their users | +| `GET`, `POST /login` · `POST /logout`, `/logout/all` · `POST /refresh` | sessions, with `m.login.password`, `m.login.application_service` and refresh tokens | | `GET /account/whoami` · `GET /capabilities` | who am I, and what this build can do | +| `POST /account/password` · `POST /account/deactivate` | changing your password and closing your account, both behind the password | +| `POST /_matrix/client/v1/appservice/{id}/ping` | an application service checking this server can reach it; see [docs/appservices.md](docs/appservices.md) | | `GET /devices`, `GET`/`PUT`/`DELETE /devices/{id}` · `POST /delete_devices` | device management; deletion re-asks for the password | | `GET /profile/{user}` and its `displayname` / `avatar_url` sub-resources | profiles — reads are public, remote ones asked of their server, writes are your own | | `GET`/`PUT /user/{user}/account_data/{type}` and the per-room variant | account data, stored and returned byte for byte | @@ -146,10 +167,12 @@ Client-Server API: | anything else under `/_matrix/` | `404 M_UNRECOGNIZED`, as the spec requires | Admin API under `/_warp/admin/v1`: the federation and registration -switches, per-server federation health, user listing and creation, the -room list and detail, purging a room, removing a user from one, and the -dashboard counters. See -[docs/admin-api.md](docs/admin-api.md). +switches and the allow/deny list editor, per-server federation health, +users (listing, creation, the admin flag, deactivation, password resets, +devices, media quotas), registration tokens, server notices, the +application services and their delivery, the room list and detail, +purging a room, removing a user from one, the audit log and the dashboard +counters. See [docs/admin-api.md](docs/admin-api.md). Also in place: argon2id passwords, hashed access and refresh tokens, per-endpoint rate limiting, YAML + environment configuration, structured @@ -287,10 +310,12 @@ server/ the homeserver (Go module) federation/ the Server-Server protocol: gate, X-Matrix, discovery, client, keys federationapi/ federation endpoints, the outbound queue, the switchable Module (docs/federation.md) settings/ runtime settings service (the switches) - userapi/ accounts, passwords, devices, tokens, account data + userapi/ accounts, passwords, devices, tokens, account data, deactivation, registration tokens roomserver/ rooms: the input pipeline (docs/rooms.md) mediaapi/ uploads, downloads, thumbnails, remote media (docs/media.md) keyserver/ device keys, one-time keys, cross-signing, key backup (docs/e2ee.md) + appservice/ application services: registrations, namespaces, delivery (docs/appservices.md) + servernotices/ messages from the server to its users clientapi/ the Client-Server API handlers adminapi/ /_warp/admin/v1 setup/ wiring: storage, routes, graceful shutdown diff --git a/docs/README.md b/docs/README.md index 547a7bb..c194f4b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ | [media.md](media.md) | `internal/mediaapi`: uploads, downloads, thumbnails, other servers' media, where the files go | | [e2ee.md](e2ee.md) | `internal/keyserver`: device and one-time keys, cross-signing, device list tracking, key backup | | [push.md](push.md) | `internal/pushrules` and `internal/pushserver`: push rules, notification counts, pushers, the push gateway client | +| [appservices.md](appservices.md) | `internal/appservice`: registration files, namespaces, acting as users, delivering events to bridges | | [admin-api.md](admin-api.md) | `/_warp/admin/v1`: the operator API behind the admin panel | The design and roadmap are in [PLAN.md](../PLAN.md) at the repository root. diff --git a/docs/admin-api.md b/docs/admin-api.md index d35cafa..50bed53 100644 --- a/docs/admin-api.md +++ b/docs/admin-api.md @@ -44,7 +44,8 @@ The runtime federation switch of PLAN.md section 3. "enabled": false, "toggleable": true, "allowlist": [], - "denylist": [] + "denylist": [], + "lists_edited": false } ``` @@ -54,8 +55,28 @@ and answers with the same document. `toggleable` is `false` when `PUT` then returns `409` and the panel should grey the switch out rather than offer something that will bounce. -Editing the allow and deny lists is Phase 9. They are reported now because -the dashboard has to show which servers a running Warp will talk to. +The lists are lower-cased and sorted. `lists_edited` is `true` when they +were set through `PUT /federation/lists` and replace the config file's. + +### `PUT /federation/lists`, `DELETE /federation/lists` + +```json +{"allowlist": ["friend.example"], "denylist": []} +``` + +Replaces both lists. They take effect on the next request in either +direction, and are kept in `server_settings` in place of the config +file's, across restarts. Both have to be in the body (`[]` for an empty +one): a panel that sent one and had the other silently kept would be one +bug away from leaving a server it meant to cut off on the allowlist. A +name that is not a server name is `400 M_INVALID_PARAM`. + +`DELETE` forgets the edited lists and puts the config file's back. Both +answer with the `GET /federation` document. + +Unlike the switch, the lists are editable whatever +`federation.allow_runtime_toggle` says: they are not a switch the file can +pin. ### `GET /federation/destinations` @@ -138,6 +159,259 @@ the admin's own action, not a public sign-up. `username` is normalised (lower-cased) like any other localpart. A taken name is `400 M_USER_IN_USE`, an unusable one `400 M_INVALID_USERNAME`. +### One user: `/users/{userID}` + +`{userID}` is a full user ID of this server, `@alice:warp.example`. A user +of another server is `400 M_INVALID_PARAM`, one that does not exist +`404 M_NOT_FOUND`. + +`GET /users/{userID}` is the list row plus what it takes a count to know: + +```json +{ + "user_id": "@alice:warp.example", + "display_name": "Alice", + "admin": false, + "deactivated": false, + "created_at_ms": 1760000000000, + "devices": 2, + "joined_rooms": 5, + "media": {"uploads": 12, "bytes": 3145728, "quota_bytes": 104857600, "custom_quota": false} +} +``` + +`joined_rooms` is absent on a server without a signing key, which has no +rooms. Every change below answers with this document, so the panel can +redraw from the response. + +| Endpoint | Body | What | +|---|---|---| +| `PUT /users/{userID}/admin` | `{"admin": true}` | grant or revoke the admin flag | +| `POST /users/{userID}/deactivate` | `{"erase": false}` (optional) | deactivate the account | +| `POST /users/{userID}/reactivate` | — | let it log in again | +| `POST /users/{userID}/password` | `{"password": "...", "logout_devices": true}` | set a new password | + +**Deactivation** does what the user's own `POST /account/deactivate` does. +Every token stops working at once; the devices go, with their keys and +pushers; the user leaves every room, rejects every invite and withdraws +every knock. With `erase`, the display name and avatar are cleared first, +so the leave events carry neither. The name stays taken: a new user with +the old user's ID would inherit what other servers and clients remember +about the old one. Deactivating a deactivated account changes nothing. + +**Reactivation** gives back the ability to log in, with the old password +unless a new one is set. Nothing else comes back: the rooms were left and +the devices are gone. + +**A password reset** logs every device out unless `logout_devices` is +`false`. An admin resetting a password is usually doing it because +somebody else has the account. + +An admin cannot revoke their own admin flag or deactivate themselves +(`403`). Both are one click from locking the last admin out of the panel, +and both are easy to do by accident on a table of users. Another admin can +still do either. + +### `GET /users/{userID}/devices`, `DELETE /users/{userID}/devices[/{deviceID}]` + +```json +{"devices": [{"device_id": "QWERTYUIOP", "display_name": "Phone", "created_at_ms": 1760000000000, + "last_seen_ms": 1760600000000, "last_seen_ip": "203.0.113.4", "user_agent": "Element/1.11"}]} +``` + +`DELETE` on one device logs that session out, taking its keys and pushers +with it. An unknown device is `404`. On the collection it logs out every +device. Unlike the client's own device deletion, no password is asked for: +the admin token is the authority here. + +### `GET /users/{userID}/media`, `PUT`/`DELETE /users/{userID}/media/quota` + +```json +{"uploads": 12, "bytes": 3145728, "quota_bytes": 104857600, "custom_quota": false} +``` + +What the user has uploaded, against their quota. Every upload counts at +its own size, even when two are the same bytes and the server keeps one +copy: a quota is about what a user asked this server to keep. +`quota_bytes` is the quota in force, `0` for none. `custom_quota` says +whether it was set for this user or comes from +`media.max_user_storage_bytes`. + +`PUT {"max_bytes": 1073741824}` gives the user a quota of their own (`0` +for no limit). `DELETE` puts them back on the configured one. A quota below +what the user already has deletes nothing; it stops further uploads until +they are back under it. + +An upload that would go over is refused as it arrives, with +`413 M_TOO_LARGE`. Every client already tells the user an upload was too +big, and that is what it was. Two uploads racing for the last of a quota +can both get in: a quota bounds what a user keeps, it is not an +accounting system. + +### Registration tokens + +Tokens with a use limit and an expiry, for the `m.login.registration_token` +stage of sign-up. The stage is in the flow when `registration.tokens` is +non-empty or `registration.require_token` is on. The config file's tokens +and these are both accepted. The config file's are never listed here: +they are in the config file, and this would be the one place a secret +from it is served over HTTP. + +``` +GET /registration_tokens +POST /registration_tokens {"token": "...", "length": 16, "uses_allowed": 5, "expiry_time_ms": 1760000000000} +GET /registration_tokens/{token} +PUT /registration_tokens/{token} {"uses_allowed": null, "expiry_time_ms": 1760000000000} +DELETE /registration_tokens/{token} +``` + +A token: + +```json +{ + "token": "invite-1", + "uses_allowed": 5, + "completed": 2, + "expiry_time_ms": null, + "created_at_ms": 1760000000000, + "valid": true +} +``` + +`uses_allowed` and `expiry_time_ms` are `null` for no limit. `valid` is +whether a registration could use the token now. The list also carries +`"required"`, whether sign-up asks for a token at all. A token made on a +server that does not ask is never needed. + +`POST` with no `token` generates one of `length` characters (16 by +default) from `A-Za-z0-9`. A chosen token is 1 to 64 of the characters +`A-Za-z0-9._~-`, the spec's grammar. A taken or malformed one is +`400 M_INVALID_PARAM`. It answers `201`. + +`PUT` changes only the fields in the body; `null` removes a limit. Raising +`uses_allowed` on a used-up token makes it usable again. `DELETE` does not +affect accounts registered with the token. + +A use is spent when the account is created, not when the stage is passed, +so an abandoned sign-up costs nothing. Checking and spending are one +statement, so two sign-ups racing for the last use cannot both have it. +The one that loses is refused with `403 M_FORBIDDEN` at the end of its +flow. + +### `POST /server_notices` + +```json +{"user_id": "@alice:warp.example", "content": {"msgtype": "m.text", "body": "Maintenance at 22:00 UTC."}} +``` + +Sends a message from the server to one of its users, and answers +`{"room_id": "...", "event_id": "..."}`. Needs `server_notices.enabled` +and a signing key. Without them it is `409`, the server's configuration +refusing. + +Each user gets one notice room. It is made the first time, by the notices +user (`server_notices.localpart`, created at startup), and carries the +`m.server_notice` tag in the user's room account data, so a client can +show it apart from other rooms. A second notice goes to the same room. A +user who left is invited back. The user cannot answer: the power levels +give them none of the levels a message needs. The user does not have to be +online: the invite and the message wait for them. + +`content.body` is required. A missing `msgtype` becomes `m.text`. An +unknown user is `404`; a deactivated one `403`. + +### `GET /appservices` + +The registered application services and how delivery to each is going. +Registrations are files read at startup, so this is a view, not an editor. +Tokens are never shown. + +```json +{ + "appservices": [ + { + "id": "telegram", + "url": "http://localhost:29317", + "sender": "@telegrambot:warp.example", + "rate_limited": false, + "protocols": [], + "namespaces": { + "users": [{"regex": "@telegram_.*:warp\\.example", "exclusive": true}], + "aliases": [], + "rooms": [] + }, + "delivery": { + "stream_pos": 1204, + "behind": 0, + "txn_id": 88, + "events_delivered": 412, + "failures": 0, + "last_success_ms": 1760600000000, + "last_failure_ms": 0, + "retry_at_ms": 0 + } + } + ] +} +``` + +`delivery` is absent for a service without a URL, which is sent nothing. +`behind` is how many stream positions it has still to read: not all of +them will concern it. While `failures` is above zero the service is being +backed off, `last_error` says why, and `retry_at_ms` is the next attempt. +See [appservices.md](appservices.md). + +### `GET /audit_log` + +``` +GET /audit_log?limit=50&before=1234&actor=@root:warp.example&action=user.deactivate +``` + +Every change made through the admin API, newest first: + +```json +{ + "entries": [ + { + "id": 1235, + "ts_ms": 1760600000000, + "actor": "@root:warp.example", + "action": "user.deactivate", + "target": "@mallory:warp.example", + "details": {"erase": true} + } + ], + "next_before": 1235 +} +``` + +The log is paged by ID, not by offset. It grows at the top while an admin +reads it, and an offset would shift under them. Pass `next_before` as +`?before=` for the next page; it is absent on the last one. `actor` and +`action` filter exactly. + +The actions: + +| Action | Target | Details | +|---|---|---| +| `federation.switch` · `registration.switch` | — | `enabled` | +| `federation.lists` · `federation.lists_reset` | — | `allowlist`, `denylist` | +| `federation.destination_reset` | server name | — | +| `user.create` | user | `admin` | +| `user.admin` | user | `admin` | +| `user.deactivate` · `user.reactivate` | user | `erase` | +| `user.password` | user | `logout_devices` | +| `user.device_delete` | user | `device_id`, or `all` | +| `user.media_quota` | user | `max_bytes` (`null` for a reset) | +| `room.purge` · `room.force_leave` | room | `user_id` | +| `registration_token.create` · `.update` · `.delete` | token | `uses_allowed`, `expiry_time_ms` | +| `server_notice.send` | user | `room_id`, `event_id` | + +An entry is written after the change has been made. If the write fails, +that is logged and the request still succeeds: the change has happened, +and a `500` for something that worked would have the panel retry it. The +same information is in the server log either way. + ### `GET /rooms` ``` @@ -259,14 +533,12 @@ configuration is what refuses it. ## Not here yet -Phase 9 completes the admin API: deactivating users, resetting passwords, -revoking devices, granting the admin flag, registration token management, -the federation allow/deny list editor, server notices, media quotas and an -audit log of admin actions. +The room endpoints are still the Phase 3 set. Still missing: a room's +state and recent events, for an operator looking at a report, and a way to +publish or withdraw a room from the directory without borrowing a member's +token. Media has quotas but no purge or quarantine yet; see +[media.md](media.md#known-gaps). -The room endpoints will grow with it: a room's state and its recent events -for an operator looking at a report, and a way to publish or withdraw a room -from the directory without borrowing a member's token. - -A server started without a signing key has no roomserver, and the four room -endpoints then answer `503` rather than failing on the first call. +A server started without a signing key has no roomserver. The four room +endpoints then answer `503` rather than failing on the first call, and +server notices answer `409`. diff --git a/docs/appservices.md b/docs/appservices.md new file mode 100644 index 0000000..6a0ea5e --- /dev/null +++ b/docs/appservices.md @@ -0,0 +1,253 @@ +# Application services + +`server/internal/appservice`, and the Client-Server half of it in +`server/internal/clientapi/appservice.go`: PLAN.md Phase 9. An application +service is a bridge or a bot with a namespace of its own: users, aliases +and rooms it may act for, and the events about them delivered to it as +they happen. This is the +[Application Service API](https://spec.matrix.org/latest/application-service-api/). + +| Package | What | +|---|---| +| `appservice` | Registration files, namespace matching, the registry, the HTTP client that talks to services, and the sender that delivers events to them. | +| `clientapi` | A service's token as an access token, acting as its users with `?user_id=`, registering and logging in its users, the `M_EXCLUSIVE` refusals, the ping endpoint, and asking services about users and aliases that do not exist. | + +## Registering a service + +Every bridge writes a registration file. Point Warp at it: + +```yaml +appservice: + config_files: + - /etc/warp/appservices/telegram.yaml + timeout: 30s +``` + +or `WARP_APPSERVICE_CONFIG_FILES=/etc/warp/appservices/telegram.yaml`. +Files are read at startup, and a file that does not parse stops the server +from starting, as a bad config file does. Adding or changing one needs a +restart. The admin API shows what is registered (`GET /appservices`) but +does not edit it. + +```yaml +id: telegram +url: http://localhost:29317 # null: the service is sent nothing +as_token: # what the service authenticates with +hs_token: # what Warp authenticates to the service with +sender_localpart: telegrambot +rate_limited: false # absent means true +namespaces: + users: + - exclusive: true + regex: '@telegram_.*:warp\.example' + aliases: + - exclusive: true + regex: '#telegram_.*:warp\.example' + rooms: [] +``` + +What is checked: + +- `id`, `as_token`, `hs_token` and `sender_localpart` are required, and the + sender has to be a valid localpart. +- `as_token` and `hs_token` must differ. No two services may share an `id` + or an `as_token`, and no service's `as_token` may be another service's + `hs_token`. +- `url` is empty, `null` or an http(s) URL. +- Every namespace regex compiles. + +Keys Warp does not know are ignored rather than refused. This is unlike +the main config file. Bridges add their own keys and unstable ones +(`push_ephemeral`, the MSC3202 keys), and a bridge upgrade must not stop +the homeserver. + +The sender user is created at startup if it does not exist. It has no +password. + +### How namespaces match + +A namespace regex is anchored at the start and not at the end, the way +Synapse matches. Registration files in the wild are written against +Synapse. `@_irc_` matches every user ID that begins with it. `@bot` does +not match `@notabot:…`. Anchoring both ends would quietly match nothing +for many existing files. + +Namespaces match full IDs: `@localpart:server` for users, +`#alias:server` for aliases, `!id:server` for rooms. + +## A service acting as its users + +A request carrying a service's `as_token` acts as the service's sender. +With `?user_id=` it acts as that user instead. The user has to be: + +- the sender, or in one of the service's user namespaces (exclusive or + not, as the spec says) +- a user that exists — a service registers its users first + +Otherwise the answer is `403 M_FORBIDDEN`. A deactivated user is +`403 M_USER_DEACTIVATED`. + +Such a request has no device. `device_id` in `/whoami` is empty, and there +is no device to record activity on. A service that needs devices for its +users (for end-to-end encryption, say) logs them in with +`m.login.application_service` and uses their access tokens. + +### Rate limiting + +A service's own requests are never rate limited. Requests it makes as its +users are exempt when its registration says `rate_limited: false`. +Without that, a bridge puppeting a thousand users from one address would +put all of them in one bucket. The limiter runs before the token is +resolved, so the exemption is a map lookup on the token. It needs no +database query. + +### Registering and logging in its users + +```http +POST /_matrix/client/v3/register +Authorization: Bearer + +{"type": "m.login.application_service", "username": "telegram_12345"} +``` + +The user must be in the service's namespace (`400 M_EXCLUSIVE` otherwise). +There is no flow and no password, and the registration switch does not +apply: this is the service's own action, not a sign-up. `inhibit_login` +is honoured. Without it, the answer carries an access token for a new +device. + +`POST /login` with `"type": "m.login.application_service"` and the +service's token logs in one of its users and returns a device. The flow +is listed by `GET /login` while any service is registered. + +The service's users can change their password and deactivate their +account without the password re-authentication other users go through. +They have no password to re-enter, and the service's token is already the +strongest credential involved. + +### Exclusive namespaces + +A namespace marked `exclusive` belongs to its service alone: + +| Who | What | Answer | +|---|---|---| +| anyone else | registering a user in it, or asking `/register/available` | `400 M_EXCLUSIVE` | +| anyone else | `PUT /directory/room/{alias}`, or `createRoom` with a `room_alias_name` that lands in it | `400 M_EXCLUSIVE` | +| the service | any of the above | allowed | + +A service's sender counts as claimed by it, even outside its namespaces. + +Exclusive room namespaces are not enforced. Nothing in the +Client-Server API creates a room with a chosen ID. + +## Delivering events + +The sender reads the roomserver's event stream from a cursor kept per +service in `appservice_state` (migration 0011). Events are not queued as +they are persisted. The stream is already durable and ordered, so a cursor +is all a service needs to catch up after being down for a day. The emit +stage only wakes the sender's workers; it hands nothing over, and it does +no work under the room lock. + +One worker per service with a URL: + +1. Read up to 100 events after the cursor. +2. Keep the ones that concern the service (see below). +3. If none do, move the cursor past them. Otherwise record a transaction + as in flight (its ID and the top of its events), and + `PUT /_matrix/app/v1/transactions/{txnId}` until the service accepts + it. +4. Move the cursor and clear the transaction. + +A transaction that fails is retried with the same ID and the same events. +The backoff doubles from 2 seconds to 5 minutes and stays there. A service +that answers a ping has its waiting transaction sent at once. + +After a restart, a transaction left in flight is sent again with its old +ID. Its events are read again from the stream range it covered, so the +service can deduplicate by ID as the spec intends. One caveat: whether +events concern the service is decided again on re-read. A room the +service's users left in the meantime can make the resent transaction +smaller than the first attempt. Within one run, retries reuse the exact +same body. + +A service seen for the first time starts at the top of the stream. It is +told what happens from then on, not the server's history. Changing a +service's `id` starts it over in the same way. + +### Which events concern a service + +An event is delivered when any of these hold: + +- its sender is one of the service's users (its sender or a user + namespace match) +- it is an `m.room.member` event whose target is one of them +- its room ID is in a room namespace +- its room has one of the service's users joined to it +- its room has one of the service's aliases pointing at it + +The last two are what let a bridge see the conversations its puppets are +in. They are read once per room per batch, as the room is when the batch +is read. An event read late, from a backlog, is judged by who is in the +room then, not when the event happened. + +Events are in the client format, with `room_id`, as the spec has it. +Backfilled history (negative stream positions) is never delivered. + +The request carries `Authorization: Bearer `. The deprecated +`?access_token=` form is not sent: it was deprecated in v1.4, and a token +in a URL ends up in the service's access log. Redirects are not followed, +because they would take the `hs_token` with them. + +## Asking services about users and aliases + +When something needs a local user or alias that does not exist, the +services whose namespaces match are asked first. They answer by creating +it, and Warp looks again: + +| When | Query | +|---|---| +| `GET /directory/room/{alias}` and `POST /join/{alias}` for an unknown local alias | `GET /_matrix/app/v1/rooms/{alias}` | +| `GET /profile/{userId}` and `POST /rooms/{id}/invite` for an unknown local user | `GET /_matrix/app/v1/users/{userId}` | + +This is how a bridge creates a portal room the first time somebody looks +for it, and a puppet the first time somebody invites it. Services are +asked in registration order; the first yes ends it. A service that fails +to answer counts as a no and is logged. The request that wanted the user +goes on without it rather than failing. + +## The ping + +`POST /_matrix/client/v1/appservice/{appserviceId}/ping` is a service +asking Warp to ping it back. It is authenticated with the service's own +token, and `{appserviceId}` has to be that service. Warp sends +`POST /_matrix/app/v1/ping` with the `transaction_id`, and answers: + +| Outcome | Answer | +|---|---| +| the service answered 2xx | `200 {"duration_ms": …}` | +| no URL registered | `400 M_URL_NOT_SET` | +| the service answered something else | `502 M_BAD_STATUS`, with its `status` and `body` | +| it could not be reached | `502 M_CONNECTION_FAILED` | +| it did not answer within `appservice.timeout` | `504 M_CONNECTION_TIMEOUT` | +| another token | `403 M_FORBIDDEN` | + +## Watching delivery + +`GET /_warp/admin/v1/appservices` lists each service: its namespaces, +sender and URL, never its tokens. For a service with a URL it also shows +the cursor, how far behind the top of the stream it is, the last +transaction ID, the events delivered since startup, and the failure +count, last error and next retry while it is failing. See +[admin-api.md](admin-api.md#get-appservices). + +## Not implemented + +| Thing | Why | +|---|---| +| Ephemeral events and to-device messages in transactions (MSC2409, MSC3202) | unstable; bridges that want them fall back to `/sync` as a user | +| Device masquerading with `?device_id=` (MSC3202) | as above; log the user in instead | +| Third-party lookups (`/thirdparty/*`) | needs the protocol metadata exchange, which no first-party client uses | +| Network-specific room directories (`PUT /directory/list/appservice/{networkId}/{roomId}`) | not served | +| Appservice key claims and queries (MSC3983, MSC3984) | unstable | +| Editing registrations at runtime | registrations are files; restart to change them | diff --git a/docs/configuration.md b/docs/configuration.md index a299315..479edfc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -66,6 +66,7 @@ List-valued variables are comma-separated. A single `-` clears a list: | `federation.tls_ca_file` | `WARP_FEDERATION_TLS_CA_FILE` | empty | Extra CA certificates (PEM) for outbound federation, on top of the system roots. | | `media.base_path` | `WARP_MEDIA_BASE_PATH` | `./media_store` | Where media content is kept. Created on the first upload. | | `media.max_upload_bytes` | `WARP_MEDIA_MAX_UPLOAD_BYTES` | `52428800` | The largest upload (50 MiB), and the largest file fetched from another server. Reported to clients as `m.upload.size`. | +| `media.max_user_storage_bytes` | `WARP_MEDIA_MAX_USER_STORAGE_BYTES` | `0` | The most one user may have uploaded in all. `0` = no limit. The admin API can give a user a quota of their own. | | `media.max_thumbnail_pixels` | `WARP_MEDIA_MAX_THUMBNAIL_PIXELS` | `32000000` | The largest image, in pixels, decoded to make a thumbnail. | | `media.thumbnail_sizes` | — | five sizes, see [media.md](media.md) | The thumbnails this server makes, as `{width, height, method}`. Empty = no thumbnails. | | `media.pending_upload_expiry` | `WARP_MEDIA_PENDING_UPLOAD_EXPIRY` | `24h` | How long an ID reserved with `POST /media/v1/create` waits for its content. | @@ -77,7 +78,14 @@ List-valued variables are comma-separated. A single `-` clears a list: | `push.notification_retention` | `WARP_PUSH_NOTIFICATION_RETENTION` | `720h` | How long a read notification is kept for `GET /notifications`. Unread ones are kept until read. | | `registration.enabled` | `WARP_REGISTRATION_ENABLED` | `false` | | | `registration.allow_runtime_toggle` | `WARP_REGISTRATION_ALLOW_RUNTIME_TOGGLE` | `true` | | -| `registration.tokens` | `WARP_REGISTRATION_TOKENS` | `[]` | Non-empty = sign-up also needs one of these, as the `m.login.registration_token` stage. | +| `registration.tokens` | `WARP_REGISTRATION_TOKENS` | `[]` | Non-empty = sign-up also needs a token, as the `m.login.registration_token` stage. These never run out; tokens with a use limit or an expiry are made through the admin API, and both kinds are accepted. | +| `registration.require_token` | `WARP_REGISTRATION_REQUIRE_TOKEN` | `false` | Ask for a token even with no `tokens` above, for a server whose tokens all come from the admin API. | +| `appservice.config_files` | `WARP_APPSERVICE_CONFIG_FILES` | `[]` | Application service registration files, read at startup. A bad one stops the server. See [appservices.md](appservices.md). | +| `appservice.timeout` | `WARP_APPSERVICE_TIMEOUT` | `30s` | One request to an application service: a transaction, a query or a ping. | +| `server_notices.enabled` | `WARP_SERVER_NOTICES_ENABLED` | `false` | Create the notices user at startup and serve `POST /_warp/admin/v1/server_notices`. Needs a signing key. | +| `server_notices.localpart` | `WARP_SERVER_NOTICES_LOCALPART` | `server-notices` | The user notices come from. Created if it does not exist, so pick a name no real user has. | +| `server_notices.display_name` | `WARP_SERVER_NOTICES_DISPLAY_NAME` | `Server Notices` | Its display name, kept in step with this setting at startup. | +| `server_notices.room_name` | `WARP_SERVER_NOTICES_ROOM_NAME` | `Server Notices` | The name of the room each user gets their notices in. | | `logging.level` | `WARP_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`. | | `logging.format` | `WARP_LOG_FORMAT` | `text` | `text` or `json`. | | `logging.add_source` | `WARP_LOG_ADD_SOURCE` | `false` | File and line in every record. | diff --git a/docs/e2ee.md b/docs/e2ee.md index ca44781..10488d4 100644 --- a/docs/e2ee.md +++ b/docs/e2ee.md @@ -250,7 +250,7 @@ not exist, `404`. A version's algorithm cannot be changed. | Thing | Why | |---|---| | Dehydrated devices (MSC3814) | — | -| Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984) | application services are Phase 9 | +| Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984) | unstable; application services are in, without these ([appservices.md](appservices.md)) | | Purging a deleted device's to-device queue | the messages stay queued; a new device later logged in under the same ID is handed them, encrypted to the old device's keys and so unreadable to it | | Deleting a device's keys in the same transaction as the device | the two are separate stores; keys left behind by a crash between them are never served, because a query reads the device list first | diff --git a/docs/media.md b/docs/media.md index 263be6f..931be0a 100644 --- a/docs/media.md +++ b/docs/media.md @@ -213,7 +213,7 @@ with a message saying federation is disabled. | | | |---|---| | `404 M_NOT_FOUND` | no such media, here or on its server | -| `413 M_TOO_LARGE` | larger than `media.max_upload_bytes`, uploaded or fetched | +| `413 M_TOO_LARGE` | larger than `media.max_upload_bytes`, uploaded or fetched; or over the uploader's quota | | `504 M_NOT_YET_UPLOADED` | reserved, not uploaded within `timeout_ms` | | `409 M_CANNOT_OVERWRITE_MEDIA` | a reserved URI that already has content | | `403 M_FORBIDDEN` | someone else's reservation; or remote media with federation off | @@ -227,11 +227,12 @@ with a message saying federation is disabled. `M_UNRECOGNIZED`. Fetching arbitrary URLs for clients needs the same network guard as media redirects plus an HTML parser, and is its own piece of work. -- **No deletion or quotas yet.** Nothing removes media: no admin purge, no - retention for the remote cache, no per-user quota. Those are the media - parts of Phase 9's admin work, and content-addressed storage means a - purge has to check that no other row still uses a blob before deleting - it. +- **No deletion yet.** Nothing removes media: no admin purge, and no + retention for the remote cache. Content-addressed storage means a purge + has to check that no other row still uses a blob before deleting it. + Per-user quotas are in (Phase 9): `media.max_user_storage_bytes`, and a + quota per user through the admin API. An upload over the quota is + `413 M_TOO_LARGE`. See [admin-api.md](admin-api.md). - **No quarantine**, which belongs with the same admin work. - **No animated thumbnails**; `animated=true` gets a still image. - **No S3** until the PocketBase driver brings its filesystem abstraction; diff --git a/docs/rooms.md b/docs/rooms.md index bc0c3f3..8c39c22 100644 --- a/docs/rooms.md +++ b/docs/rooms.md @@ -174,7 +174,7 @@ authorisation rules — a knock-rule room needs an invite to join — but `POST /knock/{roomIdOrAlias}` is not served. **No third-party invites.** They go through an identity server, which Warp -does not talk to (Phase 9). `createRoom` with `invite_3pid`, and `/invite` +does not talk to: identity servers are a non-goal (PLAN.md section 1). `createRoom` with `invite_3pid`, and `/invite` with `id_server`, are refused rather than silently dropped. **No room upgrades.** `POST /rooms/{roomID}/upgrade` and the tombstone diff --git a/server/internal/adminapi/adminapi.go b/server/internal/adminapi/adminapi.go index 7c3e27b..ec35fdb 100644 --- a/server/internal/adminapi/adminapi.go +++ b/server/internal/adminapi/adminapi.go @@ -8,21 +8,26 @@ // with the admin flag, so the panel logs in the way any client does and // there is no second credential to manage. // -// Phase 1 is the skeleton PLAN.md section 7 asks for: the two runtime -// switches, user listing and creation, and the dashboard counters. Full user -// management, registration tokens, the federation allow/deny list editor and -// the audit log arrive in Phase 9. +// Phase 1 built the skeleton PLAN.md section 7 asks for: the two runtime +// switches, user listing and creation, and the dashboard counters. Phase 9 +// completed it: full user management, registration tokens, server notices, +// media quotas, the federation allow/deny list editor, the application +// service status view, and an audit log of every change made through it. package adminapi import ( "errors" "log/slog" "net/http" + "time" + "github.com/dariuszjaniak/warp/server/internal/appservice" "github.com/dariuszjaniak/warp/server/internal/buildinfo" "github.com/dariuszjaniak/warp/server/internal/federation" "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/mediaapi" "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/servernotices" "github.com/dariuszjaniak/warp/server/internal/settings" "github.com/dariuszjaniak/warp/server/internal/storage" "github.com/dariuszjaniak/warp/server/internal/userapi" @@ -41,6 +46,13 @@ type AdminAPI struct { database storage.Database fedStatus FederationStatus log *slog.Logger + now func() time.Time + + media *mediaapi.API + appservices *appservice.API + notices *servernotices.API + tokens *userapi.RegistrationTokens + adminStore storage.AdminStore } // Options configure the component. @@ -60,11 +72,31 @@ type Options struct { // without one, and the destinations endpoints then say so. FederationStatus FederationStatus - // Database is used for the driver name the dashboard shows and for the - // migration state; the admin API does not write to it directly. + // Database is used for the driver name the dashboard shows, and its + // admin store for the audit log; nothing else here writes to it + // directly. Database storage.Database + // Media is the media repository, for the quota endpoints. Nil leaves + // them answering 404. + Media *mediaapi.API + + // AppServices is the application service component, for the status + // view. Nil means none are registered. + AppServices *appservice.API + + // Notices sends server notices. Nil means server notices are not + // configured, and the endpoint says so. + Notices *servernotices.API + + // RegistrationTokens manages the tokens made through the admin API. + // Nil builds one on the Database's admin store. + RegistrationTokens *userapi.RegistrationTokens + Logger *slog.Logger + + // Now is the clock, for the audit log. + Now func() time.Time } // New builds the component. @@ -83,15 +115,41 @@ func New(opts Options) (*AdminAPI, error) { if opts.Logger == nil { opts.Logger = slog.Default() } + if opts.Now == nil { + opts.Now = time.Now + } + var adminStore storage.AdminStore + if opts.Database != nil { + adminStore = opts.Database.Admin() + } + if opts.RegistrationTokens == nil { + now := opts.Now + opts.RegistrationTokens = userapi.NewRegistrationTokens(userapi.RegistrationTokensOptions{ + Store: adminStore, + NowMS: func() int64 { return now().UnixMilli() }, + }) + } + if opts.AppServices == nil { + var err error + if opts.AppServices, err = appservice.New(appservice.Options{Logger: opts.Logger}); err != nil { + return nil, err + } + } return &AdminAPI{ - serverName: opts.ServerName, - users: opts.Users, - rooms: opts.Rooms, - settings: opts.Settings, - federation: opts.Federation, - database: opts.Database, - fedStatus: opts.FederationStatus, - log: opts.Logger, + serverName: opts.ServerName, + users: opts.Users, + rooms: opts.Rooms, + settings: opts.Settings, + federation: opts.Federation, + database: opts.Database, + fedStatus: opts.FederationStatus, + log: opts.Logger, + now: opts.Now, + media: opts.Media, + appservices: opts.AppServices, + notices: opts.Notices, + tokens: opts.RegistrationTokens, + adminStore: adminStore, }, nil } @@ -103,14 +161,39 @@ func (a *AdminAPI) RegisterRoutes(r *httputil.Router) { a.handle(r, "POST "+Prefix+"/federation/destinations/{serverName}/reset", a.handleResetDestination) a.handle(r, "GET "+Prefix+"/registration", a.handleGetRegistration) a.handle(r, "PUT "+Prefix+"/registration", a.handlePutRegistration) + a.handle(r, "PUT "+Prefix+"/federation/lists", a.handlePutFederationLists) + a.handle(r, "DELETE "+Prefix+"/federation/lists", a.handleDeleteFederationLists) a.handle(r, "GET "+Prefix+"/users", a.handleGetUsers) a.handle(r, "POST "+Prefix+"/users", a.handlePostUsers) a.handle(r, "GET "+Prefix+"/stats", a.handleGetStats) + // One user. {userID} is a full user ID, escaped in the path. + a.handle(r, "GET "+Prefix+"/users/{userID}", a.handleGetUser) + a.handle(r, "PUT "+Prefix+"/users/{userID}/admin", a.handlePutUserAdmin) + a.handle(r, "POST "+Prefix+"/users/{userID}/deactivate", a.handleDeactivateUser) + a.handle(r, "POST "+Prefix+"/users/{userID}/reactivate", a.handleReactivateUser) + a.handle(r, "POST "+Prefix+"/users/{userID}/password", a.handleResetPassword) + a.handle(r, "GET "+Prefix+"/users/{userID}/devices", a.handleGetUserDevices) + a.handle(r, "DELETE "+Prefix+"/users/{userID}/devices", a.handleDeleteUserDevices) + a.handle(r, "DELETE "+Prefix+"/users/{userID}/devices/{deviceID}", a.handleDeleteUserDevice) + a.handle(r, "GET "+Prefix+"/users/{userID}/media", a.handleGetUserMedia) + a.handle(r, "PUT "+Prefix+"/users/{userID}/media/quota", a.handlePutMediaQuota) + a.handle(r, "DELETE "+Prefix+"/users/{userID}/media/quota", a.handleDeleteMediaQuota) + a.handle(r, "GET "+Prefix+"/rooms", a.handleGetRooms) a.handle(r, "GET "+Prefix+"/rooms/{roomID}", a.handleGetRoom) a.handle(r, "DELETE "+Prefix+"/rooms/{roomID}", a.handleDeleteRoom) a.handle(r, "POST "+Prefix+"/rooms/{roomID}/force_leave", a.handleForceLeave) + + a.handle(r, "GET "+Prefix+"/registration_tokens", a.handleGetRegistrationTokens) + a.handle(r, "POST "+Prefix+"/registration_tokens", a.handlePostRegistrationToken) + a.handle(r, "GET "+Prefix+"/registration_tokens/{token}", a.handleGetRegistrationToken) + a.handle(r, "PUT "+Prefix+"/registration_tokens/{token}", a.handlePutRegistrationToken) + a.handle(r, "DELETE "+Prefix+"/registration_tokens/{token}", a.handleDeleteRegistrationToken) + + a.handle(r, "POST "+Prefix+"/server_notices", a.handlePostServerNotice) + a.handle(r, "GET "+Prefix+"/appservices", a.handleGetAppServices) + a.handle(r, "GET "+Prefix+"/audit_log", a.handleGetAuditLog) } // adminHandler is a handler that has already been shown to be an admin. @@ -214,6 +297,10 @@ type federationResponse struct { switchResponse Allowlist []string `json:"allowlist"` Denylist []string `json:"denylist"` + + // ListsEdited is true when the lists were set through the admin API + // and replace the config file's. + ListsEdited bool `json:"lists_edited"` } // handleGetFederation implements GET /_warp/admin/v1/federation. @@ -224,14 +311,65 @@ func (a *AdminAPI) handleGetFederation(w http.ResponseWriter, _ *http.Request, _ Enabled: a.settings.FederationEnabled(), Toggleable: a.settings.FederationToggleable(), }, - // Editing these is Phase 9; they are reported now because the - // dashboard has to show which servers a running Warp will talk to. - Allowlist: allowlist, - Denylist: denylist, + Allowlist: allowlist, + Denylist: denylist, + ListsEdited: a.settings.FederationLists().Edited, }) return nil } +// federationListsRequest is the PUT /federation/lists body. +type federationListsRequest struct { + Allowlist *[]string `json:"allowlist"` + Denylist *[]string `json:"denylist"` +} + +// handlePutFederationLists implements PUT /_warp/admin/v1/federation/lists. +// +// Both lists are replaced together, and both have to be in the body: a +// panel that sent one and had the other silently kept would be one bug +// away from leaving a server it meant to cut off on the allowlist. They +// take effect at once, on the next request in either direction, and are +// kept across restarts in place of the config file's. +func (a *AdminAPI) handlePutFederationLists(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + var req federationListsRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + if req.Allowlist == nil || req.Denylist == nil { + return httputil.MissingParam("allowlist and denylist are both required (send [] for an empty list)") + } + for _, list := range [][]string{*req.Allowlist, *req.Denylist} { + for _, name := range list { + if _, err := federation.ParseServerName(name); err != nil { + return httputil.InvalidParam("%q is not a server name", name) + } + } + } + if err := a.settings.SetFederationLists(r.Context(), *req.Allowlist, *req.Denylist); err != nil { + return err + } + lists := a.settings.FederationLists() + a.federation.Policy().Replace(lists.Allowlist, lists.Denylist) + a.audit(r.Context(), auth, auditFederationLists, "", map[string]any{ + "allowlist": lists.Allowlist, "denylist": lists.Denylist, + }) + return a.handleGetFederation(w, r, auth) +} + +// handleDeleteFederationLists implements DELETE +// /_warp/admin/v1/federation/lists: forget the edited lists and go back to +// the config file's. +func (a *AdminAPI) handleDeleteFederationLists(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + lists, err := a.settings.ResetFederationLists(r.Context()) + if err != nil { + return err + } + a.federation.Policy().Replace(lists.Allowlist, lists.Denylist) + a.audit(r.Context(), auth, auditFederationReset, "", nil) + return a.handleGetFederation(w, r, auth) +} + // handlePutFederation implements PUT /_warp/admin/v1/federation, the switch // PLAN.md section 3 describes: no restart, the gate picks it up on the next // request. @@ -243,8 +381,7 @@ func (a *AdminAPI) handlePutFederation(w http.ResponseWriter, r *http.Request, a if err := a.settings.SetFederationEnabled(r.Context(), enabled); err != nil { return toggleError(err, "federation") } - a.log.Info("federation switch changed through the admin API", - "enabled", enabled, "by", auth.UserID) + a.audit(r.Context(), auth, auditFederationSwitch, "", map[string]any{"enabled": enabled}) return a.handleGetFederation(w, r, auth) } @@ -266,8 +403,7 @@ func (a *AdminAPI) handlePutRegistration(w http.ResponseWriter, r *http.Request, if err := a.settings.SetRegistrationEnabled(r.Context(), enabled); err != nil { return toggleError(err, "registration") } - a.log.Info("registration switch changed through the admin API", - "enabled", enabled, "by", auth.UserID) + a.audit(r.Context(), auth, auditRegistrationSwitch, "", map[string]any{"enabled": enabled}) return a.handleGetRegistration(w, r, auth) } diff --git a/server/internal/adminapi/appservices.go b/server/internal/adminapi/appservices.go new file mode 100644 index 0000000..373506a --- /dev/null +++ b/server/internal/adminapi/appservices.go @@ -0,0 +1,96 @@ +package adminapi + +import ( + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/appservice" + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// appServiceNamespace is one namespace in the status view. +type appServiceNamespace struct { + Regex string `json:"regex"` + Exclusive bool `json:"exclusive"` +} + +// appServiceDelivery is a service's delivery health. It is absent for a +// service without a URL, which is sent nothing. +type appServiceDelivery struct { + StreamPos int64 `json:"stream_pos"` + Behind int64 `json:"behind"` + TxnID int64 `json:"txn_id"` + EventsDelivered int64 `json:"events_delivered"` + Failures int `json:"failures"` + LastError string `json:"last_error,omitempty"` + LastSuccessMS int64 `json:"last_success_ms"` + LastFailureMS int64 `json:"last_failure_ms"` + RetryAtMS int64 `json:"retry_at_ms"` +} + +// appServiceResponse is one service in GET /appservices. +type appServiceResponse struct { + ID string `json:"id"` + URL string `json:"url,omitempty"` + Sender string `json:"sender"` + RateLimited bool `json:"rate_limited"` + Protocols []string `json:"protocols"` + Namespaces map[string][]appServiceNamespace `json:"namespaces"` + Delivery *appServiceDelivery `json:"delivery,omitempty"` +} + +// handleGetAppServices implements GET /_warp/admin/v1/appservices: what is +// registered, and how delivery to each is going. Registrations are read +// from files at startup, so this is a view, not an editor; the tokens are +// never shown. +func (a *AdminAPI) handleGetAppServices(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + var top int64 + if a.rooms != nil { + var err error + if top, err = a.rooms.StreamTop(r.Context()); err != nil { + return err + } + } + statuses := map[string]appServiceDelivery{} + for _, s := range a.appservices.Statuses() { + statuses[s.ID] = appServiceDelivery{ + StreamPos: s.StreamPos, Behind: max(top-s.StreamPos, 0), TxnID: s.TxnID, + EventsDelivered: s.EventsDelivered, Failures: s.Failures, LastError: s.LastError, + LastSuccessMS: s.LastSuccessMS, LastFailureMS: s.LastFailureMS, RetryAtMS: s.RetryAtMS, + } + } + + out := []appServiceResponse{} + for _, svc := range a.appservices.Registry().Services() { + resp := appServiceResponse{ + ID: svc.ID, + URL: svc.URL, + Sender: svc.SenderUserID(), + RateLimited: svc.RateLimitedUsers(), + Protocols: svc.Protocols, + Namespaces: map[string][]appServiceNamespace{ + "users": namespacesOf(svc.Namespaces.Users), + "aliases": namespacesOf(svc.Namespaces.Aliases), + "rooms": namespacesOf(svc.Namespaces.Rooms), + }, + } + if resp.Protocols == nil { + resp.Protocols = []string{} + } + if d, ok := statuses[svc.ID]; ok { + resp.Delivery = &d + } + out = append(out, resp) + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, map[string]any{"appservices": out}) + return nil +} + +func namespacesOf(list []appservice.Namespace) []appServiceNamespace { + out := make([]appServiceNamespace, 0, len(list)) + for _, ns := range list { + out = append(out, appServiceNamespace{Regex: ns.Regex, Exclusive: ns.Exclusive}) + } + return out +} diff --git a/server/internal/adminapi/audit.go b/server/internal/adminapi/audit.go new file mode 100644 index 0000000..ca42f73 --- /dev/null +++ b/server/internal/adminapi/audit.go @@ -0,0 +1,141 @@ +package adminapi + +import ( + "context" + "encoding/json" + "net/http" + "strconv" + + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/storage" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// The actions the audit log records, one per kind of change the admin API +// can make. They are part of the API: the panel filters on them. +const ( + auditFederationSwitch = "federation.switch" + auditFederationLists = "federation.lists" + auditFederationReset = "federation.lists_reset" + auditDestinationReset = "federation.destination_reset" + auditRegistrationSwitch = "registration.switch" + auditUserCreate = "user.create" + auditUserAdmin = "user.admin" + auditUserDeactivate = "user.deactivate" + auditUserReactivate = "user.reactivate" + auditUserPassword = "user.password" + auditUserDeviceDelete = "user.device_delete" + auditUserMediaQuota = "user.media_quota" + auditRoomPurge = "room.purge" + auditRoomForceLeave = "room.force_leave" + auditTokenCreate = "registration_token.create" + auditTokenUpdate = "registration_token.update" + auditTokenDelete = "registration_token.delete" + auditServerNotice = "server_notice.send" +) + +// audit records a change made through the admin API, after it has been +// made. +// +// A failure to write the entry is logged and does not fail the request: +// the change has happened, and answering 500 for something that worked +// would have the panel retry it. The log line carries the same +// information, so nothing is lost but the convenience of the table. +func (a *AdminAPI) audit(ctx context.Context, auth userapi.Authenticated, action, target string, details map[string]any) { + raw := []byte("{}") + if len(details) > 0 { + var err error + if raw, err = json.Marshal(details); err != nil { + raw = []byte("{}") + } + } + a.log.Info("admin action", "action", action, "target", target, "by", auth.UserID, "details", string(raw)) + if a.adminStore == nil { + return + } + if _, err := a.adminStore.AppendAudit(context.WithoutCancel(ctx), storage.AuditEntry{ + TSMS: a.now().UnixMilli(), + Actor: auth.UserID, + Action: action, + Target: target, + Details: raw, + }); err != nil { + a.log.Error("could not write the audit log", "action", action, "target", target, "error", err) + } +} + +// auditEntry is one entry as the API shows it. +type auditEntry struct { + ID int64 `json:"id"` + TSMS int64 `json:"ts_ms"` + Actor string `json:"actor"` + Action string `json:"action"` + Target string `json:"target,omitempty"` + Details json.RawMessage `json:"details"` +} + +type auditResponse struct { + Entries []auditEntry `json:"entries"` + + // NextBefore is the ?before= of the next page, absent on the last. + NextBefore int64 `json:"next_before,omitempty"` +} + +// defaultAuditPageLimit is what a request without ?limit= gets, and +// maxAuditPageLimit the most it may ask for. +const ( + defaultAuditPageLimit = 50 + maxAuditPageLimit = 1000 +) + +// handleGetAuditLog implements GET /_warp/admin/v1/audit_log. +// +// It pages backwards from the newest entry by ID rather than by offset: +// the log grows at the top while an admin reads it, and an offset would +// shift under them. +func (a *AdminAPI) handleGetAuditLog(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + if a.adminStore == nil { + return httputil.NotFound("This server keeps no audit log") + } + query := r.URL.Query() + limit, err := intParam(query.Get("limit"), defaultAuditPageLimit) + if err != nil || limit < 0 { + return httputil.InvalidParam("limit must be a non-negative number") + } + // Clamped here rather than left to the store, so that "a full page" + // below means the same thing the store does. + if limit == 0 { + limit = defaultAuditPageLimit + } + limit = min(limit, maxAuditPageLimit) + var before int64 + if raw := query.Get("before"); raw != "" { + if before, err = strconv.ParseInt(raw, 10, 64); err != nil || before < 0 { + return httputil.InvalidParam("before must be an entry ID") + } + } + + entries, err := a.adminStore.AuditLog(r.Context(), storage.AuditFilter{ + Before: before, + Limit: limit, + Actor: query.Get("actor"), + Action: query.Get("action"), + }) + if err != nil { + return err + } + resp := auditResponse{Entries: make([]auditEntry, 0, len(entries))} + for _, e := range entries { + resp.Entries = append(resp.Entries, auditEntry{ + ID: e.ID, TSMS: e.TSMS, Actor: e.Actor, Action: e.Action, Target: e.Target, + Details: json.RawMessage(e.Details), + }) + } + // A full page may have more behind it; a short one cannot. + if len(entries) == limit { + resp.NextBefore = entries[len(entries)-1].ID + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, resp) + return nil +} diff --git a/server/internal/adminapi/federation.go b/server/internal/adminapi/federation.go index ef8b877..64d35da 100644 --- a/server/internal/adminapi/federation.go +++ b/server/internal/adminapi/federation.go @@ -96,7 +96,7 @@ func (a *AdminAPI) handleResetDestination(w http.ResponseWriter, r *http.Request if err := a.fedStatus.ResetBackoff(r.Context(), serverName); err != nil { return err } - a.log.Info("federation backoff reset through the admin API", "server_name", serverName, "by", auth.UserID) + a.audit(r.Context(), auth, auditDestinationReset, serverName, nil) httputil.WriteJSON(w, http.StatusOK, struct{}{}) return nil } diff --git a/server/internal/adminapi/notices.go b/server/internal/adminapi/notices.go new file mode 100644 index 0000000..37e5336 --- /dev/null +++ b/server/internal/adminapi/notices.go @@ -0,0 +1,62 @@ +package adminapi + +import ( + "errors" + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/servernotices" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// serverNoticeRequest is the POST /server_notices body. +type serverNoticeRequest struct { + UserID string `json:"user_id"` + + // Content is an m.room.message content. A body is required; a missing + // msgtype becomes m.text. + Content map[string]any `json:"content"` +} + +// handlePostServerNotice implements POST /_warp/admin/v1/server_notices: +// send a message from the server to one of its users. +// +// The notice goes into the user's server notice room, which is made the +// first time and which the user is invited back to if they left it. The +// user does not have to be online or to accept anything first: the +// invite and the message wait for them. +func (a *AdminAPI) handlePostServerNotice(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + if a.notices == nil { + return httputil.NewError(http.StatusConflict, httputil.ErrCodeForbidden, + "Server notices are not configured (server_notices.enabled is false, or the server has no signing key)") + } + var req serverNoticeRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + if req.UserID == "" { + return httputil.MissingParam("user_id is required") + } + if body, _ := req.Content["body"].(string); body == "" { + return httputil.MissingParam("content.body is required") + } + localpart, err := a.users.ParseLocalUserID(req.UserID) + if err != nil { + return httputil.InvalidParam("%s is not a user of this server", req.UserID) + } + + roomID, eventID, err := a.notices.Send(r.Context(), localpart, req.Content) + switch { + case errors.Is(err, servernotices.ErrNoSuchUser): + return httputil.NotFound("No such user") + case errors.Is(err, servernotices.ErrDeactivated): + return httputil.Forbidden("%s is deactivated", req.UserID) + case err != nil: + return roomError(err) + } + a.audit(r.Context(), auth, auditServerNotice, a.users.UserID(localpart), map[string]any{ + "room_id": roomID, "event_id": eventID, + }) + httputil.WriteJSON(w, http.StatusOK, map[string]string{"room_id": roomID, "event_id": eventID}) + return nil +} diff --git a/server/internal/adminapi/rooms.go b/server/internal/adminapi/rooms.go index ac38596..01f977a 100644 --- a/server/internal/adminapi/rooms.go +++ b/server/internal/adminapi/rooms.go @@ -141,8 +141,7 @@ func (a *AdminAPI) handleDeleteRoom(w http.ResponseWriter, r *http.Request, auth if err := a.rooms.PurgeRoom(r.Context(), roomID); err != nil { return roomError(err) } - a.log.Warn("room purged through the admin API", - "room_id", roomID, "by", auth.UserID) + a.audit(r.Context(), auth, auditRoomPurge, roomID, nil) httputil.WriteJSON(w, http.StatusOK, struct{}{}) return nil } @@ -176,8 +175,7 @@ func (a *AdminAPI) handleForceLeave(w http.ResponseWriter, r *http.Request, auth if err := a.rooms.ForceLeave(r.Context(), roomID, req.UserID); err != nil { return roomError(err) } - a.log.Warn("user removed from a room through the admin API", - "room_id", roomID, "user_id", req.UserID, "by", auth.UserID) + a.audit(r.Context(), auth, auditRoomForceLeave, roomID, map[string]any{"user_id": req.UserID}) httputil.WriteJSON(w, http.StatusOK, struct{}{}) return nil } diff --git a/server/internal/adminapi/tokens.go b/server/internal/adminapi/tokens.go new file mode 100644 index 0000000..3e59f92 --- /dev/null +++ b/server/internal/adminapi/tokens.go @@ -0,0 +1,198 @@ +package adminapi + +import ( + "encoding/json" + "errors" + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/storage" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// registrationToken is a token as the API shows it. +type registrationToken struct { + Token string `json:"token"` + + // UsesAllowed is null for a token with no use limit. + UsesAllowed *int `json:"uses_allowed"` + Completed int `json:"completed"` + + // ExpiryTimeMS is null for a token that never expires. + ExpiryTimeMS *int64 `json:"expiry_time_ms"` + + CreatedAtMS int64 `json:"created_at_ms"` + + // Valid is whether a registration could use the token now. + Valid bool `json:"valid"` +} + +func (a *AdminAPI) registrationTokenOf(t storage.RegistrationToken) registrationToken { + out := registrationToken{ + Token: t.Token, + UsesAllowed: t.UsesAllowed, + Completed: t.Completed, + CreatedAtMS: t.CreatedAtMS, + Valid: t.Usable(a.now().UnixMilli()), + } + if t.ExpiresAtMS != 0 { + expiry := t.ExpiresAtMS + out.ExpiryTimeMS = &expiry + } + return out +} + +// handleGetRegistrationTokens implements GET /_warp/admin/v1/registration_tokens. +// +// The tokens of the config file are not listed. They are in the config +// file, and listing them here would be the one place a secret from it is +// served over HTTP. +func (a *AdminAPI) handleGetRegistrationTokens(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + tokens, err := a.tokens.List(r.Context()) + if err != nil { + return err + } + out := make([]registrationToken, 0, len(tokens)) + for _, t := range tokens { + out = append(out, a.registrationTokenOf(t)) + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, map[string]any{ + "registration_tokens": out, + // Whether the sign-up flow asks for a token at all. A token made + // on a server that does not ask for one is never needed. + "required": a.tokens.Required(), + }) + return nil +} + +// createTokenRequest is the POST /registration_tokens body. +type createTokenRequest struct { + // Token is the token to make; empty generates one of Length + // characters, 16 if that is absent too. + Token string `json:"token"` + Length int `json:"length"` + + UsesAllowed *int `json:"uses_allowed"` + ExpiryTimeMS *int64 `json:"expiry_time_ms"` +} + +// handlePostRegistrationToken implements POST /_warp/admin/v1/registration_tokens. +func (a *AdminAPI) handlePostRegistrationToken(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + var req createTokenRequest + if _, err := httputil.DecodeOptionalJSON(w, r, &req); err != nil { + return err + } + var expiry int64 + if req.ExpiryTimeMS != nil { + expiry = *req.ExpiryTimeMS + } + created, err := a.tokens.CreateRegistrationToken(r.Context(), userapi.NewRegistrationToken{ + Token: req.Token, + Length: req.Length, + UsesAllowed: req.UsesAllowed, + ExpiresAtMS: expiry, + }) + switch { + case errors.Is(err, userapi.ErrInvalidRegistrationToken): + return httputil.InvalidParam("%s", err.Error()) + case errors.Is(err, userapi.ErrRegistrationTokenExists): + return httputil.InvalidParam("That registration token already exists") + case err != nil: + return err + } + a.audit(r.Context(), auth, auditTokenCreate, created.Token, map[string]any{ + "uses_allowed": created.UsesAllowed, "expiry_time_ms": created.ExpiresAtMS, + }) + httputil.WriteJSON(w, http.StatusCreated, a.registrationTokenOf(created)) + return nil +} + +// handleGetRegistrationToken implements GET /_warp/admin/v1/registration_tokens/{token}. +func (a *AdminAPI) handleGetRegistrationToken(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + t, err := a.tokens.RegistrationToken(r.Context(), r.PathValue("token")) + if err != nil { + return tokenError(err) + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, a.registrationTokenOf(t)) + return nil +} + +// updateTokenRequest is the PUT /registration_tokens/{token} body. Each +// field is raw so that "absent" (keep it) can be told from null (remove +// the limit). +type updateTokenRequest struct { + UsesAllowed json.RawMessage `json:"uses_allowed"` + ExpiryTimeMS json.RawMessage `json:"expiry_time_ms"` +} + +// handlePutRegistrationToken implements PUT /_warp/admin/v1/registration_tokens/{token}. +// +// A field left out keeps its value; a field set to null removes the +// limit. Raising uses_allowed on a used-up token makes it usable again. +func (a *AdminAPI) handlePutRegistrationToken(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + token := r.PathValue("token") + current, err := a.tokens.RegistrationToken(r.Context(), token) + if err != nil { + return tokenError(err) + } + var req updateTokenRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + + uses, expiry := current.UsesAllowed, current.ExpiresAtMS + if len(req.UsesAllowed) > 0 { + if string(req.UsesAllowed) == "null" { + uses = nil + } else { + var n int + if err := json.Unmarshal(req.UsesAllowed, &n); err != nil { + return httputil.InvalidParam("uses_allowed must be a number or null") + } + uses = &n + } + } + if len(req.ExpiryTimeMS) > 0 { + if string(req.ExpiryTimeMS) == "null" { + expiry = 0 + } else if err := json.Unmarshal(req.ExpiryTimeMS, &expiry); err != nil { + return httputil.InvalidParam("expiry_time_ms must be a number or null") + } + } + + updated, err := a.tokens.Update(r.Context(), token, uses, expiry) + if err != nil { + return tokenError(err) + } + a.audit(r.Context(), auth, auditTokenUpdate, token, map[string]any{ + "uses_allowed": updated.UsesAllowed, "expiry_time_ms": updated.ExpiresAtMS, + }) + httputil.WriteJSON(w, http.StatusOK, a.registrationTokenOf(updated)) + return nil +} + +// handleDeleteRegistrationToken implements DELETE +// /_warp/admin/v1/registration_tokens/{token}. Accounts registered with +// the token are not affected. +func (a *AdminAPI) handleDeleteRegistrationToken(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + token := r.PathValue("token") + if err := a.tokens.Delete(r.Context(), token); err != nil { + return tokenError(err) + } + a.audit(r.Context(), auth, auditTokenDelete, token, nil) + httputil.WriteJSON(w, http.StatusOK, struct{}{}) + return nil +} + +func tokenError(err error) error { + switch { + case errors.Is(err, userapi.ErrNoSuchRegistrationToken): + return httputil.NotFound("No such registration token") + case errors.Is(err, userapi.ErrInvalidRegistrationToken): + return httputil.InvalidParam("%s", err.Error()) + default: + return err + } +} diff --git a/server/internal/adminapi/user.go b/server/internal/adminapi/user.go new file mode 100644 index 0000000..d2a53e8 --- /dev/null +++ b/server/internal/adminapi/user.go @@ -0,0 +1,415 @@ +package adminapi + +import ( + "errors" + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/storage" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// The endpoints about one user, addressed by full user ID. + +// userDetail is GET /users/{userID}: the list row and what it takes a +// count to know. +type userDetail struct { + adminUser + + Devices int `json:"devices"` + + // JoinedRooms is absent on a server without rooms. + JoinedRooms *int `json:"joined_rooms,omitempty"` + + // Media is absent on a server without a media repository. + Media *mediaResponse `json:"media,omitempty"` +} + +// mediaResponse is a user's uploads against their quota. +type mediaResponse struct { + Uploads int `json:"uploads"` + Bytes int64 `json:"bytes"` + + // QuotaBytes is the quota in force, 0 for no limit; CustomQuota says + // whether it was set for this user rather than taken from the config. + QuotaBytes int64 `json:"quota_bytes"` + CustomQuota bool `json:"custom_quota"` +} + +// pathUser reads the {userID} path value: a user of this server. +func (a *AdminAPI) pathUser(r *http.Request) (string, error) { + raw := r.PathValue("userID") + localpart, err := a.users.ParseLocalUserID(raw) + switch { + case errors.Is(err, userapi.ErrForeignServer): + return "", httputil.InvalidParam("%s is not a user of this server", raw) + case err != nil: + return "", httputil.InvalidParam("%s is not a valid user ID", raw) + } + return localpart, nil +} + +// pathAccount reads the {userID} path value and the account it names. +func (a *AdminAPI) pathAccount(r *http.Request) (storage.Account, error) { + localpart, err := a.pathUser(r) + if err != nil { + return storage.Account{}, err + } + account, err := a.users.Account(r.Context(), localpart) + switch { + case errors.Is(err, userapi.ErrNoSuchUser): + return storage.Account{}, httputil.NotFound("No such user") + case err != nil: + return storage.Account{}, err + } + return account, nil +} + +func (a *AdminAPI) adminUserOf(acc storage.Account) adminUser { + return adminUser{ + UserID: a.users.UserID(acc.Localpart), + DisplayName: acc.DisplayName, + AvatarURL: acc.AvatarURL, + Admin: acc.Admin, + Deactivated: acc.Deactivated, + CreatedAtMS: acc.CreatedAtMS, + } +} + +// handleGetUser implements GET /_warp/admin/v1/users/{userID}. +func (a *AdminAPI) handleGetUser(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + return a.writeUser(w, r, account) +} + +// writeUser answers with a user's detail, which is also what every change +// to a user answers with, so the panel can redraw from the response. +func (a *AdminAPI) writeUser(w http.ResponseWriter, r *http.Request, account storage.Account) error { + devices, err := a.users.Devices(r.Context(), account.Localpart) + if err != nil { + return err + } + resp := userDetail{adminUser: a.adminUserOf(account), Devices: len(devices)} + userID := a.users.UserID(account.Localpart) + if a.rooms != nil { + joined, err := a.rooms.JoinedRooms(r.Context(), userID) + if err != nil { + return err + } + n := len(joined) + resp.JoinedRooms = &n + } + if a.media != nil { + quota, err := a.media.Quota(r.Context(), userID) + if err != nil { + return err + } + resp.Media = &mediaResponse{ + Uploads: quota.Usage.Count, Bytes: quota.Usage.Bytes, + QuotaBytes: quota.MaxBytes, CustomQuota: quota.Custom, + } + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, resp) + return nil +} + +// refuseSelf stops an admin taking their own admin flag or deactivating +// themselves. Both are one click from locking the last admin out of the +// panel, and both are easy to do by accident on a table of users; another +// admin can still do either. +func refuseSelf(auth userapi.Authenticated, localpart, what string) error { + if auth.Localpart == localpart { + return httputil.Forbidden("You cannot %s your own account; ask another admin", what) + } + return nil +} + +// adminFlagRequest is the PUT /users/{userID}/admin body. +type adminFlagRequest struct { + Admin *bool `json:"admin"` +} + +// handlePutUserAdmin implements PUT /_warp/admin/v1/users/{userID}/admin. +func (a *AdminAPI) handlePutUserAdmin(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + var req adminFlagRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + if req.Admin == nil { + return httputil.MissingParam("admin is required") + } + if !*req.Admin { + if err := refuseSelf(auth, account.Localpart, "remove the admin flag from"); err != nil { + return err + } + } + if account.Admin != *req.Admin { + if err := a.users.SetAdmin(r.Context(), account.Localpart, *req.Admin); err != nil { + return err + } + account.Admin = *req.Admin + a.audit(r.Context(), auth, auditUserAdmin, a.users.UserID(account.Localpart), + map[string]any{"admin": *req.Admin}) + } + return a.writeUser(w, r, account) +} + +// deactivateRequest is the POST /users/{userID}/deactivate body. +type deactivateRequest struct { + // Erase forgets the profile as well. + Erase bool `json:"erase"` +} + +// handleDeactivateUser implements POST /_warp/admin/v1/users/{userID}/deactivate. +// +// It does what the user's own POST /account/deactivate does: every token +// stops working, the devices and their keys and pushers go, the user +// leaves every room, and with erase the display name and avatar are +// cleared first so the leave events carry neither. The name stays taken. +// Deactivating a deactivated account changes nothing. +func (a *AdminAPI) handleDeactivateUser(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + var req deactivateRequest + if _, err := httputil.DecodeOptionalJSON(w, r, &req); err != nil { + return err + } + if err := refuseSelf(auth, account.Localpart, "deactivate"); err != nil { + return err + } + if !account.Deactivated { + if err := a.users.DeactivateAccount(r.Context(), account.Localpart, req.Erase); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserDeactivate, a.users.UserID(account.Localpart), + map[string]any{"erase": req.Erase}) + } + if account, err = a.users.Account(r.Context(), account.Localpart); err != nil { + return err + } + return a.writeUser(w, r, account) +} + +// handleReactivateUser implements POST /_warp/admin/v1/users/{userID}/reactivate. +// +// The account can log in again, with its old password unless an admin +// sets a new one. Nothing else comes back: the rooms were left and the +// devices are gone. +func (a *AdminAPI) handleReactivateUser(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + if account.Deactivated { + if err := a.users.ReactivateAccount(r.Context(), account.Localpart); err != nil { + return err + } + account.Deactivated = false + a.audit(r.Context(), auth, auditUserReactivate, a.users.UserID(account.Localpart), nil) + } + return a.writeUser(w, r, account) +} + +// resetPasswordRequest is the POST /users/{userID}/password body. +type resetPasswordRequest struct { + Password string `json:"password"` + + // LogoutDevices defaults to true: an admin resetting a password is + // usually doing it because somebody else has the account. + LogoutDevices *bool `json:"logout_devices"` +} + +// handleResetPassword implements POST /_warp/admin/v1/users/{userID}/password. +func (a *AdminAPI) handleResetPassword(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + var req resetPasswordRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + if req.Password == "" { + return httputil.MissingParam("password is required") + } + if len(req.Password) > userapi.MaxPasswordLength { + return httputil.NewError(http.StatusBadRequest, httputil.ErrCodeWeakPassword, + "Password must be at most %d bytes", userapi.MaxPasswordLength) + } + logout := req.LogoutDevices == nil || *req.LogoutDevices + if err := a.users.ChangePassword(r.Context(), account.Localpart, req.Password, logout, ""); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserPassword, a.users.UserID(account.Localpart), + map[string]any{"logout_devices": logout}) + return a.writeUser(w, r, account) +} + +// adminDevice is one device in the device list. +type adminDevice struct { + DeviceID string `json:"device_id"` + DisplayName string `json:"display_name,omitempty"` + CreatedAtMS int64 `json:"created_at_ms"` + LastSeenMS int64 `json:"last_seen_ms"` + LastSeenIP string `json:"last_seen_ip,omitempty"` + UserAgent string `json:"user_agent,omitempty"` +} + +// handleGetUserDevices implements GET /_warp/admin/v1/users/{userID}/devices. +func (a *AdminAPI) handleGetUserDevices(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + devices, err := a.users.Devices(r.Context(), account.Localpart) + if err != nil { + return err + } + out := make([]adminDevice, 0, len(devices)) + for _, d := range devices { + out = append(out, adminDevice{ + DeviceID: d.ID, DisplayName: d.DisplayName, CreatedAtMS: d.CreatedAtMS, + LastSeenMS: d.LastSeenMS, LastSeenIP: d.LastSeenIP, UserAgent: d.UserAgent, + }) + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, map[string]any{"devices": out}) + return nil +} + +// handleDeleteUserDevice implements +// DELETE /_warp/admin/v1/users/{userID}/devices/{deviceID}: log one +// session out, taking its keys and pushers with it. +func (a *AdminAPI) handleDeleteUserDevice(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + deviceID := r.PathValue("deviceID") + if _, err := a.users.Device(r.Context(), account.Localpart, deviceID); errors.Is(err, userapi.ErrNoSuchDevice) { + return httputil.NotFound("No such device") + } else if err != nil { + return err + } + if err := a.users.DeleteDevices(r.Context(), account.Localpart, []string{deviceID}); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserDeviceDelete, a.users.UserID(account.Localpart), + map[string]any{"device_id": deviceID}) + httputil.WriteJSON(w, http.StatusOK, struct{}{}) + return nil +} + +// handleDeleteUserDevices implements +// DELETE /_warp/admin/v1/users/{userID}/devices: log every session out. +func (a *AdminAPI) handleDeleteUserDevices(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + account, err := a.pathAccount(r) + if err != nil { + return err + } + if err := a.users.DeleteAllDevices(r.Context(), account.Localpart); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserDeviceDelete, a.users.UserID(account.Localpart), + map[string]any{"all": true}) + httputil.WriteJSON(w, http.StatusOK, struct{}{}) + return nil +} + +// requireMedia refuses the media endpoints on a server without a media +// repository. +func (a *AdminAPI) requireMedia() error { + if a.media == nil { + return httputil.NotFound("This server has no media repository") + } + return nil +} + +// handleGetUserMedia implements GET /_warp/admin/v1/users/{userID}/media. +func (a *AdminAPI) handleGetUserMedia(w http.ResponseWriter, r *http.Request, _ userapi.Authenticated) error { + if err := a.requireMedia(); err != nil { + return err + } + account, err := a.pathAccount(r) + if err != nil { + return err + } + return a.writeMedia(w, r, account) +} + +func (a *AdminAPI) writeMedia(w http.ResponseWriter, r *http.Request, account storage.Account) error { + quota, err := a.media.Quota(r.Context(), a.users.UserID(account.Localpart)) + if err != nil { + return err + } + httputil.NoCache(w) + httputil.WriteJSON(w, http.StatusOK, mediaResponse{ + Uploads: quota.Usage.Count, Bytes: quota.Usage.Bytes, + QuotaBytes: quota.MaxBytes, CustomQuota: quota.Custom, + }) + return nil +} + +// quotaRequest is the PUT /users/{userID}/media/quota body. +type quotaRequest struct { + MaxBytes *int64 `json:"max_bytes"` +} + +// handlePutMediaQuota implements PUT /_warp/admin/v1/users/{userID}/media/quota. +// +// A quota below what the user already has does not delete anything; it +// stops further uploads until they are back under it. +func (a *AdminAPI) handlePutMediaQuota(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + if err := a.requireMedia(); err != nil { + return err + } + account, err := a.pathAccount(r) + if err != nil { + return err + } + var req quotaRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + if req.MaxBytes == nil { + return httputil.MissingParam("max_bytes is required (0 for no limit)") + } + if *req.MaxBytes < 0 { + return httputil.InvalidParam("max_bytes must not be negative") + } + userID := a.users.UserID(account.Localpart) + if err := a.media.SetQuota(r.Context(), userID, *req.MaxBytes); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserMediaQuota, userID, map[string]any{"max_bytes": *req.MaxBytes}) + return a.writeMedia(w, r, account) +} + +// handleDeleteMediaQuota implements DELETE +// /_warp/admin/v1/users/{userID}/media/quota: back to the configured +// default. +func (a *AdminAPI) handleDeleteMediaQuota(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + if err := a.requireMedia(); err != nil { + return err + } + account, err := a.pathAccount(r) + if err != nil { + return err + } + userID := a.users.UserID(account.Localpart) + if err := a.media.ResetQuota(r.Context(), userID); err != nil { + return err + } + a.audit(r.Context(), auth, auditUserMediaQuota, userID, map[string]any{"max_bytes": nil}) + return a.writeMedia(w, r, account) +} diff --git a/server/internal/adminapi/users.go b/server/internal/adminapi/users.go index e59a0bc..20c19a9 100644 --- a/server/internal/adminapi/users.go +++ b/server/internal/adminapi/users.go @@ -128,8 +128,8 @@ func (a *AdminAPI) handlePostUsers(w http.ResponseWriter, r *http.Request, auth return err } - a.log.Info("account created through the admin API", - "user_id", a.users.UserID(account.Localpart), "admin", account.Admin, "by", auth.UserID) + a.audit(r.Context(), auth, auditUserCreate, a.users.UserID(account.Localpart), + map[string]any{"admin": account.Admin}) httputil.WriteJSON(w, http.StatusCreated, adminUser{ UserID: a.users.UserID(account.Localpart), diff --git a/server/internal/appservice/appservice.go b/server/internal/appservice/appservice.go new file mode 100644 index 0000000..a3f9d0c --- /dev/null +++ b/server/internal/appservice/appservice.go @@ -0,0 +1,246 @@ +// Package appservice is the homeserver's side of the Application Service +// API: the registry of application services (bridges and bots with a +// namespace of users, aliases and rooms of their own), and the sender that +// delivers them the events they are interested in. +// +// The Client-Server half — an application service acting as one of its +// users, registering them, the M_EXCLUSIVE refusals — is in clientapi, +// which asks the Registry here. What this package does on its own is +// talk to the services: transactions, the user and alias queries, and the +// ping. +// +// Delivery reads the roomserver's event stream from a cursor kept per +// service in the database, rather than queueing events as they are +// persisted. The stream is already durable and ordered, so a cursor is all +// a service that has been down for a day needs to be caught up, and the +// emit stage only has to say "there is more" rather than hand anything +// over. +package appservice + +import ( + "context" + "errors" + "fmt" + "log/slog" + "sync" + "time" + + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// Rooms is what the sender reads from the roomserver. *roomserver.API +// satisfies it. +type Rooms interface { + StreamTop(ctx context.Context) (int64, error) + StreamEvents(ctx context.Context, after, upTo int64, limit int) ([]roomserver.StreamEvent, error) + JoinedUserIDs(ctx context.Context, roomID string) ([]string, error) + LocalAliases(ctx context.Context, roomID string) ([]string, error) +} + +// DefaultTimeout bounds one request to a service. +const DefaultTimeout = 30 * time.Second + +// Options configure the component. +type Options struct { + Registry *Registry + Store storage.AppServiceStore + + // Rooms is the roomserver. Without one there is no stream to deliver + // and the sender does not start; the registry and the queries still + // work. + Rooms Rooms + + // Timeout bounds one request to a service; zero is DefaultTimeout. + Timeout time.Duration + + Logger *slog.Logger + + // Now is the clock, for the status timestamps. + Now func() time.Time +} + +// API is the component. +type API struct { + registry *Registry + store storage.AppServiceStore + rooms Rooms + client *Client + log *slog.Logger + now func() time.Time + + workers map[string]*worker + + mu sync.Mutex + cancel context.CancelFunc + wg sync.WaitGroup + started bool +} + +// New builds the component. It does not start delivering; Start does. +func New(opts Options) (*API, error) { + if opts.Registry == nil { + var err error + if opts.Registry, err = NewRegistry("", nil); err != nil { + return nil, err + } + } + if opts.Store == nil && opts.Rooms != nil { + return nil, errors.New("appservice: Store must not be nil when there is a roomserver") + } + if opts.Timeout <= 0 { + opts.Timeout = DefaultTimeout + } + if opts.Logger == nil { + opts.Logger = slog.Default() + } + if opts.Now == nil { + opts.Now = time.Now + } + a := &API{ + registry: opts.Registry, + store: opts.Store, + rooms: opts.Rooms, + client: NewClient(opts.Timeout), + log: opts.Logger, + now: opts.Now, + workers: map[string]*worker{}, + } + for _, svc := range opts.Registry.Services() { + // A service with no URL is never sent anything, so it has no + // worker and no cursor. + if svc.URL != "" { + a.workers[svc.ID] = newWorker(svc) + } + } + return a, nil +} + +// Registry is every registered service. +func (a *API) Registry() *Registry { return a.registry } + +// Start starts a delivery worker per service with a URL. Calling it twice +// is harmless. +func (a *API) Start(ctx context.Context) { + a.mu.Lock() + defer a.mu.Unlock() + if a.started || a.rooms == nil || len(a.workers) == 0 { + return + } + a.started = true + ctx, a.cancel = context.WithCancel(context.WithoutCancel(ctx)) + for _, w := range a.workers { + a.wg.Add(1) + go a.run(ctx, w) + } + a.log.Info("application service delivery started", "services", len(a.workers)) +} + +// Close stops the workers and waits for them. A transaction being sent is +// abandoned and sent again, with the same ID, on the next start. +func (a *API) Close() { + a.mu.Lock() + cancel := a.cancel + a.mu.Unlock() + if cancel != nil { + cancel() + } + a.wg.Wait() +} + +// EventPersisted implements roomserver.Notifier: it wakes every worker. +// Deciding whether the event concerns a service is the worker's job, done +// off the room's write lock. +func (a *API) EventPersisted(context.Context, roomserver.Notification) { + for _, w := range a.workers { + w.poke() + } +} + +// Statuses reports the delivery state of every service with a URL, in +// registration order. +func (a *API) Statuses() []Status { + out := make([]Status, 0, len(a.workers)) + for _, svc := range a.registry.Services() { + if w, ok := a.workers[svc.ID]; ok { + out = append(out, w.snapshot()) + } + } + return out +} + +// QueryUser asks the services whose namespaces match a user whether it +// exists, stopping at the first that says yes. A service that fails to +// answer counts as a no, and is logged: the request that wanted the user +// goes on without it rather than failing. +func (a *API) QueryUser(ctx context.Context, userID string) bool { + for _, svc := range a.registry.UserOwners(userID) { + ok, err := a.client.QueryUser(ctx, svc, userID) + if err != nil { + a.log.Warn("an application service failed a user query", "appservice", svc.ID, "user_id", userID, "error", err) + continue + } + if ok { + return true + } + } + return false +} + +// QueryAlias is QueryUser for a room alias. +func (a *API) QueryAlias(ctx context.Context, alias string) bool { + for _, svc := range a.registry.AliasOwners(alias) { + ok, err := a.client.QueryAlias(ctx, svc, alias) + if err != nil { + a.log.Warn("an application service failed an alias query", "appservice", svc.ID, "alias", alias, "error", err) + continue + } + if ok { + return true + } + } + return false +} + +// Ping makes the homeserver ping a service, for the service's own +// POST /_matrix/client/v1/appservice/{appserviceId}/ping. A service that +// answers has any transaction waiting out a backoff sent at once. +func (a *API) Ping(ctx context.Context, svc *Service, transactionID string) (time.Duration, error) { + took, err := a.client.Ping(ctx, svc, transactionID) + if err != nil { + return 0, err + } + if w, ok := a.workers[svc.ID]; ok { + w.retryNow() + } + return took, nil +} + +// WaitIdle blocks until every worker has caught up with the top of the +// stream, or ctx ends. Tests use it to know a transaction has been sent. +func (a *API) WaitIdle(ctx context.Context) error { + if a.rooms == nil { + return nil + } + for { + top, err := a.rooms.StreamTop(ctx) + if err != nil { + return err + } + idle := true + for _, w := range a.workers { + if w.snapshot().StreamPos < top { + idle = false + break + } + } + if idle { + return nil + } + select { + case <-ctx.Done(): + return fmt.Errorf("application services are still delivering: %w", ctx.Err()) + case <-time.After(10 * time.Millisecond): + } + } +} diff --git a/server/internal/appservice/appservice_test.go b/server/internal/appservice/appservice_test.go new file mode 100644 index 0000000..9c572b5 --- /dev/null +++ b/server/internal/appservice/appservice_test.go @@ -0,0 +1,435 @@ +package appservice + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +const ircRegistration = ` +id: irc +url: %URL% +as_token: as-secret +hs_token: hs-secret +sender_localpart: ircbot +rate_limited: false +namespaces: + users: + - exclusive: true + regex: '@_irc_.*' + aliases: + - exclusive: false + regex: '#_irc_.*:hs' + rooms: [] +de.sorunome.msc2409.push_ephemeral: true +` + +func parse(t *testing.T, doc, url string) *Registration { + t.Helper() + reg, err := ParseRegistration([]byte(strings.ReplaceAll(doc, "%URL%", url)), "hs") + if err != nil { + t.Fatalf("ParseRegistration: %v", err) + } + return reg +} + +func TestParseRegistration(t *testing.T) { + reg := parse(t, ircRegistration, "http://localhost:9000") + if reg.ID != "irc" || reg.SenderLocalpart != "ircbot" || reg.RateLimitedUsers() { + t.Errorf("registration = %+v", reg) + } + + // The namespace is anchored at the start only, the way Synapse matches. + ns := reg.Namespaces.Users[0] + for id, want := range map[string]bool{ + "@_irc_alice:hs": true, + "@_irc_:other": true, + "@alice_irc_x:hs": false, + "@notbot_irc_x:hs": false, + } { + if got := ns.Matches(id); got != want { + t.Errorf("Matches(%q) = %v", id, got) + } + } + + // A url of null is a service that receives nothing. + noURL := parse(t, strings.Replace(ircRegistration, "url: %URL%", "url: null", 1), "") + if noURL.URL != "" { + t.Errorf("url: null = %q", noURL.URL) + } + + for name, doc := range map[string]string{ + "no id": strings.Replace(ircRegistration, "id: irc", "id: ''", 1), + "same tokens": strings.Replace(ircRegistration, "hs_token: hs-secret", "hs_token: as-secret", 1), + "bad regex": strings.Replace(ircRegistration, "'@_irc_.*'", "'@_irc_(.*'", 1), + "bad sender": strings.Replace(ircRegistration, "sender_localpart: ircbot", "sender_localpart: 'irc bot'", 1), + "not http": strings.Replace(ircRegistration, "url: %URL%", "url: ftp://x", 1), + "no as_token": strings.Replace(ircRegistration, "as_token: as-secret", "as_token: ''", 1), + "empty regex": strings.Replace(ircRegistration, "'@_irc_.*'", "''", 1), + "not a document": "- just\n- a list\n", + } { + if _, err := ParseRegistration([]byte(doc), "hs"); err == nil { + t.Errorf("%s: accepted", name) + } + } +} + +func TestRegistry(t *testing.T) { + irc := parse(t, ircRegistration, "http://localhost:9000") + other := parse(t, strings.NewReplacer( + "id: irc", "id: slack", + "as_token: as-secret", "as_token: as-slack", + "hs_token: hs-secret", "hs_token: hs-slack", + "sender_localpart: ircbot", "sender_localpart: slackbot", + "'@_irc_.*'", "'@_slack_.*'", + "exclusive: true", "exclusive: false", + ).Replace(ircRegistration), "") + + reg, err := NewRegistry("hs", []*Registration{irc, other}) + if err != nil { + t.Fatal(err) + } + if reg.ByToken("as-secret").ID != "irc" || reg.ByToken("hs-secret") != nil || reg.ByToken("") != nil { + t.Error("ByToken") + } + if reg.ByID("slack").SenderUserID() != "@slackbot:hs" { + t.Error("ByID / SenderUserID") + } + if owner := reg.ExclusiveUserOwner("@_irc_bob:hs"); owner == nil || owner.ID != "irc" { + t.Error("the exclusive namespace has no owner") + } + if reg.ExclusiveUserOwner("@_slack_bob:hs") != nil { + t.Error("a non-exclusive namespace was claimed") + } + if reg.ExclusiveUserOwner("@slackbot:hs") == nil { + t.Error("a sender is not claimed by its service") + } + svc := reg.ByID("irc") + if !svc.OwnsUser("@ircbot:hs") || !svc.OwnsUser("@_irc_x:hs") || svc.OwnsUser("@_slack_x:hs") { + t.Error("OwnsUser") + } + if !svc.OwnsAlias("#_irc_chan:hs") || svc.ClaimsAliasExclusively("#_irc_chan:hs") { + t.Error("alias namespaces") + } + if owners := reg.UserOwners("@_slack_x:hs"); len(owners) != 0 { + t.Errorf("a service without a URL is asked about users: %v", owners) + } + + if _, err := NewRegistry("hs", []*Registration{irc, irc}); err == nil { + t.Error("a duplicate ID was accepted") + } + clash := parse(t, strings.NewReplacer("id: irc", "id: clash", "as_token: as-secret", "as_token: hs-secret", + "hs_token: hs-secret", "hs_token: other").Replace(ircRegistration), "http://x") + if _, err := NewRegistry("hs", []*Registration{irc, clash}); err == nil { + t.Error("an as_token equal to another service's hs_token was accepted") + } +} + +// fakeRooms is a roomserver with a stream held in memory. +type fakeRooms struct { + mu sync.Mutex + events []roomserver.StreamEvent + members map[string][]string + aliases map[string][]string +} + +func (f *fakeRooms) add(roomID, eventType, sender string, stateKey *string) { + f.mu.Lock() + defer f.mu.Unlock() + pos := int64(len(f.events) + 1) + f.events = append(f.events, roomserver.StreamEvent{Position: pos, Event: roomserver.ClientEvent{ + EventID: "$" + string(rune('a'+pos)), RoomID: roomID, Type: eventType, Sender: sender, + StateKey: stateKey, Content: json.RawMessage(`{}`), + }}) +} + +func (f *fakeRooms) StreamTop(context.Context) (int64, error) { + f.mu.Lock() + defer f.mu.Unlock() + return int64(len(f.events)), nil +} + +func (f *fakeRooms) StreamEvents(_ context.Context, after, upTo int64, limit int) ([]roomserver.StreamEvent, error) { + f.mu.Lock() + defer f.mu.Unlock() + var out []roomserver.StreamEvent + for _, e := range f.events { + if e.Position > after && (upTo == 0 || e.Position <= upTo) && len(out) < limit { + out = append(out, e) + } + } + return out, nil +} + +func (f *fakeRooms) JoinedUserIDs(_ context.Context, roomID string) ([]string, error) { + f.mu.Lock() + defer f.mu.Unlock() + return f.members[roomID], nil +} + +func (f *fakeRooms) LocalAliases(_ context.Context, roomID string) ([]string, error) { + f.mu.Lock() + defer f.mu.Unlock() + return f.aliases[roomID], nil +} + +// memStore is an AppServiceStore in memory. +type memStore struct { + mu sync.Mutex + state map[string]storage.AppServiceState +} + +func (m *memStore) AppServiceState(_ context.Context, id string, start int64) (storage.AppServiceState, error) { + m.mu.Lock() + defer m.mu.Unlock() + st, ok := m.state[id] + if !ok { + st = storage.AppServiceState{ID: id, StreamPos: start} + m.state[id] = st + } + return st, nil +} + +func (m *memStore) BeginTransaction(_ context.Context, id string, txnID, to int64) error { + m.mu.Lock() + defer m.mu.Unlock() + st := m.state[id] + st.TxnID, st.TxnTo = txnID, to + m.state[id] = st + return nil +} + +func (m *memStore) CompleteTransaction(_ context.Context, id string, to int64) error { + m.mu.Lock() + defer m.mu.Unlock() + st := m.state[id] + st.StreamPos, st.TxnTo = to, 0 + m.state[id] = st + return nil +} + +// fakeService records the transactions it is sent, and fails the first +// failFirst of them. +type fakeService struct { + mu sync.Mutex + txns map[string][]string // txn ID -> event IDs + order []string + failFirst int + auth []string +} + +func (f *fakeService) ServeHTTP(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + f.auth = append(f.auth, r.Header.Get("Authorization")) + switch { + case strings.HasPrefix(r.URL.Path, "/_matrix/app/v1/transactions/"): + if f.failFirst > 0 { + f.failFirst-- + http.Error(w, `{"errcode":"M_UNKNOWN"}`, http.StatusInternalServerError) + return + } + txnID := strings.TrimPrefix(r.URL.Path, "/_matrix/app/v1/transactions/") + var body struct { + Events []struct { + EventID string `json:"event_id"` + } `json:"events"` + } + raw, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(raw, &body) + var ids []string + for _, e := range body.Events { + ids = append(ids, e.EventID) + } + if _, seen := f.txns[txnID]; !seen { + f.order = append(f.order, txnID) + } + f.txns[txnID] = ids + _, _ = w.Write([]byte(`{}`)) + case r.URL.Path == "/_matrix/app/v1/users/@_irc_known:hs": + _, _ = w.Write([]byte(`{}`)) + case strings.HasPrefix(r.URL.Path, "/_matrix/app/v1/users/"): + http.Error(w, `{"errcode":"M_NOT_FOUND"}`, http.StatusNotFound) + case r.URL.Path == "/_matrix/app/v1/ping": + _, _ = w.Write([]byte(`{}`)) + default: + http.NotFound(w, r) + } +} + +func (f *fakeService) delivered() (map[string][]string, []string) { + f.mu.Lock() + defer f.mu.Unlock() + out := map[string][]string{} + for k, v := range f.txns { + out[k] = v + } + return out, append([]string(nil), f.order...) +} + +func strPtr(s string) *string { return &s } + +func TestSenderDeliversWhatConcernsTheService(t *testing.T) { + service := &fakeService{txns: map[string][]string{}, failFirst: 1} + srv := httptest.NewServer(service) + defer srv.Close() + + reg, err := NewRegistry("hs", []*Registration{parse(t, ircRegistration, srv.URL)}) + if err != nil { + t.Fatal(err) + } + rooms := &fakeRooms{ + members: map[string][]string{"!bridged:hs": {"@alice:hs", "@_irc_bob:hs"}}, + aliases: map[string][]string{"!aliased:hs": {"#_irc_chan:hs"}}, + } + // Before the service is first seen: it starts at the top, and is not + // told about this. + rooms.add("!other:hs", "m.room.message", "@alice:hs", nil) + + store := &memStore{state: map[string]storage.AppServiceState{}} + api, err := New(Options{Registry: reg, Store: store, Rooms: rooms}) + if err != nil { + t.Fatal(err) + } + // The first failure is retried after minBackoff; a ping cuts that short. + api.Start(t.Context()) + defer api.Close() + waitFor(t, func() bool { return api.Statuses()[0].StreamPos == 1 }) + + rooms.add("!other:hs", "m.room.message", "@alice:hs", nil) // 2: no + rooms.add("!other:hs", "m.room.message", "@_irc_carol:hs", nil) // 3: sender + rooms.add("!other:hs", "m.room.member", "@alice:hs", strPtr("@_irc_dave:hs")) // 4: target + rooms.add("!bridged:hs", "m.room.message", "@alice:hs", nil) // 5: member + rooms.add("!aliased:hs", "m.room.message", "@alice:hs", nil) // 6: alias + rooms.add("!other:hs", "m.room.message", "@ircbot:hs", nil) // 7: the sender itself + api.EventPersisted(t.Context(), roomserver.Notification{}) + + waitFor(t, func() bool { return api.Statuses()[0].Failures > 0 }) + if _, err := api.Ping(t.Context(), reg.ByID("irc"), "ping-1"); err != nil { + t.Fatalf("Ping: %v", err) + } + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + if err := api.WaitIdle(ctx); err != nil { + t.Fatal(err) + } + + txns, order := service.delivered() + if len(order) != 1 { + t.Fatalf("transactions = %v", txns) + } + got := strings.Join(txns[order[0]], ",") + if want := "$d,$e,$f,$g,$h"; got != want { + t.Errorf("delivered %s, want %s", got, want) + } + if order[0] != "1" { + t.Errorf("first transaction ID = %s", order[0]) + } + status := api.Statuses()[0] + if status.Failures != 0 || status.EventsDelivered != 5 || status.TxnID != 1 { + t.Errorf("status = %+v", status) + } + for _, a := range service.auth { + if a != "Bearer hs-secret" { + t.Errorf("Authorization = %q", a) + } + } + + // Queries: yes for a user the service knows, no otherwise. + if !api.QueryUser(t.Context(), "@_irc_known:hs") || api.QueryUser(t.Context(), "@_irc_unknown:hs") { + t.Error("QueryUser") + } + if api.QueryUser(t.Context(), "@alice:hs") { + t.Error("a user outside every namespace was queried") + } +} + +func TestSenderResumesATransactionInFlight(t *testing.T) { + service := &fakeService{txns: map[string][]string{}} + srv := httptest.NewServer(service) + defer srv.Close() + reg, _ := NewRegistry("hs", []*Registration{parse(t, ircRegistration, srv.URL)}) + + rooms := &fakeRooms{} + rooms.add("!r:hs", "m.room.message", "@_irc_a:hs", nil) // 1 + rooms.add("!r:hs", "m.room.message", "@_irc_b:hs", nil) // 2 + rooms.add("!r:hs", "m.room.message", "@_irc_c:hs", nil) // 3 + + // An earlier run began transaction 7 over (0, 2] and stopped. + store := &memStore{state: map[string]storage.AppServiceState{ + "irc": {ID: "irc", StreamPos: 0, TxnID: 7, TxnTo: 2}, + }} + api, _ := New(Options{Registry: reg, Store: store, Rooms: rooms}) + api.Start(t.Context()) + defer api.Close() + + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + if err := api.WaitIdle(ctx); err != nil { + t.Fatal(err) + } + txns, order := service.delivered() + if len(order) != 2 || order[0] != "7" || order[1] != "8" { + t.Fatalf("transactions = %v (%v)", txns, order) + } + if strings.Join(txns["7"], ",") != "$b,$c" || strings.Join(txns["8"], ",") != "$d" { + t.Errorf("transactions = %v", txns) + } +} + +func TestPingErrors(t *testing.T) { + client := NewClient(200 * time.Millisecond) + + noURL := &Service{Registration: &Registration{ID: "x", HSToken: "t"}} + if _, err := client.Ping(t.Context(), noURL, ""); !errors.Is(err, ErrNoURL) { + t.Errorf("no URL: %v", err) + } + + bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "nope", http.StatusForbidden) + })) + defer bad.Close() + var status *BadStatusError + if _, err := client.Ping(t.Context(), &Service{Registration: &Registration{URL: bad.URL, HSToken: "t"}}, "x"); !errors.As(err, &status) || status.Status != http.StatusForbidden { + t.Errorf("bad status: %v", err) + } + + slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(2 * time.Second): + } + })) + defer slow.Close() + if _, err := client.Ping(t.Context(), &Service{Registration: &Registration{URL: slow.URL, HSToken: "t"}}, ""); !errors.Is(err, ErrConnectionTimeout) { + t.Errorf("slow: %v", err) + } + + closed := httptest.NewServer(http.NotFoundHandler()) + closedURL := closed.URL + closed.Close() + if _, err := client.Ping(t.Context(), &Service{Registration: &Registration{URL: closedURL, HSToken: "t"}}, ""); !errors.Is(err, ErrConnectionFailed) { + t.Errorf("closed: %v", err) + } +} + +func waitFor(t *testing.T, cond func() bool) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for !cond() { + if time.Now().After(deadline) { + t.Fatal("timed out") + } + time.Sleep(5 * time.Millisecond) + } +} diff --git a/server/internal/appservice/client.go b/server/internal/appservice/client.go new file mode 100644 index 0000000..c4a2bd6 --- /dev/null +++ b/server/internal/appservice/client.go @@ -0,0 +1,170 @@ +package appservice + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strconv" + "strings" + "time" +) + +// Errors of the Application Service API client. The ping endpoint reports +// each with its own Matrix error code. +var ( + // ErrNoURL is a service registered without a URL, which cannot be + // sent anything. + ErrNoURL = errors.New("application service has no URL") + + // ErrConnectionFailed means the service could not be reached. + ErrConnectionFailed = errors.New("could not connect to the application service") + + // ErrConnectionTimeout means it did not answer in time. + ErrConnectionTimeout = errors.New("the application service did not answer in time") +) + +// BadStatusError is an answer from the service that was not a success. +type BadStatusError struct { + Status int + + // Body is the start of what the service answered, for whoever has to + // work out why. + Body string +} + +func (e *BadStatusError) Error() string { + return fmt.Sprintf("the application service answered %d: %s", e.Status, e.Body) +} + +// maxErrorBody caps how much of a failed answer is kept. +const maxErrorBody = 4 << 10 + +// Client speaks the Application Service API: the requests a homeserver +// makes to a service. +type Client struct { + http *http.Client + timeout time.Duration +} + +// NewClient builds a client. The timeout bounds one request; transactions +// that time out are retried like any other failure. +func NewClient(timeout time.Duration) *Client { + return &Client{ + http: &http.Client{ + // A service is reached at the URL its registration names and + // nowhere else: a redirect would take the hs_token with it. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + }, + timeout: timeout, + } +} + +// transactionBody is PUT /_matrix/app/v1/transactions/{txnId}. +type transactionBody struct { + Events []json.RawMessage `json:"events"` +} + +// SendTransaction delivers a batch of events. The same transaction ID is +// resent with the same events until the service accepts it, and the +// service deduplicates by ID. +func (c *Client) SendTransaction(ctx context.Context, svc *Service, txnID int64, events []json.RawMessage) error { + body, err := json.Marshal(transactionBody{Events: events}) + if err != nil { + return err + } + _, err = c.do(ctx, svc, http.MethodPut, "/_matrix/app/v1/transactions/"+strconv.FormatInt(txnID, 10), body) + return err +} + +// QueryUser asks a service whether a user in its namespace exists, which +// it answers by creating the user if it wants it to. It reports whether +// the service said yes. +func (c *Client) QueryUser(ctx context.Context, svc *Service, userID string) (bool, error) { + return c.query(ctx, svc, "/_matrix/app/v1/users/"+url.PathEscape(userID)) +} + +// QueryAlias asks a service whether an alias in its namespace exists, +// which it answers by creating the room and the alias. +func (c *Client) QueryAlias(ctx context.Context, svc *Service, alias string) (bool, error) { + return c.query(ctx, svc, "/_matrix/app/v1/rooms/"+url.PathEscape(alias)) +} + +// query makes a GET whose answer is yes (200) or no (404). +func (c *Client) query(ctx context.Context, svc *Service, path string) (bool, error) { + _, err := c.do(ctx, svc, http.MethodGet, path, nil) + var status *BadStatusError + switch { + case err == nil: + return true, nil + case errors.As(err, &status) && status.Status == http.StatusNotFound: + return false, nil + default: + return false, err + } +} + +// Ping makes POST /_matrix/app/v1/ping and reports how long it took. +func (c *Client) Ping(ctx context.Context, svc *Service, transactionID string) (time.Duration, error) { + body, err := json.Marshal(map[string]string{"transaction_id": transactionID}) + if transactionID == "" { + body, err = []byte(`{}`), nil + } + if err != nil { + return 0, err + } + start := time.Now() + if _, err := c.do(ctx, svc, http.MethodPost, "/_matrix/app/v1/ping", body); err != nil { + return 0, err + } + return time.Since(start), nil +} + +// do makes one request to a service, authenticated with its hs_token. +func (c *Client) do(ctx context.Context, svc *Service, method, path string, body []byte) ([]byte, error) { + if svc.URL == "" { + return nil, ErrNoURL + } + ctx, cancel := context.WithTimeout(ctx, c.timeout) + defer cancel() + + var reader io.Reader + if body != nil { + reader = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(svc.URL, "/")+path, reader) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrConnectionFailed, err) + } + // The header form only. The spec deprecated ?access_token= for this in + // v1.4, and a token in a URL ends up in the service's access log. + req.Header.Set("Authorization", "Bearer "+svc.HSToken) + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + + resp, err := c.http.Do(req) + if err != nil { + var netErr net.Error + if errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &netErr) && netErr.Timeout()) { + return nil, fmt.Errorf("%w: %v", ErrConnectionTimeout, err) + } + return nil, fmt.Errorf("%w: %v", ErrConnectionFailed, err) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode < 200 || resp.StatusCode > 299 { + head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody)) + return nil, &BadStatusError{Status: resp.StatusCode, Body: string(head)} + } + out, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrConnectionFailed, err) + } + return out, nil +} diff --git a/server/internal/appservice/registration.go b/server/internal/appservice/registration.go new file mode 100644 index 0000000..8a526b4 --- /dev/null +++ b/server/internal/appservice/registration.go @@ -0,0 +1,178 @@ +package appservice + +import ( + "errors" + "fmt" + "net/url" + "os" + "regexp" + "strings" + + "gopkg.in/yaml.v3" + + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// Registration is one application service's registration file, the YAML +// document the Application Service API defines and every bridge writes. +// +// Unknown keys are ignored rather than refused, unlike Warp's own config +// file: registration files are written by the bridges, which add their own +// and unstable keys freely (push_ephemeral, the MSC3202 ones), and a bridge +// upgrade that adds one must not stop the homeserver from starting. +type Registration struct { + // ID names the application service. It is what the ping endpoint is + // addressed by and what its delivery cursor is stored under, so + // changing it starts the service over from the top of the stream. + ID string `yaml:"id"` + + // URL is where transactions and queries are sent. Empty (or null in + // the file) is a service that receives nothing, and only uses the + // Client-Server API. + URL string `yaml:"url"` + + // ASToken is what the service authenticates to this server with; + // HSToken is what this server authenticates to the service with. + ASToken string `yaml:"as_token"` + HSToken string `yaml:"hs_token"` + + // SenderLocalpart is the localpart of the service's own user, which it + // acts as when it does not ask to act as somebody else. + SenderLocalpart string `yaml:"sender_localpart"` + + Namespaces Namespaces `yaml:"namespaces"` + + // RateLimited is whether the users the service acts as are rate + // limited. Absent is true, as the spec has it. The sender itself never + // is. + RateLimited *bool `yaml:"rate_limited"` + + // Protocols are the third-party protocols the service provides. + Protocols []string `yaml:"protocols"` + + // path is the file the registration was read from, for messages. + path string +} + +// Namespaces are the users, aliases and rooms a service is interested in. +type Namespaces struct { + Users []Namespace `yaml:"users"` + Aliases []Namespace `yaml:"aliases"` + Rooms []Namespace `yaml:"rooms"` +} + +// Namespace is one regular expression and whether the service claims what +// it matches for itself alone. +type Namespace struct { + Exclusive bool `yaml:"exclusive"` + Regex string `yaml:"regex"` + + re *regexp.Regexp +} + +// Matches reports whether an ID is in the namespace. +func (n Namespace) Matches(id string) bool { return n.re != nil && n.re.MatchString(id) } + +// LoadRegistration reads and validates a registration file. +func LoadRegistration(path, serverName string) (*Registration, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read application service registration: %w", err) + } + reg, err := ParseRegistration(data, serverName) + if err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + reg.path = path + return reg, nil +} + +// ParseRegistration parses and validates a registration document. +func ParseRegistration(data []byte, serverName string) (*Registration, error) { + var reg Registration + if err := yaml.Unmarshal(data, ®); err != nil { + return nil, fmt.Errorf("parse application service registration: %w", err) + } + if err := reg.validate(serverName); err != nil { + return nil, err + } + return ®, nil +} + +// validate checks a registration and compiles its namespaces, reporting +// every problem rather than the first. +func (r *Registration) validate(serverName string) error { + var errs []error + bad := func(format string, args ...any) { errs = append(errs, fmt.Errorf(format, args...)) } + + if strings.TrimSpace(r.ID) == "" { + bad("id: must not be empty") + } + if r.ASToken == "" { + bad("as_token: must not be empty") + } + if r.HSToken == "" { + bad("hs_token: must not be empty") + } + if r.ASToken != "" && r.ASToken == r.HSToken { + // Either side could then impersonate the other. + bad("as_token and hs_token must differ") + } + if r.URL != "" { + u, err := url.Parse(r.URL) + switch { + case err != nil: + bad("url: %v", err) + case u.Scheme != "http" && u.Scheme != "https": + bad("url: must be an http(s) URL, got %q", r.URL) + case u.Host == "": + bad("url: has no host") + } + } + r.SenderLocalpart = userapi.NormaliseLocalpart(r.SenderLocalpart) + if err := userapi.ValidateLocalpart(r.SenderLocalpart, serverName); err != nil { + bad("sender_localpart: %v", err) + } + for _, group := range []struct { + name string + list []Namespace + }{ + {"users", r.Namespaces.Users}, + {"aliases", r.Namespaces.Aliases}, + {"rooms", r.Namespaces.Rooms}, + } { + for i := range group.list { + ns := &group.list[i] + if ns.Regex == "" { + bad("namespaces.%s[%d].regex: must not be empty", group.name, i) + continue + } + re, err := compileNamespace(ns.Regex) + if err != nil { + bad("namespaces.%s[%d].regex: %v", group.name, i, err) + continue + } + ns.re = re + } + } + return errors.Join(errs...) +} + +// compileNamespace compiles a namespace regular expression, anchored at +// the start. +// +// The spec does not say how a namespace matches, and the registration +// files in the wild are written against Synapse, which anchors at the +// start and not at the end — so "@_irc_" matches every user whose ID +// begins with it. Anchoring both ends would quietly match nothing for +// those files; anchoring neither would let "@bot" match "@notabot". +func compileNamespace(expr string) (*regexp.Regexp, error) { + return regexp.Compile(`^(?:` + expr + `)`) +} + +// RateLimitedUsers reports whether the users the service acts as are rate +// limited. +func (r *Registration) RateLimitedUsers() bool { return r.RateLimited == nil || *r.RateLimited } + +// Path is the file the registration came from, or "". +func (r *Registration) Path() string { return r.path } diff --git a/server/internal/appservice/registry.go b/server/internal/appservice/registry.go new file mode 100644 index 0000000..af1de5f --- /dev/null +++ b/server/internal/appservice/registry.go @@ -0,0 +1,180 @@ +package appservice + +import ( + "crypto/sha256" + "fmt" +) + +// Service is one registered application service. +type Service struct { + *Registration + + // sender is the full user ID of SenderLocalpart. + sender string +} + +// SenderUserID is the service's own user. +func (s *Service) SenderUserID() string { return s.sender } + +// OwnsUser reports whether the service may act as a user: its own sender, +// or anybody in its user namespaces, exclusive or not. +func (s *Service) OwnsUser(userID string) bool { + return userID == s.sender || matchesAny(s.Namespaces.Users, userID) +} + +// ClaimsUserExclusively reports whether a user ID is in one of the +// service's exclusive user namespaces, which nobody else may register. +func (s *Service) ClaimsUserExclusively(userID string) bool { + return matchesExclusive(s.Namespaces.Users, userID) +} + +// OwnsAlias reports whether an alias is in the service's alias +// namespaces. +func (s *Service) OwnsAlias(alias string) bool { return matchesAny(s.Namespaces.Aliases, alias) } + +// ClaimsAliasExclusively reports whether an alias is in an exclusive alias +// namespace of the service. +func (s *Service) ClaimsAliasExclusively(alias string) bool { + return matchesExclusive(s.Namespaces.Aliases, alias) +} + +// InterestedInUser reports whether events by or about a user concern the +// service. +func (s *Service) InterestedInUser(userID string) bool { return s.OwnsUser(userID) } + +// InterestedInRoom reports whether a room ID is in the service's room +// namespaces. +func (s *Service) InterestedInRoom(roomID string) bool { return matchesAny(s.Namespaces.Rooms, roomID) } + +func matchesAny(list []Namespace, id string) bool { + for _, ns := range list { + if ns.Matches(id) { + return true + } + } + return false +} + +func matchesExclusive(list []Namespace, id string) bool { + for _, ns := range list { + if ns.Exclusive && ns.Matches(id) { + return true + } + } + return false +} + +// Registry is every application service this server knows, looked up by +// ID and by token. +type Registry struct { + services []*Service + byID map[string]*Service + + // byToken is keyed by the SHA-256 of the as_token, not the token: a + // map lookup's timing depends on the key, and the key an attacker + // controls should not be the secret itself. + byToken map[[sha256.Size]byte]*Service +} + +// NewRegistry builds a registry, refusing two services with one ID or one +// token, or a token shared with another service's hs_token. +func NewRegistry(serverName string, regs []*Registration) (*Registry, error) { + r := &Registry{ + byID: map[string]*Service{}, + byToken: map[[sha256.Size]byte]*Service{}, + } + hsTokens := map[string]string{} + for _, reg := range regs { + hsTokens[reg.HSToken] = reg.ID + } + for _, reg := range regs { + if _, dup := r.byID[reg.ID]; dup { + return nil, fmt.Errorf("application service %q is registered twice", reg.ID) + } + key := sha256.Sum256([]byte(reg.ASToken)) + if other, dup := r.byToken[key]; dup { + return nil, fmt.Errorf("application services %q and %q have the same as_token", other.ID, reg.ID) + } + if other, ok := hsTokens[reg.ASToken]; ok { + return nil, fmt.Errorf("the as_token of application service %q is the hs_token of %q", reg.ID, other) + } + svc := &Service{Registration: reg, sender: "@" + reg.SenderLocalpart + ":" + serverName} + r.services = append(r.services, svc) + r.byID[reg.ID] = svc + r.byToken[key] = svc + } + return r, nil +} + +// Services lists every service, in the order they were registered. +func (r *Registry) Services() []*Service { + if r == nil { + return nil + } + return r.services +} + +// Empty reports whether no service is registered. +func (r *Registry) Empty() bool { return r == nil || len(r.services) == 0 } + +// ByID returns a service by ID, or nil. +func (r *Registry) ByID(id string) *Service { + if r == nil { + return nil + } + return r.byID[id] +} + +// ByToken returns the service an as_token belongs to, or nil. +func (r *Registry) ByToken(token string) *Service { + if r == nil || token == "" { + return nil + } + return r.byToken[sha256.Sum256([]byte(token))] +} + +// ExclusiveUserOwner returns the service that claims a user ID +// exclusively, or nil. The sender of a service counts as claimed by it. +func (r *Registry) ExclusiveUserOwner(userID string) *Service { + for _, svc := range r.Services() { + if svc.sender == userID || svc.ClaimsUserExclusively(userID) { + return svc + } + } + return nil +} + +// ExclusiveAliasOwner returns the service that claims an alias +// exclusively, or nil. +func (r *Registry) ExclusiveAliasOwner(alias string) *Service { + for _, svc := range r.Services() { + if svc.ClaimsAliasExclusively(alias) { + return svc + } + } + return nil +} + +// UserOwners lists the services with a URL whose user namespaces match a +// user: the ones to ask whether it exists. +func (r *Registry) UserOwners(userID string) []*Service { + var out []*Service + for _, svc := range r.Services() { + if svc.URL != "" && matchesAny(svc.Namespaces.Users, userID) { + out = append(out, svc) + } + } + return out +} + +// AliasOwners lists the services with a URL whose alias namespaces match +// an alias. +func (r *Registry) AliasOwners(alias string) []*Service { + var out []*Service + for _, svc := range r.Services() { + if svc.URL != "" && svc.OwnsAlias(alias) { + out = append(out, svc) + } + } + return out +} diff --git a/server/internal/appservice/sender.go b/server/internal/appservice/sender.go new file mode 100644 index 0000000..1dbedcb --- /dev/null +++ b/server/internal/appservice/sender.go @@ -0,0 +1,365 @@ +package appservice + +import ( + "context" + "encoding/json" + "errors" + "sync" + "time" + + "github.com/dariuszjaniak/warp/server/internal/matrix/event" + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// transactionSize is the most events one transaction carries, which is +// also how far the stream is read ahead at a time. Synapse sends 100. +const transactionSize = 100 + +// The backoff after a failed transaction: it doubles from the first value +// to the second and stays there. A bridge that is down for maintenance is +// tried every few minutes, not hammered, and is caught up within a minute +// or so of coming back. +const ( + minBackoff = 2 * time.Second + maxBackoff = 5 * time.Minute +) + +// idleRecheck is how often an idle worker looks at the stream without +// being woken. Wake-ups come from the emit stage and are never missed in +// a running server; this is the safety net for one that is. +const idleRecheck = 30 * time.Second + +// Status is what the admin API shows of one service's delivery. +type Status struct { + ID string + URL string + SenderUserID string + StreamPos int64 + TxnID int64 + Failures int + LastError string + LastSuccessMS int64 + LastFailureMS int64 + RetryAtMS int64 + EventsDelivered int64 +} + +// worker delivers to one service. +type worker struct { + svc *Service + wake chan struct{} + + // retry cuts a backoff short: a service that has just answered a ping + // is up, and making it wait out minutes of backoff helps nobody. + retry chan struct{} + + mu sync.Mutex + status Status +} + +func newWorker(svc *Service) *worker { + return &worker{ + svc: svc, + wake: make(chan struct{}, 1), + retry: make(chan struct{}, 1), + status: Status{ + ID: svc.ID, + URL: svc.URL, + SenderUserID: svc.SenderUserID(), + }, + } +} + +// poke wakes the worker without blocking; one pending wake-up is as good +// as many. +func (w *worker) poke() { + select { + case w.wake <- struct{}{}: + default: + } +} + +// retryNow cuts the current backoff short, without blocking. +func (w *worker) retryNow() { + select { + case w.retry <- struct{}{}: + default: + } +} + +func (w *worker) update(fn func(*Status)) { + w.mu.Lock() + defer w.mu.Unlock() + fn(&w.status) +} + +func (w *worker) snapshot() Status { + w.mu.Lock() + defer w.mu.Unlock() + return w.status +} + +// run is a worker's loop: read the stream from the cursor, keep what the +// service is interested in, send it as a transaction until the service +// takes it, move the cursor, repeat. +// +// The cursor is in the database, and so is the transaction in flight, so +// a restart resumes where it stopped — with the same transaction ID for +// the same events, if one was being sent — and nothing is lost or sent +// twice under two IDs. +func (a *API) run(ctx context.Context, w *worker) { + defer a.wg.Done() + log := a.log.With("appservice", w.svc.ID) + + state, ok := a.loadState(ctx, w) + if !ok { + return + } + + var pending []json.RawMessage + for ctx.Err() == nil { + if state.TxnTo != 0 { + if pending == nil { + // A transaction left in flight by an earlier run: read its + // events again. + events, err := a.collect(ctx, w.svc, state.StreamPos, state.TxnTo) + if err != nil { + log.Warn("could not read the events of a transaction in flight", "txn_id", state.TxnID, "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + pending = events + } + if !a.deliver(ctx, w, state.TxnID, pending) { + return + } + if err := a.store.CompleteTransaction(ctx, w.svc.ID, state.TxnTo); err != nil { + log.Warn("could not record a delivered transaction", "txn_id", state.TxnID, "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + delivered := int64(len(pending)) + state.StreamPos, state.TxnTo, pending = state.TxnTo, 0, nil + w.update(func(s *Status) { + s.StreamPos, s.TxnID = state.StreamPos, state.TxnID + s.EventsDelivered += delivered + }) + continue + } + + batch, err := a.rooms.StreamEvents(ctx, state.StreamPos, 0, transactionSize) + if err != nil { + log.Warn("could not read the event stream", "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + if len(batch) == 0 { + select { + case <-ctx.Done(): + return + case <-w.wake: + case <-time.After(idleRecheck): + } + continue + } + + to := batch[len(batch)-1].Position + events, err := a.interesting(ctx, w.svc, batch) + if err != nil { + log.Warn("could not decide which events concern the service", "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + if len(events) == 0 { + // Nothing here for the service: move past it without a + // transaction. + if err := a.store.CompleteTransaction(ctx, w.svc.ID, to); err != nil { + log.Warn("could not advance the cursor", "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + state.StreamPos = to + w.update(func(s *Status) { s.StreamPos = to }) + continue + } + + txnID := state.TxnID + 1 + if err := a.store.BeginTransaction(ctx, w.svc.ID, txnID, to); err != nil { + log.Warn("could not record a transaction", "error", err) + if !sleep(ctx, minBackoff) { + return + } + continue + } + state.TxnID, state.TxnTo, pending = txnID, to, events + } +} + +// loadState reads (or starts) the service's cursor. A service seen for the +// first time starts at the top of the stream: it is told what happens from +// now on, not the server's whole history. +func (a *API) loadState(ctx context.Context, w *worker) (storage.AppServiceState, bool) { + for { + top, err := a.rooms.StreamTop(ctx) + if err == nil { + var state storage.AppServiceState + if state, err = a.store.AppServiceState(ctx, w.svc.ID, top); err == nil { + w.update(func(s *Status) { s.StreamPos, s.TxnID = state.StreamPos, state.TxnID }) + return state, true + } + } + a.log.Warn("could not read an application service's cursor", "appservice", w.svc.ID, "error", err) + if !sleep(ctx, minBackoff) { + return storage.AppServiceState{}, false + } + } +} + +// collect reads the events of (from, to] the service is interested in. +func (a *API) collect(ctx context.Context, svc *Service, from, to int64) ([]json.RawMessage, error) { + var out []json.RawMessage + for from < to { + batch, err := a.rooms.StreamEvents(ctx, from, to, transactionSize) + if err != nil { + return nil, err + } + if len(batch) == 0 { + break + } + events, err := a.interesting(ctx, svc, batch) + if err != nil { + return nil, err + } + out = append(out, events...) + from = batch[len(batch)-1].Position + } + if out == nil { + out = []json.RawMessage{} + } + return out, nil +} + +// interesting keeps the events of a batch that concern a service, encoded +// for a transaction. +// +// An event concerns a service when its sender or (for a membership) its +// target is one of the service's users, when its room is in the service's +// room namespaces, or when its room has one of the service's users in it +// or one of the service's aliases pointing at it. The last two are the +// rules that make a bridge see the conversations its puppets are in, and +// they are read once per room per batch, as the room is now: an event +// read late, from a backlog, is judged by who is in the room when it is +// read rather than when it happened. +func (a *API) interesting(ctx context.Context, svc *Service, batch []roomserver.StreamEvent) ([]json.RawMessage, error) { + rooms := map[string]bool{} + var out []json.RawMessage + for _, se := range batch { + e := se.Event + keep := svc.InterestedInUser(e.Sender) || svc.InterestedInRoom(e.RoomID) + if !keep && e.Type == event.TypeMember && e.StateKey != nil { + keep = svc.InterestedInUser(*e.StateKey) + } + if !keep { + roomInterest, seen := rooms[e.RoomID] + if !seen { + var err error + if roomInterest, err = a.roomConcerns(ctx, svc, e.RoomID); err != nil { + return nil, err + } + rooms[e.RoomID] = roomInterest + } + keep = roomInterest + } + if !keep { + continue + } + raw, err := json.Marshal(e) + if err != nil { + return nil, err + } + out = append(out, raw) + } + return out, nil +} + +// roomConcerns reports whether a room has one of a service's users in it +// or one of its aliases pointing at it. +func (a *API) roomConcerns(ctx context.Context, svc *Service, roomID string) (bool, error) { + members, err := a.rooms.JoinedUserIDs(ctx, roomID) + if err != nil && !errors.Is(err, storage.ErrNoSuchRoom) { + return false, err + } + for _, m := range members { + if svc.OwnsUser(m) { + return true, nil + } + } + if len(svc.Namespaces.Aliases) == 0 { + return false, nil + } + aliases, err := a.rooms.LocalAliases(ctx, roomID) + if err != nil && !errors.Is(err, storage.ErrNoSuchRoom) { + return false, err + } + for _, alias := range aliases { + if svc.OwnsAlias(alias) { + return true, nil + } + } + return false, nil +} + +// deliver sends one transaction until the service accepts it. It returns +// false only when the worker is being stopped. +func (a *API) deliver(ctx context.Context, w *worker, txnID int64, events []json.RawMessage) bool { + backoff := minBackoff + for { + err := a.client.SendTransaction(ctx, w.svc, txnID, events) + now := a.now().UnixMilli() + if err == nil { + w.update(func(s *Status) { + s.Failures, s.LastError, s.RetryAtMS, s.LastSuccessMS = 0, "", 0, now + }) + return true + } + if ctx.Err() != nil { + return false + } + w.update(func(s *Status) { + s.Failures++ + s.LastError = err.Error() + s.LastFailureMS = now + s.RetryAtMS = now + backoff.Milliseconds() + }) + a.log.Warn("could not deliver a transaction to an application service", + "appservice", w.svc.ID, "txn_id", txnID, "events", len(events), "retry_in", backoff, "error", err) + select { + case <-ctx.Done(): + return false + case <-time.After(backoff): + case <-w.retry: + } + backoff = min(backoff*2, maxBackoff) + } +} + +// sleep waits for d or for ctx, reporting whether the wait ran its course. +func sleep(ctx context.Context, d time.Duration) bool { + select { + case <-ctx.Done(): + return false + case <-time.After(d): + return true + } +} diff --git a/server/internal/clientapi/account.go b/server/internal/clientapi/account.go new file mode 100644 index 0000000..0b81c07 --- /dev/null +++ b/server/internal/clientapi/account.go @@ -0,0 +1,111 @@ +package clientapi + +import ( + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// changePasswordRequest is the body of POST /account/password. +type changePasswordRequest struct { + NewPassword string `json:"new_password"` + + // LogoutDevices defaults to true, which is why it is a pointer: a + // client that says nothing gets every other session logged out. + LogoutDevices *bool `json:"logout_devices"` + + Auth *uiaRequest `json:"auth"` +} + +// handleChangePassword implements POST /_matrix/client/v3/account/password. +// +// The current password is asked for again first, through the same +// interactive authentication as deleting a device, and for the same +// reason: a stolen access token must not be enough to lock the owner out. +// The session the change is made from stays logged in; with +// logout_devices, every other one is logged out. +// +// An application service acting as one of its users skips the +// re-authentication: its users have no password to ask for, and the +// service's token is already the strongest credential involved. +func (c *ClientAPI) handleChangePassword(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + var req changePasswordRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + + sessionID := "" + if auth.AppService == "" { + flows := []uiaFlow{{Stages: []string{stagePassword}}} + var done bool + var err error + sessionID, done, err = c.beginOrAdvanceUIA(w, flows, req.Auth, + func(_ string, stage *uiaRequest) (*httputil.MatrixError, error) { + return c.passwordStage(r, auth, stage) + }) + if err != nil || !done { + return err + } + } + + if req.NewPassword == "" { + return httputil.MissingParam("new_password is required") + } + if len(req.NewPassword) > userapi.MaxPasswordLength { + return httputil.NewError(http.StatusBadRequest, httputil.ErrCodeWeakPassword, + "Password must be at most %d bytes", userapi.MaxPasswordLength) + } + logout := req.LogoutDevices == nil || *req.LogoutDevices + + if err := c.users.ChangePassword(r.Context(), auth.Localpart, req.NewPassword, logout, auth.DeviceID); err != nil { + return err + } + c.uia.remove(sessionID) + httputil.WriteJSON(w, http.StatusOK, struct{}{}) + return nil +} + +// deactivateRequest is the body of POST /account/deactivate. +type deactivateRequest struct { + // Erase asks for the account's profile to be forgotten as well. + Erase bool `json:"erase"` + + Auth *uiaRequest `json:"auth"` +} + +// deactivateResponse is its answer. Warp keeps no third-party identifiers +// and talks to no identity server, so there was nothing to unbind. +type deactivateResponse struct { + IDServerUnbindResult string `json:"id_server_unbind_result"` +} + +// handleDeactivate implements POST /_matrix/client/v3/account/deactivate: +// the user closing their own account, behind a re-entered password. +func (c *ClientAPI) handleDeactivate(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + var req deactivateRequest + if err := httputil.DecodeJSON(w, r, &req); err != nil { + return err + } + + sessionID := "" + if auth.AppService == "" { + flows := []uiaFlow{{Stages: []string{stagePassword}}} + var done bool + var err error + sessionID, done, err = c.beginOrAdvanceUIA(w, flows, req.Auth, + func(_ string, stage *uiaRequest) (*httputil.MatrixError, error) { + return c.passwordStage(r, auth, stage) + }) + if err != nil || !done { + return err + } + } + + if err := c.users.DeactivateAccount(r.Context(), auth.Localpart, req.Erase); err != nil { + return err + } + c.uia.remove(sessionID) + httputil.WriteJSON(w, http.StatusOK, deactivateResponse{IDServerUnbindResult: "no-support"}) + return nil +} diff --git a/server/internal/clientapi/appservice.go b/server/internal/clientapi/appservice.go new file mode 100644 index 0000000..1e3ac03 --- /dev/null +++ b/server/internal/clientapi/appservice.go @@ -0,0 +1,161 @@ +package clientapi + +import ( + "context" + "errors" + "net/http" + + "github.com/dariuszjaniak/warp/server/internal/appservice" + "github.com/dariuszjaniak/warp/server/internal/httputil" + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// The Client-Server half of the Application Service API: a service's +// as_token as an access token, acting as one of its users with +// ?user_id=, and the namespaces other users are kept out of. + +// authenticateAppService turns a request carrying a service's as_token +// into an identity: the service's sender, or the user ?user_id= names. +// +// The user has to be the sender or in one of the service's user +// namespaces — exclusive or not, the spec does not distinguish — and has +// to exist: a service registers its users first, and acting as a user +// nobody created would hand out an identity with no account behind it. +func (c *ClientAPI) authenticateAppService(r *http.Request, svc *appservice.Service) (userapi.Authenticated, error) { + userID := r.URL.Query().Get("user_id") + if userID == "" { + userID = svc.SenderUserID() + } + localpart, err := c.users.ParseLocalUserID(userID) + switch { + case errors.Is(err, userapi.ErrForeignServer): + return userapi.Authenticated{}, httputil.Forbidden("An application service cannot act as a user of another server") + case err != nil: + return userapi.Authenticated{}, httputil.InvalidParam("%s is not a valid user ID", userID) + } + canonical := c.users.UserID(localpart) + if !svc.OwnsUser(canonical) { + return userapi.Authenticated{}, httputil.Forbidden("%s is not in the namespace of application service %s", canonical, svc.ID) + } + + account, err := c.users.Account(r.Context(), localpart) + switch { + case errors.Is(err, userapi.ErrNoSuchUser): + return userapi.Authenticated{}, httputil.Forbidden("Application service %s has not registered %s", svc.ID, canonical) + case err != nil: + return userapi.Authenticated{}, err + case account.Deactivated: + return userapi.Authenticated{}, httputil.UserDeactivated() + } + return userapi.Authenticated{ + UserID: canonical, + Localpart: localpart, + Admin: account.Admin, + AppService: svc.ID, + }, nil +} + +// rateLimitExempt lets an application service's own requests past the +// rate limiter, and the requests of the users it acts as when its +// registration says rate_limited: false. +// +// A bridge puppeting a thousand users from one address would otherwise +// share one bucket among all of them. The check is a map lookup on the +// token, so it is cheap enough to run ahead of the limiter. +func (c *ClientAPI) rateLimitExempt(r *http.Request) bool { + svc := c.appservices.Registry().ByToken(accessTokenFrom(r)) + if svc == nil { + return false + } + userID := r.URL.Query().Get("user_id") + return userID == "" || userID == svc.SenderUserID() || !svc.RateLimitedUsers() +} + +// checkExclusiveUser refuses a user ID in a namespace an application +// service has claimed, unless the request comes from that service. +func (c *ClientAPI) checkExclusiveUser(userID, asking string) error { + if owner := c.appservices.Registry().ExclusiveUserOwner(userID); owner != nil && owner.ID != asking { + return httputil.Exclusive("%s is reserved by an application service", userID) + } + return nil +} + +// checkExclusiveAlias is checkExclusiveUser for a room alias. +func (c *ClientAPI) checkExclusiveAlias(alias string, auth userapi.Authenticated) error { + if owner := c.appservices.Registry().ExclusiveAliasOwner(alias); owner != nil && owner.ID != auth.AppService { + return httputil.Exclusive("%s is reserved by an application service", alias) + } + return nil +} + +// resolveAlias resolves an alias, asking the application services whose +// namespace it is in when this server does not know it — which is how a +// bridge creates a portal room the first time somebody looks for it. +func (c *ClientAPI) resolveAlias(ctx context.Context, alias string) (roomserver.ResolvedAlias, error) { + resolved, err := c.rooms.ResolveAlias(ctx, alias) + if !errors.Is(err, roomserver.ErrNoSuchAlias) { + return resolved, err + } + if !c.appservices.QueryAlias(ctx, alias) { + return resolved, err + } + return c.rooms.ResolveAlias(ctx, alias) +} + +// ensureAppServiceUser gives the application services a chance to create +// a local user that does not exist, before something that needs the user +// gives up on it. It reports whether the user exists afterwards. +func (c *ClientAPI) ensureAppServiceUser(ctx context.Context, localpart string) (bool, error) { + exists, err := c.users.AccountExists(ctx, localpart) + if err != nil || exists { + return exists, err + } + if !c.appservices.QueryUser(ctx, c.users.UserID(localpart)) { + return false, nil + } + return c.users.AccountExists(ctx, localpart) +} + +// pingRequest is the body of the ping endpoint. +type pingRequest struct { + TransactionID string `json:"transaction_id"` +} + +// handleAppServicePing implements +// POST /_matrix/client/v1/appservice/{appserviceId}/ping: a service asking +// this server to ping it back, to check the two can reach each other. +func (c *ClientAPI) handleAppServicePing(w http.ResponseWriter, r *http.Request, auth userapi.Authenticated) error { + svc := c.appservices.Registry().ByID(r.PathValue("appserviceID")) + if svc == nil || auth.AppService != svc.ID { + return httputil.Forbidden("Only the application service itself may ask for its ping") + } + var req pingRequest + if _, err := httputil.DecodeOptionalJSON(w, r, &req); err != nil { + return err + } + + took, err := c.appservices.Ping(r.Context(), svc, req.TransactionID) + var status *appservice.BadStatusError + switch { + case err == nil: + case errors.Is(err, appservice.ErrNoURL): + return httputil.NewError(http.StatusBadRequest, httputil.ErrCodeURLNotSet, + "Application service %s has no URL", svc.ID) + case errors.As(err, &status): + e := httputil.NewError(http.StatusBadGateway, httputil.ErrCodeBadStatus, + "The application service answered the ping with %d", status.Status) + e.RemoteStatus, e.RemoteBody = status.Status, status.Body + return e + case errors.Is(err, appservice.ErrConnectionTimeout): + return httputil.NewError(http.StatusGatewayTimeout, httputil.ErrCodeConnectionTimeout, + "The application service did not answer the ping in time") + case errors.Is(err, appservice.ErrConnectionFailed): + return httputil.NewError(http.StatusBadGateway, httputil.ErrCodeConnectionFailed, + "Could not connect to the application service") + default: + return err + } + httputil.WriteJSON(w, http.StatusOK, map[string]int64{"duration_ms": took.Milliseconds()}) + return nil +} diff --git a/server/internal/clientapi/auth.go b/server/internal/clientapi/auth.go index fdabe01..12a00c7 100644 --- a/server/internal/clientapi/auth.go +++ b/server/internal/clientapi/auth.go @@ -22,7 +22,10 @@ func (c *ClientAPI) withAuth(h AuthedHandler) httputil.Handler { if err != nil { return err } - c.users.RecordActivity(r.Context(), auth, c.clientIP(r), r.UserAgent()) + // An application service has no device to record a visit on. + if auth.AppService == "" { + c.users.RecordActivity(r.Context(), auth, c.clientIP(r), r.UserAgent()) + } return h(w, r, auth) } } @@ -33,6 +36,9 @@ func (c *ClientAPI) authenticate(r *http.Request) (userapi.Authenticated, error) if token == "" { return userapi.Authenticated{}, httputil.MissingToken() } + if svc := c.appservices.Registry().ByToken(token); svc != nil { + return c.authenticateAppService(r, svc) + } auth, err := c.users.Authenticate(r.Context(), token) switch { diff --git a/server/internal/clientapi/capabilities.go b/server/internal/clientapi/capabilities.go index 60f7a97..08fb717 100644 --- a/server/internal/clientapi/capabilities.go +++ b/server/internal/clientapi/capabilities.go @@ -58,9 +58,9 @@ type capabilities struct { // you are, and the spec makes it authenticated for that reason. // // Every flag here is the truth about this build, not an aspiration. -// Password changes and third-party identifiers arrive with the rest of -// account management in Phase 9, so they report false and clients hide the -// buttons rather than offering something that 404s. +// Third-party identifiers need an identity server, which Warp does not +// talk to, so they report false and clients hide the buttons rather than +// offering something that 404s. func (c *ClientAPI) handleCapabilities(w http.ResponseWriter, _ *http.Request, _ userapi.Authenticated) error { httputil.WriteJSON(w, http.StatusOK, capabilitiesResponse{ Capabilities: capabilities{ @@ -68,7 +68,7 @@ func (c *ClientAPI) handleCapabilities(w http.ResponseWriter, _ *http.Request, _ Default: DefaultRoomVersion, Available: availableRoomVersions(), }, - ChangePassword: booleanCapability{Enabled: false}, + ChangePassword: booleanCapability{Enabled: true}, SetDisplayName: booleanCapability{Enabled: true}, SetAvatarURL: booleanCapability{Enabled: true}, ThreePIDChanges: booleanCapability{Enabled: false}, diff --git a/server/internal/clientapi/clientapi.go b/server/internal/clientapi/clientapi.go index 0915645..3c74099 100644 --- a/server/internal/clientapi/clientapi.go +++ b/server/internal/clientapi/clientapi.go @@ -17,6 +17,7 @@ import ( "log/slog" "time" + "github.com/dariuszjaniak/warp/server/internal/appservice" "github.com/dariuszjaniak/warp/server/internal/config" "github.com/dariuszjaniak/warp/server/internal/federation" "github.com/dariuszjaniak/warp/server/internal/httputil" @@ -44,7 +45,8 @@ type ClientAPI struct { settings *settings.Service federation *federation.Gate remoteProfile RemoteProfiles - registration config.Registration + regTokens *userapi.RegistrationTokens + appservices *appservice.API log *slog.Logger now func() time.Time @@ -127,6 +129,15 @@ type Options struct { // Registration is the file configuration of the registration flow. Registration config.Registration + // RegistrationTokens answers the m.login.registration_token stage. + // Nil builds one from Registration's static tokens alone. + RegistrationTokens *userapi.RegistrationTokens + + // AppServices is the application service component: who may act as + // whom, the exclusive namespaces, and the queries. Nil means no + // application services are registered. + AppServices *appservice.API + // RateLimit configures the per-endpoint buckets. RateLimit config.RateLimit @@ -159,6 +170,20 @@ func New(opts Options) (*ClientAPI, error) { if opts.Now == nil { opts.Now = time.Now } + if opts.RegistrationTokens == nil { + now := opts.Now + opts.RegistrationTokens = userapi.NewRegistrationTokens(userapi.RegistrationTokensOptions{ + Static: opts.Registration.Tokens, + Required: opts.Registration.RequireToken, + NowMS: func() int64 { return now().UnixMilli() }, + }) + } + if opts.AppServices == nil { + var err error + if opts.AppServices, err = appservice.New(appservice.Options{Logger: opts.Logger}); err != nil { + return nil, err + } + } c := &ClientAPI{ serverName: opts.ServerName, @@ -173,7 +198,8 @@ func New(opts Options) (*ClientAPI, error) { settings: opts.Settings, federation: opts.Federation, remoteProfile: opts.RemoteProfiles, - registration: opts.Registration, + regTokens: opts.RegistrationTokens, + appservices: opts.AppServices, log: opts.Logger, now: opts.Now, uia: newUIAStore(uiaSessionTTL, opts.Now), @@ -232,6 +258,13 @@ func (c *ClientAPI) RegisterRoutes(r *httputil.Router) { // Account. c.authed(r, "GET "+v3+"/account/whoami", c.limitDefault, c.handleWhoami) + c.authed(r, "POST "+v3+"/account/password", c.limitLogin, c.handleChangePassword) + c.authed(r, "POST "+v3+"/account/deactivate", c.limitLogin, c.handleDeactivate) + + // Application services. The ping is the service checking that this + // server can reach it, so it is authenticated with the service's own + // token. + c.authed(r, "POST "+v1+"/appservice/{appserviceID}/ping", c.limitDefault, c.handleAppServicePing) c.authed(r, "GET "+v3+"/capabilities", c.limitDefault, c.handleCapabilities) // Devices. @@ -372,7 +405,7 @@ func (c *ClientAPI) registerRoomRoutes(r *httputil.Router) { // public registers an endpoint that needs no access token. func (c *ClientAPI) public(r *httputil.Router, pattern string, limiter *ratelimit.Limiter, h httputil.Handler) { - r.Handle(pattern, limiter.Middleware(c.clientIP)(httputil.Wrap(h))) + r.Handle(pattern, limiter.MiddlewareUnless(c.clientIP, c.rateLimitExempt)(httputil.Wrap(h))) } // authed registers an endpoint that needs one. @@ -381,5 +414,5 @@ func (c *ClientAPI) public(r *httputil.Router, pattern string, limiter *ratelimi // it runs before the token is resolved, and a limiter that had to hit the // database to find its own key would be a denial of service of its own. func (c *ClientAPI) authed(r *httputil.Router, pattern string, limiter *ratelimit.Limiter, h AuthedHandler) { - r.Handle(pattern, limiter.Middleware(c.clientIP)(httputil.Wrap(c.withAuth(h)))) + r.Handle(pattern, limiter.MiddlewareUnless(c.clientIP, c.rateLimitExempt)(httputil.Wrap(c.withAuth(h)))) } diff --git a/server/internal/clientapi/directory.go b/server/internal/clientapi/directory.go index e3d7909..a8d40b5 100644 --- a/server/internal/clientapi/directory.go +++ b/server/internal/clientapi/directory.go @@ -25,7 +25,7 @@ func (c *ClientAPI) handleGetAlias(w http.ResponseWriter, r *http.Request) error if err != nil { return err } - resolved, err := c.rooms.ResolveAlias(r.Context(), alias) + resolved, err := c.resolveAlias(r.Context(), alias) if err != nil { return roomError(err) } @@ -54,6 +54,9 @@ func (c *ClientAPI) handlePutAlias(w http.ResponseWriter, r *http.Request, auth if req.RoomID == "" { return httputil.MissingParam("A room ID is required") } + if err := c.checkExclusiveAlias(alias, auth); err != nil { + return err + } if err := c.rooms.CreateAlias(r.Context(), alias, req.RoomID, auth.UserID); err != nil { return roomError(err) diff --git a/server/internal/clientapi/login.go b/server/internal/clientapi/login.go index 413e6dd..1f54f19 100644 --- a/server/internal/clientapi/login.go +++ b/server/internal/clientapi/login.go @@ -77,9 +77,11 @@ type wellKnownBaseURL struct { // handleLoginFlows implements GET /_matrix/client/v3/login. func (c *ClientAPI) handleLoginFlows(w http.ResponseWriter, _ *http.Request) error { - httputil.WriteJSON(w, http.StatusOK, loginFlowsResponse{ - Flows: []loginFlow{{Type: stagePassword, Stages: []string{stagePassword}}}, - }) + flows := []loginFlow{{Type: stagePassword, Stages: []string{stagePassword}}} + if !c.appservices.Registry().Empty() { + flows = append(flows, loginFlow{Type: loginAppService, Stages: []string{loginAppService}}) + } + httputil.WriteJSON(w, http.StatusOK, loginFlowsResponse{Flows: flows}) return nil } @@ -94,7 +96,7 @@ func (c *ClientAPI) handleLogin(w http.ResponseWriter, r *http.Request) error { if err := httputil.DecodeJSON(w, r, &req); err != nil { return err } - if req.Type != stagePassword { + if req.Type != stagePassword && req.Type != loginAppService { return httputil.Unknown(http.StatusBadRequest, "Login type %q is not supported", req.Type) } @@ -119,16 +121,36 @@ func (c *ClientAPI) handleLogin(w http.ResponseWriter, r *http.Request) error { return httputil.Forbidden("Invalid username or password") } - switch err := c.users.CheckPassword(r.Context(), localpart, req.Password); { - case err == nil: - case errors.Is(err, userapi.ErrWrongPassword), errors.Is(err, userapi.ErrNoPassword): - return httputil.Forbidden("Invalid username or password") - default: - return err + if req.Type == loginAppService { + // The credential is the application service's token, and the + // user has to be one of its own. + svc := c.appservices.Registry().ByToken(accessTokenFrom(r)) + switch { + case svc == nil && accessTokenFrom(r) == "": + return httputil.MissingToken() + case svc == nil: + return httputil.UnknownToken(false) + case !svc.OwnsUser(c.users.UserID(localpart)): + return httputil.Forbidden("%s is not in the namespace of application service %s", + c.users.UserID(localpart), svc.ID) + } + } else { + switch err := c.users.CheckPassword(r.Context(), localpart, req.Password); { + case err == nil: + case errors.Is(err, userapi.ErrWrongPassword), errors.Is(err, userapi.ErrNoPassword): + return httputil.Forbidden("Invalid username or password") + default: + return err + } } account, err := c.users.Account(r.Context(), localpart) - if err != nil { + switch { + case errors.Is(err, userapi.ErrNoSuchUser): + // Only reachable for an application service login: a password + // check has already failed for a user who does not exist. + return httputil.Forbidden("%s does not exist", c.users.UserID(localpart)) + case err != nil: return err } if account.Deactivated { diff --git a/server/internal/clientapi/membership.go b/server/internal/clientapi/membership.go index 0837e77..1b8059b 100644 --- a/server/internal/clientapi/membership.go +++ b/server/internal/clientapi/membership.go @@ -45,7 +45,7 @@ func (c *ClientAPI) handleJoinRoom(w http.ResponseWriter, r *http.Request, auth return c.join(w, r, auth, target) case strings.HasPrefix(target, "#"): - resolved, err := c.rooms.ResolveAlias(r.Context(), target) + resolved, err := c.resolveAlias(r.Context(), target) if err != nil { return roomError(err) } @@ -129,6 +129,15 @@ func (c *ClientAPI) handleInvite(w http.ResponseWriter, r *http.Request, auth us "Third-party invites are not supported by this server") } + // A user of this server that does not exist may be one an + // application service creates on demand: a bridge's puppet for + // somebody on the other network is invited before it is registered. + if localpart, err := c.users.ParseLocalUserID(req.UserID); err == nil { + if _, err := c.ensureAppServiceUser(r.Context(), localpart); err != nil { + return err + } + } + if err := c.rooms.InviteUser(r.Context(), roomserver.InviteRequest{ RoomID: roomID, Sender: auth.UserID, diff --git a/server/internal/clientapi/profile.go b/server/internal/clientapi/profile.go index 5c7ddb1..14ac00a 100644 --- a/server/internal/clientapi/profile.go +++ b/server/internal/clientapi/profile.go @@ -88,6 +88,9 @@ func (c *ClientAPI) lookupProfile(r *http.Request) (storage.Account, error) { return storage.Account{}, httputil.InvalidParam("%s is not a valid user ID", userID) } + if _, err := c.ensureAppServiceUser(r.Context(), localpart); err != nil { + return storage.Account{}, err + } account, err := c.users.Account(r.Context(), localpart) switch { case errors.Is(err, userapi.ErrNoSuchUser): diff --git a/server/internal/clientapi/register.go b/server/internal/clientapi/register.go index f1abdd2..c7a65f5 100644 --- a/server/internal/clientapi/register.go +++ b/server/internal/clientapi/register.go @@ -11,6 +11,10 @@ import ( // registerRequest is the body of POST /register. type registerRequest struct { + // Type is m.login.application_service for an application service + // registering one of its users, and otherwise absent. + Type string `json:"type"` + Username string `json:"username"` Password string `json:"password"` @@ -65,6 +69,12 @@ func (c *ClientAPI) handleRegister(w http.ResponseWriter, r *http.Request) error return httputil.InvalidParam("Unknown registration kind %q", kind) } + // An application service registering one of its users is not a + // sign-up: it needs neither the registration switch nor the flow. + if req.Type == loginAppService || (req.Auth != nil && req.Auth.Type == loginAppService) { + return c.registerAppServiceUser(w, r, req) + } + if !c.settings.RegistrationEnabled() { return httputil.Forbidden("Registration is disabled on this server") } @@ -81,10 +91,16 @@ func (c *ClientAPI) handleRegister(w http.ResponseWriter, r *http.Request) error if taken { return httputil.UserInUse() } + if err := c.checkExclusiveUser(c.users.UserID(localpart), ""); err != nil { + return err + } } flows := c.registrationFlows() - sessionID, done, err := c.beginOrAdvanceUIA(w, flows, req.Auth, c.registrationStage) + sessionID, done, err := c.beginOrAdvanceUIA(w, flows, req.Auth, + func(stage string, auth *uiaRequest) (*httputil.MatrixError, error) { + return c.registrationStage(r, stage, auth) + }) if err != nil || !done { return err } @@ -111,10 +127,30 @@ func (c *ClientAPI) handleRegister(w http.ResponseWriter, r *http.Request) error } } + // The token is spent now that an account is about to exist, not when + // the stage was passed: a sign-up abandoned halfway costs no use. A + // token that ran out in between is refused here. + release := func() {} + if c.regTokens.Required() { + var token string + if session := c.uia.get(sessionID); session != nil { + token = session.registrationToken + } + release, err = c.regTokens.Spend(r.Context(), token) + switch { + case errors.Is(err, userapi.ErrNoSuchRegistrationToken): + c.uia.remove(sessionID) + return httputil.Forbidden("The registration token is no longer valid") + case err != nil: + return err + } + } + if _, err := c.users.CreateAccount(r.Context(), userapi.NewAccountRequest{ Localpart: localpart, Password: req.Password, }); err != nil { + release() switch { case errors.Is(err, userapi.ErrUserExists): // Lost the race against another registration of the same name @@ -167,14 +203,14 @@ func (c *ClientAPI) handleRegister(w http.ResponseWriter, r *http.Request) error // interactive, and a server with no real requirements still has to offer // something for the client to complete. func (c *ClientAPI) registrationFlows() []uiaFlow { - if c.registration.RequiresToken() { + if c.regTokens.Required() { return []uiaFlow{{Stages: []string{stageRegistrationToken, stageDummy}}} } return []uiaFlow{{Stages: []string{stageDummy}}} } // registrationStage checks one stage of the registration flow. -func (c *ClientAPI) registrationStage(stage string, auth *uiaRequest) (*httputil.MatrixError, error) { +func (c *ClientAPI) registrationStage(r *http.Request, stage string, auth *uiaRequest) (*httputil.MatrixError, error) { switch stage { case stageDummy: return nil, nil @@ -182,9 +218,14 @@ func (c *ClientAPI) registrationStage(stage string, auth *uiaRequest) (*httputil if auth.Token == "" { return httputil.MissingParam("A registration token is required"), nil } - if !c.registration.TokenValid(auth.Token) { + valid, err := c.regTokens.Valid(r.Context(), auth.Token) + if err != nil { + return nil, err + } + if !valid { return httputil.Forbidden("Invalid registration token"), nil } + c.uia.setRegistrationToken(auth.Session, auth.Token) return nil, nil default: // beginOrAdvanceUIA only calls this for stages it offered, so @@ -212,6 +253,9 @@ func (c *ClientAPI) handleRegisterAvailable(w http.ResponseWriter, r *http.Reque if taken { return httputil.UserInUse() } + if err := c.checkExclusiveUser(c.users.UserID(localpart), ""); err != nil { + return err + } httputil.WriteJSON(w, http.StatusOK, map[string]bool{"available": true}) return nil @@ -226,7 +270,78 @@ func (c *ClientAPI) handleRegistrationTokenValidity(w http.ResponseWriter, r *ht if token == "" { return httputil.MissingParam("The token query parameter is required") } - httputil.WriteJSON(w, http.StatusOK, map[string]bool{"valid": c.registration.TokenValid(token)}) + valid, err := c.regTokens.Valid(r.Context(), token) + if err != nil { + return err + } + httputil.WriteJSON(w, http.StatusOK, map[string]bool{"valid": valid}) + return nil +} + +// registerAppServiceUser registers a user for the application service +// whose token the request carries. +// +// The user has to be in the service's user namespace. There is no +// password: an application service's users are only ever used through +// the service, which needs none, or through an m.login.application_service +// login, which needs the service's token rather than a password. +func (c *ClientAPI) registerAppServiceUser(w http.ResponseWriter, r *http.Request, req registerRequest) error { + token := accessTokenFrom(r) + if token == "" { + return httputil.MissingToken() + } + svc := c.appservices.Registry().ByToken(token) + if svc == nil { + return httputil.UnknownToken(false) + } + if req.Username == "" { + return httputil.MissingParam("An application service has to name the user it registers") + } + localpart := userapi.NormaliseLocalpart(req.Username) + if err := userapi.ValidateLocalpart(localpart, c.serverName); err != nil { + return httputil.InvalidUsername("%s", err.Error()) + } + userID := c.users.UserID(localpart) + if !svc.OwnsUser(userID) { + return httputil.Exclusive("%s is not in the namespace of application service %s", userID, svc.ID) + } + if req.DeviceID != "" { + if err := userapi.ValidateDeviceID(req.DeviceID); err != nil { + return httputil.InvalidParam("%s", err.Error()) + } + } + + _, err := c.users.CreateAccount(r.Context(), userapi.NewAccountRequest{Localpart: localpart}) + switch { + case errors.Is(err, userapi.ErrUserExists): + return httputil.UserInUse() + case errors.Is(err, userapi.ErrInvalidLocalpart): + return httputil.InvalidUsername("%s", err.Error()) + case err != nil: + return err + } + + resp := registerResponse{UserID: userID, HomeServer: c.serverName} + if req.InhibitLogin { + httputil.WriteJSON(w, http.StatusOK, resp) + return nil + } + session, err := c.users.CreateSession(r.Context(), userapi.SessionRequest{ + Localpart: localpart, + DeviceID: req.DeviceID, + DisplayName: req.InitialDeviceDisplayName, + WithRefreshToken: req.RefreshToken, + IP: c.clientIP(r), + UserAgent: r.UserAgent(), + }) + if err != nil { + return err + } + resp.AccessToken = session.AccessToken + resp.DeviceID = session.DeviceID + resp.RefreshToken = session.RefreshToken + resp.ExpiresInMS = session.ExpiresInMS + httputil.WriteJSON(w, http.StatusOK, resp) return nil } diff --git a/server/internal/clientapi/rooms.go b/server/internal/clientapi/rooms.go index a9f897e..26c6b2f 100644 --- a/server/internal/clientapi/rooms.go +++ b/server/internal/clientapi/rooms.go @@ -100,6 +100,12 @@ func (c *ClientAPI) handleCreateRoom(w http.ResponseWriter, r *http.Request, aut "Third-party invites are not supported by this server") } + if req.RoomAliasName != "" { + if err := c.checkExclusiveAlias("#"+req.RoomAliasName+":"+c.serverName, auth); err != nil { + return err + } + } + resp, err := c.rooms.CreateRoom(r.Context(), roomserver.CreateRoomRequest{ Creator: auth.UserID, Version: req.RoomVersion, diff --git a/server/internal/clientapi/uia.go b/server/internal/clientapi/uia.go index f8be648..8980bf9 100644 --- a/server/internal/clientapi/uia.go +++ b/server/internal/clientapi/uia.go @@ -19,6 +19,11 @@ const ( stageDummy = "m.login.dummy" stagePassword = "m.login.password" stageRegistrationToken = "m.login.registration_token" + + // loginAppService is not an interactive stage but a login and + // registration type: an application service's own token is the + // credential. + loginAppService = "m.login.application_service" ) // uiaFlow is one route through the stages. A client may satisfy any flow. @@ -54,6 +59,11 @@ type uiaSession struct { id string completed []string expiresAt time.Time + + // registrationToken is the token a registration's + // m.login.registration_token stage was passed with. It is spent when + // the account is created, so it has to be remembered until then. + registrationToken string } // uiaStore holds the in-flight authentication sessions. @@ -115,7 +125,22 @@ func (s *uiaStore) get(id string) *uiaSession { } // Each copy handed out is independent, so a caller cannot mutate the // stored progress by accident. - return &uiaSession{id: session.id, completed: slices.Clone(session.completed), expiresAt: session.expiresAt} + return &uiaSession{ + id: session.id, + completed: slices.Clone(session.completed), + expiresAt: session.expiresAt, + registrationToken: session.registrationToken, + } +} + +// setRegistrationToken remembers the token a session's registration token +// stage was passed with. +func (s *uiaStore) setRegistrationToken(id, token string) { + s.mu.Lock() + defer s.mu.Unlock() + if session, ok := s.sessions[id]; ok { + session.registrationToken = token + } } // complete records a finished stage and returns the session's new progress. diff --git a/server/internal/config/config.go b/server/internal/config/config.go index 18310e8..40c54d5 100644 --- a/server/internal/config/config.go +++ b/server/internal/config/config.go @@ -18,17 +18,19 @@ type Config struct { // Version is the config schema version; bumped only on breaking changes. Version int `yaml:"version"` - Global Global `yaml:"global"` - Listen Listen `yaml:"listen"` - Database Database `yaml:"database"` - ClientAPI ClientAPI `yaml:"client_api"` - Sync Sync `yaml:"sync"` - Federation Federation `yaml:"federation"` - Media Media `yaml:"media"` - Push Push `yaml:"push"` - Registration Registration `yaml:"registration"` - Logging Logging `yaml:"logging"` - Metrics Metrics `yaml:"metrics"` + Global Global `yaml:"global"` + Listen Listen `yaml:"listen"` + Database Database `yaml:"database"` + ClientAPI ClientAPI `yaml:"client_api"` + Sync Sync `yaml:"sync"` + Federation Federation `yaml:"federation"` + Media Media `yaml:"media"` + Push Push `yaml:"push"` + Registration Registration `yaml:"registration"` + AppService AppService `yaml:"appservice"` + ServerNotices ServerNotices `yaml:"server_notices"` + Logging Logging `yaml:"logging"` + Metrics Metrics `yaml:"metrics"` // path is the file this config was read from, if any. path string @@ -248,6 +250,10 @@ type Media struct { // server. It is what /media/config reports as m.upload.size. MaxUploadBytes int64 `yaml:"max_upload_bytes" env:"WARP_MEDIA_MAX_UPLOAD_BYTES"` + // MaxUserStorageBytes caps how much one user may have uploaded in all; + // 0 is no limit. The admin API can give a user a quota of their own. + MaxUserStorageBytes int64 `yaml:"max_user_storage_bytes" env:"WARP_MEDIA_MAX_USER_STORAGE_BYTES"` + // MaxThumbnailPixels is the largest image, in pixels, that will be // decoded to make a thumbnail. Decoding allocates four bytes a pixel // whatever the file size, and a 100 KiB PNG can claim to be 30000 × @@ -316,15 +322,19 @@ type Registration struct { // Tokens, when non-empty, adds m.login.registration_token as a required // stage to the registration flow and lists the tokens that satisfy it. // - // Phase 1 keeps the tokens in the config file: per-token use counts and - // expiry belong to the admin token management of Phase 9, and a table - // nothing can write to yet would be dead weight. + // These never run out. Tokens with a use count or an expiry are made + // through the admin API instead, and both kinds are accepted. Tokens []string `yaml:"tokens" env:"WARP_REGISTRATION_TOKENS"` + + // RequireToken adds the m.login.registration_token stage even when + // Tokens is empty, for a server whose tokens are all made through the + // admin API, with use counts and expiry. + RequireToken bool `yaml:"require_token" env:"WARP_REGISTRATION_REQUIRE_TOKEN"` } // RequiresToken reports whether the registration flow includes the // m.login.registration_token stage. -func (r Registration) RequiresToken() bool { return len(r.Tokens) > 0 } +func (r Registration) RequiresToken() bool { return r.RequireToken || len(r.Tokens) > 0 } // TokenValid reports whether token is one of the configured registration // tokens. Every candidate is compared in constant time, and the loop does @@ -338,6 +348,36 @@ func (r Registration) TokenValid(token string) bool { return found == 1 } +// AppService configures the application services: bridges and bots that +// own a namespace of users and aliases. +type AppService struct { + // ConfigFiles are the services' registration files, the YAML document + // every bridge writes. A file that does not parse stops the server + // from starting, as a bad config file does. + ConfigFiles []string `yaml:"config_files" env:"WARP_APPSERVICE_CONFIG_FILES"` + + // Timeout bounds one request to a service: a transaction, a query or + // a ping. A transaction that times out is retried. + Timeout time.Duration `yaml:"timeout" env:"WARP_APPSERVICE_TIMEOUT"` +} + +// ServerNotices configures messages from the server to its users, sent +// through the admin API. +type ServerNotices struct { + // Enabled creates the notices user at startup and turns the admin + // endpoint on. Off by default: the user is created with the localpart + // below, and a server with a real user of that name must pick another + // before turning this on. + Enabled bool `yaml:"enabled" env:"WARP_SERVER_NOTICES_ENABLED"` + + // Localpart and DisplayName are the user notices come from. + Localpart string `yaml:"localpart" env:"WARP_SERVER_NOTICES_LOCALPART"` + DisplayName string `yaml:"display_name" env:"WARP_SERVER_NOTICES_DISPLAY_NAME"` + + // RoomName names the room each user gets their notices in. + RoomName string `yaml:"room_name" env:"WARP_SERVER_NOTICES_ROOM_NAME"` +} + // Logging configures slog. type Logging struct { Level string `yaml:"level" env:"WARP_LOG_LEVEL"` @@ -430,6 +470,15 @@ func Default() *Config { Enabled: false, AllowRuntimeToggle: true, }, + AppService: AppService{ + Timeout: 30 * time.Second, + }, + ServerNotices: ServerNotices{ + Enabled: false, + Localpart: "server-notices", + DisplayName: "Server Notices", + RoomName: "Server Notices", + }, Logging: Logging{Level: "info", Format: "text"}, Metrics: Metrics{Enabled: true, Path: "/metrics"}, } @@ -563,6 +612,9 @@ func (c *Config) Validate() error { if c.Media.MaxUploadBytes < 1 { bad("media.max_upload_bytes: must be positive") } + if c.Media.MaxUserStorageBytes < 0 { + bad("media.max_user_storage_bytes: must not be negative") + } if c.Media.MaxThumbnailPixels < 1 { bad("media.max_thumbnail_pixels: must be positive") } @@ -594,6 +646,23 @@ func (c *Config) Validate() error { } } + for i, path := range c.AppService.ConfigFiles { + if strings.TrimSpace(path) == "" { + bad("appservice.config_files[%d]: must not be empty", i) + } + } + if c.AppService.Timeout <= 0 { + bad("appservice.timeout: must be positive") + } + if c.ServerNotices.Enabled { + if c.ServerNotices.Localpart == "" { + bad("server_notices.localpart: must not be empty") + } + if c.ServerNotices.RoomName == "" { + bad("server_notices.room_name: must not be empty") + } + } + for i, s := range c.Federation.Allowlist { if err := validateServerName(s); err != nil { bad("federation.allowlist[%d]: %w", i, err) diff --git a/server/internal/federation/gate.go b/server/internal/federation/gate.go index 1fa6126..8cb6de6 100644 --- a/server/internal/federation/gate.go +++ b/server/internal/federation/gate.go @@ -10,7 +10,9 @@ package federation import ( + "maps" "net/http" + "slices" "strings" "sync" @@ -133,11 +135,12 @@ func toSet(names []string) map[string]struct{} { return out } +// setToSlice lists a set in order, so that the admin panel shows the +// same list the same way every time. func setToSlice(set map[string]struct{}) []string { - out := make([]string, 0, len(set)) - for name := range set { - out = append(out, name) - } + // Never nil: an empty list is [] in the admin API, not null. + out := slices.AppendSeq(make([]string, 0, len(set)), maps.Keys(set)) + slices.Sort(out) return out } diff --git a/server/internal/httputil/errors.go b/server/internal/httputil/errors.go index 0c40163..ff0345e 100644 --- a/server/internal/httputil/errors.go +++ b/server/internal/httputil/errors.go @@ -43,6 +43,12 @@ const ( // End-to-end encryption codes, added with the keyserver in Phase 7. ErrCodeInvalidSignature = "M_INVALID_SIGNATURE" ErrCodeWrongRoomKeysVersion = "M_WRONG_ROOM_KEYS_VERSION" + + // Application service and admin codes, added in Phase 9. + ErrCodeURLNotSet = "M_URL_NOT_SET" + ErrCodeConnectionFailed = "M_CONNECTION_FAILED" + ErrCodeConnectionTimeout = "M_CONNECTION_TIMEOUT" + ErrCodeBadStatus = "M_BAD_STATUS" ) // MatrixError is the standard Matrix error envelope. Every non-2xx response @@ -66,6 +72,11 @@ type MatrixError struct { // version the client should be writing to instead. CurrentVersion string `json:"current_version,omitempty"` + // RemoteStatus and RemoteBody accompany M_BAD_STATUS: what an + // application service answered a ping with. + RemoteStatus int `json:"status,omitempty"` + RemoteBody string `json:"body,omitempty"` + // status is the HTTP status to send; not part of the JSON body. status int } @@ -164,6 +175,12 @@ func InvalidUsername(format string, args ...any) *MatrixError { return NewError(http.StatusBadRequest, ErrCodeInvalidUsername, format, args...) } +// Exclusive is M_EXCLUSIVE (400): a user ID or alias in a namespace an +// application service has claimed for itself. +func Exclusive(format string, args ...any) *MatrixError { + return NewError(http.StatusBadRequest, ErrCodeExclusive, format, args...) +} + // UserDeactivated is M_USER_DEACTIVATED (403). func UserDeactivated() *MatrixError { return NewError(http.StatusForbidden, ErrCodeUserDeactivated, "This account has been deactivated") diff --git a/server/internal/mediaapi/http.go b/server/internal/mediaapi/http.go index a5d1e0c..2479e20 100644 --- a/server/internal/mediaapi/http.go +++ b/server/internal/mediaapi/http.go @@ -185,6 +185,11 @@ func HTTPError(err error) error { return httputil.NotFound("Media not found") case errors.Is(err, ErrTooLarge): return httputil.NewError(http.StatusRequestEntityTooLarge, httputil.ErrCodeTooLarge, "Media is larger than this server allows") + case errors.Is(err, ErrQuotaExceeded): + // M_TOO_LARGE rather than a code of its own: every client already + // tells the user an upload was too big, and that is what it was. + return httputil.NewError(http.StatusRequestEntityTooLarge, httputil.ErrCodeTooLarge, + "This upload would take you over your media quota") case errors.Is(err, ErrNotYetUploaded): return httputil.NewError(http.StatusGatewayTimeout, httputil.ErrCodeNotYetUploaded, "Media has not been uploaded yet") case errors.Is(err, ErrCannotOverwrite): diff --git a/server/internal/mediaapi/mediaapi.go b/server/internal/mediaapi/mediaapi.go index f3571cd..bd92149 100644 --- a/server/internal/mediaapi/mediaapi.go +++ b/server/internal/mediaapi/mediaapi.go @@ -40,6 +40,10 @@ var ( // ErrTooLarge means content is larger than media.max_upload_bytes. ErrTooLarge = errors.New("media is too large") + // ErrQuotaExceeded means the upload would take the user past their + // media quota. + ErrQuotaExceeded = errors.New("media quota exceeded") + // ErrNotYetUploaded means the media ID was reserved and its content // has not arrived within the time the request was willing to wait. ErrNotYetUploaded = errors.New("media has not been uploaded yet") @@ -325,9 +329,24 @@ func (a *API) UploadTo(ctx context.Context, origin, mediaID string, up Upload) e } // storeContent writes an upload's body to the blob store. +// +// The user's quota is applied by lowering the limit the content is +// written with to what the quota has left, so an upload that would go +// over is cut off as it arrives rather than stored and then refused. Two +// uploads racing for the last of a quota can both get in; a quota is a +// bound on what a user keeps here, not an accounting system. func (a *API) storeContent(ctx context.Context, up Upload) (storage.MediaContent, error) { - blob, err := a.blobs.Put(ctx, up.Body, a.cfg.MaxUploadBytes) + limit, quotaBound, err := a.uploadLimit(ctx, up.UserID) + if err != nil { + return storage.MediaContent{}, err + } + if limit <= 0 { + return storage.MediaContent{}, ErrQuotaExceeded + } + blob, err := a.blobs.Put(ctx, up.Body, limit) switch { + case errors.Is(err, storage.ErrBlobTooLarge) && quotaBound: + return storage.MediaContent{}, ErrQuotaExceeded case errors.Is(err, storage.ErrBlobTooLarge): return storage.MediaContent{}, ErrTooLarge case err != nil: @@ -341,6 +360,63 @@ func (a *API) storeContent(ctx context.Context, up Upload) (storage.MediaContent }, nil } +// uploadLimit is the most one upload by a user may be: the server's +// upload limit, or what is left of the user's quota if that is less, in +// which case quotaBound is set. +func (a *API) uploadLimit(ctx context.Context, userID string) (limit int64, quotaBound bool, err error) { + quota, err := a.Quota(ctx, userID) + if err != nil { + return 0, false, err + } + if quota.MaxBytes == 0 { + return a.cfg.MaxUploadBytes, false, nil + } + left := quota.MaxBytes - quota.Usage.Bytes + if left < a.cfg.MaxUploadBytes { + return left, true, nil + } + return a.cfg.MaxUploadBytes, false, nil +} + +// Quota is a user's media quota and what they are using of it. +type Quota struct { + Usage storage.MediaUsage + + // MaxBytes is the quota in force; 0 is no limit. + MaxBytes int64 + + // Custom reports a quota set for this user through the admin API, + // rather than media.max_user_storage_bytes. + Custom bool +} + +// Quota reads a user's quota and usage. +func (a *API) Quota(ctx context.Context, userID string) (Quota, error) { + usage, err := a.store.MediaUsage(ctx, a.serverName, userID) + if err != nil { + return Quota{}, err + } + q := Quota{Usage: usage, MaxBytes: a.cfg.MaxUserStorageBytes} + custom, ok, err := a.store.MediaQuota(ctx, userID) + if err != nil { + return Quota{}, err + } + if ok { + q.MaxBytes, q.Custom = custom, true + } + return q, nil +} + +// SetQuota gives a user a quota of their own; 0 is no limit. +func (a *API) SetQuota(ctx context.Context, userID string, maxBytes int64) error { + return a.store.SetMediaQuota(ctx, userID, maxBytes) +} + +// ResetQuota puts a user back on media.max_user_storage_bytes. +func (a *API) ResetQuota(ctx context.Context, userID string) error { + return a.store.DeleteMediaQuota(ctx, userID) +} + // normaliseContentType keeps a Content-Type that parses and replaces one // that does not. What is stored is what downloads are served with, and a // header that does not parse is one every client would guess at. diff --git a/server/internal/ratelimit/ratelimit.go b/server/internal/ratelimit/ratelimit.go index 692f7ef..2c8fcb3 100644 --- a/server/internal/ratelimit/ratelimit.go +++ b/server/internal/ratelimit/ratelimit.go @@ -121,8 +121,21 @@ type KeyFunc func(*http.Request) string // Middleware rejects requests over the limit with M_LIMIT_EXCEEDED. func (l *Limiter) Middleware(key KeyFunc) httputil.Middleware { + return l.MiddlewareUnless(key, nil) +} + +// MiddlewareUnless is Middleware with an exemption: a request exempt says +// true for is let through without being counted. Nil exempts nothing. +// +// The exemption runs before anything else, so it must be cheap — an +// in-memory lookup, never a database query. +func (l *Limiter) MiddlewareUnless(key KeyFunc, exempt func(*http.Request) bool) httputil.Middleware { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if exempt != nil && exempt(r) { + next.ServeHTTP(w, r) + return + } if retryAfter := l.Allow(key(r)); retryAfter > 0 { // Rounded up: a client that waits exactly retry_after_ms // and is told to wait again learns nothing useful. diff --git a/server/internal/roomserver/stream.go b/server/internal/roomserver/stream.go index fae915e..3590aa5 100644 --- a/server/internal/roomserver/stream.go +++ b/server/internal/roomserver/stream.go @@ -592,3 +592,48 @@ func (r *API) JoinedUserIDs(ctx context.Context, roomID string) ([]string, error } return out, nil } + +// StreamEvent is one event of the server-wide stream, at its position. +type StreamEvent struct { + Position int64 + Event ClientEvent +} + +// StreamEvents reads the server's event stream across every room: the +// events after position `after`, up to `upTo` (0 for no bound), at most +// limit of them, in the client shape with their room IDs. +// +// Nothing is filtered by who may see what. Its one reader is the +// application service sender, which decides for itself which of them an +// application service is told about. +func (r *API) StreamEvents(ctx context.Context, after, upTo int64, limit int) ([]StreamEvent, error) { + stored, err := r.store.StreamEvents(ctx, after, upTo, limit) + if err != nil { + return nil, err + } + rules := map[string]roomversion.Rules{} + out := make([]StreamEvent, 0, len(stored)) + for _, s := range stored { + roomRules, ok := rules[s.RoomID] + if !ok { + _, roomRules, err = r.roomAndRules(ctx, s.RoomID) + if err != nil { + return nil, err + } + rules[s.RoomID] = roomRules + } + e, err := parseEvent(s, roomRules) + if err != nil { + return nil, err + } + out = append(out, StreamEvent{Position: s.StreamOrdering, Event: r.clientEvent(s, e, s.RoomID)}) + } + return out, nil +} + +// LocalAliases lists the aliases on this server that point at a room, +// without asking who wants to know: the application service sender matches +// them against its alias namespaces. +func (r *API) LocalAliases(ctx context.Context, roomID string) ([]string, error) { + return r.store.AliasesForRoom(ctx, roomID) +} diff --git a/server/internal/servernotices/servernotices.go b/server/internal/servernotices/servernotices.go new file mode 100644 index 0000000..6e85ca7 --- /dev/null +++ b/server/internal/servernotices/servernotices.go @@ -0,0 +1,245 @@ +// Package servernotices sends server notices: messages from the server +// itself to one of its users, in a room the server keeps for the purpose. +// +// The spec has little to say about how. A notice is an ordinary +// m.room.message, sent by a user the server owns, into a room the target +// user has been invited to and which carries the m.server_notice tag in +// their account data, so that a client can show it apart from their other +// rooms. Each user gets one such room. A second notice goes into the same +// room; a user who left it is invited back. +package servernotices + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "sync" + + "github.com/dariuszjaniak/warp/server/internal/matrix/event" + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/storage" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// Errors callers have to tell apart. +var ( + // ErrNoSuchUser is a target that does not exist. + ErrNoSuchUser = userapi.ErrNoSuchUser + + // ErrDeactivated is a target whose account is deactivated, who would + // never read the notice. + ErrDeactivated = errors.New("the account is deactivated") +) + +// TagServerNotice is the room tag a server notice room carries. +const TagServerNotice = "m.server_notice" + +// Accounts is what the sender needs of the userapi. *userapi.API +// satisfies it. +type Accounts interface { + Account(ctx context.Context, localpart string) (storage.Account, error) + CreateAccount(ctx context.Context, req userapi.NewAccountRequest) (storage.Account, error) + SetDisplayName(ctx context.Context, localpart, displayName string) error + AccountData(ctx context.Context, localpart, roomID, dataType string) ([]byte, error) + SetAccountData(ctx context.Context, localpart, roomID, dataType string, content []byte) error + UserID(localpart string) string +} + +// Rooms is what it needs of the roomserver. *roomserver.API satisfies it. +type Rooms interface { + CreateRoom(ctx context.Context, req roomserver.CreateRoomRequest) (roomserver.CreateRoomResponse, error) + Membership(ctx context.Context, roomID, userID string) (string, error) + InviteUser(ctx context.Context, req roomserver.InviteRequest) error + SendEvent(ctx context.Context, req roomserver.SendEventRequest) (string, error) +} + +// Options configure the sender. +type Options struct { + // Localpart is the server's own user, which sends the notices. + Localpart string + DisplayName string + + // RoomName names the room each user gets their notices in. + RoomName string + + Accounts Accounts + Rooms Rooms + Store storage.AdminStore + + // WakeUser wakes a user's blocked /sync after their account data + // changed. Nil wakes nobody. + WakeUser func(userID string) + + Logger *slog.Logger +} + +// API sends server notices. +type API struct { + opts Options + log *slog.Logger + + // mu serialises sends, so that two notices to a user who has no room + // yet do not make two rooms. + mu sync.Mutex +} + +// New builds the sender. +func New(opts Options) (*API, error) { + switch { + case opts.Localpart == "": + return nil, errors.New("servernotices: Localpart must not be empty") + case opts.Accounts == nil, opts.Rooms == nil, opts.Store == nil: + return nil, errors.New("servernotices: Accounts, Rooms and Store must not be nil") + } + if opts.WakeUser == nil { + opts.WakeUser = func(string) {} + } + if opts.Logger == nil { + opts.Logger = slog.Default() + } + return &API{opts: opts, log: opts.Logger}, nil +} + +// UserID is the user notices come from. +func (a *API) UserID() string { return a.opts.Accounts.UserID(a.opts.Localpart) } + +// EnsureUser creates the notices user if it does not exist and keeps its +// display name in step with the configuration. The account has no +// password: nobody logs in as it. +func (a *API) EnsureUser(ctx context.Context) error { + account, err := a.opts.Accounts.Account(ctx, a.opts.Localpart) + if errors.Is(err, userapi.ErrNoSuchUser) { + account, err = a.opts.Accounts.CreateAccount(ctx, userapi.NewAccountRequest{ + Localpart: a.opts.Localpart, + DisplayName: a.opts.DisplayName, + }) + } + if err != nil { + return fmt.Errorf("the server notices user: %w", err) + } + if account.DisplayName != a.opts.DisplayName { + return a.opts.Accounts.SetDisplayName(ctx, a.opts.Localpart, a.opts.DisplayName) + } + return nil +} + +// Send sends a notice to a local user and returns the room and the event. +// content is an m.room.message content; one without a msgtype is sent as +// m.text. +func (a *API) Send(ctx context.Context, localpart string, content map[string]any) (roomID, eventID string, err error) { + account, err := a.opts.Accounts.Account(ctx, localpart) + switch { + case err != nil: + return "", "", err + case account.Deactivated: + return "", "", ErrDeactivated + } + target := a.opts.Accounts.UserID(localpart) + + a.mu.Lock() + defer a.mu.Unlock() + + roomID, err = a.room(ctx, localpart, target) + if err != nil { + return "", "", err + } + + membership, err := a.opts.Rooms.Membership(ctx, roomID, target) + if err != nil { + return "", "", err + } + if membership != event.MembershipJoin && membership != event.MembershipInvite { + if err := a.opts.Rooms.InviteUser(ctx, roomserver.InviteRequest{ + RoomID: roomID, Sender: a.UserID(), Target: target, + }); err != nil { + return "", "", fmt.Errorf("invite %s to their server notice room: %w", target, err) + } + } + if err := a.tag(ctx, localpart, target, roomID); err != nil { + return "", "", err + } + + if _, ok := content["msgtype"]; !ok { + content["msgtype"] = "m.text" + } + raw, err := json.Marshal(content) + if err != nil { + return "", "", err + } + eventID, err = a.opts.Rooms.SendEvent(ctx, roomserver.SendEventRequest{ + RoomID: roomID, Sender: a.UserID(), Type: roomserver.TypeMessage, Content: raw, + }) + if err != nil { + return "", "", err + } + a.log.Info("sent a server notice", "user_id", target, "room_id", roomID, "event_id", eventID) + return roomID, eventID, nil +} + +// room returns the user's notice room, creating it the first time. +// +// The room is private, and only the notices user may send in it: the +// power levels give everybody else none of the levels a message needs. +// A notice room the user can talk back in would be a support channel +// nobody reads. +func (a *API) room(ctx context.Context, localpart, target string) (string, error) { + roomID, err := a.opts.Store.ServerNoticeRoom(ctx, localpart) + if err == nil { + return roomID, nil + } + if !errors.Is(err, storage.ErrNoServerNoticeRoom) { + return "", err + } + + powerLevels, _ := json.Marshal(map[string]any{ + "events_default": 100, + "users": map[string]int{a.UserID(): 100}, + }) + created, err := a.opts.Rooms.CreateRoom(ctx, roomserver.CreateRoomRequest{ + Creator: a.UserID(), + Preset: roomserver.PresetPrivateChat, + Name: a.opts.RoomName, + Invite: []string{target}, + PowerLevelContentOverride: powerLevels, + }) + if err != nil { + return "", fmt.Errorf("create the server notice room of %s: %w", target, err) + } + if err := a.opts.Store.SetServerNoticeRoom(ctx, localpart, created.RoomID); err != nil { + return "", err + } + return created.RoomID, nil +} + +// tag puts the m.server_notice tag on the room in the user's account +// data, keeping whatever other tags they gave it. +func (a *API) tag(ctx context.Context, localpart, target, roomID string) error { + tags := map[string]map[string]json.RawMessage{"tags": {}} + raw, err := a.opts.Accounts.AccountData(ctx, localpart, roomID, "m.tag") + switch { + case errors.Is(err, userapi.ErrNoSuchAccountData): + case err != nil: + return err + default: + // Tags a client wrote that do not parse are replaced; the server + // notice tag is the one that has to be there. + if json.Unmarshal(raw, &tags) != nil || tags["tags"] == nil { + tags = map[string]map[string]json.RawMessage{"tags": {}} + } + } + if _, ok := tags["tags"][TagServerNotice]; ok { + return nil + } + tags["tags"][TagServerNotice] = json.RawMessage(`{}`) + encoded, err := json.Marshal(tags) + if err != nil { + return err + } + if err := a.opts.Accounts.SetAccountData(ctx, localpart, roomID, "m.tag", encoded); err != nil { + return err + } + a.opts.WakeUser(target) + return nil +} diff --git a/server/internal/settings/settings.go b/server/internal/settings/settings.go index fdd826e..c15e35f 100644 --- a/server/internal/settings/settings.go +++ b/server/internal/settings/settings.go @@ -13,9 +13,11 @@ package settings import ( "context" + "encoding/json" "errors" "fmt" "log/slog" + "slices" "strconv" "sync" @@ -26,6 +28,11 @@ import ( const ( KeyFederationEnabled = "federation.enabled" KeyRegistrationEnabled = "registration.enabled" + + // KeyFederationLists holds the allow and deny lists an admin set + // through the admin API, as one JSON object, so that the two are + // always written and read together. + KeyFederationLists = "federation.lists" ) // ErrNotToggleable is returned when a setting is pinned by the config file. @@ -38,6 +45,21 @@ type Defaults struct { RegistrationEnabled bool RegistrationAllowRuntimeToggle bool + + // FederationAllowlist and FederationDenylist are the config file's + // lists, in force until an admin sets others. + FederationAllowlist []string + FederationDenylist []string +} + +// FederationLists are the federation allow and deny lists in force. +type FederationLists struct { + Allowlist []string `json:"allowlist"` + Denylist []string `json:"denylist"` + + // Edited reports lists set through the admin API, which replace the + // config file's until they are reset. + Edited bool `json:"-"` } // Snapshot is a consistent view of every runtime setting. @@ -54,6 +76,7 @@ type Service struct { mu sync.RWMutex current Snapshot + lists FederationLists listeners []func(Snapshot) // writeMu serialises Set calls; see set. @@ -88,6 +111,20 @@ func New(ctx context.Context, store storage.SettingsStore, defaults Defaults, lo s.current.RegistrationEnabled = s.boolOr(stored, KeyRegistrationEnabled, defaults.RegistrationEnabled) } + // The lists an admin edited win over the file's whatever the toggle + // settings say: they are not a switch the file can pin, and an admin + // who set them expects them to survive a restart. + s.lists = FederationLists{Allowlist: defaults.FederationAllowlist, Denylist: defaults.FederationDenylist} + if raw, ok := stored[KeyFederationLists]; ok { + var edited FederationLists + if err := json.Unmarshal([]byte(raw), &edited); err != nil { + s.log.Warn("ignoring unparseable stored federation lists", "error", err) + } else { + edited.Edited = true + s.lists = edited + } + } + s.log.Info("runtime settings loaded", "federation_enabled", s.current.FederationEnabled, "registration_enabled", s.current.RegistrationEnabled) @@ -151,6 +188,65 @@ func (s *Service) SetRegistrationEnabled(ctx context.Context, enabled bool) erro return s.set(ctx, KeyRegistrationEnabled, enabled, func(snap *Snapshot) { snap.RegistrationEnabled = enabled }) } +// FederationLists returns the allow and deny lists in force. +func (s *Service) FederationLists() FederationLists { + s.mu.RLock() + defer s.mu.RUnlock() + return FederationLists{ + Allowlist: slices.Clone(s.lists.Allowlist), + Denylist: slices.Clone(s.lists.Denylist), + Edited: s.lists.Edited, + } +} + +// SetFederationLists persists new allow and deny lists. Applying them to +// the federation policy is the caller's job. +func (s *Service) SetFederationLists(ctx context.Context, allowlist, denylist []string) error { + s.writeMu.Lock() + defer s.writeMu.Unlock() + + lists := FederationLists{Allowlist: nonNil(allowlist), Denylist: nonNil(denylist), Edited: true} + raw, err := json.Marshal(lists) + if err != nil { + return err + } + if err := s.store.Set(ctx, KeyFederationLists, string(raw)); err != nil { + return err + } + s.mu.Lock() + s.lists = lists + s.mu.Unlock() + s.log.Info("federation lists changed", "allowlist", lists.Allowlist, "denylist", lists.Denylist) + return nil +} + +// ResetFederationLists forgets the edited lists and puts the config file's +// back in force, returning them. +func (s *Service) ResetFederationLists(ctx context.Context) (FederationLists, error) { + s.writeMu.Lock() + defer s.writeMu.Unlock() + + if err := s.store.Delete(ctx, KeyFederationLists); err != nil { + return FederationLists{}, err + } + lists := FederationLists{ + Allowlist: nonNil(s.defaults.FederationAllowlist), + Denylist: nonNil(s.defaults.FederationDenylist), + } + s.mu.Lock() + s.lists = lists + s.mu.Unlock() + s.log.Info("federation lists reset to the configuration file's") + return lists, nil +} + +func nonNil(list []string) []string { + if list == nil { + return []string{} + } + return slices.Clone(list) +} + // set writes the store first: if persistence fails the in-memory value must // not change, or a restart would silently undo what the admin API reported // as done. diff --git a/server/internal/settings/settings_test.go b/server/internal/settings/settings_test.go index 0391e87..14dfdc3 100644 --- a/server/internal/settings/settings_test.go +++ b/server/internal/settings/settings_test.go @@ -198,3 +198,50 @@ func TestConcurrentReadsAndWrites(t *testing.T) { } wg.Wait() } + +func TestFederationListsPersist(t *testing.T) { + ctx := t.Context() + store := newMemStore(nil) + defaults := settings.Defaults{ + FederationAllowlist: []string{"a.example"}, + FederationDenylist: []string{"bad.example"}, + } + + s, err := settings.New(ctx, store, defaults, nil) + if err != nil { + t.Fatal(err) + } + if got := s.FederationLists(); got.Edited || len(got.Allowlist) != 1 || got.Denylist[0] != "bad.example" { + t.Fatalf("defaults = %+v", got) + } + if err := s.SetFederationLists(ctx, []string{"b.example"}, nil); err != nil { + t.Fatal(err) + } + + // A restart keeps the edited lists over the file's. + again, err := settings.New(ctx, store, defaults, nil) + if err != nil { + t.Fatal(err) + } + got := again.FederationLists() + if !got.Edited || len(got.Allowlist) != 1 || got.Allowlist[0] != "b.example" || got.Denylist == nil || len(got.Denylist) != 0 { + t.Fatalf("after restart = %+v", got) + } + + reset, err := again.ResetFederationLists(ctx) + if err != nil || reset.Edited || reset.Allowlist[0] != "a.example" { + t.Fatalf("reset = %+v, %v", reset, err) + } + if _, ok, _ := store.Get(ctx, settings.KeyFederationLists); ok { + t.Error("the stored lists survived a reset") + } + + // A failed write changes nothing. + store.failSet = errors.New("disk full") + if err := again.SetFederationLists(ctx, []string{"c.example"}, nil); err == nil { + t.Fatal("a failed write succeeded") + } + if got := again.FederationLists(); got.Edited || got.Allowlist[0] != "a.example" { + t.Errorf("after a failed write = %+v", got) + } +} diff --git a/server/internal/setup/appservices.go b/server/internal/setup/appservices.go new file mode 100644 index 0000000..8c57770 --- /dev/null +++ b/server/internal/setup/appservices.go @@ -0,0 +1,72 @@ +package setup + +import ( + "context" + "errors" + "fmt" + "log/slog" + + "github.com/dariuszjaniak/warp/server/internal/appservice" + "github.com/dariuszjaniak/warp/server/internal/config" + "github.com/dariuszjaniak/warp/server/internal/matrix/event" + "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// loadAppServices reads the registration files the config names. +func loadAppServices(cfg *config.Config) (*appservice.Registry, error) { + regs := make([]*appservice.Registration, 0, len(cfg.AppService.ConfigFiles)) + for _, path := range cfg.AppService.ConfigFiles { + reg, err := appservice.LoadRegistration(path, cfg.Global.ServerName) + if err != nil { + return nil, err + } + regs = append(regs, reg) + } + return appservice.NewRegistry(cfg.Global.ServerName, regs) +} + +// ensureAppServiceSenders creates the sender user of every application +// service that does not have one yet. A service acts as its sender +// whenever it does not ask to act as somebody else, so the account has to +// exist before the first request; it has no password, because nobody logs +// in as it. +func ensureAppServiceSenders(ctx context.Context, users *userapi.API, registry *appservice.Registry, log *slog.Logger) error { + for _, svc := range registry.Services() { + _, err := users.CreateAccount(ctx, userapi.NewAccountRequest{Localpart: svc.SenderLocalpart}) + switch { + case err == nil: + log.Info("created the sender of an application service", "appservice", svc.ID, "user_id", svc.SenderUserID()) + case errors.Is(err, userapi.ErrUserExists): + default: + return fmt.Errorf("create the sender of application service %q: %w", svc.ID, err) + } + } + return nil +} + +// roomLeaver is the roomserver's side of a deactivation: the user leaves +// every room they are in, rejects every invite and withdraws every knock. +// A room that refuses is logged and skipped; the account is already +// deactivated, and one stuck room must not keep the user in all the +// others. +type roomLeaver struct { + rooms *roomserver.API + log *slog.Logger +} + +// AccountDeactivated implements userapi.AccountObserver. +func (l roomLeaver) AccountDeactivated(ctx context.Context, localpart string) { + userID := "@" + localpart + ":" + l.rooms.ServerName() + memberships, err := l.rooms.UserMemberships(ctx, userID, + []string{event.MembershipJoin, event.MembershipInvite, event.MembershipKnock}) + if err != nil { + l.log.Error("could not list the rooms of a deactivated user", "user_id", userID, "error", err) + return + } + for _, m := range memberships { + if err := l.rooms.LeaveRoom(ctx, m.RoomID, userID, ""); err != nil { + l.log.Warn("a deactivated user could not leave a room", "user_id", userID, "room_id", m.RoomID, "error", err) + } + } +} diff --git a/server/internal/setup/phase9_test.go b/server/internal/setup/phase9_test.go new file mode 100644 index 0000000..56190ab --- /dev/null +++ b/server/internal/setup/phase9_test.go @@ -0,0 +1,711 @@ +package setup_test + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/dariuszjaniak/warp/server/internal/config" + "github.com/dariuszjaniak/warp/server/internal/setup" + "github.com/dariuszjaniak/warp/server/internal/userapi" +) + +// The Phase 9 tests: application services, account management and the +// completed admin API, through the whole stack. + +// adminToken creates an admin account and returns a token for it. +func adminToken(t *testing.T, srv *setup.Server, localpart string) string { + t.Helper() + account, err := srv.Users().CreateAccount(t.Context(), userapi.NewAccountRequest{ + Localpart: localpart, Password: "sUp3rs3kr1t", Admin: true, + }) + if err != nil { + t.Fatalf("CreateAccount: %v", err) + } + session, err := srv.Users().CreateSession(t.Context(), userapi.SessionRequest{Localpart: account.Localpart}) + if err != nil { + t.Fatalf("CreateSession: %v", err) + } + return session.AccessToken +} + +// errcode reads the Matrix error code of a response. +func errcode(t *testing.T, rec *httptest.ResponseRecorder) string { + t.Helper() + code, _ := decode[map[string]any](t, rec)["errcode"].(string) + return code +} + +func expectError(t *testing.T, rec *httptest.ResponseRecorder, status int, code, what string) { + t.Helper() + if rec.Code != status || errcode(t, rec) != code { + t.Errorf("%s: %d %s, want %d %s", what, rec.Code, rec.Body.String(), status, code) + } +} + +func adminPath(parts ...string) string { + return "/_warp/admin/v1/" + strings.Join(parts, "/") +} + +// fakeBridge is an application service: it records the transactions it is +// sent, and answers user and alias queries by creating what was asked for, +// through the homeserver's own API, the way a bridge does. +type fakeBridge struct { + t *testing.T + srv *setup.Server + asToken string + + mu sync.Mutex + txnIDs []string + events []map[string]any + pings int +} + +func (b *fakeBridge) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer hs-bridge" { + http.Error(w, `{"errcode":"M_FORBIDDEN"}`, http.StatusForbidden) + return + } + switch { + case strings.HasPrefix(r.URL.Path, "/_matrix/app/v1/transactions/"): + var body struct { + Events []map[string]any `json:"events"` + } + raw, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(raw, &body) + b.mu.Lock() + b.txnIDs = append(b.txnIDs, strings.TrimPrefix(r.URL.Path, "/_matrix/app/v1/transactions/")) + b.events = append(b.events, body.Events...) + b.mu.Unlock() + _, _ = w.Write([]byte(`{}`)) + + case strings.HasPrefix(r.URL.Path, "/_matrix/app/v1/users/"): + userID, _ := url.PathUnescape(strings.TrimPrefix(r.URL.Path, "/_matrix/app/v1/users/")) + localpart := strings.TrimPrefix(strings.Split(userID, ":")[0], "@") + rec := post(b.t, b.srv, "/_matrix/client/v3/register", map[string]any{ + "type": "m.login.application_service", "username": localpart, "inhibit_login": true, + }, b.asToken) + if rec.Code != http.StatusOK { + http.Error(w, rec.Body.String(), http.StatusInternalServerError) + return + } + _, _ = w.Write([]byte(`{}`)) + + case strings.HasPrefix(r.URL.Path, "/_matrix/app/v1/rooms/"): + alias, _ := url.PathUnescape(strings.TrimPrefix(r.URL.Path, "/_matrix/app/v1/rooms/")) + localpart := strings.TrimPrefix(strings.Split(alias, ":")[0], "#") + rec := post(b.t, b.srv, "/_matrix/client/v3/createRoom", map[string]any{ + "preset": "public_chat", "room_alias_name": localpart, + }, b.asToken) + if rec.Code != http.StatusOK { + http.Error(w, rec.Body.String(), http.StatusInternalServerError) + return + } + _, _ = w.Write([]byte(`{}`)) + + case r.URL.Path == "/_matrix/app/v1/ping": + b.mu.Lock() + b.pings++ + b.mu.Unlock() + _, _ = w.Write([]byte(`{}`)) + + default: + http.NotFound(w, r) + } +} + +// eventIDs lists the event IDs the bridge has been sent, of one type. +func (b *fakeBridge) eventIDs(eventType string) []string { + b.mu.Lock() + defer b.mu.Unlock() + var out []string + for _, e := range b.events { + if e["type"] == eventType { + id, _ := e["event_id"].(string) + out = append(out, id) + } + } + return out +} + +const bridgeRegistration = ` +id: bridge +url: %s +as_token: as-bridge +hs_token: hs-bridge +sender_localpart: bridgebot +rate_limited: false +namespaces: + users: + - exclusive: true + regex: '@_bridge_.*:warp\.example' + aliases: + - exclusive: true + regex: '#_bridge_.*:warp\.example' +` + +// newBridgedServer starts a server with the fake bridge registered. +func newBridgedServer(t *testing.T, tweak func(*config.Config)) (*setup.Server, *fakeBridge) { + t.Helper() + bridge := &fakeBridge{t: t, asToken: "as-bridge"} + hs := httptest.NewServer(bridge) + t.Cleanup(hs.Close) + + regPath := filepath.Join(t.TempDir(), "bridge.yaml") + if err := os.WriteFile(regPath, []byte(strings.Replace(bridgeRegistration, "%s", hs.URL, 1)), 0o600); err != nil { + t.Fatal(err) + } + srv := newSigningServer(t, func(c *config.Config) { + c.AppService.ConfigFiles = []string{regPath} + c.AppService.Timeout = 5 * time.Second + if tweak != nil { + tweak(c) + } + }) + bridge.srv = srv + return srv, bridge +} + +func waitForAppServices(t *testing.T, srv *setup.Server) { + t.Helper() + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) + defer cancel() + if err := srv.AppServices().WaitIdle(ctx); err != nil { + t.Fatal(err) + } +} + +func TestAppServicesThroughTheAPI(t *testing.T) { + srv, bridge := newBridgedServer(t, nil) + const as = "as-bridge" + + // The sender exists from the start, and is who the token is. + who := mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", as), "whoami as the sender") + if who["user_id"] != "@bridgebot:warp.example" { + t.Errorf("whoami = %v", who) + } + + // Registering a puppet: the service's token, no password, no flow. + reg := mustOK(t, post(t, srv, "/_matrix/client/v3/register", map[string]any{ + "type": "m.login.application_service", "username": "_bridge_alice", "inhibit_login": true, + }, as), "register a puppet") + if reg["user_id"] != "@_bridge_alice:warp.example" || reg["access_token"] != nil { + t.Errorf("register = %v", reg) + } + expectError(t, post(t, srv, "/_matrix/client/v3/register", map[string]any{ + "type": "m.login.application_service", "username": "carol", + }, as), http.StatusBadRequest, "M_EXCLUSIVE", "register outside the namespace") + expectError(t, post(t, srv, "/_matrix/client/v3/register", map[string]any{ + "type": "m.login.application_service", "username": "_bridge_x", + }, "nope"), http.StatusUnauthorized, "M_UNKNOWN_TOKEN", "register with a bad token") + + // Nobody else may have a name in the exclusive namespace. + expectError(t, post(t, srv, "/_matrix/client/v3/register", map[string]any{ + "username": "_bridge_bob", "password": "sUp3rs3kr1t", + }, ""), http.StatusBadRequest, "M_EXCLUSIVE", "a user registering in the namespace") + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/register/available?username=_bridge_bob", ""), + http.StatusBadRequest, "M_EXCLUSIVE", "register/available in the namespace") + + // Acting as a puppet. + asAlice := "?user_id=" + url.QueryEscape("@_bridge_alice:warp.example") + who = mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami"+asAlice, as), "whoami as a puppet") + if who["user_id"] != "@_bridge_alice:warp.example" { + t.Errorf("whoami = %v", who) + } + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami?user_id=%40carol%3Awarp.example", as), + http.StatusForbidden, "M_FORBIDDEN", "acting as a user outside the namespace") + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami?user_id=%40_bridge_nobody%3Awarp.example", as), + http.StatusForbidden, "M_FORBIDDEN", "acting as a puppet that does not exist") + + // A real user talks to the puppet; the bridge hears about it. + alice := registerThrough(t, srv, "alice") + roomID := createPublicRoom(t, srv, alice) + mustOK(t, post(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/invite", + map[string]any{"user_id": "@_bridge_alice:warp.example"}, alice), "invite the puppet") + mustOK(t, post(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/join"+asAlice, + map[string]any{}, as), "the puppet joins") + msgID := sendText(t, srv, alice, roomID, "t1", map[string]any{"msgtype": "m.text", "body": "hello bridge"}) + + // A room the bridge has nothing to do with stays private. + otherRoom := createPublicRoom(t, srv, alice) + privateID := sendText(t, srv, alice, otherRoom, "t2", map[string]any{"msgtype": "m.text", "body": "not for you"}) + + waitForAppServices(t, srv) + messages := bridge.eventIDs("m.room.message") + if !contains(messages, msgID) || contains(messages, privateID) { + t.Errorf("the bridge was sent %v; want %s and not %s", messages, msgID, privateID) + } + + // Inviting a puppet that does not exist yet asks the bridge, which + // registers it. + mustOK(t, post(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/invite", + map[string]any{"user_id": "@_bridge_carol:warp.example"}, alice), "invite a puppet on demand") + if ok, _ := srv.Users().AccountExists(t.Context(), "_bridge_carol"); !ok { + t.Error("the bridge was not asked to create the puppet") + } + + // Aliases: reserved for the bridge, and created by it on demand. + expectError(t, put(t, srv, "/_matrix/client/v3/directory/room/"+url.PathEscape("#_bridge_mine:warp.example"), + map[string]any{"room_id": roomID}, alice), http.StatusBadRequest, "M_EXCLUSIVE", "an alias in the namespace") + expectError(t, post(t, srv, "/_matrix/client/v3/createRoom", map[string]any{"room_alias_name": "_bridge_x"}, alice), + http.StatusBadRequest, "M_EXCLUSIVE", "createRoom with an alias in the namespace") + resolved := mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/directory/room/"+url.PathEscape("#_bridge_portal:warp.example"), ""), + "resolve a portal alias") + if id, _ := resolved["room_id"].(string); !strings.HasPrefix(id, "!") { + t.Errorf("resolved = %v", resolved) + } + + // Logging in as a puppet, with the service's token as the credential. + login := mustOK(t, post(t, srv, "/_matrix/client/v3/login", map[string]any{ + "type": "m.login.application_service", + "identifier": map[string]any{"type": "m.id.user", "user": "_bridge_alice"}, + }, as), "application service login") + if login["user_id"] != "@_bridge_alice:warp.example" || login["access_token"] == nil { + t.Errorf("login = %v", login) + } + expectError(t, post(t, srv, "/_matrix/client/v3/login", map[string]any{ + "type": "m.login.application_service", + "identifier": map[string]any{"type": "m.id.user", "user": "alice"}, + }, as), http.StatusForbidden, "M_FORBIDDEN", "application service login outside the namespace") + flows := mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/login", ""), "login flows") + if !strings.Contains(mustJSON(t, flows), "m.login.application_service") { + t.Errorf("login flows = %v", flows) + } + + // The ping. + ping := mustOK(t, post(t, srv, "/_matrix/client/v1/appservice/bridge/ping", + map[string]any{"transaction_id": "p1"}, as), "ping") + if _, ok := ping["duration_ms"]; !ok { + t.Errorf("ping = %v", ping) + } + expectError(t, post(t, srv, "/_matrix/client/v1/appservice/bridge/ping", map[string]any{}, alice), + http.StatusForbidden, "M_FORBIDDEN", "a user asking for the ping") + + // The admin view. + root := adminToken(t, srv, "root") + list := mustOK(t, getWithToken(t, srv, adminPath("appservices"), root), "GET /appservices") + services, _ := list["appservices"].([]any) + if len(services) != 1 { + t.Fatalf("appservices = %v", list) + } + svc := services[0].(map[string]any) + delivery, _ := svc["delivery"].(map[string]any) + if svc["id"] != "bridge" || svc["sender"] != "@bridgebot:warp.example" || delivery == nil || + delivery["failures"] != float64(0) || strings.Contains(mustJSON(t, list), "as-bridge") { + t.Errorf("appservice = %v", svc) + } +} + +func mustJSON(t *testing.T, v any) string { + t.Helper() + raw, err := json.Marshal(v) + if err != nil { + t.Fatal(err) + } + return string(raw) +} + +// passwordAuth answers a UIA challenge with a password stage. +func passwordAuth(t *testing.T, rec *httptest.ResponseRecorder, user, password string) map[string]any { + t.Helper() + if rec.Code != http.StatusUnauthorized { + t.Fatalf("want a UIA challenge, got %d %s", rec.Code, rec.Body.String()) + } + return map[string]any{ + "type": "m.login.password", + "session": decode[map[string]any](t, rec)["session"], + "identifier": map[string]any{"type": "m.id.user", "user": user}, + "password": password, + } +} + +func login(t *testing.T, srv *setup.Server, user, password string) *httptest.ResponseRecorder { + t.Helper() + return post(t, srv, "/_matrix/client/v3/login", map[string]any{ + "type": "m.login.password", + "identifier": map[string]any{"type": "m.id.user", "user": user}, + "password": password, + }, "") +} + +func TestPasswordChangeAndDeactivation(t *testing.T) { + srv := newSigningServer(t, nil) + alice := registerThrough(t, srv, "alice") + other := decode[map[string]any](t, login(t, srv, "alice", "sUp3rs3kr1t"))["access_token"].(string) + + caps := mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/capabilities", alice), "capabilities") + if !strings.Contains(mustJSON(t, caps), `"m.change_password":{"enabled":true}`) { + t.Errorf("capabilities = %v", caps) + } + + // Changing the password asks for the old one first. + body := map[string]any{"new_password": "n3w-s3cr3t"} + rec := post(t, srv, "/_matrix/client/v3/account/password", body, alice) + body["auth"] = passwordAuth(t, rec, "alice", "wrong") + if rec := post(t, srv, "/_matrix/client/v3/account/password", body, alice); rec.Code != http.StatusUnauthorized { + t.Fatalf("a wrong password: %d %s", rec.Code, rec.Body.String()) + } + body["auth"] = passwordAuth(t, post(t, srv, "/_matrix/client/v3/account/password", map[string]any{"new_password": "x"}, alice), + "alice", "sUp3rs3kr1t") + mustOK(t, post(t, srv, "/_matrix/client/v3/account/password", body, alice), "change password") + + // The old password is gone, the other session with it; this one stays. + expectError(t, login(t, srv, "alice", "sUp3rs3kr1t"), http.StatusForbidden, "M_FORBIDDEN", "the old password") + mustOK(t, login(t, srv, "alice", "n3w-s3cr3t"), "the new password") + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", other), + http.StatusUnauthorized, "M_UNKNOWN_TOKEN", "another session after the change") + mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", alice), "this session after the change") + + // Deactivation: same re-authentication, then the user is gone from + // their rooms and cannot log in again. + bob := registerThrough(t, srv, "bob") + roomID := createPublicRoom(t, srv, bob) + mustOK(t, post(t, srv, "/_matrix/client/v3/join/"+url.PathEscape(roomID), map[string]any{}, alice), "join") + mustOK(t, put(t, srv, "/_matrix/client/v3/profile/@alice:warp.example/displayname", + map[string]any{"displayname": "Alice"}, alice), "set displayname") + + deactivate := map[string]any{"erase": true} + deactivate["auth"] = passwordAuth(t, post(t, srv, "/_matrix/client/v3/account/deactivate", deactivate, alice), + "alice", "n3w-s3cr3t") + resp := mustOK(t, post(t, srv, "/_matrix/client/v3/account/deactivate", deactivate, alice), "deactivate") + if resp["id_server_unbind_result"] != "no-support" { + t.Errorf("deactivate = %v", resp) + } + + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", alice), + http.StatusUnauthorized, "M_UNKNOWN_TOKEN", "a token of a deactivated account") + expectError(t, login(t, srv, "alice", "n3w-s3cr3t"), http.StatusForbidden, "M_USER_DEACTIVATED", "logging in when deactivated") + members := mustOK(t, getWithToken(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/joined_members", bob), "joined_members") + if joined, _ := members["joined"].(map[string]any); joined["@alice:warp.example"] != nil { + t.Errorf("a deactivated user is still in the room: %v", members) + } + profile := getWithToken(t, srv, "/_matrix/client/v3/profile/@alice:warp.example", bob) + if strings.Contains(profile.Body.String(), "Alice") { + t.Errorf("an erased profile = %s", profile.Body.String()) + } + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/register/available?username=alice", ""), + http.StatusBadRequest, "M_USER_IN_USE", "the name of a deactivated account") +} + +func TestAdminUserManagement(t *testing.T) { + srv := newSigningServer(t, func(c *config.Config) { + c.Media.BasePath = t.TempDir() + c.Media.MaxUserStorageBytes = 1000 + }) + root := adminToken(t, srv, "root") + alice := registerThrough(t, srv, "alice") + aliceID := url.PathEscape("@alice:warp.example") + + user := mustOK(t, getWithToken(t, srv, adminPath("users", aliceID), root), "GET user") + if user["user_id"] != "@alice:warp.example" || user["devices"] != float64(1) || user["joined_rooms"] != float64(0) { + t.Errorf("user = %v", user) + } + if media, _ := user["media"].(map[string]any); media["quota_bytes"] != float64(1000) || media["custom_quota"] != false { + t.Errorf("media = %v", user["media"]) + } + expectError(t, getWithToken(t, srv, adminPath("users", url.PathEscape("@nobody:warp.example")), root), + http.StatusNotFound, "M_NOT_FOUND", "an unknown user") + expectError(t, getWithToken(t, srv, adminPath("users", url.PathEscape("@x:elsewhere")), root), + http.StatusBadRequest, "M_INVALID_PARAM", "a user of another server") + expectError(t, getWithToken(t, srv, adminPath("users", aliceID), alice), + http.StatusForbidden, "M_FORBIDDEN", "a non-admin") + + // The admin flag, and the one change an admin may not make to themselves. + if got := mustOK(t, put(t, srv, adminPath("users", aliceID, "admin"), map[string]any{"admin": true}, root), "grant"); got["admin"] != true { + t.Errorf("grant = %v", got) + } + mustOK(t, getWithToken(t, srv, adminPath("stats"), alice), "alice is an admin now") + mustOK(t, put(t, srv, adminPath("users", aliceID, "admin"), map[string]any{"admin": false}, root), "revoke") + expectError(t, put(t, srv, adminPath("users", url.PathEscape("@root:warp.example"), "admin"), map[string]any{"admin": false}, root), + http.StatusForbidden, "M_FORBIDDEN", "demoting yourself") + expectError(t, post(t, srv, adminPath("users", url.PathEscape("@root:warp.example"), "deactivate"), map[string]any{}, root), + http.StatusForbidden, "M_FORBIDDEN", "deactivating yourself") + + // Devices. + devices := mustOK(t, getWithToken(t, srv, adminPath("users", aliceID, "devices"), root), "devices") + list, _ := devices["devices"].([]any) + if len(list) != 1 { + t.Fatalf("devices = %v", devices) + } + deviceID := list[0].(map[string]any)["device_id"].(string) + expectError(t, sendJSON(t, srv, http.MethodDelete, adminPath("users", aliceID, "devices", "NOPE"), nil, root), + http.StatusNotFound, "M_NOT_FOUND", "deleting an unknown device") + mustOK(t, sendJSON(t, srv, http.MethodDelete, adminPath("users", aliceID, "devices", deviceID), nil, root), "delete device") + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", alice), + http.StatusUnauthorized, "M_UNKNOWN_TOKEN", "the deleted device") + + // A password reset logs everything out by default. + alice = decode[map[string]any](t, login(t, srv, "alice", "sUp3rs3kr1t"))["access_token"].(string) + mustOK(t, post(t, srv, adminPath("users", aliceID, "password"), map[string]any{"password": "r3s3t-pw"}, root), "reset password") + expectError(t, getWithToken(t, srv, "/_matrix/client/v3/account/whoami", alice), + http.StatusUnauthorized, "M_UNKNOWN_TOKEN", "a session after the reset") + alice = decode[map[string]any](t, login(t, srv, "alice", "r3s3t-pw"))["access_token"].(string) + + // Media quota: the configured one, then one of her own. + if rec := send(t, srv, http.MethodPost, "/_matrix/media/v3/upload", "text/plain", make([]byte, 600), alice, nil); rec.Code != http.StatusOK { + t.Fatalf("first upload: %d %s", rec.Code, rec.Body.String()) + } + expectError(t, send(t, srv, http.MethodPost, "/_matrix/media/v3/upload", "text/plain", make([]byte, 600), alice, nil), + http.StatusRequestEntityTooLarge, "M_TOO_LARGE", "an upload over the quota") + quota := mustOK(t, put(t, srv, adminPath("users", aliceID, "media", "quota"), map[string]any{"max_bytes": 0}, root), "lift the quota") + if quota["quota_bytes"] != float64(0) || quota["custom_quota"] != true || quota["bytes"] != float64(600) { + t.Errorf("quota = %v", quota) + } + if rec := send(t, srv, http.MethodPost, "/_matrix/media/v3/upload", "text/plain", make([]byte, 600), alice, nil); rec.Code != http.StatusOK { + t.Errorf("upload with no limit: %d %s", rec.Code, rec.Body.String()) + } + quota = mustOK(t, sendJSON(t, srv, http.MethodDelete, adminPath("users", aliceID, "media", "quota"), nil, root), "reset the quota") + if quota["quota_bytes"] != float64(1000) || quota["custom_quota"] != false || quota["uploads"] != float64(2) { + t.Errorf("quota = %v", quota) + } + + // Deactivation and reactivation. + user = mustOK(t, post(t, srv, adminPath("users", aliceID, "deactivate"), map[string]any{"erase": false}, root), "deactivate") + if user["deactivated"] != true || user["devices"] != float64(0) { + t.Errorf("deactivated user = %v", user) + } + expectError(t, login(t, srv, "alice", "r3s3t-pw"), http.StatusForbidden, "M_USER_DEACTIVATED", "logging in when deactivated") + user = mustOK(t, post(t, srv, adminPath("users", aliceID, "reactivate"), map[string]any{}, root), "reactivate") + if user["deactivated"] != false { + t.Errorf("reactivated user = %v", user) + } + mustOK(t, login(t, srv, "alice", "r3s3t-pw"), "logging in after reactivation") + + // Every change is in the audit log, newest first. + logResp := mustOK(t, getWithToken(t, srv, adminPath("audit_log")+"?actor="+url.QueryEscape("@root:warp.example"), root), "audit log") + var actions []string + for _, e := range logResp["entries"].([]any) { + entry := e.(map[string]any) + actions = append(actions, entry["action"].(string)) + if entry["actor"] != "@root:warp.example" { + t.Errorf("entry = %v", entry) + } + } + want := []string{"user.reactivate", "user.deactivate", "user.media_quota", "user.media_quota", + "user.password", "user.device_delete", "user.admin", "user.admin"} + if strings.Join(actions, ",") != strings.Join(want, ",") { + t.Errorf("audit actions = %v, want %v", actions, want) + } + page := mustOK(t, getWithToken(t, srv, adminPath("audit_log")+"?limit=2", root), "a page of the audit log") + if entries, _ := page["entries"].([]any); len(entries) != 2 || page["next_before"] == nil { + t.Errorf("page = %v", page) + } +} + +func TestRegistrationTokensThroughTheAdminAPI(t *testing.T) { + srv := newSigningServer(t, func(c *config.Config) { c.Registration.RequireToken = true }) + root := adminToken(t, srv, "root") + + created := httptestCreated(t, post(t, srv, adminPath("registration_tokens"), + map[string]any{"token": "invite-1", "uses_allowed": 1}, root)) + if created["token"] != "invite-1" || created["uses_allowed"] != float64(1) || created["valid"] != true || created["expiry_time_ms"] != nil { + t.Errorf("created = %v", created) + } + generated := httptestCreated(t, post(t, srv, adminPath("registration_tokens"), map[string]any{"length": 24}, root)) + if tok, _ := generated["token"].(string); len(tok) != 24 || generated["uses_allowed"] != nil { + t.Errorf("generated = %v", generated) + } + expectError(t, post(t, srv, adminPath("registration_tokens"), map[string]any{"token": "invite-1"}, root), + http.StatusBadRequest, "M_INVALID_PARAM", "a duplicate token") + expectError(t, post(t, srv, adminPath("registration_tokens"), map[string]any{"token": "not valid!"}, root), + http.StatusBadRequest, "M_INVALID_PARAM", "a malformed token") + + validity := func() any { + return mustOK(t, getWithToken(t, srv, "/_matrix/client/v1/register/m.login.registration_token/validity?token=invite-1", ""), + "validity")["valid"] + } + if validity() != true { + t.Error("a fresh token is not valid") + } + + // Signing up with it spends its one use. + register := func(username string) *httptest.ResponseRecorder { + body := map[string]any{"username": username, "password": "sUp3rs3kr1t"} + rec := post(t, srv, "/_matrix/client/v3/register", body, "") + session := decode[map[string]any](t, rec)["session"] + body["auth"] = map[string]any{"type": "m.login.registration_token", "token": "invite-1", "session": session} + rec = post(t, srv, "/_matrix/client/v3/register", body, "") + if rec.Code != http.StatusUnauthorized { + return rec + } + body["auth"] = map[string]any{"type": "m.login.dummy", "session": session} + return post(t, srv, "/_matrix/client/v3/register", body, "") + } + mustOK(t, register("alice"), "register with the token") + if validity() != false { + t.Error("a used-up token is still valid") + } + if rec := register("bob"); rec.Code == http.StatusOK { + t.Errorf("a used-up token registered bob: %s", rec.Body.String()) + } + + got := mustOK(t, getWithToken(t, srv, adminPath("registration_tokens", "invite-1"), root), "GET token") + if got["completed"] != float64(1) || got["valid"] != false { + t.Errorf("token = %v", got) + } + updated := mustOK(t, put(t, srv, adminPath("registration_tokens", "invite-1"), map[string]any{"uses_allowed": nil}, root), "lift the limit") + if updated["uses_allowed"] != nil || updated["valid"] != true || updated["completed"] != float64(1) { + t.Errorf("updated = %v", updated) + } + mustOK(t, register("bob"), "register after the limit was lifted") + + list := mustOK(t, getWithToken(t, srv, adminPath("registration_tokens"), root), "list tokens") + if tokens, _ := list["registration_tokens"].([]any); len(tokens) != 2 || list["required"] != true { + t.Errorf("list = %v", list) + } + mustOK(t, sendJSON(t, srv, http.MethodDelete, adminPath("registration_tokens", "invite-1"), nil, root), "delete") + expectError(t, getWithToken(t, srv, adminPath("registration_tokens", "invite-1"), root), + http.StatusNotFound, "M_NOT_FOUND", "a deleted token") + if validity() != false { + t.Error("a deleted token is still valid") + } +} + +func httptestCreated(t *testing.T, rec *httptest.ResponseRecorder) map[string]any { + t.Helper() + if rec.Code != http.StatusCreated { + t.Fatalf("want 201, got %d %s", rec.Code, rec.Body.String()) + } + return decode[map[string]any](t, rec) +} + +// noticeSync is the part of a sync the server notice test reads. +type noticeSync struct { + Rooms struct { + Invite map[string]json.RawMessage `json:"invite"` + Join map[string]struct { + AccountData struct { + Events []struct { + Type string `json:"type"` + Content json.RawMessage `json:"content"` + } `json:"events"` + } `json:"account_data"` + Timeline struct { + Events []struct { + Sender string `json:"sender"` + Content map[string]any `json:"content"` + } `json:"events"` + } `json:"timeline"` + } `json:"join"` + } `json:"rooms"` +} + +func TestServerNotices(t *testing.T) { + srv := newSigningServer(t, func(c *config.Config) { c.ServerNotices.Enabled = true }) + root := adminToken(t, srv, "root") + alice := registerThrough(t, srv, "alice") + + notice := func(body string) map[string]any { + return mustOK(t, post(t, srv, adminPath("server_notices"), map[string]any{ + "user_id": "@alice:warp.example", + "content": map[string]any{"body": body}, + }, root), "send a notice") + } + first := notice("Maintenance tonight") + roomID, _ := first["room_id"].(string) + + var s noticeSync + rec := getWithToken(t, srv, "/_matrix/client/v3/sync?timeout=0", alice) + if err := json.Unmarshal(rec.Body.Bytes(), &s); err != nil { + t.Fatal(err) + } + if _, ok := s.Rooms.Invite[roomID]; !ok { + t.Fatalf("alice has no invite to %s: %s", roomID, rec.Body.String()) + } + + // A second notice goes to the same room. + if second := notice("Done"); second["room_id"] != roomID { + t.Errorf("second notice went to %v, want %s", second["room_id"], roomID) + } + + mustOK(t, post(t, srv, "/_matrix/client/v3/join/"+url.PathEscape(roomID), map[string]any{}, alice), "join the notice room") + rec = getWithToken(t, srv, "/_matrix/client/v3/sync?timeout=0", alice) + s = noticeSync{} + _ = json.Unmarshal(rec.Body.Bytes(), &s) + joined := s.Rooms.Join[roomID] + var bodies []string + for _, e := range joined.Timeline.Events { + if body, _ := e.Content["body"].(string); body != "" { + bodies = append(bodies, body) + if e.Sender != "@server-notices:warp.example" || e.Content["msgtype"] != "m.text" { + t.Errorf("notice event = %+v", e) + } + } + } + if strings.Join(bodies, "|") != "Maintenance tonight|Done" { + t.Errorf("notices = %v (%s)", bodies, rec.Body.String()) + } + var tagged bool + for _, e := range joined.AccountData.Events { + if e.Type == "m.tag" && strings.Contains(string(e.Content), "m.server_notice") { + tagged = true + } + } + if !tagged { + t.Errorf("the notice room has no m.server_notice tag: %s", rec.Body.String()) + } + + // The user cannot answer, and a user who leaves is invited back. + expectError(t, put(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/send/m.room.message/x", + map[string]any{"msgtype": "m.text", "body": "hi"}, alice), http.StatusForbidden, "M_FORBIDDEN", "replying to a notice") + mustOK(t, post(t, srv, "/_matrix/client/v3/rooms/"+url.PathEscape(roomID)+"/leave", map[string]any{}, alice), "leave") + notice("Again") + if membership, _ := srv.Rooms().Membership(t.Context(), roomID, "@alice:warp.example"); membership != "invite" { + t.Errorf("after leaving, membership = %q", membership) + } + + expectError(t, post(t, srv, adminPath("server_notices"), map[string]any{ + "user_id": "@nobody:warp.example", "content": map[string]any{"body": "x"}, + }, root), http.StatusNotFound, "M_NOT_FOUND", "a notice to nobody") + expectError(t, post(t, srv, adminPath("server_notices"), map[string]any{ + "user_id": "@alice:warp.example", "content": map[string]any{}, + }, root), http.StatusBadRequest, "M_MISSING_PARAM", "a notice without a body") +} + +func TestServerNoticesOffByDefault(t *testing.T) { + srv := newSigningServer(t, nil) + root := adminToken(t, srv, "root") + expectError(t, post(t, srv, adminPath("server_notices"), map[string]any{ + "user_id": "@root:warp.example", "content": map[string]any{"body": "x"}, + }, root), http.StatusConflict, "M_FORBIDDEN", "server notices when they are off") + if ok, _ := srv.Users().AccountExists(t.Context(), "server-notices"); ok { + t.Error("the notices user exists although notices are off") + } +} + +func TestFederationListsThroughTheAdminAPI(t *testing.T) { + srv := newSigningServer(t, func(c *config.Config) { c.Federation.Denylist = []string{"bad.example"} }) + root := adminToken(t, srv, "root") + + got := mustOK(t, getWithToken(t, srv, adminPath("federation"), root), "GET federation") + if got["lists_edited"] != false || mustJSON(t, got["denylist"]) != `["bad.example"]` { + t.Errorf("federation = %v", got) + } + + got = mustOK(t, put(t, srv, adminPath("federation", "lists"), map[string]any{ + "allowlist": []string{"friend.example", "Other.Example"}, "denylist": []string{}, + }, root), "PUT lists") + if got["lists_edited"] != true || mustJSON(t, got["allowlist"]) != `["friend.example","other.example"]` || + mustJSON(t, got["denylist"]) != `[]` { + t.Errorf("after PUT = %v", got) + } + expectError(t, put(t, srv, adminPath("federation", "lists"), map[string]any{"allowlist": []string{}}, root), + http.StatusBadRequest, "M_MISSING_PARAM", "only one list") + expectError(t, put(t, srv, adminPath("federation", "lists"), map[string]any{ + "allowlist": []string{"not a name"}, "denylist": []string{}, + }, root), http.StatusBadRequest, "M_INVALID_PARAM", "a bad server name") + + got = mustOK(t, sendJSON(t, srv, http.MethodDelete, adminPath("federation", "lists"), nil, root), "DELETE lists") + if got["lists_edited"] != false || mustJSON(t, got["allowlist"]) != `[]` || mustJSON(t, got["denylist"]) != `["bad.example"]` { + t.Errorf("after DELETE = %v", got) + } +} diff --git a/server/internal/setup/server.go b/server/internal/setup/server.go index 5763223..e6d2453 100644 --- a/server/internal/setup/server.go +++ b/server/internal/setup/server.go @@ -15,6 +15,7 @@ import ( "time" "github.com/dariuszjaniak/warp/server/internal/adminapi" + "github.com/dariuszjaniak/warp/server/internal/appservice" "github.com/dariuszjaniak/warp/server/internal/buildinfo" "github.com/dariuszjaniak/warp/server/internal/clientapi" "github.com/dariuszjaniak/warp/server/internal/config" @@ -27,6 +28,7 @@ import ( "github.com/dariuszjaniak/warp/server/internal/metrics" "github.com/dariuszjaniak/warp/server/internal/pushserver" "github.com/dariuszjaniak/warp/server/internal/roomserver" + "github.com/dariuszjaniak/warp/server/internal/servernotices" "github.com/dariuszjaniak/warp/server/internal/settings" "github.com/dariuszjaniak/warp/server/internal/storage" _ "github.com/dariuszjaniak/warp/server/internal/storage/drivers" // link in the storage drivers @@ -52,7 +54,12 @@ type Server struct { push *pushserver.API media *mediaapi.API clientAPI *clientapi.ClientAPI - adminAPI *adminapi.AdminAPI + + appservices *appservice.API + notices *servernotices.API + regTokens *userapi.RegistrationTokens + + adminAPI *adminapi.AdminAPI // The federation components. All nil when the server started without // a signing key, because nothing that federates can work without one. @@ -151,6 +158,8 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) FederationAllowRuntimeToggle: cfg.Federation.AllowRuntimeToggle, RegistrationEnabled: cfg.Registration.Enabled, RegistrationAllowRuntimeToggle: cfg.Registration.AllowRuntimeToggle, + FederationAllowlist: cfg.Federation.Allowlist, + FederationDenylist: cfg.Federation.Denylist, }, log) if err != nil { _ = s.closeDB() @@ -158,9 +167,12 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) } s.settings = settingsService + // The lists in force are the settings service's: the config file's, + // unless an admin has replaced them through the admin API. + lists := settingsService.FederationLists() s.federation = federation.NewGate( settingsService.FederationEnabled, - federation.NewPolicy(cfg.Federation.Allowlist, cfg.Federation.Denylist), + federation.NewPolicy(lists.Allowlist, lists.Denylist), ) // The switch starts and stops the federation module. The gate and the // client read the switch on every request and need no telling. @@ -192,6 +204,29 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) return nil, err } + // Registration tokens come from the config file and from the admin + // API; the sign-up flow and the admin endpoints share one view of both. + s.regTokens = userapi.NewRegistrationTokens(userapi.RegistrationTokensOptions{ + Store: s.db.Admin(), + Static: cfg.Registration.Tokens, + Required: cfg.Registration.RequireToken, + NowMS: func() int64 { return time.Now().UnixMilli() }, + }) + + // Application services are read from their registration files now, + // so that a bad one stops the server before it listens. Their senders + // are created here too: a service acts as its sender from its first + // request. + registry, err := loadAppServices(cfg) + if err != nil { + _ = s.closeDB() + return nil, err + } + if err := ensureAppServiceSenders(ctx, s.users, registry, log); err != nil { + _ = s.closeDB() + return nil, err + } + // The media repository needs no key: uploads and downloads of local // media are not signed. Fetching other servers' media is, and is // handed the federation client below once there is one. @@ -342,6 +377,47 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) } s.sync.SetOutbound(federationapi.NewEDUs(queue, s.rooms)) + // Application services read the event stream from a cursor of + // their own, and only need waking when it moves: they are one + // more listener on the emit stage. + s.appservices, err = appservice.New(appservice.Options{ + Registry: registry, + Store: s.db.AppServices(), + Rooms: s.rooms, + Timeout: cfg.AppService.Timeout, + Logger: log, + }) + if err != nil { + _ = s.closeDB() + return nil, err + } + emit.add(s.appservices) + + // Deactivating an account leaves its rooms, which only the + // roomserver can do. + s.users.SetAccountObserver(roomLeaver{rooms: s.rooms, log: log}) + + // Server notices are room messages, so they come with the rooms. + if cfg.ServerNotices.Enabled { + s.notices, err = servernotices.New(servernotices.Options{ + Localpart: cfg.ServerNotices.Localpart, + DisplayName: cfg.ServerNotices.DisplayName, + RoomName: cfg.ServerNotices.RoomName, + Accounts: s.users, + Rooms: s.rooms, + Store: s.db.Admin(), + WakeUser: notifier.NotifyUser, + Logger: log, + }) + if err == nil { + err = s.notices.EnsureUser(ctx) + } + if err != nil { + _ = s.closeDB() + return nil, err + } + } + s.fedAPI, err = federationapi.NewServer(federationapi.ServerOptions{ ServerName: cfg.Global.ServerName, Gate: s.federation, @@ -360,6 +436,16 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) } } else { log.Warn("started without a signing key: rooms, /sync and end-to-end encryption are unavailable") + // Without rooms there is no stream to deliver; the registry still + // lets the services use the Client-Server API. + if s.appservices, err = appservice.New(appservice.Options{ + Registry: registry, + Timeout: cfg.AppService.Timeout, + Logger: log, + }); err != nil { + _ = s.closeDB() + return nil, err + } } // The interfaces below are left nil rather than set to a nil pointer @@ -386,6 +472,8 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) Federation: s.federation, RemoteProfiles: remoteProfiles, Registration: cfg.Registration, + RegistrationTokens: s.regTokens, + AppServices: s.appservices, RateLimit: cfg.ClientAPI.RateLimit, TrustXForwardedFor: cfg.ClientAPI.TrustXForwardedFor, Logger: log, @@ -396,20 +484,30 @@ func New(ctx context.Context, cfg *config.Config, opts Options) (*Server, error) } s.adminAPI, err = adminapi.New(adminapi.Options{ - ServerName: cfg.Global.ServerName, - Users: s.users, - Rooms: s.rooms, - Settings: settingsService, - Federation: s.federation, - FederationStatus: fedStatus, - Database: s.db, - Logger: log, + ServerName: cfg.Global.ServerName, + Users: s.users, + Rooms: s.rooms, + Settings: settingsService, + Federation: s.federation, + FederationStatus: fedStatus, + Database: s.db, + Media: s.media, + AppServices: s.appservices, + Notices: s.notices, + RegistrationTokens: s.regTokens, + Logger: log, }) if err != nil { _ = s.closeDB() return nil, err } + // Delivery to application services starts with the server rather + // than with Run, like the push worker: it only reads the database and + // talks to the services, and tests that drive the handler directly + // need it running. + s.appservices.Start(ctx) + s.handler = httputil.Chain(s.routes(), httputil.Recoverer, httputil.RequestLogger(log), @@ -477,6 +575,13 @@ func (s *Server) Keys() *keyserver.API { return s.keys } // signing key. func (s *Server) Push() *pushserver.API { return s.push } +// AppServices exposes the application service component. +func (s *Server) AppServices() *appservice.API { return s.appservices } + +// ServerNotices exposes the server notice sender, or nil when server +// notices are off or the server has no signing key. +func (s *Server) ServerNotices() *servernotices.API { return s.notices } + // Media exposes the media repository. func (s *Server) Media() *mediaapi.API { return s.media } @@ -624,6 +729,11 @@ func (s *Server) Close() error { // to close. s.push.Close() } + if s.appservices != nil { + // So do the application service workers, which move their + // cursors. + s.appservices.Close() + } return s.closeDB() } diff --git a/server/internal/storage/admin.go b/server/internal/storage/admin.go new file mode 100644 index 0000000..afcac4b --- /dev/null +++ b/server/internal/storage/admin.go @@ -0,0 +1,156 @@ +package storage + +import ( + "context" + "errors" +) + +// Errors of the admin store. +var ( + // ErrNoSuchRegistrationToken is an unknown registration token — or, + // from UseRegistrationToken, one that is known but expired or used + // up. A client is told the same thing either way. + ErrNoSuchRegistrationToken = errors.New("no such registration token") + + // ErrRegistrationTokenExists is CreateRegistrationToken's answer for a + // token that is already there. + ErrRegistrationTokenExists = errors.New("registration token already exists") + + // ErrNoServerNoticeRoom is a user who has never been sent a server + // notice. + ErrNoServerNoticeRoom = errors.New("no server notice room") +) + +// RegistrationToken is a token made through the admin API. +type RegistrationToken struct { + Token string + + // UsesAllowed is how many registrations may spend the token; nil is + // no limit. + UsesAllowed *int + + // Completed is how many registrations have spent it. + Completed int + + // ExpiresAtMS is when the token stops working, or 0 for never. + ExpiresAtMS int64 + + CreatedAtMS int64 +} + +// Usable reports whether the token can still be spent at nowMS. +func (t RegistrationToken) Usable(nowMS int64) bool { + if t.ExpiresAtMS != 0 && t.ExpiresAtMS <= nowMS { + return false + } + return t.UsesAllowed == nil || t.Completed < *t.UsesAllowed +} + +// AuditEntry is one change made through the admin API. +type AuditEntry struct { + // ID orders the log; it is assigned by AppendAudit. + ID int64 + + TSMS int64 + + // Actor is the full user ID of the admin. + Actor string + + // Action names what was done, e.g. "user.deactivate". + Action string + + // Target is what it was done to, or empty for a server-wide change. + Target string + + // Details is a JSON object with whatever else is worth keeping: the + // new value of a switch, whether a deactivation erased the profile. + Details []byte +} + +// AuditFilter narrows and pages the audit log, newest first. +type AuditFilter struct { + // Before returns entries with an ID below it; 0 starts at the newest. + Before int64 + + // Limit caps the returned entries; 0 means the store's own default. + Limit int + + // Actor and Action, when set, must match exactly. + Actor string + Action string +} + +// AdminStore keeps the state of the admin API's own: registration tokens, +// the room each user's server notices go to, and the audit log. +type AdminStore interface { + // CreateRegistrationToken stores a new token, returning + // ErrRegistrationTokenExists when it is taken. + CreateRegistrationToken(ctx context.Context, t RegistrationToken) error + + // RegistrationToken reads one token. + RegistrationToken(ctx context.Context, token string) (RegistrationToken, error) + + // RegistrationTokens lists every token, ordered by token. + RegistrationTokens(ctx context.Context) ([]RegistrationToken, error) + + // UpdateRegistrationToken replaces a token's use limit and expiry; the + // count of uses it has had stays. + UpdateRegistrationToken(ctx context.Context, token string, usesAllowed *int, expiresAtMS int64) error + + // DeleteRegistrationToken removes a token. + DeleteRegistrationToken(ctx context.Context, token string) error + + // UseRegistrationToken spends one use of a token, if it is usable at + // nowMS, in a single statement: two registrations racing for the last + // use cannot both have it. A token that is not usable is + // ErrNoSuchRegistrationToken. + UseRegistrationToken(ctx context.Context, token string, nowMS int64) error + + // ReleaseRegistrationToken gives back a use taken by a registration + // that then failed. + ReleaseRegistrationToken(ctx context.Context, token string) error + + // AppendAudit records an entry and returns its ID. + AppendAudit(ctx context.Context, e AuditEntry) (int64, error) + + // AuditLog reads a page of the log, newest first. + AuditLog(ctx context.Context, f AuditFilter) ([]AuditEntry, error) + + // ServerNoticeRoom returns the room a user's server notices go to, + // or ErrNoServerNoticeRoom. + ServerNoticeRoom(ctx context.Context, localpart string) (string, error) + + // SetServerNoticeRoom records it, replacing any earlier one. + SetServerNoticeRoom(ctx context.Context, localpart, roomID string) error +} + +// AppServiceState is where an application service is in the event stream. +type AppServiceState struct { + ID string + + // StreamPos is the last position whose events have been delivered or + // skipped. + StreamPos int64 + + // TxnID is the ID of the last transaction begun. While TxnTo is + // non-zero that transaction is in flight and holds the events in + // (StreamPos, TxnTo]. + TxnID int64 + TxnTo int64 +} + +// AppServiceStore keeps the application services' delivery cursors. +type AppServiceStore interface { + // AppServiceState reads an application service's cursor, creating it + // at startPos the first time the service is seen: a newly registered + // service is sent what happens from now on, not the server's history. + AppServiceState(ctx context.Context, id string, startPos int64) (AppServiceState, error) + + // BeginTransaction records a transaction as in flight. + BeginTransaction(ctx context.Context, id string, txnID, to int64) error + + // CompleteTransaction moves the cursor to `to` and clears the + // transaction in flight. It is also how a stretch of the stream with + // nothing of interest in it is skipped. + CompleteTransaction(ctx context.Context, id string, to int64) error +} diff --git a/server/internal/storage/admin_test.go b/server/internal/storage/admin_test.go new file mode 100644 index 0000000..1261f8e --- /dev/null +++ b/server/internal/storage/admin_test.go @@ -0,0 +1,258 @@ +package storage_test + +import ( + "database/sql" + "errors" + "os" + "testing" + + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// migratedAdminDB opens a migrated database with empty Phase 9 tables. +func migratedAdminDB(t *testing.T, driver string) storage.Database { + t.Helper() + db := openTestDB(t, driver) + if err := db.Migrate(t.Context()); err != nil { + t.Fatalf("Migrate: %v", err) + } + if driver == storage.DriverPostgres { + pool, err := sql.Open("pgx", os.Getenv("WARP_TEST_POSTGRES_DSN")) + if err != nil { + t.Fatalf("open postgres to reset the admin tables: %v", err) + } + defer func() { _ = pool.Close() }() + for _, stmt := range []string{ + `TRUNCATE admin_registration_tokens, admin_server_notice_rooms, admin_audit_log, + media_quotas, appservice_state`, + `UPDATE admin_sequences SET value = 0`, + } { + if _, err := pool.ExecContext(t.Context(), stmt); err != nil { + t.Fatalf("reset the admin tables: %v", err) + } + } + } + return db +} + +func TestRegistrationTokens(t *testing.T) { + for _, driver := range testDrivers { + t.Run(driver, func(t *testing.T) { + ctx := t.Context() + admin := migratedAdminDB(t, driver).Admin() + + two := 2 + if err := admin.CreateRegistrationToken(ctx, storage.RegistrationToken{ + Token: "abc", UsesAllowed: &two, CreatedAtMS: 1, + }); err != nil { + t.Fatal(err) + } + if err := admin.CreateRegistrationToken(ctx, storage.RegistrationToken{Token: "abc", CreatedAtMS: 1}); !errors.Is(err, storage.ErrRegistrationTokenExists) { + t.Errorf("duplicate token: %v", err) + } + if err := admin.CreateRegistrationToken(ctx, storage.RegistrationToken{ + Token: "open", CreatedAtMS: 1, ExpiresAtMS: 1000, + }); err != nil { + t.Fatal(err) + } + + // Two uses, then no more; giving one back makes room again. + for i := range 2 { + if err := admin.UseRegistrationToken(ctx, "abc", 10); err != nil { + t.Fatalf("use %d: %v", i, err) + } + } + if err := admin.UseRegistrationToken(ctx, "abc", 10); !errors.Is(err, storage.ErrNoSuchRegistrationToken) { + t.Errorf("third use: %v", err) + } + if err := admin.ReleaseRegistrationToken(ctx, "abc"); err != nil { + t.Fatal(err) + } + got, err := admin.RegistrationToken(ctx, "abc") + if err != nil || got.Completed != 1 || got.UsesAllowed == nil || *got.UsesAllowed != 2 || !got.Usable(10) { + t.Errorf("after release = %+v, %v", got, err) + } + + // An unlimited token stops at its expiry. + if err := admin.UseRegistrationToken(ctx, "open", 999); err != nil { + t.Errorf("before expiry: %v", err) + } + if err := admin.UseRegistrationToken(ctx, "open", 1000); !errors.Is(err, storage.ErrNoSuchRegistrationToken) { + t.Errorf("at expiry: %v", err) + } + + // Updating replaces the limit and the expiry and keeps the count. + if err := admin.UpdateRegistrationToken(ctx, "open", &two, 0); err != nil { + t.Fatal(err) + } + got, _ = admin.RegistrationToken(ctx, "open") + if got.Completed != 1 || got.ExpiresAtMS != 0 || *got.UsesAllowed != 2 { + t.Errorf("after update = %+v", got) + } + if err := admin.UpdateRegistrationToken(ctx, "nope", nil, 0); !errors.Is(err, storage.ErrNoSuchRegistrationToken) { + t.Errorf("update unknown: %v", err) + } + + all, err := admin.RegistrationTokens(ctx) + if err != nil || len(all) != 2 || all[0].Token != "abc" || all[1].Token != "open" { + t.Errorf("RegistrationTokens = %+v, %v", all, err) + } + if err := admin.DeleteRegistrationToken(ctx, "abc"); err != nil { + t.Fatal(err) + } + if err := admin.DeleteRegistrationToken(ctx, "abc"); !errors.Is(err, storage.ErrNoSuchRegistrationToken) { + t.Errorf("second delete: %v", err) + } + if _, err := admin.RegistrationToken(ctx, "abc"); !errors.Is(err, storage.ErrNoSuchRegistrationToken) { + t.Errorf("read deleted: %v", err) + } + }) + } +} + +func TestAuditLog(t *testing.T) { + for _, driver := range testDrivers { + t.Run(driver, func(t *testing.T) { + ctx := t.Context() + admin := migratedAdminDB(t, driver).Admin() + + for i, e := range []storage.AuditEntry{ + {TSMS: 1, Actor: "@root:hs", Action: "user.create", Target: "@a:hs"}, + {TSMS: 2, Actor: "@root:hs", Action: "user.deactivate", Target: "@a:hs", Details: []byte(`{"erase":true}`)}, + {TSMS: 3, Actor: "@ops:hs", Action: "user.create", Target: "@b:hs"}, + } { + id, err := admin.AppendAudit(ctx, e) + if err != nil || id != int64(i+1) { + t.Fatalf("AppendAudit %d = %d, %v", i, id, err) + } + } + + all, err := admin.AuditLog(ctx, storage.AuditFilter{}) + if err != nil || len(all) != 3 || all[0].ID != 3 || all[2].ID != 1 { + t.Fatalf("AuditLog = %+v, %v", all, err) + } + if string(all[1].Details) != `{"erase":true}` || string(all[0].Details) != `{}` { + t.Errorf("details = %s, %s", all[1].Details, all[0].Details) + } + + page, _ := admin.AuditLog(ctx, storage.AuditFilter{Limit: 1, Before: 3}) + if len(page) != 1 || page[0].ID != 2 { + t.Errorf("paged = %+v", page) + } + byActor, _ := admin.AuditLog(ctx, storage.AuditFilter{Actor: "@root:hs"}) + byAction, _ := admin.AuditLog(ctx, storage.AuditFilter{Action: "user.create"}) + if len(byActor) != 2 || len(byAction) != 2 || byAction[0].Target != "@b:hs" { + t.Errorf("filtered = %+v / %+v", byActor, byAction) + } + }) + } +} + +func TestServerNoticeRooms(t *testing.T) { + for _, driver := range testDrivers { + t.Run(driver, func(t *testing.T) { + ctx := t.Context() + admin := migratedAdminDB(t, driver).Admin() + + if _, err := admin.ServerNoticeRoom(ctx, "alice"); !errors.Is(err, storage.ErrNoServerNoticeRoom) { + t.Errorf("unset: %v", err) + } + for _, room := range []string{"!one:hs", "!two:hs"} { + if err := admin.SetServerNoticeRoom(ctx, "alice", room); err != nil { + t.Fatal(err) + } + } + if room, err := admin.ServerNoticeRoom(ctx, "alice"); err != nil || room != "!two:hs" { + t.Errorf("ServerNoticeRoom = %q, %v", room, err) + } + }) + } +} + +func TestAppServiceState(t *testing.T) { + for _, driver := range testDrivers { + t.Run(driver, func(t *testing.T) { + ctx := t.Context() + apps := migratedAdminDB(t, driver).AppServices() + + st, err := apps.AppServiceState(ctx, "irc", 42) + if err != nil || st.StreamPos != 42 || st.TxnTo != 0 { + t.Fatalf("first read = %+v, %v", st, err) + } + // The start position only applies the first time. + if err := apps.BeginTransaction(ctx, "irc", 1, 50); err != nil { + t.Fatal(err) + } + st, _ = apps.AppServiceState(ctx, "irc", 99) + if st.StreamPos != 42 || st.TxnID != 1 || st.TxnTo != 50 { + t.Errorf("in flight = %+v", st) + } + if err := apps.CompleteTransaction(ctx, "irc", 50); err != nil { + t.Fatal(err) + } + st, _ = apps.AppServiceState(ctx, "irc", 0) + if st.StreamPos != 50 || st.TxnID != 1 || st.TxnTo != 0 { + t.Errorf("completed = %+v", st) + } + }) + } +} + +func TestMediaUsageAndQuota(t *testing.T) { + for _, driver := range testDrivers { + t.Run(driver, func(t *testing.T) { + ctx := t.Context() + db := migratedAdminDB(t, driver) + if driver == storage.DriverPostgres { + pool, err := sql.Open("pgx", os.Getenv("WARP_TEST_POSTGRES_DSN")) + if err != nil { + t.Fatal(err) + } + defer func() { _ = pool.Close() }() + if _, err := pool.ExecContext(ctx, `TRUNCATE media_repository`); err != nil { + t.Fatal(err) + } + } + media := db.Media() + + for _, m := range []storage.Media{ + {Origin: "hs", MediaID: "a", Size: 100, SHA256: "x", UserID: "@alice:hs", CreatedAtMS: 1}, + {Origin: "hs", MediaID: "b", Size: 100, SHA256: "x", UserID: "@alice:hs", CreatedAtMS: 1}, + {Origin: "hs", MediaID: "c", UserID: "@alice:hs", CreatedAtMS: 1, Pending: true, ExpiresAtMS: 10}, + {Origin: "hs", MediaID: "d", Size: 7, SHA256: "y", UserID: "@bob:hs", CreatedAtMS: 1}, + {Origin: "remote", MediaID: "e", Size: 5000, SHA256: "z", CreatedAtMS: 1}, + } { + if err := media.CreateMedia(ctx, m); err != nil { + t.Fatal(err) + } + } + // The same bytes twice count twice, and a reservation counts + // for nothing. + usage, err := media.MediaUsage(ctx, "hs", "@alice:hs") + if err != nil || usage.Count != 2 || usage.Bytes != 200 { + t.Errorf("alice = %+v, %v", usage, err) + } + if usage, _ := media.MediaUsage(ctx, "hs", "@carol:hs"); usage.Count != 0 || usage.Bytes != 0 { + t.Errorf("carol = %+v", usage) + } + + if _, ok, err := media.MediaQuota(ctx, "@alice:hs"); ok || err != nil { + t.Errorf("unset quota: %v, %v", ok, err) + } + for _, q := range []int64{1000, 500} { + if err := media.SetMediaQuota(ctx, "@alice:hs", q); err != nil { + t.Fatal(err) + } + } + if q, ok, err := media.MediaQuota(ctx, "@alice:hs"); !ok || q != 500 || err != nil { + t.Errorf("quota = %d, %v, %v", q, ok, err) + } + if err := media.DeleteMediaQuota(ctx, "@alice:hs"); err != nil { + t.Fatal(err) + } + if _, ok, _ := media.MediaQuota(ctx, "@alice:hs"); ok { + t.Error("quota survived its deletion") + } + }) + } +} diff --git a/server/internal/storage/media.go b/server/internal/storage/media.go index afa84da..70f2f54 100644 --- a/server/internal/storage/media.go +++ b/server/internal/storage/media.go @@ -109,6 +109,26 @@ type MediaStore interface { // StoreThumbnail records a thumbnail, replacing one stored for the // same size. StoreThumbnail(ctx context.Context, t Thumbnail) error + + // MediaUsage totals what a user has uploaded to this server. Pending + // reservations count for nothing until their content arrives. + MediaUsage(ctx context.Context, origin, userID string) (MediaUsage, error) + + // MediaQuota returns the quota the admin API set for a user, and + // whether one is set. + MediaQuota(ctx context.Context, userID string) (int64, bool, error) + + // SetMediaQuota sets a user's quota; 0 is no limit. + SetMediaQuota(ctx context.Context, userID string, maxBytes int64) error + + // DeleteMediaQuota returns a user to the configured default. + DeleteMediaQuota(ctx context.Context, userID string) error +} + +// MediaUsage is what one user's uploads add up to. +type MediaUsage struct { + Count int + Bytes int64 } // ErrNoSuchBlob means the blob store holds nothing under a hash. diff --git a/server/internal/storage/pocketbase/pocketbase.go b/server/internal/storage/pocketbase/pocketbase.go index ee6412c..67cc1fc 100644 --- a/server/internal/storage/pocketbase/pocketbase.go +++ b/server/internal/storage/pocketbase/pocketbase.go @@ -72,6 +72,12 @@ func (*Database) Keys() storage.KeyStore { return notImplementedKeys{} } // Push implements storage.Database. func (*Database) Push() storage.PushStore { return notImplementedPush{} } +// Admin implements storage.Database. +func (*Database) Admin() storage.AdminStore { return notImplementedAdmin{} } + +// AppServices implements storage.Database. +func (*Database) AppServices() storage.AppServiceStore { return notImplementedAppServices{} } + // Close implements storage.Database. func (*Database) Close() error { return nil } @@ -125,6 +131,10 @@ func (notImplementedUsers) SetDisplayName(context.Context, string, string) error return storage.ErrNotImplemented } +func (notImplementedUsers) SetDeactivated(context.Context, string, bool) error { + return storage.ErrNotImplemented +} + func (notImplementedUsers) SetAvatarURL(context.Context, string, string) error { return storage.ErrNotImplemented } @@ -254,6 +264,10 @@ func (notImplementedRooms) MaxStreamOrdering(context.Context) (int64, error) { return 0, storage.ErrNotImplemented } +func (notImplementedRooms) StreamEvents(context.Context, int64, int64, int) ([]storage.StoredEvent, error) { + return nil, storage.ErrNotImplemented +} + func (notImplementedRooms) Timeline(context.Context, storage.TimelineRequest) ([]storage.StoredEvent, error) { return nil, storage.ErrNotImplemented } @@ -477,6 +491,22 @@ func (notImplementedMedia) Thumbnail(context.Context, string, string, int, int, return storage.Thumbnail{}, storage.ErrNotImplemented } +func (notImplementedMedia) MediaUsage(context.Context, string, string) (storage.MediaUsage, error) { + return storage.MediaUsage{}, storage.ErrNotImplemented +} + +func (notImplementedMedia) MediaQuota(context.Context, string) (int64, bool, error) { + return 0, false, storage.ErrNotImplemented +} + +func (notImplementedMedia) SetMediaQuota(context.Context, string, int64) error { + return storage.ErrNotImplemented +} + +func (notImplementedMedia) DeleteMediaQuota(context.Context, string) error { + return storage.ErrNotImplemented +} + func (notImplementedMedia) StoreThumbnail(context.Context, storage.Thumbnail) error { return storage.ErrNotImplemented } @@ -655,3 +685,66 @@ func (notImplementedPush) Notifications(context.Context, string, int64, int, boo func (notImplementedPush) PruneNotifications(context.Context, int64) (int64, error) { return 0, storage.ErrNotImplemented } + +// notImplementedAdmin is the admin store until the driver has one. +type notImplementedAdmin struct{} + +func (notImplementedAdmin) CreateRegistrationToken(context.Context, storage.RegistrationToken) error { + return storage.ErrNotImplemented +} + +func (notImplementedAdmin) RegistrationToken(context.Context, string) (storage.RegistrationToken, error) { + return storage.RegistrationToken{}, storage.ErrNotImplemented +} + +func (notImplementedAdmin) RegistrationTokens(context.Context) ([]storage.RegistrationToken, error) { + return nil, storage.ErrNotImplemented +} + +func (notImplementedAdmin) UpdateRegistrationToken(context.Context, string, *int, int64) error { + return storage.ErrNotImplemented +} + +func (notImplementedAdmin) DeleteRegistrationToken(context.Context, string) error { + return storage.ErrNotImplemented +} + +func (notImplementedAdmin) UseRegistrationToken(context.Context, string, int64) error { + return storage.ErrNotImplemented +} + +func (notImplementedAdmin) ReleaseRegistrationToken(context.Context, string) error { + return storage.ErrNotImplemented +} + +func (notImplementedAdmin) AppendAudit(context.Context, storage.AuditEntry) (int64, error) { + return 0, storage.ErrNotImplemented +} + +func (notImplementedAdmin) AuditLog(context.Context, storage.AuditFilter) ([]storage.AuditEntry, error) { + return nil, storage.ErrNotImplemented +} + +func (notImplementedAdmin) ServerNoticeRoom(context.Context, string) (string, error) { + return "", storage.ErrNotImplemented +} + +func (notImplementedAdmin) SetServerNoticeRoom(context.Context, string, string) error { + return storage.ErrNotImplemented +} + +// notImplementedAppServices is the application service store until the +// driver has one. +type notImplementedAppServices struct{} + +func (notImplementedAppServices) AppServiceState(context.Context, string, int64) (storage.AppServiceState, error) { + return storage.AppServiceState{}, storage.ErrNotImplemented +} + +func (notImplementedAppServices) BeginTransaction(context.Context, string, int64, int64) error { + return storage.ErrNotImplemented +} + +func (notImplementedAppServices) CompleteTransaction(context.Context, string, int64) error { + return storage.ErrNotImplemented +} diff --git a/server/internal/storage/postgres/migrations/0011_admin_appservices.sql b/server/internal/storage/postgres/migrations/0011_admin_appservices.sql new file mode 100644 index 0000000..5d51eae --- /dev/null +++ b/server/internal/storage/postgres/migrations/0011_admin_appservices.sql @@ -0,0 +1,82 @@ +-- Phase 9: application services, and the parts of the admin API that keep +-- state of their own — registration tokens, server notice rooms, media +-- quotas and the audit log. + +-- Registration tokens made through the admin API. The tokens in the config +-- file are not here: they have no use count or expiry to keep. +-- +-- uses_allowed is NULL for a token that may be used any number of times. +-- completed counts the registrations that spent it; a registration takes +-- its use when the account is created, not when the stage is passed, so an +-- abandoned sign-up costs nothing. +CREATE TABLE IF NOT EXISTS admin_registration_tokens ( + token TEXT NOT NULL PRIMARY KEY, + uses_allowed BIGINT, + completed BIGINT NOT NULL DEFAULT 0, + -- 0 means the token never expires. + expires_at_ms BIGINT NOT NULL DEFAULT 0, + created_at_ms BIGINT NOT NULL +); + +-- The room each user gets their server notices in. One per user: a second +-- notice goes into the same room, and a user who left it is invited back. +CREATE TABLE IF NOT EXISTS admin_server_notice_rooms ( + localpart TEXT NOT NULL PRIMARY KEY, + room_id TEXT NOT NULL +); + +-- A user's media quota, where the admin API set one. Users without a row +-- get media.max_user_storage_bytes. max_bytes 0 is no limit, the same as +-- the config value. +CREATE TABLE IF NOT EXISTS media_quotas ( + user_id TEXT NOT NULL PRIMARY KEY, + max_bytes BIGINT NOT NULL +); + +-- Usage is summed over a user's uploads, so it wants an index that starts +-- with the uploader. +CREATE INDEX IF NOT EXISTS media_repository_user_idx + ON media_repository (user_id, media_origin); + +-- The audit log: one row per change made through the admin API. id is +-- handed out by admin_sequences inside the writing transaction, for the +-- reason the roomserver's stream positions are (0003): no engine-specific +-- autoincrement, and no gaps. +CREATE TABLE IF NOT EXISTS admin_sequences ( + name TEXT NOT NULL PRIMARY KEY, + value BIGINT NOT NULL +); + +INSERT INTO admin_sequences (name, value) VALUES ('audit', 0) ON CONFLICT (name) DO NOTHING; + +CREATE TABLE IF NOT EXISTS admin_audit_log ( + id BIGINT NOT NULL PRIMARY KEY, + ts_ms BIGINT NOT NULL, + -- The admin who made the change, as a full user ID. + actor TEXT NOT NULL, + -- What was done, e.g. "user.deactivate"; see docs/admin-api.md. + action TEXT NOT NULL, + -- What it was done to: a user ID, a room ID, a token. Empty for a + -- server-wide change. + target TEXT NOT NULL DEFAULT '', + details_json TEXT NOT NULL DEFAULT '{}' +); + +CREATE INDEX IF NOT EXISTS admin_audit_log_actor_idx ON admin_audit_log (actor, id); +CREATE INDEX IF NOT EXISTS admin_audit_log_action_idx ON admin_audit_log (action, id); + +-- Where each application service is in the event stream. +-- +-- stream_pos is the last stream position whose events have been delivered +-- (or found to be of no interest). txn_to is non-zero while a transaction +-- is in flight: it is the top of the events in it, and txn_id its ID, so +-- that after a restart the same transaction — same ID, same events — is +-- sent again rather than a new one that overlaps it. An application +-- service deduplicates by transaction ID, and only a retry that is +-- identical can be deduplicated. +CREATE TABLE IF NOT EXISTS appservice_state ( + as_id TEXT NOT NULL PRIMARY KEY, + stream_pos BIGINT NOT NULL, + txn_id BIGINT NOT NULL DEFAULT 0, + txn_to BIGINT NOT NULL DEFAULT 0 +); diff --git a/server/internal/storage/rooms.go b/server/internal/storage/rooms.go index a4c67a4..6d35efc 100644 --- a/server/internal/storage/rooms.go +++ b/server/internal/storage/rooms.go @@ -383,6 +383,13 @@ type RoomStore interface { // TimelineRequest: the bounds are exclusive and the limit is required. Timeline(ctx context.Context, req TimelineRequest) ([]StoredEvent, error) + // StreamEvents reads the server's whole event stream after a position, + // in order, across every room: the events with a stream ordering above + // after and at or below upTo (0 for no upper bound), outliers left out, + // at most limit of them. It is what the application services are fed + // from. + StreamEvents(ctx context.Context, after, upTo int64, limit int) ([]StoredEvent, error) + // RoomsWithEvents returns the subset of roomIDs that have an event // after from and at or before to. // diff --git a/server/internal/storage/sqlite/migrations/0011_admin_appservices.sql b/server/internal/storage/sqlite/migrations/0011_admin_appservices.sql new file mode 100644 index 0000000..5d51eae --- /dev/null +++ b/server/internal/storage/sqlite/migrations/0011_admin_appservices.sql @@ -0,0 +1,82 @@ +-- Phase 9: application services, and the parts of the admin API that keep +-- state of their own — registration tokens, server notice rooms, media +-- quotas and the audit log. + +-- Registration tokens made through the admin API. The tokens in the config +-- file are not here: they have no use count or expiry to keep. +-- +-- uses_allowed is NULL for a token that may be used any number of times. +-- completed counts the registrations that spent it; a registration takes +-- its use when the account is created, not when the stage is passed, so an +-- abandoned sign-up costs nothing. +CREATE TABLE IF NOT EXISTS admin_registration_tokens ( + token TEXT NOT NULL PRIMARY KEY, + uses_allowed BIGINT, + completed BIGINT NOT NULL DEFAULT 0, + -- 0 means the token never expires. + expires_at_ms BIGINT NOT NULL DEFAULT 0, + created_at_ms BIGINT NOT NULL +); + +-- The room each user gets their server notices in. One per user: a second +-- notice goes into the same room, and a user who left it is invited back. +CREATE TABLE IF NOT EXISTS admin_server_notice_rooms ( + localpart TEXT NOT NULL PRIMARY KEY, + room_id TEXT NOT NULL +); + +-- A user's media quota, where the admin API set one. Users without a row +-- get media.max_user_storage_bytes. max_bytes 0 is no limit, the same as +-- the config value. +CREATE TABLE IF NOT EXISTS media_quotas ( + user_id TEXT NOT NULL PRIMARY KEY, + max_bytes BIGINT NOT NULL +); + +-- Usage is summed over a user's uploads, so it wants an index that starts +-- with the uploader. +CREATE INDEX IF NOT EXISTS media_repository_user_idx + ON media_repository (user_id, media_origin); + +-- The audit log: one row per change made through the admin API. id is +-- handed out by admin_sequences inside the writing transaction, for the +-- reason the roomserver's stream positions are (0003): no engine-specific +-- autoincrement, and no gaps. +CREATE TABLE IF NOT EXISTS admin_sequences ( + name TEXT NOT NULL PRIMARY KEY, + value BIGINT NOT NULL +); + +INSERT INTO admin_sequences (name, value) VALUES ('audit', 0) ON CONFLICT (name) DO NOTHING; + +CREATE TABLE IF NOT EXISTS admin_audit_log ( + id BIGINT NOT NULL PRIMARY KEY, + ts_ms BIGINT NOT NULL, + -- The admin who made the change, as a full user ID. + actor TEXT NOT NULL, + -- What was done, e.g. "user.deactivate"; see docs/admin-api.md. + action TEXT NOT NULL, + -- What it was done to: a user ID, a room ID, a token. Empty for a + -- server-wide change. + target TEXT NOT NULL DEFAULT '', + details_json TEXT NOT NULL DEFAULT '{}' +); + +CREATE INDEX IF NOT EXISTS admin_audit_log_actor_idx ON admin_audit_log (actor, id); +CREATE INDEX IF NOT EXISTS admin_audit_log_action_idx ON admin_audit_log (action, id); + +-- Where each application service is in the event stream. +-- +-- stream_pos is the last stream position whose events have been delivered +-- (or found to be of no interest). txn_to is non-zero while a transaction +-- is in flight: it is the top of the events in it, and txn_id its ID, so +-- that after a restart the same transaction — same ID, same events — is +-- sent again rather than a new one that overlaps it. An application +-- service deduplicates by transaction ID, and only a retry that is +-- identical can be deduplicated. +CREATE TABLE IF NOT EXISTS appservice_state ( + as_id TEXT NOT NULL PRIMARY KEY, + stream_pos BIGINT NOT NULL, + txn_id BIGINT NOT NULL DEFAULT 0, + txn_to BIGINT NOT NULL DEFAULT 0 +); diff --git a/server/internal/storage/sqlstore/admin.go b/server/internal/storage/sqlstore/admin.go new file mode 100644 index 0000000..86239bd --- /dev/null +++ b/server/internal/storage/sqlstore/admin.go @@ -0,0 +1,317 @@ +package sqlstore + +import ( + "context" + "database/sql" + "errors" + "fmt" + "strings" + + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// The admin store's sequence table and its one counter. +const ( + adminSequences = "admin_sequences" + seqAudit = "audit" +) + +// Admin is the shared implementation of storage.AdminStore. +type Admin struct { + db *sql.DB + rebind func(string) string + isUnique func(error) bool +} + +// NewAdmin builds an admin store. +func NewAdmin(db *sql.DB, rebind func(string) string, isUnique func(error) bool) *Admin { + if rebind == nil { + rebind = func(query string) string { return query } + } + if isUnique == nil { + isUnique = func(error) bool { return false } + } + return &Admin{db: db, rebind: rebind, isUnique: isUnique} +} + +func (s *Admin) q(query string) string { return s.rebind(query) } + +const ( + registrationTokenColumns = `token, uses_allowed, completed, expires_at_ms, created_at_ms` + + insertRegistrationTokenSQL = `INSERT INTO admin_registration_tokens (` + registrationTokenColumns + `) + VALUES (?, ?, ?, ?, ?)` + + selectRegistrationTokenSQL = `SELECT ` + registrationTokenColumns + ` + FROM admin_registration_tokens WHERE token = ?` + + selectRegistrationTokensSQL = `SELECT ` + registrationTokenColumns + ` + FROM admin_registration_tokens ORDER BY token` + + updateRegistrationTokenSQL = `UPDATE admin_registration_tokens + SET uses_allowed = ?, expires_at_ms = ? WHERE token = ?` + + deleteRegistrationTokenSQL = `DELETE FROM admin_registration_tokens WHERE token = ?` + + // The usability test is in the WHERE clause, so that checking and + // spending are one statement and two registrations cannot both take + // the last use. + useRegistrationTokenSQL = `UPDATE admin_registration_tokens SET completed = completed + 1 + WHERE token = ? + AND (uses_allowed IS NULL OR completed < uses_allowed) + AND (expires_at_ms = 0 OR expires_at_ms > ?)` + + releaseRegistrationTokenSQL = `UPDATE admin_registration_tokens SET completed = completed - 1 + WHERE token = ? AND completed > 0` +) + +// CreateRegistrationToken implements storage.AdminStore. +func (s *Admin) CreateRegistrationToken(ctx context.Context, t storage.RegistrationToken) error { + _, err := s.db.ExecContext(ctx, s.q(insertRegistrationTokenSQL), + t.Token, nullableInt(t.UsesAllowed), t.Completed, t.ExpiresAtMS, t.CreatedAtMS) + switch { + case err != nil && s.isUnique(err): + return storage.ErrRegistrationTokenExists + case err != nil: + return fmt.Errorf("create a registration token: %w", err) + } + return nil +} + +// RegistrationToken implements storage.AdminStore. +func (s *Admin) RegistrationToken(ctx context.Context, token string) (storage.RegistrationToken, error) { + t, err := scanRegistrationToken(s.db.QueryRowContext(ctx, s.q(selectRegistrationTokenSQL), token).Scan) + switch { + case errors.Is(err, sql.ErrNoRows): + return storage.RegistrationToken{}, storage.ErrNoSuchRegistrationToken + case err != nil: + return storage.RegistrationToken{}, fmt.Errorf("read a registration token: %w", err) + } + return t, nil +} + +// RegistrationTokens implements storage.AdminStore. +func (s *Admin) RegistrationTokens(ctx context.Context) ([]storage.RegistrationToken, error) { + rows, err := s.db.QueryContext(ctx, s.q(selectRegistrationTokensSQL)) + if err != nil { + return nil, fmt.Errorf("list registration tokens: %w", err) + } + defer func() { _ = rows.Close() }() + + out := []storage.RegistrationToken{} + for rows.Next() { + t, err := scanRegistrationToken(rows.Scan) + if err != nil { + return nil, fmt.Errorf("list registration tokens: %w", err) + } + out = append(out, t) + } + return out, rows.Err() +} + +// UpdateRegistrationToken implements storage.AdminStore. +func (s *Admin) UpdateRegistrationToken(ctx context.Context, token string, usesAllowed *int, expiresAtMS int64) error { + res, err := s.db.ExecContext(ctx, s.q(updateRegistrationTokenSQL), nullableInt(usesAllowed), expiresAtMS, token) + if err != nil { + return fmt.Errorf("update a registration token: %w", err) + } + return mustHaveAffected(res, storage.ErrNoSuchRegistrationToken) +} + +// DeleteRegistrationToken implements storage.AdminStore. +func (s *Admin) DeleteRegistrationToken(ctx context.Context, token string) error { + res, err := s.db.ExecContext(ctx, s.q(deleteRegistrationTokenSQL), token) + if err != nil { + return fmt.Errorf("delete a registration token: %w", err) + } + return mustHaveAffected(res, storage.ErrNoSuchRegistrationToken) +} + +// UseRegistrationToken implements storage.AdminStore. +func (s *Admin) UseRegistrationToken(ctx context.Context, token string, nowMS int64) error { + res, err := s.db.ExecContext(ctx, s.q(useRegistrationTokenSQL), token, nowMS) + if err != nil { + return fmt.Errorf("spend a registration token: %w", err) + } + return mustHaveAffected(res, storage.ErrNoSuchRegistrationToken) +} + +// ReleaseRegistrationToken implements storage.AdminStore. A token deleted +// in the meantime has nothing to give back to, which is not an error. +func (s *Admin) ReleaseRegistrationToken(ctx context.Context, token string) error { + if _, err := s.db.ExecContext(ctx, s.q(releaseRegistrationTokenSQL), token); err != nil { + return fmt.Errorf("release a registration token: %w", err) + } + return nil +} + +func scanRegistrationToken(scan func(...any) error) (storage.RegistrationToken, error) { + var ( + t storage.RegistrationToken + uses sql.NullInt64 + ) + if err := scan(&t.Token, &uses, &t.Completed, &t.ExpiresAtMS, &t.CreatedAtMS); err != nil { + return storage.RegistrationToken{}, err + } + if uses.Valid { + n := int(uses.Int64) + t.UsesAllowed = &n + } + return t, nil +} + +// nullableInt binds an optional count as NULL when it is absent. +func nullableInt(n *int) any { + if n == nil { + return nil + } + return int64(*n) +} + +// mustHaveAffected turns an UPDATE or DELETE that matched nothing into +// notFound. +func mustHaveAffected(res sql.Result, notFound error) error { + n, err := res.RowsAffected() + if err != nil { + return err + } + if n == 0 { + return notFound + } + return nil +} + +const ( + insertAuditSQL = `INSERT INTO admin_audit_log (id, ts_ms, actor, action, target, details_json) + VALUES (?, ?, ?, ?, ?, ?)` + + auditColumns = `id, ts_ms, actor, action, target, details_json` + + // defaultAuditPage and maxAuditPage bound a page of the audit log. + defaultAuditPage = 50 + maxAuditPage = 1000 +) + +// AppendAudit implements storage.AdminStore. +func (s *Admin) AppendAudit(ctx context.Context, e storage.AuditEntry) (int64, error) { + details := string(e.Details) + if details == "" { + details = "{}" + } + var id int64 + err := inTx(ctx, s.db, func(tx *sql.Tx) error { + var err error + if id, err = nextSequence(ctx, tx, s.q, adminSequences, seqAudit); err != nil { + return err + } + _, err = tx.ExecContext(ctx, s.q(insertAuditSQL), id, e.TSMS, e.Actor, e.Action, e.Target, details) + return err + }) + if err != nil { + return 0, fmt.Errorf("append to the audit log: %w", err) + } + return id, nil +} + +// AuditLog implements storage.AdminStore. +func (s *Admin) AuditLog(ctx context.Context, f storage.AuditFilter) ([]storage.AuditEntry, error) { + var ( + conditions []string + args []any + ) + if f.Before > 0 { + conditions = append(conditions, `id < ?`) + args = append(args, f.Before) + } + if f.Actor != "" { + conditions = append(conditions, `actor = ?`) + args = append(args, f.Actor) + } + if f.Action != "" { + conditions = append(conditions, `action = ?`) + args = append(args, f.Action) + } + limit := f.Limit + switch { + case limit <= 0: + limit = defaultAuditPage + case limit > maxAuditPage: + limit = maxAuditPage + } + args = append(args, limit) + + query := `SELECT ` + auditColumns + ` FROM admin_audit_log` + if len(conditions) > 0 { + query += ` WHERE ` + strings.Join(conditions, " AND ") + } + query += ` ORDER BY id DESC LIMIT ?` + + rows, err := s.db.QueryContext(ctx, s.q(query), args...) + if err != nil { + return nil, fmt.Errorf("read the audit log: %w", err) + } + defer func() { _ = rows.Close() }() + + out := []storage.AuditEntry{} + for rows.Next() { + var ( + e storage.AuditEntry + details string + ) + if err := rows.Scan(&e.ID, &e.TSMS, &e.Actor, &e.Action, &e.Target, &details); err != nil { + return nil, fmt.Errorf("read the audit log: %w", err) + } + e.Details = []byte(details) + out = append(out, e) + } + return out, rows.Err() +} + +const ( + selectServerNoticeRoomSQL = `SELECT room_id FROM admin_server_notice_rooms WHERE localpart = ?` + + upsertServerNoticeRoomSQL = ` + INSERT INTO admin_server_notice_rooms (localpart, room_id) VALUES (?, ?) + ON CONFLICT (localpart) DO UPDATE SET room_id = excluded.room_id` +) + +// ServerNoticeRoom implements storage.AdminStore. +func (s *Admin) ServerNoticeRoom(ctx context.Context, localpart string) (string, error) { + var roomID string + err := s.db.QueryRowContext(ctx, s.q(selectServerNoticeRoomSQL), localpart).Scan(&roomID) + switch { + case errors.Is(err, sql.ErrNoRows): + return "", storage.ErrNoServerNoticeRoom + case err != nil: + return "", fmt.Errorf("read the server notice room of %q: %w", localpart, err) + } + return roomID, nil +} + +// SetServerNoticeRoom implements storage.AdminStore. +func (s *Admin) SetServerNoticeRoom(ctx context.Context, localpart, roomID string) error { + if _, err := s.db.ExecContext(ctx, s.q(upsertServerNoticeRoomSQL), localpart, roomID); err != nil { + return fmt.Errorf("record the server notice room of %q: %w", localpart, err) + } + return nil +} + +// inTx runs fn in a transaction on db, rolling back on any error. +func inTx(ctx context.Context, db *sql.DB, fn func(*sql.Tx) error) error { + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("begin transaction: %w", err) + } + defer func() { _ = tx.Rollback() }() + + if err := fn(tx); err != nil { + return err + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("commit transaction: %w", err) + } + return nil +} + +// Compile-time proof that the shared implementation satisfies the interface. +var _ storage.AdminStore = (*Admin)(nil) diff --git a/server/internal/storage/sqlstore/appservices.go b/server/internal/storage/sqlstore/appservices.go new file mode 100644 index 0000000..d2fabf4 --- /dev/null +++ b/server/internal/storage/sqlstore/appservices.go @@ -0,0 +1,70 @@ +package sqlstore + +import ( + "context" + "database/sql" + "fmt" + + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// AppServices is the shared implementation of storage.AppServiceStore. +type AppServices struct { + db *sql.DB + rebind func(string) string +} + +// NewAppServices builds an application service store. +func NewAppServices(db *sql.DB, rebind func(string) string) *AppServices { + if rebind == nil { + rebind = func(query string) string { return query } + } + return &AppServices{db: db, rebind: rebind} +} + +func (s *AppServices) q(query string) string { return s.rebind(query) } + +const ( + insertAppServiceStateSQL = ` + INSERT INTO appservice_state (as_id, stream_pos) VALUES (?, ?) + ON CONFLICT (as_id) DO NOTHING` + + selectAppServiceStateSQL = ` + SELECT as_id, stream_pos, txn_id, txn_to FROM appservice_state WHERE as_id = ?` + + beginAppServiceTxnSQL = `UPDATE appservice_state SET txn_id = ?, txn_to = ? WHERE as_id = ?` + + completeAppServiceTxnSQL = `UPDATE appservice_state SET stream_pos = ?, txn_to = 0 WHERE as_id = ?` +) + +// AppServiceState implements storage.AppServiceStore. +func (s *AppServices) AppServiceState(ctx context.Context, id string, startPos int64) (storage.AppServiceState, error) { + if _, err := s.db.ExecContext(ctx, s.q(insertAppServiceStateSQL), id, startPos); err != nil { + return storage.AppServiceState{}, fmt.Errorf("start the cursor of application service %q: %w", id, err) + } + var st storage.AppServiceState + if err := s.db.QueryRowContext(ctx, s.q(selectAppServiceStateSQL), id). + Scan(&st.ID, &st.StreamPos, &st.TxnID, &st.TxnTo); err != nil { + return storage.AppServiceState{}, fmt.Errorf("read the cursor of application service %q: %w", id, err) + } + return st, nil +} + +// BeginTransaction implements storage.AppServiceStore. +func (s *AppServices) BeginTransaction(ctx context.Context, id string, txnID, to int64) error { + if _, err := s.db.ExecContext(ctx, s.q(beginAppServiceTxnSQL), txnID, to, id); err != nil { + return fmt.Errorf("begin transaction %d of application service %q: %w", txnID, id, err) + } + return nil +} + +// CompleteTransaction implements storage.AppServiceStore. +func (s *AppServices) CompleteTransaction(ctx context.Context, id string, to int64) error { + if _, err := s.db.ExecContext(ctx, s.q(completeAppServiceTxnSQL), to, id); err != nil { + return fmt.Errorf("advance the cursor of application service %q: %w", id, err) + } + return nil +} + +// Compile-time proof that the shared implementation satisfies the interface. +var _ storage.AppServiceStore = (*AppServices)(nil) diff --git a/server/internal/storage/sqlstore/media.go b/server/internal/storage/sqlstore/media.go index 3eec569..c3a9f20 100644 --- a/server/internal/storage/sqlstore/media.go +++ b/server/internal/storage/sqlstore/media.go @@ -172,3 +172,61 @@ func (s *Media) StoreThumbnail(ctx context.Context, t storage.Thumbnail) error { } return nil } + +const selectMediaUsageSQL = ` + SELECT COUNT(*), COALESCE(SUM(file_size), 0) FROM media_repository + WHERE user_id = ? AND media_origin = ? AND pending = ?` + +// MediaUsage implements storage.MediaStore. +// +// It counts every upload at its own size, even when two of them are the +// same bytes and the blob store holds one copy: a quota is about what a +// user asked this server to keep, and how the server keeps it is its own +// business. +func (s *Media) MediaUsage(ctx context.Context, origin, userID string) (storage.MediaUsage, error) { + var u storage.MediaUsage + if err := s.db.QueryRowContext(ctx, s.q(selectMediaUsageSQL), userID, origin, false). + Scan(&u.Count, &u.Bytes); err != nil { + return storage.MediaUsage{}, fmt.Errorf("total the uploads of %s: %w", userID, err) + } + return u, nil +} + +const ( + selectMediaQuotaSQL = `SELECT max_bytes FROM media_quotas WHERE user_id = ?` + + upsertMediaQuotaSQL = ` + INSERT INTO media_quotas (user_id, max_bytes) VALUES (?, ?) + ON CONFLICT (user_id) DO UPDATE SET max_bytes = excluded.max_bytes` + + deleteMediaQuotaSQL = `DELETE FROM media_quotas WHERE user_id = ?` +) + +// MediaQuota implements storage.MediaStore. +func (s *Media) MediaQuota(ctx context.Context, userID string) (int64, bool, error) { + var maxBytes int64 + err := s.db.QueryRowContext(ctx, s.q(selectMediaQuotaSQL), userID).Scan(&maxBytes) + switch { + case errors.Is(err, sql.ErrNoRows): + return 0, false, nil + case err != nil: + return 0, false, fmt.Errorf("read the media quota of %s: %w", userID, err) + } + return maxBytes, true, nil +} + +// SetMediaQuota implements storage.MediaStore. +func (s *Media) SetMediaQuota(ctx context.Context, userID string, maxBytes int64) error { + if _, err := s.db.ExecContext(ctx, s.q(upsertMediaQuotaSQL), userID, maxBytes); err != nil { + return fmt.Errorf("set the media quota of %s: %w", userID, err) + } + return nil +} + +// DeleteMediaQuota implements storage.MediaStore. +func (s *Media) DeleteMediaQuota(ctx context.Context, userID string) error { + if _, err := s.db.ExecContext(ctx, s.q(deleteMediaQuotaSQL), userID); err != nil { + return fmt.Errorf("delete the media quota of %s: %w", userID, err) + } + return nil +} diff --git a/server/internal/storage/sqlstore/sqlstore.go b/server/internal/storage/sqlstore/sqlstore.go index f1882e4..6dcb561 100644 --- a/server/internal/storage/sqlstore/sqlstore.go +++ b/server/internal/storage/sqlstore/sqlstore.go @@ -63,6 +63,8 @@ type Store struct { media *Media keys *Keys push *Push + admin *Admin + apps *AppServices } // New builds a Store. It does not touch the database. @@ -85,6 +87,8 @@ func New(opts Options) (*Store, error) { media: NewMedia(opts.DB, opts.Rebind, opts.IsUniqueViolation), keys: NewKeys(opts.DB, opts.Rebind), push: NewPush(opts.DB, opts.Rebind), + admin: NewAdmin(opts.DB, opts.Rebind, opts.IsUniqueViolation), + apps: NewAppServices(opts.DB, opts.Rebind), }, nil } @@ -124,6 +128,12 @@ func (s *Store) Keys() storage.KeyStore { return s.keys } // Push implements storage.Database. func (s *Store) Push() storage.PushStore { return s.push } +// Admin implements storage.Database. +func (s *Store) Admin() storage.AdminStore { return s.admin } + +// AppServices implements storage.Database. +func (s *Store) AppServices() storage.AppServiceStore { return s.apps } + // Migrate implements storage.Database. func (s *Store) Migrate(ctx context.Context) error { ms, err := s.migrations() diff --git a/server/internal/storage/sqlstore/timeline.go b/server/internal/storage/sqlstore/timeline.go index dbe689d..33b512e 100644 --- a/server/internal/storage/sqlstore/timeline.go +++ b/server/internal/storage/sqlstore/timeline.go @@ -142,6 +142,40 @@ func (s *Rooms) Timeline(ctx context.Context, req storage.TimelineRequest) ([]st return events, nil } +// StreamEvents implements storage.RoomStore. +// +// It reads by stream_ordering alone, which the column's own uniqueness +// index serves. Backfilled history sits below zero and a reader starting at +// a real position never reaches it, which is right for what this is for: +// an application service is told what happens, not what a backfill dug up. +func (s *Rooms) StreamEvents(ctx context.Context, after, upTo int64, limit int) ([]storage.StoredEvent, error) { + if limit <= 0 { + return nil, nil + } + conditions := []string{`e.stream_ordering > ?`, `NOT e.outlier`} + args := []any{after} + if upTo != 0 { + conditions = append(conditions, `e.stream_ordering <= ?`) + args = append(args, upTo) + } + args = append(args, limit) + + query := `SELECT ` + eventColumns + ` + FROM roomserver_events e + WHERE ` + strings.Join(conditions, " AND ") + ` + ORDER BY e.stream_ordering + LIMIT ?` + rows, err := s.db.QueryContext(ctx, s.q(query), args...) + if err != nil { + return nil, fmt.Errorf("read the event stream after %d: %w", after, err) + } + events, err := scanEvents(rows) + if err != nil { + return nil, fmt.Errorf("read the event stream after %d: %w", after, err) + } + return events, nil +} + // filterConditions renders the part of an event filter the store applies in // SQL, as a WHERE fragment and its arguments. // diff --git a/server/internal/storage/sqlstore/users.go b/server/internal/storage/sqlstore/users.go index 47afadd..47cc6db 100644 --- a/server/internal/storage/sqlstore/users.go +++ b/server/internal/storage/sqlstore/users.go @@ -140,6 +140,12 @@ func (u *Users) SetAdmin(ctx context.Context, localpart string, admin bool) erro `UPDATE user_accounts SET admin = ? WHERE localpart = ?`, admin) } +// SetDeactivated implements storage.UserStore. +func (u *Users) SetDeactivated(ctx context.Context, localpart string, deactivated bool) error { + return u.updateAccount(ctx, localpart, + `UPDATE user_accounts SET deactivated = ? WHERE localpart = ?`, deactivated) +} + // SetDisplayName implements storage.UserStore. func (u *Users) SetDisplayName(ctx context.Context, localpart, displayName string) error { return u.updateAccount(ctx, localpart, diff --git a/server/internal/storage/storage.go b/server/internal/storage/storage.go index 1599895..d2cbe0c 100644 --- a/server/internal/storage/storage.go +++ b/server/internal/storage/storage.go @@ -88,6 +88,14 @@ type Database interface { // produce with their per-room unread counts. Push() PushStore + // Admin is the admin API's own state: registration tokens, server + // notice rooms and the audit log. + Admin() AdminStore + + // AppServices is where each application service is in the event + // stream. + AppServices() AppServiceStore + // Close releases the connection pool. Close() error } diff --git a/server/internal/storage/users.go b/server/internal/storage/users.go index 75b2037..e8d0f96 100644 --- a/server/internal/storage/users.go +++ b/server/internal/storage/users.go @@ -164,6 +164,9 @@ type UserStore interface { // SetAdmin grants or revokes the admin flag. SetAdmin(ctx context.Context, localpart string, admin bool) error + // SetDeactivated sets or clears the deactivated flag. + SetDeactivated(ctx context.Context, localpart string, deactivated bool) error + // SetDisplayName updates the profile display name. SetDisplayName(ctx context.Context, localpart, displayName string) error diff --git a/server/internal/userapi/deactivate.go b/server/internal/userapi/deactivate.go new file mode 100644 index 0000000..01ef0c7 --- /dev/null +++ b/server/internal/userapi/deactivate.go @@ -0,0 +1,93 @@ +package userapi + +import "context" + +// AccountObserver is told when an account is deactivated, after the +// account itself has been dealt with. The roomserver's side of a +// deactivation — leaving every room — is done through it, because nothing +// in this package knows about rooms. +// +// Like DeviceObserver it runs on the request path and must not fail the +// deactivation: the account is already deactivated when it is called. +type AccountObserver interface { + AccountDeactivated(ctx context.Context, localpart string) +} + +// SetAccountObserver connects the observer. It is a setter for the same +// reason SetDeviceObserver is: the roomserver is built after the userapi. +func (a *API) SetAccountObserver(o AccountObserver) { a.accounts = o } + +// DeactivateAccount deactivates an account: what both the user's own +// POST /account/deactivate and the admin API's deactivation do. +// +// The order is deliberate. The flag goes first, so that every token the +// account still holds stops working at once, before anything slower runs; +// the profile is erased next, so that the leave events the observer sends +// into the user's rooms already carry no name or avatar; the devices go +// after that, taking their keys and pushers with them through the device +// observer; and the rooms are left last. +// +// The name stays taken. A deactivated account cannot be registered again, +// because a new user with the old user's ID would inherit everything other +// servers and clients remember about the old one. +func (a *API) DeactivateAccount(ctx context.Context, localpart string, erase bool) error { + if err := a.store.SetDeactivated(ctx, localpart, true); err != nil { + return err + } + if erase { + if err := a.store.SetDisplayName(ctx, localpart, ""); err != nil { + return err + } + if err := a.store.SetAvatarURL(ctx, localpart, ""); err != nil { + return err + } + } + if err := a.DeleteAllDevices(ctx, localpart); err != nil { + return err + } + if a.accounts != nil { + a.accounts.AccountDeactivated(ctx, localpart) + } + a.log.Info("deactivated account", "user_id", a.UserID(localpart), "erase", erase) + return nil +} + +// ReactivateAccount clears the deactivated flag, for the admin API. It +// gives back nothing a deactivation took — the rooms are left and the +// devices gone — only the ability to log in again. +func (a *API) ReactivateAccount(ctx context.Context, localpart string) error { + if err := a.store.SetDeactivated(ctx, localpart, false); err != nil { + return err + } + a.log.Info("reactivated account", "user_id", a.UserID(localpart)) + return nil +} + +// ChangePassword sets a new password and, when logoutDevices is set, logs +// out every device of the user except keepDeviceID (empty keeps none). +// +// A user changing their own password keeps the session they changed it +// from; an admin resetting one keeps none, since whoever is locked out of +// the account may be the one holding the sessions. +func (a *API) ChangePassword(ctx context.Context, localpart, password string, logoutDevices bool, keepDeviceID string) error { + if err := a.SetPassword(ctx, localpart, password); err != nil { + return err + } + if !logoutDevices { + return nil + } + if keepDeviceID == "" { + return a.DeleteAllDevices(ctx, localpart) + } + devices, err := a.store.Devices(ctx, localpart) + if err != nil { + return err + } + others := make([]string, 0, len(devices)) + for _, d := range devices { + if d.ID != keepDeviceID { + others = append(others, d.ID) + } + } + return a.DeleteDevices(ctx, localpart, others) +} diff --git a/server/internal/userapi/registrationtokens.go b/server/internal/userapi/registrationtokens.go new file mode 100644 index 0000000..740a1cf --- /dev/null +++ b/server/internal/userapi/registrationtokens.go @@ -0,0 +1,256 @@ +package userapi + +import ( + "context" + "crypto/rand" + "crypto/subtle" + "errors" + "fmt" + + "github.com/dariuszjaniak/warp/server/internal/storage" +) + +// Registration token errors. +var ( + ErrNoSuchRegistrationToken = storage.ErrNoSuchRegistrationToken + ErrRegistrationTokenExists = storage.ErrRegistrationTokenExists + + // ErrInvalidRegistrationToken is a token an admin asked for that does + // not fit the spec's grammar. + ErrInvalidRegistrationToken = errors.New("invalid registration token") +) + +// MaxRegistrationTokenLength is the spec's limit on a registration token. +const MaxRegistrationTokenLength = 64 + +// DefaultRegistrationTokenLength is what a generated token is when the +// admin does not say. +const DefaultRegistrationTokenLength = 16 + +// registrationTokenAlphabet is the spec's character set for a token, less +// the punctuation: a generated token is read off a screen and typed. +const registrationTokenAlphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789" + +// RegistrationTokens answers the m.login.registration_token stage from two +// places: the static tokens of the config file, which never run out, and +// the tokens an admin made through the admin API, which may. +type RegistrationTokens struct { + store storage.AdminStore + static []string + required bool + nowMS func() int64 +} + +// RegistrationTokensOptions configure them. +type RegistrationTokensOptions struct { + // Store holds the tokens made through the admin API. Nil means there + // are none and the admin endpoints cannot make any. + Store storage.AdminStore + + // Static are registration.tokens from the config file. + Static []string + + // Required puts the stage into the registration flow even with no + // static tokens — registration.require_token, for a server whose + // tokens all come from the admin API. + Required bool + + NowMS func() int64 +} + +// NewRegistrationTokens builds them. +func NewRegistrationTokens(opts RegistrationTokensOptions) *RegistrationTokens { + return &RegistrationTokens{ + store: opts.Store, + static: opts.Static, + required: opts.Required || len(opts.Static) > 0, + nowMS: opts.NowMS, + } +} + +// Required reports whether registration needs a token. +func (t *RegistrationTokens) Required() bool { return t.required } + +// Valid reports whether a token would be accepted now. It does not spend +// it: the stage is checked when it is passed and spent when the account is +// created, so a registration abandoned between the two costs no use. +func (t *RegistrationTokens) Valid(ctx context.Context, token string) (bool, error) { + if t.staticMatch(token) { + return true, nil + } + if t.store == nil || !wellFormedToken(token) { + return false, nil + } + stored, err := t.store.RegistrationToken(ctx, token) + switch { + case errors.Is(err, storage.ErrNoSuchRegistrationToken): + return false, nil + case err != nil: + return false, err + } + return stored.Usable(t.nowMS()), nil +} + +// Spend takes one use of a token for an account about to be created, and +// returns the function that gives it back if the creation then fails. A +// static token has no uses to count. A token that has run out or expired +// since the stage was passed is ErrNoSuchRegistrationToken. +func (t *RegistrationTokens) Spend(ctx context.Context, token string) (release func(), err error) { + if t.staticMatch(token) { + return func() {}, nil + } + if t.store == nil { + return nil, ErrNoSuchRegistrationToken + } + if err := t.store.UseRegistrationToken(ctx, token, t.nowMS()); err != nil { + return nil, err + } + return func() { + // The request's context may be what failed; giving the use back + // must not depend on it. + _ = t.store.ReleaseRegistrationToken(context.WithoutCancel(ctx), token) + }, nil +} + +// staticMatch compares a token against every config file token in +// constant time, and does not stop at the first match, so neither the +// value nor the position of a token leaks through the time it takes. +func (t *RegistrationTokens) staticMatch(token string) bool { + found := 0 + for _, want := range t.static { + found |= subtle.ConstantTimeCompare([]byte(token), []byte(want)) + } + return found == 1 +} + +// NewRegistrationToken is what the admin API asks for. +type NewRegistrationToken struct { + // Token is the token to make; empty generates one of Length + // characters. + Token string + Length int + + UsesAllowed *int + ExpiresAtMS int64 +} + +// CreateRegistrationToken makes a token through the admin API. +func (t *RegistrationTokens) CreateRegistrationToken(ctx context.Context, req NewRegistrationToken) (storage.RegistrationToken, error) { + if t.store == nil { + return storage.RegistrationToken{}, storage.ErrNotImplemented + } + token := req.Token + if token == "" { + length := req.Length + if length == 0 { + length = DefaultRegistrationTokenLength + } + if length < 1 || length > MaxRegistrationTokenLength { + return storage.RegistrationToken{}, fmt.Errorf("%w: length must be between 1 and %d", + ErrInvalidRegistrationToken, MaxRegistrationTokenLength) + } + var err error + if token, err = generateRegistrationToken(length); err != nil { + return storage.RegistrationToken{}, err + } + } else if !wellFormedToken(token) { + return storage.RegistrationToken{}, fmt.Errorf( + "%w: a token is 1 to %d of the characters A-Z, a-z, 0-9, '.', '_', '~' and '-'", + ErrInvalidRegistrationToken, MaxRegistrationTokenLength) + } + if req.UsesAllowed != nil && *req.UsesAllowed < 0 { + return storage.RegistrationToken{}, fmt.Errorf("%w: uses_allowed must not be negative", ErrInvalidRegistrationToken) + } + if req.ExpiresAtMS < 0 { + return storage.RegistrationToken{}, fmt.Errorf("%w: expiry_time must not be negative", ErrInvalidRegistrationToken) + } + + stored := storage.RegistrationToken{ + Token: token, + UsesAllowed: req.UsesAllowed, + ExpiresAtMS: req.ExpiresAtMS, + CreatedAtMS: t.nowMS(), + } + if err := t.store.CreateRegistrationToken(ctx, stored); err != nil { + return storage.RegistrationToken{}, err + } + return stored, nil +} + +// RegistrationToken reads one token made through the admin API. +func (t *RegistrationTokens) RegistrationToken(ctx context.Context, token string) (storage.RegistrationToken, error) { + if t.store == nil { + return storage.RegistrationToken{}, ErrNoSuchRegistrationToken + } + return t.store.RegistrationToken(ctx, token) +} + +// List lists the tokens made through the admin API. The config file's are +// not listed: they are in the config file. +func (t *RegistrationTokens) List(ctx context.Context) ([]storage.RegistrationToken, error) { + if t.store == nil { + return []storage.RegistrationToken{}, nil + } + return t.store.RegistrationTokens(ctx) +} + +// Update replaces a token's use limit and expiry. +func (t *RegistrationTokens) Update(ctx context.Context, token string, usesAllowed *int, expiresAtMS int64) (storage.RegistrationToken, error) { + if t.store == nil { + return storage.RegistrationToken{}, ErrNoSuchRegistrationToken + } + if usesAllowed != nil && *usesAllowed < 0 { + return storage.RegistrationToken{}, fmt.Errorf("%w: uses_allowed must not be negative", ErrInvalidRegistrationToken) + } + if expiresAtMS < 0 { + return storage.RegistrationToken{}, fmt.Errorf("%w: expiry_time must not be negative", ErrInvalidRegistrationToken) + } + if err := t.store.UpdateRegistrationToken(ctx, token, usesAllowed, expiresAtMS); err != nil { + return storage.RegistrationToken{}, err + } + return t.store.RegistrationToken(ctx, token) +} + +// Delete removes a token. Accounts registered with it are not affected. +func (t *RegistrationTokens) Delete(ctx context.Context, token string) error { + if t.store == nil { + return ErrNoSuchRegistrationToken + } + return t.store.DeleteRegistrationToken(ctx, token) +} + +// wellFormedToken applies the spec's grammar: 1 to 64 characters from +// [A-Za-z0-9._~-]. +func wellFormedToken(token string) bool { + if token == "" || len(token) > MaxRegistrationTokenLength { + return false + } + for i := range len(token) { + switch c := token[i]; { + case 'A' <= c && c <= 'Z', 'a' <= c && c <= 'z', '0' <= c && c <= '9': + case c == '.', c == '_', c == '~', c == '-': + default: + return false + } + } + return true +} + +// generateRegistrationToken makes a random token. Bytes that would bias +// the modulo are thrown away, as for media IDs. +func generateRegistrationToken(length int) (string, error) { + const accept = 256 - 256%len(registrationTokenAlphabet) + out := make([]byte, 0, length) + buf := make([]byte, length) + for len(out) < length { + if _, err := rand.Read(buf); err != nil { + return "", fmt.Errorf("generate a registration token: %w", err) + } + for _, c := range buf { + if int(c) < accept && len(out) < length { + out = append(out, registrationTokenAlphabet[int(c)%len(registrationTokenAlphabet)]) + } + } + } + return string(out), nil +} diff --git a/server/internal/userapi/userapi.go b/server/internal/userapi/userapi.go index 603fb41..fcc2492 100644 --- a/server/internal/userapi/userapi.go +++ b/server/internal/userapi/userapi.go @@ -72,6 +72,9 @@ type API struct { // observer hears about deleted and renamed devices; nil hears nothing. observer DeviceObserver + + // accounts hears about deactivated accounts; nil hears nothing. + accounts AccountObserver } // DeviceObserver is told about changes to devices that something else @@ -393,6 +396,12 @@ type Authenticated struct { Localpart string DeviceID string Admin bool + + // AppService is the ID of the application service making the request, + // acting as UserID, or empty for an ordinary access token. A request + // made with an application service token has no device: DeviceID is + // empty. + AppService string } // Authenticate resolves an access token. diff --git a/server/test/complement/README.md b/server/test/complement/README.md index 7ba0a1d..7675421 100644 --- a/server/test/complement/README.md +++ b/server/test/complement/README.md @@ -44,8 +44,14 @@ The switch reaches the container through `COMPLEMENT_SHARE_ENV_PREFIX`: as `WARP_FEDERATION_ENABLED`. The federation set leaves out partial-state joins (MSC3706, -`TestPartialStateJoin*`), which Warp does not implement, and anything that -needs application services (Phase 9). +`TestPartialStateJoin*`), which Warp does not implement. + +Application services arrived in Phase 9. Complement writes the +registration file of each service a blueprint asks for into +`/complement/appservice`, and `entrypoint.sh` hands every file there to +`WARP_APPSERVICE_CONFIG_FILES`. The tests that use them, and the Phase 9 +account tests (password change, deactivation), are not in either set yet. +They join once a run has shown which pass. ## The test set diff --git a/server/test/complement/entrypoint.sh b/server/test/complement/entrypoint.sh index 05798cb..4b9b581 100755 --- a/server/test/complement/entrypoint.sh +++ b/server/test/complement/entrypoint.sh @@ -59,6 +59,21 @@ if [ -f "$CA_DIR/ca.crt" ] && [ -f "$CA_DIR/ca.key" ]; then export WARP_FEDERATION_TLS_CA_FILE="$CA_DIR/ca.crt" fi +# Application services. Complement writes the registration file of every +# service a blueprint asks for into /complement/appservice; Warp reads +# them at startup like any other registration file. +AS_DIR=/complement/appservice +if [ -d "$AS_DIR" ]; then + files="" + for f in "$AS_DIR"/*.yaml; do + [ -f "$f" ] || continue + files="${files:+$files,}$f" + done + if [ -n "$files" ]; then + export WARP_APPSERVICE_CONFIG_FILES="$files" + fi +fi + # Complement captures the container's output, so text at debug level is the # most useful thing to have when a test fails. export WARP_LOG_LEVEL="${WARP_LOG_LEVEL:-debug}" diff --git a/warp.yaml.example b/warp.yaml.example index b43d2dc..a801c9d 100644 --- a/warp.yaml.example +++ b/warp.yaml.example @@ -194,6 +194,11 @@ media: # WARP_MEDIA_MAX_UPLOAD_BYTES max_upload_bytes: 52428800 + # The most one user may have uploaded in all; 0 = no limit. The admin + # API can give a user a quota of their own. + # WARP_MEDIA_MAX_USER_STORAGE_BYTES + max_user_storage_bytes: 0 + # The largest image, in pixels, decoded to make a thumbnail. Decoding # costs four bytes a pixel however small the file is. # WARP_MEDIA_MAX_THUMBNAIL_PIXELS @@ -248,12 +253,39 @@ registration: enabled: false # WARP_REGISTRATION_ENABLED allow_runtime_toggle: true # WARP_REGISTRATION_ALLOW_RUNTIME_TOGGLE - # Non-empty = registration also requires one of these tokens, as the - # m.login.registration_token stage of the sign-up flow. Per-token use - # counts and expiry arrive with admin token management in Phase 9. - # WARP_REGISTRATION_TOKENS + # Non-empty = registration also requires a token, as the + # m.login.registration_token stage of the sign-up flow. These never run + # out; tokens with a use limit or an expiry are made through the admin + # API (POST /_warp/admin/v1/registration_tokens), and both kinds are + # accepted. WARP_REGISTRATION_TOKENS tokens: [] + # Ask for a token even with none listed above, for a server whose + # tokens all come from the admin API. WARP_REGISTRATION_REQUIRE_TOKEN + require_token: false + +appservice: + # Application service registration files: the YAML every bridge + # writes. Read at startup; a bad one stops the server. See + # docs/appservices.md. WARP_APPSERVICE_CONFIG_FILES + config_files: [] + + # One request to a service: a transaction, a query or a ping. + # WARP_APPSERVICE_TIMEOUT + timeout: 30s + +server_notices: + # Messages from the server to its users, sent through the admin API + # (POST /_warp/admin/v1/server_notices). Needs a signing key. + # WARP_SERVER_NOTICES_ENABLED + enabled: false + + # The user notices come from, created at startup if it does not exist — + # pick a name no real user has. WARP_SERVER_NOTICES_LOCALPART + localpart: server-notices + display_name: Server Notices # WARP_SERVER_NOTICES_DISPLAY_NAME + room_name: Server Notices # WARP_SERVER_NOTICES_ROOM_NAME + logging: level: info # debug, info, warn, error WARP_LOG_LEVEL format: text # text or json WARP_LOG_FORMAT