Warp/docs/appservices.md

Application services

server/internal/appservice, and the Client-Server half of it in server/internal/clientapi/appservice.go: PLAN.md Phase 9. An application service is a bridge or a bot with a namespace of its own: users, aliases and rooms it may act for, and the events about them delivered to it as they happen. This is the Application Service API.

Package	What
appservice	Registration files, namespace matching, the registry, the HTTP client that talks to services, and the sender that delivers events to them.
clientapi	A service's token as an access token, acting as its users with ?user_id=, registering and logging in its users, the M_EXCLUSIVE refusals, the ping endpoint, and asking services about users and aliases that do not exist.

Registering a service

Every bridge writes a registration file. Point Warp at it:

appservice:
  config_files:
    - /etc/warp/appservices/telegram.yaml
  timeout: 30s

or WARP_APPSERVICE_CONFIG_FILES=/etc/warp/appservices/telegram.yaml. Files are read at startup, and a file that does not parse stops the server from starting, as a bad config file does. Adding or changing one needs a restart. The admin API shows what is registered (GET /appservices) but does not edit it.

id: telegram
url: http://localhost:29317       # null: the service is sent nothing
as_token: <random>                # what the service authenticates with
hs_token: <random>                # what Warp authenticates to the service with
sender_localpart: telegrambot
rate_limited: false               # absent means true
namespaces:
  users:
    - exclusive: true
      regex: '@telegram_.*:warp\.example'
  aliases:
    - exclusive: true
      regex: '#telegram_.*:warp\.example'
  rooms: []

What is checked:

• id, as_token, hs_token and sender_localpart are required, and the sender has to be a valid localpart.
• as_token and hs_token must differ. No two services may share an id or an as_token, and no service's as_token may be another service's hs_token.
• url is empty, null or an http(s) URL.
• Every namespace regex compiles.

Keys Warp does not know are ignored rather than refused. This is unlike the main config file. Bridges add their own keys and unstable ones (push_ephemeral, the MSC3202 keys), and a bridge upgrade must not stop the homeserver.

The sender user is created at startup if it does not exist. It has no password.

How namespaces match

A namespace regex is anchored at the start and not at the end, the way Synapse matches. Registration files in the wild are written against Synapse. @_irc_ matches every user ID that begins with it. @bot does not match @notabot:…. Anchoring both ends would quietly match nothing for many existing files.

Namespaces match full IDs: @localpart:server for users, #alias:server for aliases, !id:server for rooms.

A service acting as its users

A request carrying a service's as_token acts as the service's sender. With ?user_id= it acts as that user instead. The user has to be:

• the sender, or in one of the service's user namespaces (exclusive or not, as the spec says)
• a user that exists — a service registers its users first

Otherwise the answer is 403 M_FORBIDDEN. A deactivated user is 403 M_USER_DEACTIVATED.

Such a request has no device. device_id in /whoami is empty, and there is no device to record activity on. A service that needs devices for its users (for end-to-end encryption, say) logs them in with m.login.application_service and uses their access tokens.

Rate limiting

A service's own requests are never rate limited. Requests it makes as its users are exempt when its registration says rate_limited: false. Without that, a bridge puppeting a thousand users from one address would put all of them in one bucket. The limiter runs before the token is resolved, so the exemption is a map lookup on the token. It needs no database query.

Registering and logging in its users

POST /_matrix/client/v3/register
Authorization: Bearer <as_token>

{"type": "m.login.application_service", "username": "telegram_12345"}

The user must be in the service's namespace (400 M_EXCLUSIVE otherwise). There is no flow and no password, and the registration switch does not apply: this is the service's own action, not a sign-up. inhibit_login is honoured. Without it, the answer carries an access token for a new device.

POST /login with "type": "m.login.application_service" and the service's token logs in one of its users and returns a device. The flow is listed by GET /login while any service is registered.

The service's users can change their password and deactivate their account without the password re-authentication other users go through. They have no password to re-enter, and the service's token is already the strongest credential involved.

Exclusive namespaces

A namespace marked exclusive belongs to its service alone:

Who	What	Answer
anyone else	registering a user in it, or asking /register/available	400 M_EXCLUSIVE
anyone else	PUT /directory/room/{alias}, or createRoom with a room_alias_name that lands in it	400 M_EXCLUSIVE
the service	any of the above	allowed

A service's sender counts as claimed by it, even outside its namespaces.

Exclusive room namespaces are not enforced. Nothing in the Client-Server API creates a room with a chosen ID.

Delivering events

The sender reads the roomserver's event stream from a cursor kept per service in appservice_state (migration 0011). Events are not queued as they are persisted. The stream is already durable and ordered, so a cursor is all a service needs to catch up after being down for a day. The emit stage only wakes the sender's workers; it hands nothing over, and it does no work under the room lock.

One worker per service with a URL:

1. Read up to 100 events after the cursor.
2. Keep the ones that concern the service (see below).
3. If none do, move the cursor past them. Otherwise record a transaction as in flight (its ID and the top of its events), and PUT /_matrix/app/v1/transactions/{txnId} until the service accepts it.
4. Move the cursor and clear the transaction.

A transaction that fails is retried with the same ID and the same events. The backoff doubles from 2 seconds to 5 minutes and stays there. A service that answers a ping has its waiting transaction sent at once.

After a restart, a transaction left in flight is sent again with its old ID. Its events are read again from the stream range it covered, so the service can deduplicate by ID as the spec intends. One caveat: whether events concern the service is decided again on re-read. A room the service's users left in the meantime can make the resent transaction smaller than the first attempt. Within one run, retries reuse the exact same body.

A service seen for the first time starts at the top of the stream. It is told what happens from then on, not the server's history. Changing a service's id starts it over in the same way.

Which events concern a service

An event is delivered when any of these hold:

• its sender is one of the service's users (its sender or a user namespace match)
• it is an m.room.member event whose target is one of them
• its room ID is in a room namespace
• its room has one of the service's users joined to it
• its room has one of the service's aliases pointing at it

The last two are what let a bridge see the conversations its puppets are in. They are read once per room per batch, as the room is when the batch is read. An event read late, from a backlog, is judged by who is in the room then, not when the event happened.

Events are in the client format, with room_id, as the spec has it. Backfilled history (negative stream positions) is never delivered.

The request carries Authorization: Bearer <hs_token>. The deprecated ?access_token= form is not sent: it was deprecated in v1.4, and a token in a URL ends up in the service's access log. Redirects are not followed, because they would take the hs_token with them.

Asking services about users and aliases

When something needs a local user or alias that does not exist, the services whose namespaces match are asked first. They answer by creating it, and Warp looks again:

When	Query
GET /directory/room/{alias} and POST /join/{alias} for an unknown local alias	GET /_matrix/app/v1/rooms/{alias}
GET /profile/{userId} and POST /rooms/{id}/invite for an unknown local user	GET /_matrix/app/v1/users/{userId}

This is how a bridge creates a portal room the first time somebody looks for it, and a puppet the first time somebody invites it. Services are asked in registration order; the first yes ends it. A service that fails to answer counts as a no and is logged. The request that wanted the user goes on without it rather than failing.

The ping

POST /_matrix/client/v1/appservice/{appserviceId}/ping is a service asking Warp to ping it back. It is authenticated with the service's own token, and {appserviceId} has to be that service. Warp sends POST /_matrix/app/v1/ping with the transaction_id, and answers:

Outcome	Answer
the service answered 2xx	200 {"duration_ms": …}
no URL registered	400 M_URL_NOT_SET
the service answered something else	502 M_BAD_STATUS, with its status and body
it could not be reached	502 M_CONNECTION_FAILED
it did not answer within appservice.timeout	504 M_CONNECTION_TIMEOUT
another token	403 M_FORBIDDEN

Watching delivery

GET /_warp/admin/v1/appservices lists each service: its namespaces, sender and URL, never its tokens. For a service with a URL it also shows the cursor, how far behind the top of the stream it is, the last transaction ID, the events delivered since startup, and the failure count, last error and next retry while it is failing. See admin-api.md.

Not implemented

Thing	Why
Ephemeral events and to-device messages in transactions (MSC2409, MSC3202)	unstable; bridges that want them fall back to /sync as a user
Device masquerading with ?device_id= (MSC3202)	as above; log the user in instead
Third-party lookups (/thirdparty/*)	needs the protocol metadata exchange, which no first-party client uses
Network-specific room directories (PUT /directory/list/appservice/{networkId}/{roomId})	not served
Appservice key claims and queries (MSC3983, MSC3984)	unstable
Editing registrations at runtime	registrations are files; restart to change them