Warp/docs/appservices.md

# Application services

`server/internal/appservice`, and the Client-Server half of it in
`server/internal/clientapi/appservice.go`: PLAN.md Phase 9. An application
service is a bridge or a bot with a namespace of its own: users, aliases
and rooms it may act for, and the events about them delivered to it as
they happen. This is the
[Application Service API](https://spec.matrix.org/latest/application-service-api/).

| Package | What |
|---|---|
| `appservice` | Registration files, namespace matching, the registry, the HTTP client that talks to services, and the sender that delivers events to them. |
| `clientapi` | A service's token as an access token, acting as its users with `?user_id=`, registering and logging in its users, the `M_EXCLUSIVE` refusals, the ping endpoint, and asking services about users and aliases that do not exist. |

## Registering a service

Every bridge writes a registration file. Point Warp at it:

```yaml
appservice:
  config_files:
    - /etc/warp/appservices/telegram.yaml
  timeout: 30s
```

or `WARP_APPSERVICE_CONFIG_FILES=/etc/warp/appservices/telegram.yaml`.
Files are read at startup, and a file that does not parse stops the server
from starting, as a bad config file does. Adding or changing one needs a
restart. The admin API shows what is registered (`GET /appservices`) but
does not edit it.

```yaml
id: telegram
url: http://localhost:29317       # null: the service is sent nothing
as_token: <random>                # what the service authenticates with
hs_token: <random>                # what Warp authenticates to the service with
sender_localpart: telegrambot
rate_limited: false               # absent means true
namespaces:
  users:
    - exclusive: true
      regex: '@telegram_.*:warp\.example'
  aliases:
    - exclusive: true
      regex: '#telegram_.*:warp\.example'
  rooms: []
```

What is checked:

- `id`, `as_token`, `hs_token` and `sender_localpart` are required, and the
  sender has to be a valid localpart.
- `as_token` and `hs_token` must differ. No two services may share an `id`
  or an `as_token`, and no service's `as_token` may be another service's
  `hs_token`.
- `url` is empty, `null` or an http(s) URL.
- Every namespace regex compiles.

Keys Warp does not know are ignored rather than refused. This is unlike
the main config file. Bridges add their own keys and unstable ones
(`push_ephemeral`, the MSC3202 keys), and a bridge upgrade must not stop
the homeserver.

The sender user is created at startup if it does not exist. It has no
password.

### How namespaces match

A namespace regex is anchored at the start and not at the end, the way
Synapse matches. Registration files in the wild are written against
Synapse. `@_irc_` matches every user ID that begins with it. `@bot` does
not match `@notabot:…`. Anchoring both ends would quietly match nothing
for many existing files.

Namespaces match full IDs: `@localpart:server` for users,
`#alias:server` for aliases, `!id:server` for rooms.

## A service acting as its users

A request carrying a service's `as_token` acts as the service's sender.
With `?user_id=` it acts as that user instead. The user has to be:

- the sender, or in one of the service's user namespaces (exclusive or
  not, as the spec says)
- a user that exists — a service registers its users first

Otherwise the answer is `403 M_FORBIDDEN`. A deactivated user is
`403 M_USER_DEACTIVATED`.

Such a request has no device. `device_id` in `/whoami` is empty, and there
is no device to record activity on. A service that needs devices for its
users (for end-to-end encryption, say) logs them in with
`m.login.application_service` and uses their access tokens.

### Rate limiting

A service's own requests are never rate limited. Requests it makes as its
users are exempt when its registration says `rate_limited: false`.
Without that, a bridge puppeting a thousand users from one address would
put all of them in one bucket. The limiter runs before the token is
resolved, so the exemption is a map lookup on the token. It needs no
database query.

### Registering and logging in its users

```http
POST /_matrix/client/v3/register
Authorization: Bearer <as_token>

{"type": "m.login.application_service", "username": "telegram_12345"}
```

The user must be in the service's namespace (`400 M_EXCLUSIVE` otherwise).
There is no flow and no password, and the registration switch does not
apply: this is the service's own action, not a sign-up. `inhibit_login`
is honoured. Without it, the answer carries an access token for a new
device.

`POST /login` with `"type": "m.login.application_service"` and the
service's token logs in one of its users and returns a device. The flow
is listed by `GET /login` while any service is registered.

The service's users can change their password and deactivate their
account without the password re-authentication other users go through.
They have no password to re-enter, and the service's token is already the
strongest credential involved.

### Exclusive namespaces

A namespace marked `exclusive` belongs to its service alone:

| Who | What | Answer |
|---|---|---|
| anyone else | registering a user in it, or asking `/register/available` | `400 M_EXCLUSIVE` |
| anyone else | `PUT /directory/room/{alias}`, or `createRoom` with a `room_alias_name` that lands in it | `400 M_EXCLUSIVE` |
| the service | any of the above | allowed |

A service's sender counts as claimed by it, even outside its namespaces.

Exclusive room namespaces are not enforced. Nothing in the
Client-Server API creates a room with a chosen ID.

## Delivering events

The sender reads the roomserver's event stream from a cursor kept per
service in `appservice_state` (migration 0011). Events are not queued as
they are persisted. The stream is already durable and ordered, so a cursor
is all a service needs to catch up after being down for a day. The emit
stage only wakes the sender's workers; it hands nothing over, and it does
no work under the room lock.

One worker per service with a URL:

1. Read up to 100 events after the cursor.
2. Keep the ones that concern the service (see below).
3. If none do, move the cursor past them. Otherwise record a transaction
   as in flight (its ID and the top of its events), and
   `PUT /_matrix/app/v1/transactions/{txnId}` until the service accepts
   it.
4. Move the cursor and clear the transaction.

A transaction that fails is retried with the same ID and the same events.
The backoff doubles from 2 seconds to 5 minutes and stays there. A service
that answers a ping has its waiting transaction sent at once.

After a restart, a transaction left in flight is sent again with its old
ID. Its events are read again from the stream range it covered, so the
service can deduplicate by ID as the spec intends. One caveat: whether
events concern the service is decided again on re-read. A room the
service's users left in the meantime can make the resent transaction
smaller than the first attempt. Within one run, retries reuse the exact
same body.

A service seen for the first time starts at the top of the stream. It is
told what happens from then on, not the server's history. Changing a
service's `id` starts it over in the same way.

### Which events concern a service

An event is delivered when any of these hold:

- its sender is one of the service's users (its sender or a user
  namespace match)
- it is an `m.room.member` event whose target is one of them
- its room ID is in a room namespace
- its room has one of the service's users joined to it
- its room has one of the service's aliases pointing at it

The last two are what let a bridge see the conversations its puppets are
in. They are read once per room per batch, as the room is when the batch
is read. An event read late, from a backlog, is judged by who is in the
room then, not when the event happened.

Events are in the client format, with `room_id`, as the spec has it.
Backfilled history (negative stream positions) is never delivered.

The request carries `Authorization: Bearer <hs_token>`. The deprecated
`?access_token=` form is not sent: it was deprecated in v1.4, and a token
in a URL ends up in the service's access log. Redirects are not followed,
because they would take the `hs_token` with them.

## Asking services about users and aliases

When something needs a local user or alias that does not exist, the
services whose namespaces match are asked first. They answer by creating
it, and Warp looks again:

| When | Query |
|---|---|
| `GET /directory/room/{alias}` and `POST /join/{alias}` for an unknown local alias | `GET /_matrix/app/v1/rooms/{alias}` |
| `GET /profile/{userId}` and `POST /rooms/{id}/invite` for an unknown local user | `GET /_matrix/app/v1/users/{userId}` |

This is how a bridge creates a portal room the first time somebody looks
for it, and a puppet the first time somebody invites it. Services are
asked in registration order; the first yes ends it. A service that fails
to answer counts as a no and is logged. The request that wanted the user
goes on without it rather than failing.

## The ping

`POST /_matrix/client/v1/appservice/{appserviceId}/ping` is a service
asking Warp to ping it back. It is authenticated with the service's own
token, and `{appserviceId}` has to be that service. Warp sends
`POST /_matrix/app/v1/ping` with the `transaction_id`, and answers:

| Outcome | Answer |
|---|---|
| the service answered 2xx | `200 {"duration_ms": …}` |
| no URL registered | `400 M_URL_NOT_SET` |
| the service answered something else | `502 M_BAD_STATUS`, with its `status` and `body` |
| it could not be reached | `502 M_CONNECTION_FAILED` |
| it did not answer within `appservice.timeout` | `504 M_CONNECTION_TIMEOUT` |
| another token | `403 M_FORBIDDEN` |

## Watching delivery

`GET /_warp/admin/v1/appservices` lists each service: its namespaces,
sender and URL, never its tokens. For a service with a URL it also shows
the cursor, how far behind the top of the stream it is, the last
transaction ID, the events delivered since startup, and the failure
count, last error and next retry while it is failing. See
[admin-api.md](admin-api.md#get-appservices).

## Not implemented

| Thing | Why |
|---|---|
| Ephemeral events and to-device messages in transactions (MSC2409, MSC3202) | unstable; bridges that want them fall back to `/sync` as a user |
| Device masquerading with `?device_id=` (MSC3202) | as above; log the user in instead |
| Third-party lookups (`/thirdparty/*`) | needs the protocol metadata exchange, which no first-party client uses |
| Network-specific room directories (`PUT /directory/list/appservice/{networkId}/{roomId}`) | not served |
| Appservice key claims and queries (MSC3983, MSC3984) | unstable |
| Editing registrations at runtime | registrations are files; restart to change them |