Warp/docs/e2ee.md

# End-to-end encryption

`server/internal/keyserver` is the server half of end-to-end encryption:
everything a homeserver keeps so that its users' devices can encrypt to
each other, none of which it can read or use itself. Phase 7 of
[PLAN.md](../PLAN.md).

The client half — Olm, Megolm, verification — is not here and never will
be. Warp's own clients use `matrix-sdk-crypto-wasm` in the browser and
`goolm` on the desktop; the server only has to publish keys, hand out each
one-time key once, and tell devices when somebody's keys changed.

## What is stored

All in migration `0008_keyserver.sql`, keyed by full user ID rather than
localpart, because signatures and the device list stream are about other
servers' users too.

| Table | What |
|---|---|
| `keyserver_device_keys` | each device's identity keys, as the device uploaded and signed them, without `unsigned` |
| `keyserver_one_time_keys` | unclaimed one-time keys; a row is deleted the moment it is claimed |
| `keyserver_fallback_keys` | one per algorithm per device, marked used when claimed and kept until replaced |
| `keyserver_cross_signing_keys` | a user's master, self-signing and user-signing keys |
| `keyserver_signatures` | signatures uploaded after the key they sign, merged in when the key is read |
| `keyserver_device_lists` | each user's latest position in the device list stream, local and remote |
| `keyserver_backup_versions`, `keyserver_backup_keys` | server-side key backup |
| `keyserver_sequences` | the device list stream's counter |

Keys are stored in canonical JSON. That is what lets "the same key
uploaded again" be a byte comparison: a client that re-sends a one-time key
it already uploaded, encoded differently, is not told it conflicts.

Signatures added through `/keys/signatures/upload` are kept apart from the
key they sign rather than written into it, so a key's own JSON is always
exactly what its owner uploaded.

## Endpoints

| Endpoint | Notes |
|---|---|
| `POST /keys/upload` | device keys must be for the caller's own device; one-time and fallback keys (`fallback_keys`, or MSC2732's unstable name); answers the one-time key counts, `signed_curve25519` always included |
| `POST /keys/query` | local users from the store, remote users asked of their servers in parallel; `failures` for a server that did not answer |
| `POST /keys/claim` | a one-time key if there is one, the fallback key otherwise; remote users claimed from their servers |
| `GET /keys/changes` | whose device lists changed between two sync tokens |
| `POST /keys/device_signing/upload` | cross-signing keys; asks for the password, except where the spec says not to |
| `POST /keys/signatures/upload` | the three kinds of signature the spec describes; per-key `failures` |
| `POST`/`GET /room_keys/version` · `GET`/`PUT`/`DELETE /room_keys/version/{v}` | backup versions |
| `PUT`/`GET`/`DELETE /room_keys/keys[/{room}[/{session}]]?version=` | backed-up room keys at the three granularities |
| `PUT /sendToDevice/{type}/{txn}` | to-device messaging — the syncapi's, from Phase 4; see [sync.md](sync.md) |

Federation (behind the gate, like every federation endpoint):

| Endpoint | Notes |
|---|---|
| `POST /_matrix/federation/v1/user/keys/query` | local users only, without device display names |
| `POST /_matrix/federation/v1/user/keys/claim` | local users only |
| `GET /_matrix/federation/v1/user/devices/{userId}` | a local user's whole device list with its stream ID, for a server whose updates do not add up |

## What the server checks

It cannot tell a good key from a bad one, and does not try to vouch for
any. What it refuses is what is plainly wrong:

- **Device keys** for a user or device other than the one the access token
  belongs to. Whether they are signed by the key they say is not checked:
  every client that reads them checks that, which is the check that
  matters. Synapse and Dendrite make the same choice.
- **One-time keys** without an `algorithm:id` key ID, more than 1000 in one
  upload, or under an ID that already holds a different key
  (`M_INVALID_PARAM`).
- **Cross-signing keys** that are not the uploader's, do not have the usage
  they are uploaded as, or hold anything but one ed25519 key named by its
  own public key — and a self-signing or user-signing key the master key
  did not sign (`M_INVALID_SIGNATURE`). A new master key on its own drops
  the self-signing and user-signing keys the old one signed: they would go
  on being served, signed by a key that is no longer the user's.
- **Signatures** that do not verify against the key *as stored*. A client
  sending a modified copy with a valid signature over the modification has
  signed nothing that is here.

### When cross-signing needs a password

`/keys/device_signing/upload` goes through user-interactive authentication
(`m.login.password`) except in the two cases the spec exempts: the user has
no master key yet, or every key in the request is the one already stored.
Replacing a master key is what would let a stolen access token pass as the
user to everyone who has verified them.

### Signatures and who sees them

`/keys/signatures/upload` accepts three kinds:

| Target | Signed by | Seen by |
|---|---|---|
| one of the uploader's devices | their self-signing key | everybody |
| the uploader's master key | one of their devices | everybody |
| another user's master key | the uploader's user-signing key | **the uploader only** |

Who a user has verified is between them and their own devices, so the third
kind is returned only to the user who made it — and never sent over
federation. Signatures of anything else are ignored rather than refused:
clients send the whole signed object back, the device's own signature
included. A user-signing key is likewise only ever shown to its owner.

## Device list tracking

A client encrypting for a room needs the keys of every device of every
member, and caches them. `/sync`'s `device_lists` says when the cache is
stale:

- **changed**: a user sharing a room with the syncing user uploaded or
  deleted a device's keys, changed a device's name or their cross-signing
  keys, or gained a signature — or has only just come to share a room. The
  syncing user's own ID is in it when their own devices change, so their
  other devices start encrypting to a new one.
- **left**: a user the syncing user no longer shares any room with.

The keyserver writes the **device list stream**: every write that changes
what `/keys/query` would answer moves the user's position, in the same
transaction as the change, and wakes the long polls of everyone sharing a
room with them. The stream's position is the sixth field of a sync token
(`s…_…_…_…_…_{device_list}`; older tokens without it still parse). The
syncapi works out which moves concern a sync
(`internal/syncapi/devicelists.go`):

1. moves in the window of users who share a room with the syncer, the
   syncer included;
2. for every room whose membership of the syncer moved in the window, its
   joined members — all new to share with on a join, perhaps no longer
   shared with on a leave;
3. everybody else whose membership moved to `join`, `leave` or `ban` in a
   room the syncer is or was in.

Anyone in 2 or 3 who still shares a room is *changed*, anyone who does not
is *left*. Invites and knocks are skipped: telling a client to forget the
keys of a user it was encrypting for as an invitee would have it stop.
Both lists are empty on an initial sync, which is the client's cue to fetch
every key it needs anyway.

`device_one_time_keys_count` and `device_unused_fallback_key_types` ride on
every sync. The fallback list is sent as `[]` when empty, never left out:
a client reads its absence as a server without fallback keys, and never
uploads one.

### Over federation

| EDU | Out | In |
|---|---|---|
| `m.device_list_update` | one per device change, to every server sharing a room with the user, with the keys or `deleted: true`, a `stream_id` and the `prev_id` before it | moves the remote user's position in the device list stream |
| `m.signing_key_update` | when a user's master or self-signing key changes, or the master key gains a device's signature | likewise |

Each device of a multi-device deletion gets a stream ID of its own:
remote servers treat a stream ID they have seen as a duplicate, so three
deletions sharing one would reach them as one deletion and two repeats.

Warp does **not cache other servers' keys**. A query about a remote user is
answered by asking their server, so an incoming update has nothing to
invalidate; what it does is tell local clients, who do cache, by moving the
user in the device list stream. That only happens when a local user shares
a room with them: an update about a stranger concerns nobody here, and
recording it would let any server grow the table with users of its own
invention.

Device display names are not sent over federation, in answers or in EDUs.
Local users see them in `unsigned.device_display_name`.

With federation off, remote users in a query or claim are reported in
`failures` straight away; nothing is sent.

## Key backup

The server keeps what it is given and decides one thing: when a key for a
session that is already backed up arrives, which to keep. The spec's rule,
in `storage.BackupKey.Replaces`: a verified key beats an unverified one,
then the lower `first_message_index` wins, then the lower
`forwarded_count`. The `etag` moves only when a key actually changed.

Versions are numbers, per user, and a deleted version keeps its row so its
number is never handed out again: a client still holding the old number
must not find itself writing into a new backup encrypted to a different
key. Writing to a version that exists but is not the current one is
`403 M_WRONG_ROOM_KEYS_VERSION` with `current_version`; to one that does
not exist, `404`. A version's algorithm cannot be changed.

## What is deliberately missing

| Thing | Why |
|---|---|
| A cache of remote users' keys | Every query asks their server. Correct, and slower for a client in rooms with many remote users; a cache invalidated by the EDUs above is the obvious next step. |
| Resync on a gap in a remote server's `prev_id` | Follows from the above: with nothing cached, a missed update costs nothing. |
| Dehydrated devices (MSC3814) | — |
| Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984) | application services are Phase 9 |
| Purging a deleted device's to-device queue | the messages stay queued; a new device later logged in under the same ID is handed them, encrypted to the old device's keys and so unreadable to it |
| Deleting a device's keys in the same transaction as the device | the two are separate stores; keys left behind by a crash between them are never served, because a query reads the device list first |

## Testing

| Where | What |
|---|---|
| `internal/storage/keys_test.go` | the key store against every driver: the device list stream, claims, fallback keys, signatures, the backup replacement rule |
| `internal/keyserver/keyserver_test.go` | the checks, with real ed25519 signatures: uploads, conflicts, cross-signing, the three signature kinds and their visibility, deletions, remote updates, backups |
| `internal/setup/phase7_test.go` | through the whole HTTP stack: upload, sync, query, claim, `keys/changes`, cross-signing with and without UIA, backup, device deletion, leaving a room; and two federated servers exchanging keys and device list updates |
| Complement | `TestUploadKey*`, `TestKeyClaimOrdering`, `TestKeyChanges*`, `TestDeviceListUpdates*`, `TestE2EKeyBackup*`, `TestFallbackKeys*` and `TestCrossSigning*` are in the target sets |