Warp/docs/e2ee.md

End-to-end encryption

server/internal/keyserver is the server half of end-to-end encryption: everything a homeserver keeps so that its users' devices can encrypt to each other, none of which it can read or use itself. Phase 7 of PLAN.md.

The client half — Olm, Megolm, verification — is not here and never will be. Warp's own clients use matrix-sdk-crypto-wasm in the browser and goolm on the desktop; the server only has to publish keys, hand out each one-time key once, and tell devices when somebody's keys changed.

What is stored

All in migration 0008_keyserver.sql, keyed by full user ID rather than localpart, because signatures and the device list stream are about other servers' users too.

Table	What
keyserver_device_keys	each device's identity keys, as the device uploaded and signed them, without unsigned
keyserver_one_time_keys	unclaimed one-time keys; a row is deleted the moment it is claimed
keyserver_fallback_keys	one per algorithm per device, marked used when claimed and kept until replaced
keyserver_cross_signing_keys	a user's master, self-signing and user-signing keys
keyserver_signatures	signatures uploaded after the key they sign, merged in when the key is read
keyserver_device_lists	each user's latest position in the device list stream, local and remote
keyserver_backup_versions, keyserver_backup_keys	server-side key backup
keyserver_sequences	the device list stream's counter

Migration 0009_remote_keys.sql adds the cache of other servers' users' keys (see The remote key cache):

Table	What
keyserver_remote_users	one row per cached user: the stream_id the copy is at, and whether it is stale
keyserver_remote_device_keys	their devices' keys and display names, as their server sent them
keyserver_remote_cross_signing_keys	their master and self-signing keys

They are tables of their own rather than rows in the local ones: every write to a local user's keys moves the device list stream and goes out over federation, and keeping the copies apart means no path that announces a local change can announce a copy.

Keys are stored in canonical JSON. That is what lets "the same key uploaded again" be a byte comparison: a client that re-sends a one-time key it already uploaded, encoded differently, is not told it conflicts.

Signatures added through /keys/signatures/upload are kept apart from the key they sign rather than written into it, so a key's own JSON is always exactly what its owner uploaded.

Endpoints

Endpoint	Notes
POST /keys/upload	device keys must be for the caller's own device; one-time and fallback keys (fallback_keys, or MSC2732's unstable name); answers the one-time key counts, signed_curve25519 always included
POST /keys/query	local users from the store; remote users who share a room with somebody here from the cache, fetched on a miss; anybody else asked of their server; all servers in parallel, failures for one that did not answer
POST /keys/claim	a one-time key if there is one, the fallback key otherwise; remote users claimed from their servers
GET /keys/changes	whose device lists changed between two sync tokens
POST /keys/device_signing/upload	cross-signing keys; asks for the password, except where the spec says not to
POST /keys/signatures/upload	the three kinds of signature the spec describes; per-key failures
POST/GET /room_keys/version · GET/PUT/DELETE /room_keys/version/{v}	backup versions
PUT/GET/DELETE /room_keys/keys[/{room}[/{session}]]?version=	backed-up room keys at the three granularities
PUT /sendToDevice/{type}/{txn}	to-device messaging — the syncapi's, from Phase 4; see sync.md

Federation (behind the gate, like every federation endpoint):

Endpoint	Notes
POST /_matrix/federation/v1/user/keys/query	local users only, without device display names
POST /_matrix/federation/v1/user/keys/claim	local users only
GET /_matrix/federation/v1/user/devices/{userId}	a local user's whole device list with its stream ID, for a server whose updates do not add up

What the server checks

It cannot tell a good key from a bad one, and does not try to vouch for any. What it refuses is what is plainly wrong:

• Device keys for a user or device other than the one the access token belongs to. Whether they are signed by the key they say is not checked: every client that reads them checks that, which is the check that matters. Synapse and Dendrite make the same choice.
• One-time keys without an algorithm:id key ID, more than 1000 in one upload, or under an ID that already holds a different key (M_INVALID_PARAM).
• Cross-signing keys that are not the uploader's, do not have the usage they are uploaded as, or hold anything but one ed25519 key named by its own public key — and a self-signing or user-signing key the master key did not sign (M_INVALID_SIGNATURE). A new master key on its own drops the self-signing and user-signing keys the old one signed: they would go on being served, signed by a key that is no longer the user's.
• Signatures that do not verify against the key as stored. A client sending a modified copy with a valid signature over the modification has signed nothing that is here.

When cross-signing needs a password

/keys/device_signing/upload goes through user-interactive authentication (m.login.password) except in the two cases the spec exempts: the user has no master key yet, or every key in the request is the one already stored. Replacing a master key is what would let a stolen access token pass as the user to everyone who has verified them.

Signatures and who sees them

/keys/signatures/upload accepts three kinds:

Target	Signed by	Seen by
one of the uploader's devices	their self-signing key	everybody
the uploader's master key	one of their devices	everybody
another user's master key	the uploader's user-signing key	the uploader only

Who a user has verified is between them and their own devices, so the third kind is returned only to the user who made it — and never sent over federation. Signatures of anything else are ignored rather than refused: clients send the whole signed object back, the device's own signature included. A user-signing key is likewise only ever shown to its owner.

Device list tracking

A client encrypting for a room needs the keys of every device of every member, and caches them. /sync's device_lists says when the cache is stale:

• changed: a user sharing a room with the syncing user uploaded or deleted a device's keys, changed a device's name or their cross-signing keys, or gained a signature — or has only just come to share a room. The syncing user's own ID is in it when their own devices change, so their other devices start encrypting to a new one.
• left: a user the syncing user no longer shares any room with.

The keyserver writes the device list stream: every write that changes what /keys/query would answer moves the user's position, in the same transaction as the change, and wakes the long polls of everyone sharing a room with them. The stream's position is the sixth field of a sync token (s…_…_…_…_…_{device_list}; older tokens without it still parse). The syncapi works out which moves concern a sync (internal/syncapi/devicelists.go):

1. moves in the window of users who share a room with the syncer, the syncer included;
2. for every room whose membership of the syncer moved in the window, its joined members — all new to share with on a join, perhaps no longer shared with on a leave;
3. everybody else whose membership moved to join, leave or ban in a room the syncer is or was in.

Anyone in 2 or 3 who still shares a room is changed, anyone who does not is left. Invites and knocks are skipped: telling a client to forget the keys of a user it was encrypting for as an invitee would have it stop. Both lists are empty on an initial sync, which is the client's cue to fetch every key it needs anyway.

device_one_time_keys_count and device_unused_fallback_key_types ride on every sync. The fallback list is sent as [] when empty, never left out: a client reads its absence as a server without fallback keys, and never uploads one.

Over federation

EDU	Out	In
m.device_list_update	one per device change, to every server sharing a room with the user, with the keys or deleted: true, a stream_id and the prev_id before it	applied to the cached copy when its prev_id includes the copy's stream_id, otherwise the copy is marked stale and fetched again; moves the remote user's position in the device list stream
m.signing_key_update	when a user's master or self-signing key changes, or the master key gains a device's signature	replaces the cached master and self-signing keys; moves the position likewise

Each device of a multi-device deletion gets a stream ID of its own: remote servers treat a stream ID they have seen as a duplicate, so three deletions sharing one would reach them as one deletion and two repeats.

An incoming update about a remote user always tells local clients, who keep their own copies, by moving the user in the device list stream — and only does anything when a local user shares a room with them: an update about a stranger concerns nobody here, and recording it would let any server grow the tables with users of its own invention. A repeat of an update already applied changes nothing, not even the stream.

Device display names are not sent over federation, in answers or in EDUs. Local users see them in unsigned.device_display_name: their own devices', and those of remote devices whose server did send one.

The remote key cache

A client asks for the keys of everybody in a room before it sends an encrypted message, and again whenever a sync says one of them moved. Without a cache each of those is a federation round trip in front of the message. So the keys of remote users who share a room with a local user are kept (internal/keyserver/remotekeys.go):

• On a miss, /keys/query fetches the user's whole device list with GET /user/devices/{userId} (which also gives the stream_id the list is at), caches it and serves it. A device whose keys name another user or device is left out, and so is a cross-signing key that is not that user's key for its purpose. The next query does not leave the server.
• An m.device_list_update whose prev_id includes the cached stream_id is applied: the device's keys replaced, or deleted. One whose stream_id is at or before the copy's is a repeat and ignored. Anything else (a gap, an update with no keys for a device it says changed, keys that are not that device's) marks the copy stale, and it is fetched again in the background, not inside the transaction that brought the update, which would have the sending server wait on a request back to itself.
• A stale copy is never served. A query that finds one does what it does on a miss and asks synchronously: a device list known to be missing a change would have the sender skip the device just added, and the message would be undecryptable on it. If the user's server is down, the query reports it in failures.
• An m.signing_key_update replaces the cached master and self-signing keys with the ones it carries; one it leaves out is dropped.
• A copy goes once no local user shares a room with the user. The keyserver listens on the roomserver's emit stage: a remote user's leave or ban has their copy checked, a local user's has every copy checked, in the background. A query or update that finds a cached user has become a stranger drops the copy on the spot. Strangers are never cached; a query about one is passed to their server as before.

The cache and the updates race in one place: a snapshot fetched just before an update arrives. An update about a user with nothing cached leaves a stale marker at its stream_id, an update to a stale copy raises its stream_id, and a snapshot older than the stream_id held is not stored. So a snapshot that predates an update is never cached as current.

Local moves that Warp does not send as an m.device_list_update (a device rename, a cross-signing change, a signature) still move the user's stream_id, so a Warp server caching another Warp server's user sees the next update as a gap and fetches again. That costs one request and is always correct.

With federation off, remote users in a query or claim are reported in failures straight away; nothing is sent.

Key backup

The server keeps what it is given and decides one thing: when a key for a session that is already backed up arrives, which to keep. The spec's rule, in storage.BackupKey.Replaces: a verified key beats an unverified one, then the lower first_message_index wins, then the lower forwarded_count. The etag moves only when a key actually changed.

Versions are numbers, per user, and a deleted version keeps its row so its number is never handed out again: a client still holding the old number must not find itself writing into a new backup encrypted to a different key. Writing to a version that exists but is not the current one is 403 M_WRONG_ROOM_KEYS_VERSION with current_version; to one that does not exist, 404. A version's algorithm cannot be changed.

What is deliberately missing

Thing	Why
Dehydrated devices (MSC3814)	—
Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984)	unstable; application services are in, without these (appservices.md)
Purging a deleted device's to-device queue	the messages stay queued; a new device later logged in under the same ID is handed them, encrypted to the old device's keys and so unreadable to it
Deleting a device's keys in the same transaction as the device	the two are separate stores; keys left behind by a crash between them are never served, because a query reads the device list first

Testing

Where	What
internal/storage/keys_test.go	the key store against every driver: the device list stream, claims, fallback keys, signatures, the backup replacement rule
internal/storage/remotekeys_test.go	the remote key cache against every driver: snapshots, updates that follow on, repeats, gaps, stale markers, older snapshots refused, dropping
internal/keyserver/keyserver_test.go	the checks, with real ed25519 signatures: uploads, conflicts, cross-signing, the three signature kinds and their visibility, deletions, remote updates, backups
internal/keyserver/remotekeys_test.go	the cache: a second query not leaving the server, updates applied, a gap fetched again in the background, a stale copy not served, signing key updates, strangers dropped
internal/setup/phase7_test.go	through the whole HTTP stack: upload, sync, query, claim, keys/changes, cross-signing with and without UIA, backup, device deletion, leaving a room; and two federated servers exchanging keys and device list updates, with a second query and an update served from B's cache without a request to A
Complement	TestUploadKey*, TestKeyClaimOrdering, TestKeyChanges*, TestDeviceListUpdates*, TestE2EKeyBackup*, TestFallbackKeys* and TestCrossSigning* are in the target sets