Warp/docs/e2ee.md

End-to-end encryption

server/internal/keyserver is the server half of end-to-end encryption: everything a homeserver keeps so that its users' devices can encrypt to each other, none of which it can read or use itself. Phase 7 of PLAN.md.

The client half — Olm, Megolm, verification — is not here and never will be. Warp's own clients use matrix-sdk-crypto-wasm in the browser and goolm on the desktop; the server only has to publish keys, hand out each one-time key once, and tell devices when somebody's keys changed.

What is stored

All in migration 0008_keyserver.sql, keyed by full user ID rather than localpart, because signatures and the device list stream are about other servers' users too.

Table	What
keyserver_device_keys	each device's identity keys, as the device uploaded and signed them, without unsigned
keyserver_one_time_keys	unclaimed one-time keys; a row is deleted the moment it is claimed
keyserver_fallback_keys	one per algorithm per device, marked used when claimed and kept until replaced
keyserver_cross_signing_keys	a user's master, self-signing and user-signing keys
keyserver_signatures	signatures uploaded after the key they sign, merged in when the key is read
keyserver_device_lists	each user's latest position in the device list stream, local and remote
keyserver_backup_versions, keyserver_backup_keys	server-side key backup
keyserver_sequences	the device list stream's counter

Keys are stored in canonical JSON. That is what lets "the same key uploaded again" be a byte comparison: a client that re-sends a one-time key it already uploaded, encoded differently, is not told it conflicts.

Signatures added through /keys/signatures/upload are kept apart from the key they sign rather than written into it, so a key's own JSON is always exactly what its owner uploaded.

Endpoints

Endpoint	Notes
POST /keys/upload	device keys must be for the caller's own device; one-time and fallback keys (fallback_keys, or MSC2732's unstable name); answers the one-time key counts, signed_curve25519 always included
POST /keys/query	local users from the store, remote users asked of their servers in parallel; failures for a server that did not answer
POST /keys/claim	a one-time key if there is one, the fallback key otherwise; remote users claimed from their servers
GET /keys/changes	whose device lists changed between two sync tokens
POST /keys/device_signing/upload	cross-signing keys; asks for the password, except where the spec says not to
POST /keys/signatures/upload	the three kinds of signature the spec describes; per-key failures
POST/GET /room_keys/version · GET/PUT/DELETE /room_keys/version/{v}	backup versions
PUT/GET/DELETE /room_keys/keys[/{room}[/{session}]]?version=	backed-up room keys at the three granularities
PUT /sendToDevice/{type}/{txn}	to-device messaging — the syncapi's, from Phase 4; see sync.md

Federation (behind the gate, like every federation endpoint):

Endpoint	Notes
POST /_matrix/federation/v1/user/keys/query	local users only, without device display names
POST /_matrix/federation/v1/user/keys/claim	local users only
GET /_matrix/federation/v1/user/devices/{userId}	a local user's whole device list with its stream ID, for a server whose updates do not add up

What the server checks

It cannot tell a good key from a bad one, and does not try to vouch for any. What it refuses is what is plainly wrong:

• Device keys for a user or device other than the one the access token belongs to. Whether they are signed by the key they say is not checked: every client that reads them checks that, which is the check that matters. Synapse and Dendrite make the same choice.
• One-time keys without an algorithm:id key ID, more than 1000 in one upload, or under an ID that already holds a different key (M_INVALID_PARAM).
• Cross-signing keys that are not the uploader's, do not have the usage they are uploaded as, or hold anything but one ed25519 key named by its own public key — and a self-signing or user-signing key the master key did not sign (M_INVALID_SIGNATURE). A new master key on its own drops the self-signing and user-signing keys the old one signed: they would go on being served, signed by a key that is no longer the user's.
• Signatures that do not verify against the key as stored. A client sending a modified copy with a valid signature over the modification has signed nothing that is here.

When cross-signing needs a password

/keys/device_signing/upload goes through user-interactive authentication (m.login.password) except in the two cases the spec exempts: the user has no master key yet, or every key in the request is the one already stored. Replacing a master key is what would let a stolen access token pass as the user to everyone who has verified them.

Signatures and who sees them

/keys/signatures/upload accepts three kinds:

Target	Signed by	Seen by
one of the uploader's devices	their self-signing key	everybody
the uploader's master key	one of their devices	everybody
another user's master key	the uploader's user-signing key	the uploader only

Who a user has verified is between them and their own devices, so the third kind is returned only to the user who made it — and never sent over federation. Signatures of anything else are ignored rather than refused: clients send the whole signed object back, the device's own signature included. A user-signing key is likewise only ever shown to its owner.

Device list tracking

A client encrypting for a room needs the keys of every device of every member, and caches them. /sync's device_lists says when the cache is stale:

• changed: a user sharing a room with the syncing user uploaded or deleted a device's keys, changed a device's name or their cross-signing keys, or gained a signature — or has only just come to share a room. The syncing user's own ID is in it when their own devices change, so their other devices start encrypting to a new one.
• left: a user the syncing user no longer shares any room with.

The keyserver writes the device list stream: every write that changes what /keys/query would answer moves the user's position, in the same transaction as the change, and wakes the long polls of everyone sharing a room with them. The stream's position is the sixth field of a sync token (s…_…_…_…_…_{device_list}; older tokens without it still parse). The syncapi works out which moves concern a sync (internal/syncapi/devicelists.go):

1. moves in the window of users who share a room with the syncer, the syncer included;
2. for every room whose membership of the syncer moved in the window, its joined members — all new to share with on a join, perhaps no longer shared with on a leave;
3. everybody else whose membership moved to join, leave or ban in a room the syncer is or was in.

Anyone in 2 or 3 who still shares a room is changed, anyone who does not is left. Invites and knocks are skipped: telling a client to forget the keys of a user it was encrypting for as an invitee would have it stop. Both lists are empty on an initial sync, which is the client's cue to fetch every key it needs anyway.

device_one_time_keys_count and device_unused_fallback_key_types ride on every sync. The fallback list is sent as [] when empty, never left out: a client reads its absence as a server without fallback keys, and never uploads one.

Over federation

EDU	Out	In
m.device_list_update	one per device change, to every server sharing a room with the user, with the keys or deleted: true, a stream_id and the prev_id before it	moves the remote user's position in the device list stream
m.signing_key_update	when a user's master or self-signing key changes, or the master key gains a device's signature	likewise

Each device of a multi-device deletion gets a stream ID of its own: remote servers treat a stream ID they have seen as a duplicate, so three deletions sharing one would reach them as one deletion and two repeats.

Warp does not cache other servers' keys. A query about a remote user is answered by asking their server, so an incoming update has nothing to invalidate; what it does is tell local clients, who do cache, by moving the user in the device list stream. That only happens when a local user shares a room with them: an update about a stranger concerns nobody here, and recording it would let any server grow the table with users of its own invention.

Device display names are not sent over federation, in answers or in EDUs. Local users see them in unsigned.device_display_name.

With federation off, remote users in a query or claim are reported in failures straight away; nothing is sent.

Key backup

The server keeps what it is given and decides one thing: when a key for a session that is already backed up arrives, which to keep. The spec's rule, in storage.BackupKey.Replaces: a verified key beats an unverified one, then the lower first_message_index wins, then the lower forwarded_count. The etag moves only when a key actually changed.

Versions are numbers, per user, and a deleted version keeps its row so its number is never handed out again: a client still holding the old number must not find itself writing into a new backup encrypted to a different key. Writing to a version that exists but is not the current one is 403 M_WRONG_ROOM_KEYS_VERSION with current_version; to one that does not exist, 404. A version's algorithm cannot be changed.

What is deliberately missing

Thing	Why
A cache of remote users' keys	Every query asks their server. Correct, and slower for a client in rooms with many remote users; a cache invalidated by the EDUs above is the obvious next step.
Resync on a gap in a remote server's prev_id	Follows from the above: with nothing cached, a missed update costs nothing.
Dehydrated devices (MSC3814)	—
Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984)	application services are Phase 9
Purging a deleted device's to-device queue	the messages stay queued; a new device later logged in under the same ID is handed them, encrypted to the old device's keys and so unreadable to it
Deleting a device's keys in the same transaction as the device	the two are separate stores; keys left behind by a crash between them are never served, because a query reads the device list first

Testing

Where	What
internal/storage/keys_test.go	the key store against every driver: the device list stream, claims, fallback keys, signatures, the backup replacement rule
internal/keyserver/keyserver_test.go	the checks, with real ed25519 signatures: uploads, conflicts, cross-signing, the three signature kinds and their visibility, deletions, remote updates, backups
internal/setup/phase7_test.go	through the whole HTTP stack: upload, sync, query, claim, keys/changes, cross-signing with and without UIA, backup, device deletion, leaving a room; and two federated servers exchanging keys and device list updates
Complement	TestUploadKey*, TestKeyClaimOrdering, TestKeyChanges*, TestDeviceListUpdates*, TestE2EKeyBackup*, TestFallbackKeys* and TestCrossSigning* are in the target sets