warp/Warp
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 7: E2EE support

Browse Source 
Implements PLAN.md Phase 7: the server half of end-to-end encryption.
Devices upload identity, one-time and fallback keys; other devices query
and claim them, across servers too, and every one-time key is handed out
exactly once. Cross-signing, signature uploads, device list tracking and
server-side key backup complete the set.

internal/keyserver is the component. Keys are stored in canonical JSON
so that re-uploading the same key is a byte comparison, and signatures
uploaded later are kept apart from the key they sign and merged in on
read, so a key's own JSON stays what its owner uploaded. Device keys must
be for the caller's own device; cross-signing keys must be the
uploader's, carry the right usage and, for self- and user-signing keys,
be signed by the master key. Signatures are verified against the key as
stored. Replacing a master key asks for the password; a first upload or
an upload of the keys already stored does not, as the spec allows. A
signature of another user's master key is shown only to the user who
made it and never federated.

Every write that changes what /keys/query answers moves the device list
stream in the same transaction, one position per device, and wakes the
syncs of everyone sharing a room. The stream is the sixth field of a sync
token; older tokens still parse. The syncapi works out device_lists
changed and left from the stream and from membership moves in the
window, skipping invites and knocks, and reports
device_one_time_keys_count and device_unused_fallback_key_types, the
latter as [] when empty. GET /keys/changes uses the same logic.

Over federation: /user/keys/query, /user/keys/claim and
/user/devices/{userId}, and m.device_list_update and m.signing_key_update
both ways. Remote keys are not cached; an update about a remote user
sharing a room moves them in the stream so local clients re-query.
Device display names are not sent to other servers. With federation off,
remote users are reported in failures at once.

Deleting a device deletes its keys, through a new userapi
DeviceObserver. Key backup keeps the better copy of each session by the
spec's rule, never reuses a deleted version's number, and answers writes
to a stale version with M_WRONG_ROOM_KEYS_VERSION.

Storage is migration 0008 for sqlite and postgres, with a PocketBase
stub. The Complement target sets gain the key, device list, fallback,
cross-signing and backup tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
tajniak81andClaude Opus 5.5 committed 2026-10-17 10:34:08 +02:00
1 parent 92836f8dcd
commit fc49f56339
45 files changed
+6050 -63
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/README.md
+1
View File
@@ -9,6 +9,7 @@
| [sync.md](sync.md) | `internal/syncapi`: stream tokens, long polling, filters, the ephemeral streams |
| [federation.md](federation.md) | the Server-Server API: discovery, keys, taking in events, joins across servers, the outbound queue, the switch |
| [media.md](media.md) | `internal/mediaapi`: uploads, downloads, thumbnails, other servers' media, where the files go |
| [e2ee.md](e2ee.md) | `internal/keyserver`: device and one-time keys, cross-signing, device list tracking, key backup |
| [admin-api.md](admin-api.md) | `/_warp/admin/v1`: the operator API behind the admin panel |
The design and roadmap are in [PLAN.md](../PLAN.md) at the repository root.
docs/e2ee.md
+204
View File
@@ -0,0 +1,204 @@
# End-to-end encryption
`server/internal/keyserver` is the server half of end-to-end encryption:
everything a homeserver keeps so that its users' devices can encrypt to
each other, none of which it can read or use itself. Phase 7 of
[PLAN.md](../PLAN.md).
The client half — Olm, Megolm, verification — is not here and never will
be. Warp's own clients use `matrix-sdk-crypto-wasm` in the browser and
`goolm` on the desktop; the server only has to publish keys, hand out each
one-time key once, and tell devices when somebody's keys changed.
## What is stored
All in migration `0008_keyserver.sql`, keyed by full user ID rather than
localpart, because signatures and the device list stream are about other
servers' users too.
| Table | What |
|---|---|
| `keyserver_device_keys` | each device's identity keys, as the device uploaded and signed them, without `unsigned` |
| `keyserver_one_time_keys` | unclaimed one-time keys; a row is deleted the moment it is claimed |
| `keyserver_fallback_keys` | one per algorithm per device, marked used when claimed and kept until replaced |
| `keyserver_cross_signing_keys` | a user's master, self-signing and user-signing keys |
| `keyserver_signatures` | signatures uploaded after the key they sign, merged in when the key is read |
| `keyserver_device_lists` | each user's latest position in the device list stream, local and remote |
| `keyserver_backup_versions`, `keyserver_backup_keys` | server-side key backup |
| `keyserver_sequences` | the device list stream's counter |
Keys are stored in canonical JSON. That is what lets "the same key
uploaded again" be a byte comparison: a client that re-sends a one-time key
it already uploaded, encoded differently, is not told it conflicts.
Signatures added through `/keys/signatures/upload` are kept apart from the
key they sign rather than written into it, so a key's own JSON is always
exactly what its owner uploaded.
## Endpoints
| Endpoint | Notes |
|---|---|
| `POST /keys/upload` | device keys must be for the caller's own device; one-time and fallback keys (`fallback_keys`, or MSC2732's unstable name); answers the one-time key counts, `signed_curve25519` always included |
| `POST /keys/query` | local users from the store, remote users asked of their servers in parallel; `failures` for a server that did not answer |
| `POST /keys/claim` | a one-time key if there is one, the fallback key otherwise; remote users claimed from their servers |
| `GET /keys/changes` | whose device lists changed between two sync tokens |
| `POST /keys/device_signing/upload` | cross-signing keys; asks for the password, except where the spec says not to |
| `POST /keys/signatures/upload` | the three kinds of signature the spec describes; per-key `failures` |
| `POST`/`GET /room_keys/version` · `GET`/`PUT`/`DELETE /room_keys/version/{v}` | backup versions |
| `PUT`/`GET`/`DELETE /room_keys/keys[/{room}[/{session}]]?version=` | backed-up room keys at the three granularities |
| `PUT /sendToDevice/{type}/{txn}` | to-device messaging — the syncapi's, from Phase 4; see [sync.md](sync.md) |
Federation (behind the gate, like every federation endpoint):
| Endpoint | Notes |
|---|---|
| `POST /_matrix/federation/v1/user/keys/query` | local users only, without device display names |
| `POST /_matrix/federation/v1/user/keys/claim` | local users only |
| `GET /_matrix/federation/v1/user/devices/{userId}` | a local user's whole device list with its stream ID, for a server whose updates do not add up |
## What the server checks
It cannot tell a good key from a bad one, and does not try to vouch for
any. What it refuses is what is plainly wrong:
- **Device keys** for a user or device other than the one the access token
belongs to. Whether they are signed by the key they say is not checked:
every client that reads them checks that, which is the check that
matters. Synapse and Dendrite make the same choice.
- **One-time keys** without an `algorithm:id` key ID, more than 1000 in one
upload, or under an ID that already holds a different key
(`M_INVALID_PARAM`).
- **Cross-signing keys** that are not the uploader's, do not have the usage
they are uploaded as, or hold anything but one ed25519 key named by its
own public key — and a self-signing or user-signing key the master key
did not sign (`M_INVALID_SIGNATURE`). A new master key on its own drops
the self-signing and user-signing keys the old one signed: they would go
on being served, signed by a key that is no longer the user's.
- **Signatures** that do not verify against the key *as stored*. A client
sending a modified copy with a valid signature over the modification has
signed nothing that is here.
### When cross-signing needs a password
`/keys/device_signing/upload` goes through user-interactive authentication
(`m.login.password`) except in the two cases the spec exempts: the user has
no master key yet, or every key in the request is the one already stored.
Replacing a master key is what would let a stolen access token pass as the
user to everyone who has verified them.
### Signatures and who sees them
`/keys/signatures/upload` accepts three kinds:
| Target | Signed by | Seen by |
|---|---|---|
| one of the uploader's devices | their self-signing key | everybody |
| the uploader's master key | one of their devices | everybody |
| another user's master key | the uploader's user-signing key | **the uploader only** |
Who a user has verified is between them and their own devices, so the third
kind is returned only to the user who made it — and never sent over
federation. Signatures of anything else are ignored rather than refused:
clients send the whole signed object back, the device's own signature
included. A user-signing key is likewise only ever shown to its owner.
## Device list tracking
A client encrypting for a room needs the keys of every device of every
member, and caches them. `/sync`'s `device_lists` says when the cache is
stale:
- **changed**: a user sharing a room with the syncing user uploaded or
deleted a device's keys, changed a device's name or their cross-signing
keys, or gained a signature — or has only just come to share a room. The
syncing user's own ID is in it when their own devices change, so their
other devices start encrypting to a new one.
- **left**: a user the syncing user no longer shares any room with.
The keyserver writes the **device list stream**: every write that changes
what `/keys/query` would answer moves the user's position, in the same
transaction as the change, and wakes the long polls of everyone sharing a
room with them. The stream's position is the sixth field of a sync token
(`s…_…_…_…_…_{device_list}`; older tokens without it still parse). The
syncapi works out which moves concern a sync
(`internal/syncapi/devicelists.go`):
1. moves in the window of users who share a room with the syncer, the
syncer included;
2. for every room whose membership of the syncer moved in the window, its
joined members — all new to share with on a join, perhaps no longer
shared with on a leave;
3. everybody else whose membership moved to `join`, `leave` or `ban` in a
room the syncer is or was in.
Anyone in 2 or 3 who still shares a room is *changed*, anyone who does not
is *left*. Invites and knocks are skipped: telling a client to forget the
keys of a user it was encrypting for as an invitee would have it stop.
Both lists are empty on an initial sync, which is the client's cue to fetch
every key it needs anyway.
`device_one_time_keys_count` and `device_unused_fallback_key_types` ride on
every sync. The fallback list is sent as `[]` when empty, never left out:
a client reads its absence as a server without fallback keys, and never
uploads one.
### Over federation
| EDU | Out | In |
|---|---|---|
| `m.device_list_update` | one per device change, to every server sharing a room with the user, with the keys or `deleted: true`, a `stream_id` and the `prev_id` before it | moves the remote user's position in the device list stream |
| `m.signing_key_update` | when a user's master or self-signing key changes, or the master key gains a device's signature | likewise |
Each device of a multi-device deletion gets a stream ID of its own:
remote servers treat a stream ID they have seen as a duplicate, so three
deletions sharing one would reach them as one deletion and two repeats.
Warp does **not cache other servers' keys**. A query about a remote user is
answered by asking their server, so an incoming update has nothing to
invalidate; what it does is tell local clients, who do cache, by moving the
user in the device list stream. That only happens when a local user shares
a room with them: an update about a stranger concerns nobody here, and
recording it would let any server grow the table with users of its own
invention.
Device display names are not sent over federation, in answers or in EDUs.
Local users see them in `unsigned.device_display_name`.
With federation off, remote users in a query or claim are reported in
`failures` straight away; nothing is sent.
## Key backup
The server keeps what it is given and decides one thing: when a key for a
session that is already backed up arrives, which to keep. The spec's rule,
in `storage.BackupKey.Replaces`: a verified key beats an unverified one,
then the lower `first_message_index` wins, then the lower
`forwarded_count`. The `etag` moves only when a key actually changed.
Versions are numbers, per user, and a deleted version keeps its row so its
number is never handed out again: a client still holding the old number
must not find itself writing into a new backup encrypted to a different
key. Writing to a version that exists but is not the current one is
`403 M_WRONG_ROOM_KEYS_VERSION` with `current_version`; to one that does
not exist, `404`. A version's algorithm cannot be changed.
## What is deliberately missing
| Thing | Why |
|---|---|
| A cache of remote users' keys | Every query asks their server. Correct, and slower for a client in rooms with many remote users; a cache invalidated by the EDUs above is the obvious next step. |
| Resync on a gap in a remote server's `prev_id` | Follows from the above: with nothing cached, a missed update costs nothing. |
| Dehydrated devices (MSC3814) | — |
| Claiming several keys per device (MSC3983), appservice key claims and queries (MSC3983/MSC3984) | application services are Phase 9 |
| Purging a deleted device's to-device queue | the messages stay queued; a new device later logged in under the same ID is handed them, encrypted to the old device's keys and so unreadable to it |
| Deleting a device's keys in the same transaction as the device | the two are separate stores; keys left behind by a crash between them are never served, because a query reads the device list first |
## Testing
| Where | What |
|---|---|
| `internal/storage/keys_test.go` | the key store against every driver: the device list stream, claims, fallback keys, signatures, the backup replacement rule |
| `internal/keyserver/keyserver_test.go` | the checks, with real ed25519 signatures: uploads, conflicts, cross-signing, the three signature kinds and their visibility, deletions, remote updates, backups |
| `internal/setup/phase7_test.go` | through the whole HTTP stack: upload, sync, query, claim, `keys/changes`, cross-signing with and without UIA, backup, device deletion, leaving a room; and two federated servers exchanging keys and device list updates |
| Complement | `TestUploadKey*`, `TestKeyClaimOrdering`, `TestKeyChanges*`, `TestDeviceListUpdates*`, `TestE2EKeyBackup*`, `TestFallbackKeys*` and `TestCrossSigning*` are in the target sets |
docs/federation.md
+4 -3
View File
@@ -288,7 +288,8 @@ way rather than processed twice.
| `m.typing` | a local user's typing, to the room's servers | into the typing tracker, if the user belongs to the sending server |
| `m.receipt` | a local user's public read receipts — never private ones | into the receipts stream, likewise |
| `m.direct_to_device` | to-device messages for remote users, one EDU per server | delivered to local devices |
| `m.presence`, `m.device_list_update` | — | ignored: presence is off by default, device lists are Phase 7 |
| `m.device_list_update`, `m.signing_key_update` | a local user's key changes, to every server sharing a room with them | the remote user is marked changed for local syncs; see [e2ee.md](e2ee.md) |
| `m.presence` | — | ignored: presence is off by default |
Nothing in an EDU is signed, so the one check there is — the user must
belong to the server that sent it — is what keeps a server from making
@@ -363,8 +364,8 @@ and `GET /profile/{user}` for a remote user.
target arrives later is not applied when the target arrives.
- **Room versions 1 and 2** — event IDs carried in the event — are not
supported, so the oldest rooms on the network cannot be joined.
- **Presence and device list EDUs** are ignored on the way in and not sent
(presence is off by default; device lists are Phase 7).
- **Presence EDUs** are ignored on the way in and not sent (presence is
off by default).
- **Third-party invites** across servers (`exchange_third_party_invite`)
and **knocking** over federation (`make_knock`, `send_knock`) are not
implemented.
docs/sync.md
+4 -4
View File
@@ -401,7 +401,6 @@ no timestamp and have to decide what that meant.
| Thing | Needs | Phase |
|---|---|---|
| `unread_notifications`, `unread_thread_notifications` | push rules | 8 |
| `device_lists.changed` / `.left`, `device_one_time_keys_count` | the keyserver | 7 |
| Remote presence | presence over federation | — |
| Threads, and threaded receipts beyond storing `thread_id` | threads | 10 |
| `contains_url`, `event_fields` | nothing; they are parsed and ignored | — |
@@ -410,9 +409,10 @@ no timestamp and have to decide what that meant.
`device_lists` is sent as two empty lists rather than left out: a client
reads a missing one as a malformed response, where two empty lists say
"nothing changed". The notification counts are left out entirely instead,
because a zero there is a claim — "you have nothing unread" — and an absent
field is not.
"nothing changed". How the two lists are worked out, and the key counts
that ride along with them, are in [e2ee.md](e2ee.md). The notification
counts are left out entirely instead, because a zero there is a claim —
"you have nothing unread" — and an absent field is not.
## Endpoints