Group voice and video calls with MatrixRTC and LiveKit
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
CI / Admin panel (push) Canceled after 0s
CI / Web client (push) Canceled after 0s

Every call in the web client now goes through MatrixRTC with a LiveKit
SFU, the model Element Call and Element X use; the legacy 1:1 m.call.*
calls are gone. In a room of two the other person is rung and can answer
or decline; in a bigger room a banner offers to join the call. Calls show
a grid of participant tiles with mute, camera, screen sharing and leave,
and media is end-to-end encrypted in encrypted rooms.

Server:
- MSC4140 delayed events (send/state with a delay, plus the
  delayed_events list/restart/cancel/send API), stored and scheduled, so
  a member whose client drops off is taken out of the call.
- POST /user/{userId}/openid/request_token, which lk-jwt-service uses to
  hand out LiveKit tokens.
- org.matrix.msc4143.rtc_foci in /.well-known/matrix/client, configured
  under client_api.rtc.
- Room presets let members send call membership state.
- Migration 0014 for both SQL dialects.

Deploy: LiveKit and lk-jwt-service in compose (calls.yml) and the helm
chart. Docs: docs/calls.md, configuration and deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
tajniak81andClaude Opus 5.5 committed 2026-10-18 19:33:29 +02:00
1 parent 325431a5be
commit 76dfc98a86
49 files changed
+3849 -432

No files matched your search

+171
View File
@@ -0,0 +1,171 @@
# Voice and video calls
Every call in Warp's web client, between two people or twenty, is a
MatrixRTC call (MSC4143) with its media through a LiveKit SFU: the Element
Call model, which Element X uses too. A call of two is a call with two
members, plus a ring. The server needs a LiveKit SFU and the LiveKit JWT
service (`client_api.rtc`); without them there are no calls.
The old Matrix 1:1 VoIP (`m.call.*` events, WebRTC between the two
browsers) is not used: the web client starts the SDK with `disableVoip`, so
it neither places nor answers such calls, and shows them only as history in
the timeline. The server still serves `GET /_matrix/client/v3/voip/turnServer`
(`client_api.turn`) for other clients that do.
## How a call works
Who is in a room's call is room state: every device in it has an
`org.matrix.msc3401.call.member` state event (state key
`_@user:server_DEVICE_m.call`), with the LiveKit service it publishes to.
The media goes through a LiveKit SFU, which Warp does not run itself.
```
browser ── GET /.well-known/matrix/client ──────────▶ warp org.matrix.msc4143.rtc_foci
── GET …/org.matrix.msc4143/rtc/transports ─▶ warp (the same list, authenticated)
── PUT …/state/org.matrix.msc3401.call.member/<key>
?org.matrix.msc4140.delay=… (the delayed "left")
and then without a delay (the "joined") ──▶ warp
── POST /user/<me>/openid/request_token ────▶ warp an OpenID token
── POST <livekit_service_url>/sfu/get ──────▶ lk-jwt-service
lk-jwt-service ── GET /_matrix/federation/v1/openid/userinfo ──▶ warp
◀── {url, jwt}
── WebSocket + UDP ─────────────────────────▶ LiveKit
```
### What the server does
- **Discovery.** With `client_api.rtc.livekit_service_url` set, Warp adds
`org.matrix.msc4143.rtc_foci: [{"type": "livekit", "livekit_service_url": …}]`
to `/.well-known/matrix/client` and serves the same list at
`GET /_matrix/client/unstable/org.matrix.msc4143/rtc/transports`.
- **Delayed events (MSC4140)**, advertised as `org.matrix.msc4140` in
`/versions`. `PUT /rooms/{roomId}/state/{type}/{key}` and
`PUT /rooms/{roomId}/send/{type}/{txnId}` take
`?org.matrix.msc4140.delay=<ms>`: the event is stored instead of sent,
and the answer is `{"delay_id": …}`. Under
`/_matrix/client/unstable/org.matrix.msc4140`:
- `GET /delayed_events` lists your events as `scheduled` and `finalised`
(`?status=` and `?delay_id=` narrow it); `GET /delayed_events/{id}`
returns one.
- `POST /delayed_events/{id}/{restart|cancel|send}`, or the older
`POST /delayed_events/{id}` with `{"action": …}`. These need no access
token: the delay ID is the capability.
A scheduler goroutine sleeps until the next event is due and sends it
as the user who scheduled it, through the same pipeline as any event, so
power levels and membership still apply; a send that fails is finalised
with the error the client would have got. Someone *else* setting the
same piece of state cancels a delayed state event, so a stale "left"
can't overwrite a newer membership; the owner's own updates don't. Events
live in the database (`delayed_events`, migration 14) and survive a
restart. Finalised events are kept for a day. `client_api.delayed_events`
caps the delay (24h) and how many a user may have (100).
This is what makes a call robust: a joining member schedules its own
empty `call.member` a few seconds out and keeps restarting it while it
is in the call. A browser that crashes or loses its network stops
restarting it, and Warp removes the member.
- **OpenID.** `POST /_matrix/client/v3/user/{userId}/openid/request_token`
gives a user a one-hour token, which `GET /_matrix/federation/v1/openid/userinfo?access_token=…`
turns back into `{"sub": "@user:server"}` for whoever it was handed to.
Tokens are stored hashed (`openid_tokens`). The userinfo endpoint is
served with federation switched off too: a server that does not
federate can still have calls.
- **Power levels.** New rooms let anyone who may talk join the call:
`org.matrix.msc3401.call.member` is at level 0 in the `events` of the
default power levels. Rooms created before this keep `state_default`
(50) for it, so an ordinary member gets `M_FORBIDDEN` until a moderator
sets it to 0.
### Ringing
Starting a call in a room with one other person rings them. The caller's
MatrixRTC session sends an `org.matrix.msc4075.rtc.notification`
(`notification_type: "ring"`, a `lifetime` of a minute, `m.call.intent`
audio or video) when its membership is the first; in a bigger room it
sends `notification`, which the web client doesn't ring for, since the
"Join call" banner says enough. The other side shows an incoming-call card
for a ring that is someone else's, mentions them or the room, and is
within its lifetime, until they answer (join the call), decline (an
`org.matrix.msc4310.rtc.decline` referencing the notification, which also
stops the ring on their other devices), the caller leaves, or it expires.
The caller hears a ringback until someone joins, and gives up on a decline
or after the minute. In an encrypted room both events are encrypted like
any other.
The SDK's own validator for received notifications expects MSC4354 sticky
events and `m.rtc.slot` state, which Warp doesn't serve yet, so the web
client applies the receiving rules itself.
### What the web client does
`src/stores/call.ts` uses the SDK's MatrixRTC session
(`client.matrixRTC.getRoomSession(room)`), publishing the transport
`{type: "livekit", livekit_service_url, livekit_alias: <room ID>}` (the one
the call already uses, else this server's), gets a LiveKit token from
`/sfu/get`, and connects with `livekit-client`. In an encrypted room it
joins with `manageMediaKeys`: each member makes its own media key and
sends it to the others over encrypted to-device messages, and a
`BaseKeyProvider` hands each key to LiveKit's frame encryption for that
participant (`@user:server:DEVICE`). The settings (no shared key, no
ratcheting) match Element Call's, so the two can share a call.
### Running it
Locally, [`deploy/compose/calls.yml`](../deploy/compose/calls.yml) adds
LiveKit and lk-jwt-service to the Compose setup:
```sh
cd deploy/compose
WARP_REGISTRATION_ENABLED=true docker compose -f docker-compose.yml -f calls.yml --profile web up -d --build
```
Then sign in as different users at <http://localhost:8080>,
<http://127.0.0.1:8080> and <http://[::1]:8080> (different origins keep
separate sessions in one browser). A voice or video call in a room of two
rings the other person; in a room of three or more the others see a "Join
call" banner. `WARP_PORT` and `WARP_WEB_PORT` move the homeserver and web client
off 8008 and 8080 (then set `WARP_PUBLIC_BASEURL` to match).
The JWT service checks an OpenID token by asking the user's homeserver
over its federation port with TLS. For the server name `localhost` that
means `https://localhost:8448`, so the file gives Warp a federation
listener with a self-signed certificate (`LIVEKIT_INSECURE_SKIP_VERIFY_TLS`
accepts it) and runs LiveKit and the JWT service in Warp's network
namespace, where `localhost` is Warp. One consequence: when the `warp`
container is recreated on its own, the other two lose their network. Run
`up` for the whole file, not for one service, or recreate them with
`docker compose -f docker-compose.yml -f calls.yml up -d --force-recreate livekit livekit-jwt`.
LiveKit advertises `127.0.0.1` for media; set `LIVEKIT_NODE_IP` to the
machine's LAN address to call from other devices.
In production, give each piece its own hostname and a real certificate:
- **LiveKit**, reachable by browsers on TCP 7880 behind TLS (`wss://`),
and UDP (7882, or a port range) and TCP 7881 for media. See LiveKit's
own deployment guide or its Helm chart.
- **lk-jwt-service** behind TLS, with `LIVEKIT_URL` (the `wss://` URL),
`LIVEKIT_KEY` / `LIVEKIT_SECRET` (LiveKit's API key) and
`LIVEKIT_FULL_ACCESS_HOMESERVERS` (your server name). The Helm chart
runs it with `calls.jwtService.enabled`.
- **Warp** with `client_api.rtc.livekit_service_url` set to the JWT
service's URL. The JWT service must reach
`https://<server_name>/.well-known/matrix/server` (a delegation to the
host that serves Warp) or `<server_name>:8448`. Note that Warp serves
`/.well-known/matrix/server` only with federation on; with it off,
serve the delegation file from the server name's web server, as you
would for `/.well-known/matrix/client`.
- **warp-web**: `WARP_WEB_CALL_ORIGINS="https://livekit-jwt.example.org wss://livekit.example.org"`
lets the browser reach both through its Content-Security-Policy.
### Known gaps
- Calls only use the legacy `org.matrix.msc3401.call.member` state events,
not the sticky `m.rtc.member` events of newer MatrixRTC drafts.
- `org.matrix.msc4140.parent_delay_id` (chained delayed events) is
refused, and a delayed send's transaction ID is not deduplicated.
- No ring for calls in rooms of more than two (only the banner), no
MSC4354 sticky events, and no MSC4195 homeserver-side LiveKit token
endpoint (`rtc/livekit/get_token`).
- Calls from clients that only do legacy 1:1 VoIP (`m.call.*`) don't ring
in the web client.