Group voice and video calls with MatrixRTC and LiveKit
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
CI / Admin panel (push) Canceled after 0s
CI / Web client (push) Canceled after 0s

Every call in the web client now goes through MatrixRTC with a LiveKit
SFU, the model Element Call and Element X use; the legacy 1:1 m.call.*
calls are gone. In a room of two the other person is rung and can answer
or decline; in a bigger room a banner offers to join the call. Calls show
a grid of participant tiles with mute, camera, screen sharing and leave,
and media is end-to-end encrypted in encrypted rooms.

Server:
- MSC4140 delayed events (send/state with a delay, plus the
  delayed_events list/restart/cancel/send API), stored and scheduled, so
  a member whose client drops off is taken out of the call.
- POST /user/{userId}/openid/request_token, which lk-jwt-service uses to
  hand out LiveKit tokens.
- org.matrix.msc4143.rtc_foci in /.well-known/matrix/client, configured
  under client_api.rtc.
- Room presets let members send call membership state.
- Migration 0014 for both SQL dialects.

Deploy: LiveKit and lk-jwt-service in compose (calls.yml) and the helm
chart. Docs: docs/calls.md, configuration and deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
tajniak81andClaude Opus 5.5 committed 2026-10-18 19:33:29 +02:00
1 parent 325431a5be
commit 76dfc98a86
49 files changed
+3849 -432

No files matched your search

+18 -6
View File
@@ -63,12 +63,24 @@ that one origin.
them asks you to acknowledge it; a reset since you *verified* them says
their sessions are no longer trusted, with Verify again or Withdraw
verification.
- Voice and video calls in a room with one other person (Matrix 1:1 VoIP,
WebRTC between the browsers): ringing, answer with or without video,
mute, camera on and off, screen sharing, and a panel that stays up while
you move between rooms and expands to fill the window. In an encrypted
room the call signalling is encrypted too. Calls outside one network
need a TURN server; see `client_api.turn` in the server configuration.
- Voice and video calls in any room, all through a LiveKit SFU (MatrixRTC,
the Element Call model; the old Matrix 1:1 VoIP is not used). Voice and
video call buttons in the room header. In a room with one other person
the call rings them: an incoming-call card with ring tone, answer, answer
without video and decline, and "Calling…" with a ringback tone for the
caller until they answer, decline or a minute passes. A call of two shows
the other person large and you in the corner; a bigger one is a grid
with a tile per person, their camera or screen, and a ring around
whoever is speaking. Mute, camera, screen sharing, leave, and a panel
that stays up while you move between rooms. A "Join call" banner while
one is going on in a room you're not in it. In an encrypted room the media
is end-to-end encrypted with per-participant keys. The room list says
when a room has a call going on, and the timeline says when one started
and ended. It needs `client_api.rtc` on the server and a LiveKit JWT
service; see [docs/calls.md](../../docs/calls.md). warp-web proxies
`/.well-known/matrix/client` for this, and `-call-origins`
(`WARP_WEB_CALL_ORIGINS`) lets the browser reach the JWT service and
LiveKit through the Content-Security-Policy.
- Encryption setup in Settings: cross-signing keys, secret storage and a
key backup in one step, behind a recovery key shown once. A new session
verifies itself with the recovery key (or another session) and restores
+3 -1
View File
@@ -54,6 +54,8 @@ func run() error {
"base URL of the homeserver, proxied for the client (WARP_WEB_HOMESERVER)")
public := flag.String("public-homeserver", env("WARP_WEB_PUBLIC_HOMESERVER", ""),
"public URL of the homeserver; when set, the browser calls it directly and nothing is proxied (WARP_WEB_PUBLIC_HOMESERVER)")
callOrigins := flag.String("call-origins", env("WARP_WEB_CALL_ORIGINS", ""),
"space-separated origins of the LiveKit JWT service and SFU the browser may connect to for group calls (WARP_WEB_CALL_ORIGINS)")
logFormat := flag.String("log-format", env("WARP_WEB_LOG_FORMAT", "text"), "text or json (WARP_WEB_LOG_FORMAT)")
showVersion := flag.Bool("version", false, "print the version and exit")
flag.Parse()
@@ -74,7 +76,7 @@ func run() error {
}
slog.SetDefault(logger)
opts := host.Options{Logger: logger, Version: version}
opts := host.Options{Logger: logger, Version: version, CallOrigins: strings.Fields(*callOrigins)}
mode, target := "proxy", *homeserver
if *public != "" {
u, err := parseURL("public-homeserver", *public)
+29
View File
@@ -41,6 +41,11 @@ type Options struct {
Logger *slog.Logger
// Transport overrides the proxy's round tripper, for tests.
Transport http.RoundTripper
// CallOrigins are origins outside the homeserver the browser may
// connect to for group calls: the LiveKit JWT service and the LiveKit
// SFU's WebSocket, e.g. https://livekit-jwt.example and
// wss://livekit.example. They are added to the CSP's connect-src.
CallOrigins []string
}
// proxiedPrefixes are the parts of the homeserver a Matrix client uses: the
@@ -48,6 +53,10 @@ type Options struct {
// are not reachable through warp-web.
var proxiedPrefixes = []string{"/_matrix/client/", "/_matrix/media/"}
// wellKnownClientPath is proxied too: it is where the client finds the
// group call backend (org.matrix.msc4143.rtc_foci).
const wellKnownClientPath = "/.well-known/matrix/client"
// Config is the body of /config.json, which the client reads on start.
type Config struct {
// HomeserverURL is where the client sends Matrix requests; empty
@@ -78,6 +87,22 @@ func New(opts Options) (http.Handler, error) {
cfg.HomeserverURL = strings.TrimRight(u.String(), "/")
connect += " " + u.Scheme + "://" + u.Host
}
for _, raw := range opts.CallOrigins {
u, err := url.Parse(strings.TrimRight(raw, "/"))
if err != nil || u.Host == "" || u.Path != "" || u.RawQuery != "" ||
(u.Scheme != "http" && u.Scheme != "https" && u.Scheme != "ws" && u.Scheme != "wss") {
return nil, errors.New("host: a call origin must be an http(s) or ws(s) origin, got " + raw)
}
connect += " " + u.Scheme + "://" + u.Host
// LiveKit falls back from its WebSocket to an HTTP request on
// the same host to find out why a connection failed.
switch u.Scheme {
case "ws":
connect += " http://" + u.Host
case "wss":
connect += " https://" + u.Host
}
}
body, err := json.Marshal(cfg)
if err != nil {
return nil, err
@@ -167,6 +192,10 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("ok\n"))
return
case p == wellKnownClientPath && h.proxy != nil:
h.proxy.ServeHTTP(w, r)
return
case strings.HasPrefix(p, "/_matrix/") || strings.HasPrefix(p, "/_warp/") || strings.HasPrefix(p, "/_synapse/"):
// A path that cleans to something else (dot segments, doubled
// slashes) is refused rather than normalised: what is allowed
+32
View File
@@ -304,3 +304,35 @@ func TestNewValidatesHomeserver(t *testing.T) {
}
}
}
func TestProxiesWellKnownClient(t *testing.T) {
up := newUpstream(t)
h := newHandler(t, up, true)
rec := do(h, http.MethodGet, "/.well-known/matrix/client", nil)
if rec.Code != 200 || len(up.hits) != 1 || up.hits[0].URL.Path != "/.well-known/matrix/client" {
t.Fatalf("status %d, hits %d", rec.Code, len(up.hits))
}
// The server delegation document stays private.
if rec := do(h, http.MethodGet, "/.well-known/matrix/server", nil); rec.Code == 200 && len(up.hits) > 1 {
t.Errorf("/.well-known/matrix/server was proxied")
}
}
func TestCallOriginsJoinConnectSrc(t *testing.T) {
up := newUpstream(t)
target, _ := url.Parse(up.srv.URL)
h, err := New(Options{Homeserver: target, Client: clientFS,
CallOrigins: []string{"https://livekit-jwt.warp.example/", "wss://livekit.warp.example"}})
if err != nil {
t.Fatal(err)
}
rec := do(h, http.MethodGet, "/", nil)
want := "connect-src 'self' https://livekit-jwt.warp.example wss://livekit.warp.example https://livekit.warp.example;"
if csp := rec.Header().Get("Content-Security-Policy"); !strings.Contains(csp, want) {
t.Errorf("CSP %q", csp)
}
if _, err := New(Options{Homeserver: target, CallOrigins: []string{"https://x.example/path"}}); err == nil {
t.Error("a call origin with a path was accepted")
}
}
+1
View File
@@ -13,6 +13,7 @@
"dependencies": {
"@warp/ui": "workspace:*",
"jsqr": "1.4.0",
"livekit-client": "2.22.3",
"matrix-js-sdk": "^44.0.0",
"pinia": "^3.0.3",
"qrcode-generator": "1.4.4",
+58
View File
@@ -0,0 +1,58 @@
<script setup lang="ts">
import type { Track } from 'livekit-client'
import { computed, onBeforeUnmount, ref, watch } from 'vue'
import { WIcon } from '@warp/ui'
import MxcAvatar from '@/components/MxcAvatar.vue'
import type { Tile } from '@/stores/call'
// One person in a group call: their screen or camera, else their avatar,
// with their name, a muted microphone, and a ring while they speak.
const props = defineProps<{ tile: Tile; compact?: boolean }>()
const videoEl = ref<HTMLVideoElement>()
const audioEl = ref<HTMLAudioElement>()
const shown = computed(() => props.tile.screen ?? props.tile.camera)
// Your own camera is mirrored, the way a mirror (and every call app) shows you.
const mirrored = computed(() => props.tile.local && !props.tile.screen)
function bind(el: HTMLMediaElement | undefined, track: Track | undefined, old: Track | undefined) {
if (!el) return
if (old && old !== track) old.detach(el)
if (track) track.attach(el)
}
watch([videoEl, shown], ([el, track], [, old]) => bind(el, track, old), { flush: 'post', immediate: true })
watch([audioEl, () => props.tile.audio], ([el, track], [, old]) => bind(el, track, old), { flush: 'post', immediate: true })
onBeforeUnmount(() => {
if (videoEl.value) shown.value?.detach(videoEl.value)
if (audioEl.value) props.tile.audio?.detach(audioEl.value)
})
</script>
<template>
<div
:class="[
'relative flex min-h-0 items-center justify-center overflow-hidden rounded-md bg-[#1d2126] transition-shadow',
tile.speaking ? 'shadow-[0_0_0_3px_#3fb68b]' : '',
]"
>
<audio v-if="tile.audio" ref="audioEl" autoplay class="hidden" />
<video
v-show="shown"
ref="videoEl"
autoplay
playsinline
muted
:class="['h-full w-full', tile.screen ? 'object-contain' : 'object-cover', mirrored && '-scale-x-100']"
/>
<MxcAvatar v-if="!shown" :name="tile.name" :mxc="tile.avatarMxc" :size="compact ? 48 : 88" />
<div class="absolute bottom-2 left-2 flex max-w-[calc(100%-1rem)] items-center gap-1 rounded-sm bg-[#000000a6] px-2 py-0.5">
<WIcon v-if="tile.micMuted" name="mic-off" :size="14" label="Muted" class="shrink-0 text-[#ff8a80]" />
<WIcon v-if="tile.screen" name="screen-share" :size="14" label="Sharing their screen" class="shrink-0" />
<span class="caption truncate text-[#eceeee]">{{ tile.local ? `${tile.name} (you)` : tile.name }}</span>
</div>
</div>
</template>
+156 -130
View File
@@ -1,108 +1,74 @@
<script setup lang="ts">
import type { CallFeed } from 'matrix-js-sdk/lib/webrtc/callFeed'
import { computed, onBeforeUnmount, ref, watch, watchEffect } from 'vue'
import { computed, onBeforeUnmount, ref, watch } from 'vue'
import { useRouter } from 'vue-router'
import { WIcon } from '@warp/ui'
import { WIcon, WSpinner } from '@warp/ui'
import type { IconName } from '@warp/ui'
import CallTile from '@/components/CallTile.vue'
import MxcAvatar from '@/components/MxcAvatar.vue'
import { formatDuration } from '@/lib/format'
import { useCall } from '@/stores/call'
import { useCall, type Tile } from '@/stores/call'
// The call on screen: a ringing card for an incoming call, else a panel in
// the corner that a video call can expand to fill the window. It sits above
// every view, so a call goes on while you move between rooms.
// The call on screen: a ringing card for a call coming in, else the call
// you're in. A call of two shows the other person large and you in the
// corner; a bigger one is a grid. The panel shrinks to the corner so the
// call goes on while you read and write in your rooms, and sits above
// every view.
const call = useCall()
const router = useRouter()
const expanded = ref(false)
function feedVideo(feed: CallFeed | undefined): MediaStream | undefined {
if (!feed || feed.isVideoMuted() || !feed.stream.getVideoTracks().length) return undefined
return feed.stream
}
const view = computed(() => {
void call.version
const c = call.call
if (!c) return null
const remoteCamera = feedVideo(c.remoteUsermediaFeed)
const remoteScreen = c.remoteScreensharingStream
const localCamera = feedVideo(c.localUsermediaFeed)
const localScreen = c.localScreensharingStream
// Their screen beats their camera on the big stage; whatever is left of
// theirs or yours goes in the small picture.
const main = remoteScreen ?? remoteCamera
const pip = remoteScreen && remoteCamera ? remoteCamera : (localScreen ?? localCamera)
return {
main,
pip,
pipMirrored: pip !== undefined && pip === localCamera,
remoteAudio: c.remoteUsermediaStream,
micMuted: c.isMicrophoneMuted(),
cameraOn: Boolean(localCamera),
sharing: c.isScreensharing(),
remoteOnHold: c.isRemoteOnHold(),
localOnHold: c.isLocalOnHold(),
roomId: c.roomId,
}
})
const stage = computed(() => call.phase !== 'ringing' && Boolean(view.value?.main || view.value?.pip || call.isVideo))
const canShare = typeof navigator !== 'undefined' && Boolean(navigator.mediaDevices?.getDisplayMedia)
const status = computed(() => {
const v = view.value
switch (call.phase) {
case 'ringing':
return call.isVideo ? 'Incoming video call' : 'Incoming voice call'
case 'calling':
return 'Calling…'
case 'connecting':
return 'Connecting…'
case 'connected':
if (v?.localOnHold) return 'You’re on hold'
if (v?.remoteOnHold) return 'On hold'
return formatDuration(call.length)
case 'ended':
return call.endReason
default:
return ''
}
})
// Streams go to the media elements by hand: srcObject is not an attribute.
const mainEl = ref<HTMLVideoElement>()
const pipEl = ref<HTMLVideoElement>()
const audioEl = ref<HTMLAudioElement>()
function bind(el: HTMLMediaElement | undefined, stream: MediaStream | undefined) {
if (!el || el.srcObject === (stream ?? null)) return
el.srcObject = stream ?? null
if (stream) void el.play().catch(() => undefined)
}
watchEffect(() => bind(mainEl.value, view.value?.main), { flush: 'post' })
watchEffect(() => bind(pipEl.value, view.value?.pip), { flush: 'post' })
watchEffect(() => bind(audioEl.value, view.value?.remoteAudio), { flush: 'post' })
const expanded = ref(true)
watch(
() => call.call,
(c) => {
if (!c) expanded.value = false
() => call.roomId,
(rid) => {
// Video calls and group calls open full size; a voice call between
// two people starts as the panel in the corner.
if (rid) expanded.value = call.video || !call.callee
},
)
function onKey(e: KeyboardEvent) {
if (e.key === 'Escape' && expanded.value) expanded.value = false
}
window.addEventListener('keydown', onKey)
onBeforeUnmount(() => window.removeEventListener('keydown', onKey))
const remotes = computed(() => call.tiles.filter((t) => !t.local))
const self = computed(() => call.tiles.find((t) => t.local))
/** A call of two: the other person on the stage, you in the corner. */
const pair = computed(() => remotes.value.length === 1)
/** Ringing them, nobody here yet. */
const calling = computed(() => call.ringingOut && remotes.value.length === 0)
function openRoom() {
const roomId = view.value?.roomId
if (!roomId) return
expanded.value = false
router.push({ name: 'room', params: { roomId } })
}
// Whoever shares their screen takes the stage; otherwise the person
// speaking comes first, so the panel in the corner shows them.
const ordered = computed(() => {
const rank = (t: Tile) => (t.screen ? 0 : t.speaking && !t.local ? 1 : t.local ? 3 : 2)
return [...call.tiles].sort((a, b) => rank(a) - rank(b))
})
const shown = computed(() => (expanded.value ? ordered.value : ordered.value.slice(0, 1)))
// The rest still have to be heard while the panel shows only one.
const hidden = computed(() => (expanded.value ? [] : ordered.value.slice(1).filter((t) => t.audio)))
const columns = computed(() => {
const n = shown.value.length
if (n <= 1) return 'grid-cols-1'
if (n <= 4) return 'grid-cols-1 sm:grid-cols-2'
if (n <= 9) return 'grid-cols-2 lg:grid-cols-3'
return 'grid-cols-2 md:grid-cols-3 xl:grid-cols-4'
})
const status = computed(() => {
switch (call.phase) {
case 'joining':
return calling.value ? 'Calling…' : 'Joining…'
case 'reconnecting':
return 'Reconnecting…'
default: {
if (calling.value) return 'Calling…'
const time = formatDuration(call.length)
if (call.callee) return time
const n = call.tiles.length
return n === 1 ? `Only you so far · ${time}` : `${n} people · ${time}`
}
}
})
const canShare = typeof navigator !== 'undefined' && Boolean(navigator.mediaDevices?.getDisplayMedia)
interface Control {
icon: IconName
@@ -113,23 +79,28 @@ interface Control {
}
const controls = computed<Control[]>(() => {
const v = view.value
if (!v) return []
if (call.phase === 'ringing') {
const list: Control[] = [{ icon: 'phone-off', label: 'Decline', action: call.decline, tone: 'danger' }]
if (call.isVideo) list.push({ icon: 'phone', label: 'Answer without video', action: () => call.answer(false) })
list.push({ icon: call.isVideo ? 'video' : 'phone', label: 'Answer', action: () => call.answer(), tone: 'accept' })
return list
}
if (call.phase === 'ended') return [{ icon: 'close', label: 'Close', action: call.hangup }]
const list: Control[] = [
{ icon: v.micMuted ? 'mic-off' : 'mic', label: v.micMuted ? 'Unmute' : 'Mute', action: call.toggleMic, on: v.micMuted },
{ icon: v.cameraOn ? 'video' : 'video-off', label: v.cameraOn ? 'Turn camera off' : 'Turn camera on', action: call.toggleCamera, on: !v.cameraOn && call.isVideo },
{ icon: call.micOn ? 'mic' : 'mic-off', label: call.micOn ? 'Mute' : 'Unmute', action: () => call.toggle('mic'), on: !call.micOn },
{
icon: call.cameraOn ? 'video' : 'video-off',
label: call.cameraOn ? 'Turn camera off' : 'Turn camera on',
action: () => call.toggle('camera'),
on: !call.cameraOn,
},
]
if (canShare && call.phase === 'connected') {
list.push({ icon: 'screen-share', label: v.sharing ? 'Stop sharing' : 'Share screen', action: call.toggleScreenShare, on: v.sharing })
list.push({ icon: 'screen-share', label: call.sharing ? 'Stop sharing' : 'Share screen', action: () => call.toggle('screen'), on: call.sharing })
}
list.push({ icon: 'phone-off', label: 'Hang up', action: call.hangup, tone: 'danger' })
list.push({ icon: 'phone-off', label: calling.value ? 'Cancel call' : 'Leave call', action: () => void call.leave(), tone: 'danger' })
return list
})
const ringControls = computed<Control[]>(() => {
const c = call.incoming
if (!c) return []
const list: Control[] = [{ icon: 'phone-off', label: 'Decline', action: () => void call.decline(), tone: 'danger' }]
if (c.video) list.push({ icon: 'phone', label: 'Answer without video', action: () => void call.answer(false) })
list.push({ icon: c.video ? 'video' : 'phone', label: 'Answer', action: () => void call.answer(), tone: 'accept' })
return list
})
@@ -139,37 +110,70 @@ function controlClass(c: Control): string {
if (c.on) return 'bg-[#eceeee] text-[#111418] hover:bg-[#ffffff]'
return 'bg-[#ffffff1f] text-[#eceeee] hover:bg-[#ffffff33]'
}
function openRoom() {
const roomId = call.roomId
if (!roomId) return
expanded.value = false
router.push({ name: 'room', params: { roomId } })
}
function onKey(e: KeyboardEvent) {
if (e.key === 'Escape' && expanded.value) expanded.value = false
}
window.addEventListener('keydown', onKey)
onBeforeUnmount(() => window.removeEventListener('keydown', onKey))
</script>
<template>
<section
v-if="call.call && view"
v-if="call.incoming && !call.joined"
class="fixed right-4 bottom-4 left-4 z-40 flex flex-col overflow-hidden rounded-lg bg-[#111418] text-[#eceeee] shadow-float sm:left-auto sm:w-[360px]"
:aria-label="`Call from ${call.incoming.callerName}`"
role="alertdialog"
>
<div class="flex items-center gap-3 px-4 pt-4">
<MxcAvatar :name="call.incoming.callerName" :mxc="call.incoming.callerAvatarMxc" :size="48" />
<div class="min-w-0 flex-1">
<span class="body-strong block truncate">{{ call.incoming.callerName }}</span>
<span class="small block truncate text-[#eceeeeb3]">{{ call.incoming.video ? 'Incoming video call' : 'Incoming voice call' }}</span>
</div>
</div>
<div class="flex items-center justify-center gap-3 px-4 pt-3 pb-4">
<button
v-for="c in ringControls"
:key="c.label"
type="button"
:class="['flex h-12 w-12 cursor-pointer items-center justify-center rounded-full border-0 transition-colors', controlClass(c)]"
:aria-label="c.label"
:title="c.label"
@click="c.action()"
>
<WIcon :name="c.icon" :size="22" />
</button>
</div>
</section>
<section
v-if="call.roomId"
:class="[
'fixed z-40 flex flex-col overflow-hidden bg-[#111418] text-[#eceeee] shadow-float',
expanded ? 'inset-0' : 'right-4 bottom-4 left-4 rounded-lg sm:left-auto sm:w-[360px]',
]"
:aria-label="`Call with ${call.peerName}`"
:role="call.phase === 'ringing' ? 'alertdialog' : 'region'"
:aria-label="`Call: ${call.title}`"
role="region"
>
<audio ref="audioEl" autoplay class="hidden" />
<div v-if="stage" :class="['relative bg-[#000000]', expanded ? 'min-h-0 flex-1' : 'aspect-video']">
<video v-show="view.main" ref="mainEl" autoplay playsinline muted class="h-full w-full object-contain" />
<div v-if="!view.main" class="absolute inset-0 flex items-center justify-center">
<MxcAvatar :name="call.peerName" :mxc="call.peerAvatarMxc" :size="expanded ? 120 : 72" />
</div>
<video
v-show="view.pip"
ref="pipEl"
autoplay
playsinline
muted
:class="['absolute right-3 bottom-3 w-1/4 min-w-24 rounded-md bg-[#000000] object-cover shadow-float', view.pipMirrored && '-scale-x-100']"
/>
<div class="flex items-center gap-3 px-4 pt-3">
<button type="button" class="min-w-0 flex-1 cursor-pointer border-0 bg-transparent p-0 text-left text-[#eceeee]" @click="openRoom">
<span class="body-strong flex items-center gap-2 truncate">
{{ call.title }}
<WIcon v-if="call.encrypted" name="lock" :size="14" label="End-to-end encrypted" class="shrink-0 text-[#3fb68b]" />
</span>
<span class="small block truncate text-[#eceeeeb3] tabular-nums" aria-live="polite">{{ status }}</span>
</button>
<button
v-if="call.isVideo || view.main"
type="button"
class="absolute top-2 right-2 flex h-9 w-9 cursor-pointer items-center justify-center rounded-full border-0 bg-[#00000080] text-[#eceeee] hover:bg-[#000000b3]"
class="flex h-9 w-9 cursor-pointer items-center justify-center rounded-full border-0 bg-[#ffffff1f] text-[#eceeee] hover:bg-[#ffffff33]"
:aria-label="expanded ? 'Shrink' : 'Expand'"
:title="expanded ? 'Shrink' : 'Expand'"
@click="expanded = !expanded"
@@ -178,12 +182,34 @@ function controlClass(c: Control): string {
</button>
</div>
<div class="flex items-center gap-3 px-4 pt-4">
<MxcAvatar v-if="!stage" :name="call.peerName" :mxc="call.peerAvatarMxc" :size="48" />
<button type="button" class="min-w-0 flex-1 cursor-pointer border-0 bg-transparent p-0 text-left text-[#eceeee]" @click="openRoom">
<span class="body-strong block truncate">{{ call.peerName }}</span>
<span class="small block truncate text-[#eceeeeb3] tabular-nums" aria-live="polite">{{ status }}</span>
</button>
<div :class="['relative px-4 pt-3', expanded ? 'min-h-0 flex-1' : '']">
<!-- Joining, or ringing someone who hasn't come yet. -->
<div
v-if="call.phase === 'joining' || calling"
:class="['flex flex-col items-center justify-center gap-4', expanded ? 'h-full' : 'aspect-video']"
>
<MxcAvatar v-if="call.callee" :name="call.callee.name" :mxc="call.callee.avatarMxc" :size="expanded ? 120 : 72" />
<WSpinner v-else :size="32" />
</div>
<!-- Two of you: them on the stage, you in the corner. -->
<div v-else-if="pair && remotes[0]" :class="['relative', expanded ? 'h-full' : 'aspect-video']">
<CallTile :tile="remotes[0]" :compact="!expanded" class="h-full w-full" />
<CallTile
v-if="self"
:tile="self"
compact
class="absolute right-3 bottom-3 aspect-video w-1/4 min-w-28 shadow-float"
/>
</div>
<!-- More: a grid, or whoever is speaking when shrunk. -->
<template v-else>
<div :class="['grid gap-2', expanded ? `h-full auto-rows-fr ${columns}` : 'aspect-video grid-cols-1']">
<CallTile v-for="t in shown" :key="t.id" :tile="t" :compact="!expanded" />
</div>
<CallTile v-for="t in hidden" :key="t.id" :tile="t" class="hidden" />
</template>
</div>
<div class="flex items-center justify-center gap-3 px-4 pt-3 pb-4">
@@ -193,7 +219,7 @@ function controlClass(c: Control): string {
type="button"
:class="['flex h-12 w-12 cursor-pointer items-center justify-center rounded-full border-0 transition-colors', controlClass(c)]"
:aria-label="c.label"
:aria-pressed="c.on === undefined || c.tone ? undefined : c.on"
:aria-pressed="c.tone ? undefined : c.on"
:title="c.label"
@click="c.action()"
>
+8 -1
View File
@@ -83,6 +83,13 @@ watchEffect(() => {
document.title = `${n ? `(${n}) ` : ''}${name ? `${name} · ` : ''}Warp`
})
// A room with a call going on says so in the list, instead of its last message.
function callPreview(roomId: string): string {
if (call.roomId === roomId) return 'You’re in the call'
const n = call.inCall(roomId)
return n === 1 ? 'Call in progress: 1 person' : `Call in progress: ${n} people`
}
const ownName = computed(() => profile.displayName || localpart(session.userId))
</script>
@@ -166,7 +173,7 @@ const ownName = computed(() => profile.displayName || localpart(session.userId))
:key="r.roomId"
:name="r.name"
:avatar-mxc="r.avatarMxc"
:preview="r.preview"
:preview="call.inCall(r.roomId) ? callPreview(r.roomId) : r.preview"
:time="formatListTime(r.ts)"
:unread="r.unread"
:mention="r.highlight > 0"
+3
View File
@@ -68,6 +68,9 @@ export async function startClient(creds: Credentials): Promise<MatrixClient> {
accessToken: creds.accessToken,
timelineSupport: true,
useAuthorizationHeader: true,
// Calls are MatrixRTC with LiveKit (stores/call.ts); the SDK's own
// 1:1 VoIP, and its polling for TURN servers, stay off.
disableVoip: true,
cryptoCallbacks,
// Emoji (SAS), and QR codes both ways: shown here, or scanned with
// the camera.
+44 -1
View File
@@ -114,6 +114,7 @@ function describeCall(ev: MatrixEvent, room: Room): string | null {
case EventType.CallInvite:
return `${sender} started a ${isVideoCallInvite(ev) ? 'video' : 'voice'} call`
case EventType.CallReject:
case EventType.RTCDecline:
return `${sender} declined the call`
case EventType.CallHangup:
switch (ev.getContent().reason) {
@@ -131,6 +132,47 @@ function describeCall(ev: MatrixEvent, room: Room): string | null {
}
}
/**
* Whether a call.member state event (MatrixRTC) says its device is in the
* call. Leaving is an empty event; the legacy form lists memberships.
*/
export function isActiveCallMember(content: Record<string, unknown>): boolean {
if (Object.keys(content).length === 0) return false
if (Array.isArray(content.memberships)) return content.memberships.length > 0
return true
}
/**
* Follows who is in a room's group call along the timeline, to say when
* it started and ended. Each device in the call has a call.member state
* event of its own; a member refreshing theirs says nothing.
*/
class CallTracker {
private readonly inCall = new Set<string>()
constructor(room: Room) {
const start = room.getLiveTimeline().getState(Direction.Backward)
for (const ev of start?.getStateEvents(EventType.GroupCallMemberPrefix) ?? []) {
if (isActiveCallMember(ev.getContent())) this.inCall.add(ev.getStateKey() ?? '')
}
}
describe(ev: MatrixEvent, room: Room): string | null {
const key = ev.getStateKey() ?? ''
const was = this.inCall.has(key)
const now = isActiveCallMember(ev.getContent())
if (was === now) return null
const sender = nameOf(room, ev.getSender() ?? '')
if (now) {
const first = this.inCall.size === 0
this.inCall.add(key)
return first ? `${sender} started a call` : `${sender} joined the call`
}
this.inCall.delete(key)
return this.inCall.size === 0 ? 'The call ended' : `${sender} left the call`
}
}
/** A one-line description of a state change, or null to leave it out. */
function describeState(ev: MatrixEvent, room: Room): string | null {
const sender = nameOf(room, ev.getSender() ?? '')
@@ -285,6 +327,7 @@ export function buildTimeline(c: MatrixClient, room: Room): TimelineItem[] {
const items: TimelineItem[] = []
let lastTs = 0
let prev: MessageItem | null = null
const calls = new CallTracker(room)
for (const ev of events) {
const type = ev.getType()
@@ -310,7 +353,7 @@ export function buildTimeline(c: MatrixClient, room: Room): TimelineItem[] {
}
continue
}
const text = describeState(ev, room)
const text = type === EventType.GroupCallMemberPrefix ? calls.describe(ev, room) : describeState(ev, room)
if (text) {
items.push({ kind: 'notice', key, ts, text })
prev = null
+616 -264
View File
@@ -1,7 +1,26 @@
import { CallEvent, CallFeedEvent, createNewMatrixCall, type MatrixCall, type MatrixClient } from 'matrix-js-sdk'
import { CallDirection, CallErrorCode, CallParty, CallState, CallType, type CallError } from 'matrix-js-sdk/lib/webrtc/call'
import { CallEventHandlerEvent } from 'matrix-js-sdk/lib/webrtc/callEventHandler'
import type { CallFeed } from 'matrix-js-sdk/lib/webrtc/callFeed'
import {
ClientEvent,
EventType,
MatrixEventEvent,
RelationType,
RoomEvent as MatrixRoomEvent,
SyncState,
type IRoomTimelineData,
type MatrixClient,
type MatrixEvent,
type Room as MatrixRoom,
} from 'matrix-js-sdk'
import { MatrixRTCSessionEvent, MatrixRTCSessionManagerEvents, type MatrixRTCSession } from 'matrix-js-sdk/lib/matrixrtc'
import {
BaseKeyProvider,
createKeyMaterialFromBuffer,
DisconnectReason,
isE2EESupported,
Room,
RoomEvent,
Track,
type Participant,
} from 'livekit-client'
import { defineStore } from 'pinia'
import { computed, ref, shallowRef } from 'vue'
import { errorMessage } from '@/lib/format'
@@ -9,332 +28,665 @@ import { client } from '@/matrix/client'
import { startTone, stopTone } from '@/matrix/tones'
import { useToasts } from '@/stores/toasts'
// Voice and video calls between two people: Matrix's 1:1 VoIP (m.call.*
// events in the room, WebRTC between the browsers), which the SDK runs.
// In an encrypted room the call events are encrypted like any other; the
// media itself is always encrypted by WebRTC (DTLS-SRTP).
// Voice and video calls, all of them: MatrixRTC (MSC4143) with a LiveKit
// SFU, the way Element Call does them. A call between two people is the
// same thing as a call between ten, with a ring on top.
//
// One call at a time. A call that comes in while one is going is answered
// "busy", as a phone would.
// Who is in a room's call is Matrix state (one
// org.matrix.msc3401.call.member event per device), which the SDK's
// MatrixRTC session keeps; the media goes through LiveKit, whose token the
// LiveKit JWT service hands out for an OpenID token from the homeserver.
// The session also schedules a delayed "left" with the homeserver (MSC4140)
// and keeps pushing it back, so a member whose browser crashes or loses
// its connection drops out of the call on its own.
//
// Starting a call in a room with one other person rings them: the first
// member sends an org.matrix.msc4075.rtc.notification, and the other side
// answers by joining or declines with org.matrix.msc4310.rtc.decline.
//
// In an encrypted room the media is end-to-end encrypted too: each member
// makes its own key, shares it with the others over encrypted to-device
// messages, and LiveKit encrypts every frame with it (encryptMedia).
//
// One call at a time.
export type CallPhase = 'ringing' | 'calling' | 'connecting' | 'connected' | 'ended'
/** A person (well, a device) in the call. */
export interface Tile {
/** The LiveKit identity: "@user:server:DEVICE". */
id: string
userId: string
name: string
avatarMxc?: string
local: boolean
camera?: Track
screen?: Track
/** A remote microphone, to play. Your own is never played back. */
audio?: Track
micMuted: boolean
speaking: boolean
}
/** How long the ended call stays on screen, saying why. */
const endedLingerMs = 2500
export type CallPhase = 'joining' | 'connected' | 'reconnecting'
/** A call ringing for you. */
export interface IncomingCall {
roomId: string
/** The notification event, which a decline refers to. */
eventId: string
callerId: string
callerName: string
callerAvatarMxc?: string
roomName: string
video: boolean
/** When the ring stops on its own (ms since the epoch). */
expiresAt: number
}
/** The MatrixRTC transport this client publishes: LiveKit via the JWT service. */
interface LivekitTransport {
type: 'livekit'
livekit_service_url: string
livekit_alias: string
[key: string]: unknown
}
/** How long a call rings, unless the notification says less. */
const ringLifetimeMs = 60_000
/** The most a received notification may ring for, whatever it says. */
const maxRingMs = 120_000
/**
* Feeds the per-participant keys MatrixRTC hands out to LiveKit's frame
* encryption. Each key belongs to one participant and index; there is no
* shared key and no ratcheting, the same settings as Element Call, so the
* two can be in one call.
*/
class MatrixKeyProvider extends BaseKeyProvider {
constructor() {
super({ ratchetWindowSize: 0, keyringSize: 256 })
}
async setParticipantKey(key: Uint8Array, index: number, identity: string) {
const material = await createKeyMaterialFromBuffer(key.buffer.slice(key.byteOffset, key.byteOffset + key.byteLength) as ArrayBuffer)
this.onSetEncryptionKey(material, identity, index)
}
}
/** The user of a LiveKit identity lk-jwt-service made: "@alice:example.org:DEVICE". */
function identityUser(identity: string): string {
// The device is after the last colon; a server name may itself have a port.
const i = identity.lastIndexOf(':')
return i > 0 ? identity.slice(0, i) : identity
}
/** How many people (not devices) are in a session. */
function countMembers(session: MatrixRTCSession): number {
return new Set(session.memberships.map((m) => m.userId)).size
}
/** The other person in a room of two, or undefined. */
function otherOfTwo(room: MatrixRoom, me: string): string | undefined {
if (room.getJoinedMemberCount() !== 2) return undefined
return room.getJoinedMembers().find((m) => m.userId !== me)?.userId
}
export const useCall = defineStore('call', () => {
const call = shallowRef<MatrixCall | null>(null)
/** The room whose call this client is in. */
const roomId = ref<string | null>(null)
const phase = ref<CallPhase | null>(null)
/** Bumps on every change the call view shows. */
const version = ref(0)
/** Seconds since the call connected. */
/** How many people are in each room's call, for every room with one. */
const active = ref<Record<string, number>>({})
const lk = shallowRef<Room | null>(null)
/** Media is end-to-end encrypted. */
const encrypted = ref(false)
/** Joined with the camera on. */
const video = ref(false)
/** Seconds since you joined. */
const length = ref(0)
/** Why the call ended, once it has. */
const endReason = ref('')
/** We rang someone and they haven't come yet. */
const ringingOut = ref(false)
/** A call ringing for you. */
const incoming = ref<IncomingCall | null>(null)
let attached: MatrixClient | null = null
let clearTimer = 0
/** The call got as far as connecting media, at some point. */
let connected = false
/** The feeds whose mute state is listened to. */
let feeds: CallFeed[] = []
const phase = computed<CallPhase | null>(() => {
void version.value
const c = call.value
if (!c) return null
switch (c.state) {
case CallState.Ringing:
return 'ringing'
case CallState.Fledgling:
case CallState.WaitLocalMedia:
case CallState.CreateOffer:
case CallState.InviteSent:
return c.direction === CallDirection.Inbound ? 'connecting' : 'calling'
case CallState.CreateAnswer:
case CallState.Connecting:
return 'connecting'
case CallState.Connected:
return 'connected'
default:
return 'ended'
}
})
const isVideo = computed(() => {
void version.value
return call.value?.type === CallType.Video
})
const peerName = computed(() => {
void version.value
const c = call.value
if (!c) return ''
const member = c.getOpponentMember()
if (member) return member.name
const room = attached?.getRoom(c.roomId)
return room?.name ?? c.invitee ?? ''
})
const peerAvatarMxc = computed(() => {
void version.value
return call.value?.getOpponentMember()?.getMxcAvatarUrl() ?? undefined
})
let session: MatrixRTCSession | null = null
let keyProvider: MatrixKeyProvider | null = null
let lengthTimer = 0
let ringOutTimer = 0
let ringInTimer = 0
/** The ring notification we sent, which a decline refers to. */
let sentNotification: string | null = null
const watched = new Map<string, MatrixRTCSession>()
function bump() {
version.value++
}
// A feed muting its camera or microphone is not a change in the call's
// feeds, so each feed is listened to as well.
function watchFeeds(c: MatrixCall | null) {
for (const f of feeds) {
f.off(CallFeedEvent.MuteStateChanged, bump)
f.off(CallFeedEvent.NewStream, bump)
}
feeds = c ? c.getFeeds() : []
for (const f of feeds) {
f.on(CallFeedEvent.MuteStateChanged, bump)
f.on(CallFeedEvent.NewStream, bump)
}
const joined = computed(() => roomId.value !== null)
/** Whether this browser can take part in calls at all. */
function supported(): boolean {
return typeof window !== 'undefined' && 'RTCPeerConnection' in window
}
function onFeeds() {
watchFeeds(call.value)
bump()
}
// --- Tones --------------------------------------------------------------
function tones() {
const p = phase.value
if (p === 'ringing') startTone('ring')
else if (p === 'calling') startTone('ringback')
if (incoming.value && !joined.value) startTone('ring')
else if (ringingOut.value) startTone('ringback')
else stopTone()
}
function onState(state: CallState) {
if (state === CallState.Connected) connected = true
// --- Who is in which room's call ----------------------------------------
function refresh(rid: string) {
const s = watched.get(rid)
const n = s ? countMembers(s) : 0
const next = { ...active.value }
if (n > 0) next[rid] = n
else delete next[rid]
active.value = next
// The caller gave up, or someone else answered for you.
if (incoming.value?.roomId === rid && n === 0) clearIncoming()
// Someone came: stop ringing them.
if (rid === roomId.value && ringingOut.value && n > 1) stopRingingOut()
}
function watchSession(rid: string, s: MatrixRTCSession) {
if (watched.get(rid) === s) return
watched.get(rid)?.off(MatrixRTCSessionEvent.MembershipsChanged, onMemberships)
watched.set(rid, s)
s.on(MatrixRTCSessionEvent.MembershipsChanged, onMemberships)
refresh(rid)
// Memberships are parsed asynchronously: a call already going on when
// the room loads only shows up once that is done.
void s.initialMembershipCalculated.then(() => {
refresh(rid)
bump()
})
}
/** Follows the call of every room the client knows. */
function watchRoom(room: MatrixRoom) {
if (!attached) return
watchSession(room.roomId, attached.matrixRTC.getRoomSession(room))
}
function watchAllRooms() {
for (const r of attached?.getRooms() ?? []) watchRoom(r)
}
function onSync(state: SyncState) {
if (state === SyncState.Prepared) watchAllRooms()
}
function onMemberships() {
for (const rid of watched.keys()) refresh(rid)
bump()
tones()
}
function onLength(seconds: number) {
length.value = seconds
function onSessionStarted(rid: string, s: MatrixRTCSession) {
watchSession(rid, s)
}
function onError(err: CallError) {
// The hangup that follows says the rest.
if (err.code === CallErrorCode.NoUserMedia) {
forgetFailedMedia()
useToasts().error(isVideo.value ? 'Can’t use your camera or microphone. Check that the browser may use them.' : 'Can’t use your microphone. Check that the browser may use it.')
} else {
useToasts().error(`Call failed: ${errorMessage(err)}`)
function onSessionEnded(rid: string) {
refresh(rid)
}
// --- Ringing ------------------------------------------------------------
/**
* A ring notification or a decline arriving live. A ring counts when it
* is someone else's, still within its lifetime, mentions you (or the
* room), and you aren't in that call already.
*/
function onTimeline(ev: MatrixEvent, room: MatrixRoom | undefined, toStart: boolean | undefined, _removed: boolean, data: IRoomTimelineData) {
if (toStart || !data?.liveEvent || !room) return
// In an encrypted room this is still ciphertext; onDecrypted has it.
if (ev.isEncrypted() && ev.isBeingDecrypted()) return
onCallEvent(ev, room)
}
/** An encrypted ring or decline, once it can be read: only recent ones count. */
function onDecrypted(ev: MatrixEvent) {
if (ev.isDecryptionFailure() || Date.now() - ev.getTs() > maxRingMs) return
const room = attached?.getRoom(ev.getRoomId() ?? '')
if (room) onCallEvent(ev, room)
}
/** Event IDs already acted on, so a ring isn't taken twice. */
const seen = new Set<string>()
function onCallEvent(ev: MatrixEvent, room: MatrixRoom) {
if (!attached) return
const type = ev.getType()
if (type !== EventType.RTCNotification && type !== EventType.RTCDecline) return
const id = ev.getId()
if (id) {
if (seen.has(id)) return
seen.add(id)
}
}
function onHangup(c: MatrixCall) {
if (c !== call.value) return
endReason.value = describeHangup(c, connected)
bump()
stopTone()
window.clearTimeout(clearTimer)
clearTimer = window.setTimeout(() => {
if (call.value === c) release()
}, endedLingerMs)
}
function onReplaced(newCall: MatrixCall) {
// Both of you called at once: the SDK keeps one of the two calls.
show(newCall)
}
function listen(c: MatrixCall, on: boolean) {
const m = on ? 'on' : 'off'
c[m](CallEvent.State, onState)
c[m](CallEvent.FeedsChanged, onFeeds)
c[m](CallEvent.LocalHoldUnhold, bump)
c[m](CallEvent.RemoteHoldUnhold, bump)
c[m](CallEvent.LengthChanged, onLength)
c[m](CallEvent.Error, onError)
c[m](CallEvent.Hangup, onHangup)
c[m](CallEvent.Replaced, onReplaced)
}
function show(c: MatrixCall) {
if (call.value) listen(call.value, false)
window.clearTimeout(clearTimer)
call.value = c
connected = false
length.value = 0
endReason.value = ''
listen(c, true)
watchFeeds(c)
bump()
tones()
}
function release() {
if (call.value) listen(call.value, false)
watchFeeds(null)
window.clearTimeout(clearTimer)
call.value = null
stopTone()
bump()
}
function onIncoming(incoming: MatrixCall) {
if (call.value && phase.value !== 'ended') {
incoming.hangup(CallErrorCode.UserBusy, false)
const me = attached.getUserId() ?? ''
if (type === EventType.RTCNotification) {
if (ev.getSender() === me || roomId.value === room.roomId) return
const content = ev.getContent()
const mentions = content['m.mentions'] as { user_ids?: string[]; room?: boolean } | undefined
if (!mentions?.room && !mentions?.user_ids?.includes(me)) return
// A ring for a call of two; in a bigger room the banner says enough.
if (content.notification_type !== 'ring' && !otherOfTwo(room, me)) return
const sentAt = typeof content.sender_ts === 'number' ? Math.min(content.sender_ts, ev.getTs()) : ev.getTs()
const lifetime = Math.min(typeof content.lifetime === 'number' ? content.lifetime : ringLifetimeMs, maxRingMs)
const expiresAt = sentAt + lifetime
if (expiresAt <= Date.now()) return
if (joined.value) return // Busy: you see the banner when you're free.
const caller = room.getMember(ev.getSender() ?? '')
incoming.value = {
roomId: room.roomId,
eventId: ev.getId() ?? '',
callerId: ev.getSender() ?? '',
callerName: caller?.name ?? ev.getSender() ?? '',
callerAvatarMxc: caller?.getMxcAvatarUrl() ?? undefined,
roomName: room.name,
video: content['m.call.intent'] !== 'audio',
expiresAt,
}
window.clearTimeout(ringInTimer)
ringInTimer = window.setTimeout(clearIncoming, expiresAt - Date.now())
tones()
return
}
show(incoming)
// Hung up (or answered elsewhere) before we did anything.
if (incoming.callHasEnded()) onHangup(incoming)
if (type === EventType.RTCDecline) {
const target = ev.getContent()['m.relates_to']?.event_id
if (!target) return
// You declined on another of your devices: stop ringing here.
if (ev.getSender() === me && incoming.value?.eventId === target) {
clearIncoming()
return
}
// The one person you rang said no.
if (target === sentNotification && ringingOut.value && room.roomId === roomId.value) {
useToasts().info(`${room.getMember(ev.getSender() ?? '')?.name ?? 'They'} declined the call.`)
void leave()
}
}
}
function clearIncoming() {
window.clearTimeout(ringInTimer)
incoming.value = null
tones()
}
function stopRingingOut() {
window.clearTimeout(ringOutTimer)
ringingOut.value = false
tones()
}
function onNotificationSent(sent: { event_id: string; lifetime?: number }) {
sentNotification = sent.event_id
}
/** Answers the call ringing for you. */
async function answer(withVideo = true) {
const call = incoming.value
if (!call) return
clearIncoming()
await join(call.roomId, withVideo && call.video)
}
/** Declines the call ringing for you. The caller stops ringing when it's the two of you. */
async function decline() {
const call = incoming.value
if (!call) return
clearIncoming()
try {
await client().sendEvent(call.roomId, EventType.RTCDecline, {
'm.relates_to': { rel_type: RelationType.Reference, event_id: call.eventId },
})
} catch (e) {
console.error('call: decline: ', e)
}
}
/** Starts listening for calls on the running client. */
function attach() {
const c = client()
if (attached === c) return
if (attached) attached.off(CallEventHandlerEvent.Incoming, onIncoming)
release()
if (attached) {
attached.matrixRTC.off(MatrixRTCSessionManagerEvents.SessionStarted, onSessionStarted)
attached.matrixRTC.off(MatrixRTCSessionManagerEvents.SessionEnded, onSessionEnded)
attached.off(ClientEvent.Room, watchRoom)
attached.off(ClientEvent.Sync, onSync)
attached.off(MatrixRoomEvent.Timeline, onTimeline)
attached.off(MatrixEventEvent.Decrypted, onDecrypted)
}
for (const s of watched.values()) s.off(MatrixRTCSessionEvent.MembershipsChanged, onMemberships)
watched.clear()
active.value = {}
clearIncoming()
void leave()
attached = c
c.on(CallEventHandlerEvent.Incoming, onIncoming)
c.matrixRTC.on(MatrixRTCSessionManagerEvents.SessionStarted, onSessionStarted)
c.matrixRTC.on(MatrixRTCSessionManagerEvents.SessionEnded, onSessionEnded)
c.on(ClientEvent.Room, watchRoom)
c.on(ClientEvent.Sync, onSync)
c.on(MatrixRoomEvent.Timeline, onTimeline)
c.on(MatrixEventEvent.Decrypted, onDecrypted)
watchAllRooms()
}
/** Whether this browser can make calls at all. */
function supported(): boolean {
return Boolean(attached?.supportsVoip())
}
// --- Joining ------------------------------------------------------------
async function place(roomId: string, invitee: string | undefined, video: boolean) {
if (call.value && phase.value !== 'ended') {
useToasts().error('You’re already in a call.')
return
/** The LiveKit JWT service to use: the one the call already uses, else this server's. */
async function chooseTransport(s: MatrixRTCSession, rid: string): Promise<LivekitTransport> {
const oldest = s.memberships[0]
const theirs = oldest?.getTransport(oldest)
if (theirs && theirs.type === 'livekit' && typeof theirs.livekit_service_url === 'string') {
return { type: 'livekit', livekit_service_url: theirs.livekit_service_url, livekit_alias: rid }
}
const c = createNewMatrixCall(client(), roomId, { invitee })
if (!c) {
useToasts().error('This browser can’t make calls.')
return
}
show(c)
const c = client()
// The homeserver lists its backends (MSC4143); a server that predates
// the endpoint has them in its discovery document instead.
let foci: unknown
try {
if (video) await c.placeVideoCall()
else await c.placeVoiceCall()
} catch (e) {
// The SDK has hung up already and said why through CallEvent.Error.
console.error('call: ', e)
foci = await c._unstable_getRTCTransports()
} catch {
const res = await fetch(`${c.baseUrl.replace(/\/$/, '')}/.well-known/matrix/client`)
if (!res.ok) throw new Error('This server doesn’t offer calls.')
const doc = (await res.json()) as Record<string, unknown>
foci = doc['org.matrix.msc4143.rtc_foci']
}
const focus = Array.isArray(foci)
? (foci as Record<string, unknown>[]).find((f) => f?.type === 'livekit' && typeof f.livekit_service_url === 'string')
: undefined
if (!focus) throw new Error('This server doesn’t offer calls.')
return { type: 'livekit', livekit_service_url: String(focus.livekit_service_url), livekit_alias: rid }
}
/** Answers the ringing call; a video call with the camera off when `video` is false. */
async function answer(video = true) {
const c = call.value
if (!c || phase.value !== 'ringing') return
stopTone()
/** A LiveKit URL and token, from the JWT service, for an OpenID token. */
async function livekitToken(transport: LivekitTransport, rid: string): Promise<{ url: string; jwt: string }> {
const c = client()
const openid = await c.getOpenIdToken()
const res = await fetch(`${transport.livekit_service_url.replace(/\/$/, '')}/sfu/get`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ room: rid, openid_token: openid, device_id: c.getDeviceId() }),
})
if (!res.ok) throw new Error(`The call service refused to let you in (${res.status}).`)
const body = (await res.json()) as { url?: string; jwt?: string }
if (!body.url || !body.jwt) throw new Error('The call service gave an unusable answer.')
return { url: body.url, jwt: body.jwt }
}
function listen(r: Room) {
r.on(RoomEvent.ParticipantConnected, bump)
.on(RoomEvent.ParticipantDisconnected, bump)
.on(RoomEvent.TrackSubscribed, bump)
.on(RoomEvent.TrackUnsubscribed, bump)
.on(RoomEvent.TrackMuted, bump)
.on(RoomEvent.TrackUnmuted, bump)
.on(RoomEvent.LocalTrackPublished, bump)
.on(RoomEvent.LocalTrackUnpublished, bump)
.on(RoomEvent.ActiveSpeakersChanged, bump)
.on(RoomEvent.ParticipantEncryptionStatusChanged, bump)
.on(RoomEvent.Reconnecting, () => {
phase.value = 'reconnecting'
})
.on(RoomEvent.Reconnected, () => {
phase.value = 'connected'
})
.on(RoomEvent.Disconnected, (reason?: DisconnectReason) => {
if (lk.value !== r) return
if (reason !== DisconnectReason.CLIENT_INITIATED) useToasts().error('You were disconnected from the call.')
void leave()
})
}
function onEncryptionKey(key: Uint8Array, index: number, _membership: unknown, backendIdentity: string) {
void keyProvider?.setParticipantKey(key, index, backendIdentity)
}
function onMembershipError(e: unknown) {
useToasts().error(`The call stopped working: ${errorMessage(e)}`)
void leave()
}
/**
* Joins the call in a room, or starts one. Starting a call in a room of
* two rings the other person.
*/
async function join(rid: string, withVideo: boolean) {
if (roomId.value === rid) return
if (roomId.value) await leave()
const c = client()
const room = c.getRoom(rid)
if (!room) return
if (incoming.value?.roomId === rid) clearIncoming()
roomId.value = rid
phase.value = 'joining'
video.value = withVideo
length.value = 0
sentNotification = null
bump()
const s = c.matrixRTC.getRoomSession(room)
watchSession(rid, s)
session = s
const e2ee = room.hasEncryptionStateEvent()
encrypted.value = e2ee
const me = c.getUserId()!
const ringing = countMembers(s) === 0 && otherOfTwo(room, me) !== undefined
try {
await c.answer(true, video && c.type === CallType.Video)
if (e2ee && !isE2EESupported()) {
throw new Error('This browser can’t encrypt call media, and this room is end-to-end encrypted.')
}
const transport = await chooseTransport(s, rid)
const { url, jwt } = await livekitToken(transport, rid)
if (roomId.value !== rid) return
keyProvider = e2ee ? new MatrixKeyProvider() : null
const r = new Room({
adaptiveStream: true,
dynacast: true,
e2ee: keyProvider
? { keyProvider, worker: new Worker(new URL('livekit-client/e2ee-worker', import.meta.url), { type: 'module' }) }
: undefined,
})
lk.value = r
listen(r)
if (keyProvider) await r.setE2EEEnabled(true)
// The Matrix side: announce ourselves (and schedule our own delayed
// leave), ring the other person if we're the first, and in an
// encrypted room start sharing media keys.
const deviceId = c.getDeviceId()!
s.on(MatrixRTCSessionEvent.EncryptionKeyChanged, onEncryptionKey)
s.on(MatrixRTCSessionEvent.MembershipManagerError, onMembershipError)
s.on(MatrixRTCSessionEvent.DidSendCallNotification, onNotificationSent)
s.join({ userId: me, deviceId, memberId: `${me}:${deviceId}` }, [transport], {
manageMediaKeys: e2ee,
callIntent: withVideo ? 'video' : 'audio',
notificationType: ringing ? 'ring' : countMembers(s) === 0 ? 'notification' : undefined,
notificationLifetimeMs: ringLifetimeMs,
})
if (e2ee) s.reemitEncryptionKeys()
if (ringing) {
ringingOut.value = true
ringOutTimer = window.setTimeout(() => {
if (!ringingOut.value || roomId.value !== rid) return
useToasts().info('No answer.')
void leave()
}, ringLifetimeMs)
tones()
}
await r.connect(url, jwt)
if (roomId.value !== rid) {
await r.disconnect()
return
}
phase.value = 'connected'
window.clearInterval(lengthTimer)
const since = Date.now()
lengthTimer = window.setInterval(() => (length.value = Math.floor((Date.now() - since) / 1000)), 1000)
bump()
try {
await r.localParticipant.setMicrophoneEnabled(true)
} catch (e) {
useToasts().error(`Can’t use your microphone: ${errorMessage(e)}`)
}
if (withVideo) {
try {
await r.localParticipant.setCameraEnabled(true)
} catch (e) {
useToasts().error(`Can’t use your camera: ${errorMessage(e)}`)
}
}
// Browsers only play audio after a gesture; joining was one.
await r.startAudio().catch(() => undefined)
bump()
} catch (e) {
console.error('call: ', e)
useToasts().error(errorMessage(e))
await leave()
}
}
function decline() {
const c = call.value
if (!c) return
if (phase.value === 'ringing') c.reject()
else hangup()
}
function hangup() {
const c = call.value
if (!c) return
if (phase.value === 'ended') release()
else c.hangup(CallErrorCode.UserHangup, false)
}
async function toggleMic() {
const c = call.value
if (!c) return
await c.setMicrophoneMuted(!c.isMicrophoneMuted())
/** Leaves the call this client is in. */
async function leave() {
const s = session
const r = lk.value
session = null
lk.value = null
keyProvider = null
sentNotification = null
window.clearInterval(lengthTimer)
stopRingingOut()
const rid = roomId.value
roomId.value = null
phase.value = null
bump()
if (s) {
s.off(MatrixRTCSessionEvent.EncryptionKeyChanged, onEncryptionKey)
s.off(MatrixRTCSessionEvent.MembershipManagerError, onMembershipError)
s.off(MatrixRTCSessionEvent.DidSendCallNotification, onNotificationSent)
}
await Promise.allSettled([r?.disconnect(), s?.leave(5000)])
if (rid) refresh(rid)
}
async function toggleCamera() {
const c = call.value
if (!c) return
await c.setLocalVideoMuted(!c.isLocalVideoMuted())
bump()
}
// --- What the view shows ------------------------------------------------
async function toggleScreenShare() {
const c = call.value
if (!c) return
const tiles = computed<Tile[]>(() => {
void version.value
const r = lk.value
const rid = roomId.value
if (!r || !rid) return []
const room = attached?.getRoom(rid)
const speakers = new Set(r.activeSpeakers.map((p) => p.identity))
const tile = (p: Participant, local: boolean): Tile => {
const userId = identityUser(p.identity)
const member = room?.getMember(userId)
const cam = p.getTrackPublication(Track.Source.Camera)
const screen = p.getTrackPublication(Track.Source.ScreenShare)
const mic = p.getTrackPublication(Track.Source.Microphone)
return {
id: p.identity,
userId,
name: member?.name ?? userId,
avatarMxc: member?.getMxcAvatarUrl() ?? undefined,
local,
camera: cam?.track && !cam.isMuted ? cam.track : undefined,
screen: screen?.track && !screen.isMuted ? screen.track : undefined,
audio: !local && mic?.track ? mic.track : undefined,
micMuted: !mic || mic.isMuted,
speaking: speakers.has(p.identity),
}
}
return [tile(r.localParticipant, true), ...[...r.remoteParticipants.values()].map((p) => tile(p, false))]
})
/** The room's name, or for a call of two, the other person's. */
const title = computed(() => {
void version.value
const rid = roomId.value
const room = rid ? attached?.getRoom(rid) : undefined
if (!room) return ''
const other = otherOfTwo(room, attached?.getUserId() ?? '')
return other ? (room.getMember(other)?.name ?? room.name) : room.name
})
/** The person you're calling, for the "ringing" screen of a call of two. */
const callee = computed(() => {
void version.value
const rid = roomId.value
const room = rid ? attached?.getRoom(rid) : undefined
const other = room ? otherOfTwo(room, attached?.getUserId() ?? '') : undefined
const member = other ? room?.getMember(other) : undefined
return member ? { name: member.name, avatarMxc: member.getMxcAvatarUrl() ?? undefined } : null
})
const micOn = computed(() => {
void version.value
return lk.value?.localParticipant.isMicrophoneEnabled ?? false
})
const cameraOn = computed(() => {
void version.value
return lk.value?.localParticipant.isCameraEnabled ?? false
})
const sharing = computed(() => {
void version.value
return lk.value?.localParticipant.isScreenShareEnabled ?? false
})
async function toggle(what: 'mic' | 'camera' | 'screen') {
const p = lk.value?.localParticipant
if (!p) return
try {
await c.setScreensharingEnabled(!c.isScreensharing())
if (what === 'mic') await p.setMicrophoneEnabled(!p.isMicrophoneEnabled)
else if (what === 'camera') await p.setCameraEnabled(!p.isCameraEnabled)
else await p.setScreenShareEnabled(!p.isScreenShareEnabled, { audio: true })
} catch (e) {
useToasts().error(`Can’t share your screen: ${errorMessage(e)}`)
// Cancelling the browser's screen picker is not an error worth a toast.
if (!(e instanceof DOMException && e.name === 'NotAllowedError' && what === 'screen')) {
useToasts().error(errorMessage(e))
}
}
bump()
}
function toggleHold() {
const c = call.value
if (!c) return
c.setRemoteOnHold(!c.isRemoteOnHold())
bump()
/** How many people are in a room's call; 0 when there is none. */
function inCall(rid: string): number {
return active.value[rid] ?? 0
}
return {
call,
version,
length,
endReason,
roomId,
phase,
isVideo,
peerName,
peerAvatarMxc,
attach,
version,
active,
joined,
encrypted,
video,
length,
ringingOut,
incoming,
tiles,
title,
callee,
micOn,
cameraOn,
sharing,
supported,
place,
attach,
join,
leave,
answer,
decline,
hangup,
toggleMic,
toggleCamera,
toggleScreenShare,
toggleHold,
toggle,
inCall,
}
})
/**
* The SDK's media handler queues every request for the microphone and
* camera behind the previous one, and never lets go of a request that
* failed: once you said no, every later call would fail the same way until
* the page reloads. Dropping the failed request lets the next call ask again.
*/
function forgetFailedMedia() {
const handler = client().getMediaHandler() as unknown as { getMediaStreamPromise?: Promise<MediaStream> }
handler.getMediaStreamPromise = undefined
}
function describeHangup(c: MatrixCall, connected: boolean): string {
switch (c.hangupReason) {
case CallErrorCode.UserBusy:
return 'Busy'
case CallErrorCode.InviteTimeout:
return c.direction === CallDirection.Inbound ? 'Missed call' : 'No answer'
case CallErrorCode.AnsweredElsewhere:
return 'Answered in another session'
case CallErrorCode.IceFailed:
return 'Couldn’t connect'
case CallErrorCode.NoUserMedia:
return 'No microphone or camera'
case CallErrorCode.Replaced:
return 'Call replaced'
default:
if (!connected) {
const outbound = c.direction === CallDirection.Outbound
if (c.hangupParty === CallParty.Remote) return outbound ? 'Declined' : 'Missed call'
return outbound ? 'Call cancelled' : 'Declined'
}
return 'Call ended'
}
}
+3 -2
View File
@@ -60,8 +60,9 @@ export function describeEvent(ev: MatrixEvent, me: string, room: Room, withSende
// own placeholder text: never show that.
if (ev.isDecryptionFailure()) return withSender ? `${senderName(ev, me, room)}: Can’t decrypt this message` : 'Can’t decrypt this message'
if (type === EventType.RoomMessageEncrypted) return 'Encrypted message'
if (type === EventType.CallInvite) {
const text = isVideoCallInvite(ev) ? 'Video call' : 'Voice call'
if (type === EventType.CallInvite || type === EventType.RTCNotification) {
const video = type === EventType.CallInvite ? isVideoCallInvite(ev) : ev.getContent()['m.call.intent'] !== 'audio'
const text = video ? 'Video call' : 'Voice call'
return withSender ? `${senderName(ev, me, room)}: ${text}` : text
}
if (type !== EventType.RoomMessage && type !== EventType.Sticker) return null
+29 -13
View File
@@ -54,19 +54,25 @@ const canSend = computed(() => {
const infoOpen = ref(false)
// Calls are between two people: offered in a room with just you and one
// other person in it.
const callee = computed(() => {
// Calls, in any room with someone else in it: starting one in a room of
// two rings the other person. One call at a time, wherever it is.
const canCall = computed(() => {
void chat.version
const r = room.value
if (!r || r.getJoinedMemberCount() !== 2) return undefined
return r.getJoinedMembers().find((m) => m.userId !== me)?.userId
return Boolean(r) && membership.value === KnownMembership.Join && call.supported() && (r?.getJoinedMemberCount() ?? 0) > 1
})
const canCall = computed(() => Boolean(callee.value) && call.supported())
// One call at a time, wherever it is.
const inCall = computed(() => Boolean(call.call) && call.phase !== 'ended')
function startCall(video: boolean) {
void call.place(props.roomId, callee.value, video)
const inCall = computed(() => call.joined)
// A call going on here that you're not in.
const callHere = computed(() => call.inCall(props.roomId))
const inThisCall = computed(() => call.roomId === props.roomId)
const callPeople = computed(() => {
void call.version
void chat.version
const n = callHere.value
return n === 1 ? '1 person is' : `${n} people are`
})
function joinCall(video: boolean) {
void call.join(props.roomId, video)
}
// Someone in an encrypted room whose identity changed: said before you
@@ -294,13 +300,23 @@ const placeholder = computed(() => `Message ${summary.value?.name ?? 'this room'
<span v-if="!encrypted" class="caption hidden items-center gap-1 text-ink-muted sm:flex">
<WIcon name="shield-alert" :size="14" /> Not encrypted
</span>
<template v-if="canCall">
<WIconButton icon="phone" label="Voice call" :disabled="inCall" @click="startCall(false)" />
<WIconButton icon="video" label="Video call" :disabled="inCall" @click="startCall(true)" />
<template v-if="canCall && !callHere">
<WIconButton icon="phone" label="Voice call" :disabled="inCall" @click="joinCall(false)" />
<WIconButton icon="video" label="Video call" :disabled="inCall" @click="joinCall(true)" />
</template>
<WIconButton icon="users" :label="infoOpen ? 'Hide room info' : 'Room info'" :aria-pressed="infoOpen" @click="infoOpen = !infoOpen" />
</header>
<div v-if="callHere && canCall && !inThisCall" class="shrink-0 border-b border-line bg-surface px-4 py-2">
<WBanner tone="info" icon="video" title="A call is going on in this room">
<div class="flex flex-wrap items-center gap-2">
<span class="min-w-0 flex-1">{{ callPeople }} in it.</span>
<WButton size="sm" :disabled="inCall" @click="joinCall(false)">Join with audio</WButton>
<WButton size="sm" variant="primary" :disabled="inCall" @click="joinCall(true)">Join call</WButton>
</div>
</WBanner>
</div>
<RoomTimeline :room="room" :is-direct="summary?.isDirect ?? false" />
<div class="shrink-0 bg-surface-sunken px-4 pt-2 pb-4">
+2
View File
@@ -26,6 +26,8 @@ export default defineConfig({
proxy: {
'/_matrix/client': { target: homeserver, changeOrigin: true },
'/_matrix/media': { target: homeserver, changeOrigin: true },
// Where the client finds the group call backend (rtc_foci).
'/.well-known/matrix/client': { target: homeserver, changeOrigin: true },
},
},
})