diff --git a/apps/README.md b/apps/README.md index 9bf56bf..259e2b8 100644 --- a/apps/README.md +++ b/apps/README.md @@ -6,7 +6,7 @@ | [`admin/`](admin/README.md) | admin web panel, served by its own optional `warp-admin` binary | done for the current admin API | | [`web/`](web/README.md) | web client (Vue SPA on matrix-js-sdk), served by its own optional `warp-web` binary | rooms, timeline, sending text and files, invites; E2EE next | | `desktop/` | Wails v3 shell reusing the web frontend | next: web has a timeline | -| [`mobile/`](mobile/README.md) | Flutter app on the famedly Matrix SDK, Android first | sign-in, rooms, timeline, files, E2EE with verification, push | +| [`mobile/`](mobile/README.md) | Flutter app on the famedly Matrix SDK, Android first | rooms, timeline, files, replies, reactions, threads, spaces, E2EE with QR and emoji verification, push, calls, sliding sync | The frontend packages are a pnpm workspace (`pnpm-workspace.yaml` at the repository root); the Go hosts join `go.work`. diff --git a/apps/mobile/README.md b/apps/mobile/README.md index 0bb7a8b..12f1793 100644 --- a/apps/mobile/README.md +++ b/apps/mobile/README.md @@ -33,6 +33,18 @@ radii), the Outfit and Figtree fonts, the Warp symbol. failed one says *Not sent*, with Retry and Delete on a long press. - Typing notifications, read receipts, deleting messages, editing shown as *edited*. +- Replies (the message answered is quoted above the reply) and reactions + (six quick ones and any other; chips with counts, tap to add or take + back, long press for who reacted), from a message's long press. +- Threads: *Reply in thread* opens the thread, its root and replies, with + a composer that sends into it. Replies stay out of the main timeline; + the root shows "N replies". Room info lists a room's threads. +- Spaces: the room list's title picks a space, which narrows the list to + its rooms (and its subspaces'). A space's browser (`/hierarchy`) shows + the rooms you're in and the ones you can join; admins add existing or + new rooms. *New space* in the new-chat sheet. +- Voice and video calls, MatrixRTC with a LiveKit SFU like the web client + (see below). - New rooms (end-to-end encrypted by default, public or not), direct messages (encrypted), joining by address. Room info: members with their role and verification shield, invite, rename, topic, notification level, @@ -42,8 +54,9 @@ radii), the Outfit and Figtree fonts, the Warp symbol. backup in one step, behind a recovery key shown once. - A new session verifies itself with the recovery key (which also restores older messages from the backup) or from another session. - - Verifying people and sessions by comparing emoji, from room info, the - session list, or an incoming request. + - Verifying people and sessions from room info, the session list, or an + incoming request: by QR code (show this device's, or scan the other's + with the camera) when both sides can, else by comparing emoji. - A message this session can't decrypt says why, not just "encrypted". - *Reset encryption* for a lost recovery key. - Sessions: every session of the account, which are verified, verify one, @@ -53,10 +66,57 @@ radii), the Outfit and Figtree fonts, the Warp symbol. [UnifiedPush](https://unifiedpush.org) (below). - Settings: display name, avatar, theme (system, light, dark). -**Not yet:** replies, reactions, threads, spaces, voice and video calls, -QR-code verification, Simplified Sliding Sync (the SDK has no client for -it, so the app uses `/sync`), a per-message trust shield like the web -client's. +**Not yet:** a per-message trust shield like the web client's, screen +sharing in calls, calls that keep going with the app in the background +(Android stops the process; a foreground service would keep it), and +read receipts per thread. + +## Sync + +When the homeserver advertises `org.matrix.simplified_msc3575`, the app +uses Simplified Sliding Sync (MSC4186), as Element X does; otherwise +`/sync`. The famedly SDK only knows `/sync`, so `WarpClient` +(`lib/matrix/sliding_sync.dart`) answers the SDK's sync call with a +sliding sync request and rewrites the response into the `/sync` shape: +encryption, the database and timelines don't know the difference. + +- The first request asks for the 20 most recent rooms with a handful of + state events each, so the room list appears at once; the next ones + widen the window to every room (up to 1000). +- Two lists: rooms, and direct chats with all their members (so a DM's + name is the other person's). Members are otherwise `$LAZY`. +- Extensions: to-device, E2EE, account data, receipts, typing. +- The SDK's token becomes `ss1||`. `M_UNKNOWN_POS` + (the server forgot the connection, after a restart or 30 idle minutes) + starts a new one. + +## Calls + +Every call, between two people or more, is a MatrixRTC call (MSC4143) +with its media through a LiveKit SFU: the model of the web client and +Element Call, with the same events, so the three can share a call. See +[docs/calls.md](../../docs/calls.md) for the server side +(`client_api.rtc`, LiveKit and the JWT service). + +- Voice and video buttons in the room header. A room of two rings the + other person (`org.matrix.msc4075.rtc.notification`): an incoming-call + card with Answer, Voice only and Decline + (`org.matrix.msc4310.rtc.decline`), and an insistent ringtone + notification, full screen on a locked phone. +- Who is in a call is `org.matrix.msc3401.call.member` state, one event + per device. Before sending ours the app schedules an empty one with the + homeserver (a delayed event, MSC4140) and pushes it back every five + seconds, so a phone that dies drops out of the call by itself. +- The LiveKit token comes from the JWT service named by the call already + going on, else by the homeserver (`rtc/transports`, or + `org.matrix.msc4143.rtc_foci` in `.well-known`), for an OpenID token. +- In an encrypted room the media is end-to-end encrypted: each device + makes its own key and sends it to the others over encrypted to-device + messages (`io.element.call.encryption_keys`); keys are derived with + HKDF and never ratcheted, like the web client's livekit-client. +- A call of two shows the other person large and you in the corner; more + is a grid. Mute, camera, flip camera, leave. A "Join" banner in a room + with a call you're not in; the room list marks rooms with one. ## Push notifications @@ -139,11 +199,11 @@ and sign up on the server `http://10.0.2.2:8008`. |---|---| | `lib/main.dart` | start-up: notifications, the client, push; a push alone starts the app without UI (`--unifiedpush-bg`) | | `lib/app.dart` | the app shell: signed in or not, prompts the SDK raises (password, verification requests), opening rooms from notifications | -| `lib/matrix/` | everything around the SDK: the client and homeserver discovery (`client.dart`), the timeline projection, media, encryption setup, notifications, push | +| `lib/matrix/` | everything around the SDK: the client and homeserver discovery (`client.dart`), sliding sync, calls, the timeline projection, media, encryption setup, notifications, push | | `lib/screens/` | sign-in, sign-up, room list, room, room info, settings, encryption, sessions | | `lib/widgets/` | message bubbles, avatars, new chat, verification, password prompts | | `lib/theme.dart` | the design tokens as a Flutter theme | -| `test/` | the timeline projection, formatting | +| `test/` | the timeline projection, the sliding sync conversion, formatting | | `assets/` | the brand fonts (as TTF: Flutter can't load woff2) and the symbol | | `tool/icons/` | launcher icon sources; `dart run flutter_launcher_icons` regenerates the icons | diff --git a/apps/mobile/android/app/src/main/AndroidManifest.xml b/apps/mobile/android/app/src/main/AndroidManifest.xml index 76e6689..bac774e 100644 --- a/apps/mobile/android/app/src/main/AndroidManifest.xml +++ b/apps/mobile/android/app/src/main/AndroidManifest.xml @@ -3,6 +3,15 @@ + + + + + + + + @@ -18,6 +27,8 @@ android:name=".MainActivity" android:exported="true" android:launchMode="singleTop" + android:showWhenLocked="true" + android:turnScreenOn="true" android:taskAffinity="" android:theme="@style/LaunchTheme" android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode" diff --git a/apps/mobile/lib/app.dart b/apps/mobile/lib/app.dart index c8b649d..50ade1e 100644 --- a/apps/mobile/lib/app.dart +++ b/apps/mobile/lib/app.dart @@ -5,7 +5,9 @@ import 'package:flutter_riverpod/flutter_riverpod.dart'; import 'package:matrix/encryption.dart'; import 'package:matrix/matrix.dart'; +import 'matrix/calls.dart'; import 'matrix/notifications.dart'; +import 'screens/call_screen.dart'; import 'screens/home_screen.dart'; import 'screens/login_screen.dart'; import 'screens/room_screen.dart'; @@ -15,6 +17,7 @@ import 'widgets/uia.dart'; import 'widgets/verification_sheet.dart'; final navigatorKey = GlobalKey(); +final messengerKey = GlobalKey(); class WarpApp extends ConsumerWidget { const WarpApp({super.key}); @@ -33,6 +36,7 @@ class WarpApp extends ConsumerWidget { return MaterialApp( title: 'Warp', navigatorKey: navigatorKey, + scaffoldMessengerKey: messengerKey, debugShowCheckedModeBanner: false, theme: warpTheme(Brightness.light), darkTheme: warpTheme(Brightness.dark), @@ -70,14 +74,45 @@ class _SignedInState extends ConsumerState<_SignedIn> ..add(Notifications.opened.listen(_openRoom)) ..add(client.onUiaRequest.stream.listen(_onUia)) ..add(client.onKeyVerificationRequest.stream.listen(_onVerification)); + final calls = ref.read(callsProvider); + calls.addListener(_onCalls); + _subs.add( + calls.errors.stream.listen( + (message) => messengerKey.currentState?.showSnackBar( + SnackBar(content: Text(message)), + ), + ), + ); Notifications.launchRoomId().then((roomId) { if (roomId != null) _openRoom(roomId); }); Notifications.requestPermission(); } + IncomingCall? _shownCall; + + /// Shows the incoming call card when a call starts ringing. + void _onCalls() { + final calls = ref.read(callsProvider); + final call = calls.incoming; + if (call == null || call == _shownCall) return; + _shownCall = call; + final context = navigatorKey.currentContext; + final navigator = navigatorKey.currentState; + if (context == null || navigator == null) return; + if (!Notifications.foreground) { + Notifications.show( + roomId: call.roomId, + title: call.callerName, + body: call.video ? 'Video call' : 'Voice call', + ); + } + showIncomingCall(context, calls, navigator); + } + @override void dispose() { + ref.read(callsProvider).removeListener(_onCalls); WidgetsBinding.instance.removeObserver(this); for (final sub in _subs) { sub.cancel(); diff --git a/apps/mobile/lib/matrix/calls.dart b/apps/mobile/lib/matrix/calls.dart new file mode 100644 index 0000000..c911cf3 --- /dev/null +++ b/apps/mobile/lib/matrix/calls.dart @@ -0,0 +1,855 @@ +import 'dart:async'; +import 'dart:convert'; +import 'dart:math'; + +import 'package:flutter/foundation.dart'; +import 'package:http/http.dart' as http; +import 'package:livekit_client/livekit_client.dart' as lk; +import 'package:matrix/matrix.dart'; + +import 'notifications.dart'; +import 'package:webrtc_interface/webrtc_interface.dart' as rtc; +import 'package:flutter_webrtc/flutter_webrtc.dart' show frameCryptorFactory; + +/// Voice and video calls: MatrixRTC (MSC4143) with a LiveKit SFU, the way +/// the web client and Element Call do them, so the three can share a call. +/// +/// Who is in a room's call is room state: one +/// `org.matrix.msc3401.call.member` event per device, state key +/// `_@user:server_DEVICE_m.call`. Joining sends ours, after scheduling an +/// empty one with the homeserver (a delayed event, MSC4140) that is pushed +/// back every few seconds while the call lasts: if the app dies or loses +/// its connection, the server sends it and the device drops out. +/// +/// The media goes through LiveKit, whose token the LiveKit JWT service +/// hands out for an OpenID token from the homeserver. In an encrypted room +/// every member makes its own media key and sends it to the others over +/// encrypted to-device messages (`io.element.call.encryption_keys`); LiveKit +/// encrypts each frame with it. +/// +/// Starting a call in a room of two rings the other person +/// (`org.matrix.msc4075.rtc.notification`); declining answers it +/// (`org.matrix.msc4310.rtc.decline`). +/// +/// One call at a time. +class Calls extends ChangeNotifier { + Calls(this.client) { + _subs + ..add(client.onSync.stream.listen((_) => _refresh())) + ..add(client.onTimelineEvent.stream.listen(_onTimelineEvent)) + ..add(client.onToDeviceEvent.stream.listen(_onToDevice)); + _expiryTimer = Timer.periodic( + const Duration(seconds: 30), + (_) => _refresh(), + ); + _refresh(); + } + + final Client client; + final _subs = []; + Timer? _expiryTimer; + + static const memberType = 'org.matrix.msc3401.call.member'; + static const notificationType = 'org.matrix.msc4075.rtc.notification'; + static const declineType = 'org.matrix.msc4310.rtc.decline'; + static const keysType = 'io.element.call.encryption_keys'; + + /// How long a call rings, unless the notification says less. + static const ringLifetime = Duration(seconds: 60); + + /// The most a received notification may ring for, whatever it says. + static const maxRing = Duration(seconds: 120); + + /// A membership lasts this long unless renewed; calls are shorter. + static const membershipExpiry = Duration(hours: 4); + + static const _delayedLeave = Duration(seconds: 8); + static const _delayedLeaveRestart = Duration(seconds: 5); + + // --- What the screens show -------------------------------------------- + + /// The room whose call this device is in. + String? roomId; + CallPhase? phase; + bool video = false; + bool encrypted = false; + lk.Room? livekit; + + /// We rang someone and they haven't come yet. + bool ringingOut = false; + + /// A call ringing for us. + IncomingCall? incoming; + + /// When we joined, for the call's length. + DateTime? joinedAt; + + /// How many people (not devices) are in each room's call. + final active = {}; + + /// Errors to show once, then cleared. + final errors = StreamController.broadcast(); + + bool get inCall => roomId != null; + + int inCallCount(String roomId) => active[roomId] ?? 0; + + // --- Memberships ------------------------------------------------------ + + /// The live memberships of [room]'s call, one per device. + List memberships(Room room) { + final events = room.states[memberType]?.values ?? const []; + final now = DateTime.now(); + final out = []; + for (final e in events) { + final m = CallMember.fromState(e); + if (m != null && m.expiresAt.isAfter(now)) out.add(m); + } + out.sort((a, b) => a.createdAt.compareTo(b.createdAt)); + return out; + } + + void _refresh() { + var changed = false; + final seen = {}; + for (final room in client.rooms) { + if (room.membership != Membership.join) continue; + final users = memberships(room).map((m) => m.userId).toSet(); + final n = users.length; + if (n > 0) { + seen.add(room.id); + if (active[room.id] != n) { + active[room.id] = n; + changed = true; + } + } + } + for (final id in active.keys.toList()) { + if (!seen.contains(id)) { + active.remove(id); + changed = true; + } + } + + // The caller gave up, or someone else answered for us. + final ring = incoming; + if (ring != null && inCallCount(ring.roomId) == 0) { + _clearIncoming(); + changed = true; + } + final rid = roomId; + if (rid != null) { + final room = client.getRoomById(rid); + if (room != null) { + final members = memberships(room); + // Someone came: stop ringing them. + if (ringingOut && members.map((m) => m.userId).toSet().length > 1) { + ringingOut = false; + changed = true; + } + if (encrypted) _shareKeyWithNewDevices(room, members); + } + } + if (changed) notifyListeners(); + } + + // --- Ringing ------------------------------------------------------------ + + final _seenEvents = {}; + + void _onTimelineEvent(Event event) { + if (event.type != notificationType && event.type != declineType) return; + if (!_seenEvents.add(event.eventId)) return; + final me = client.userID; + final room = event.room; + + if (event.type == notificationType) { + if (event.senderId == me || roomId == room.id || inCall) return; + final mentions = event.content.tryGetMap('m.mentions'); + final userIds = mentions?.tryGetList('user_ids') ?? const []; + if (mentions?['room'] != true && !userIds.contains(me)) return; + // A ring for a call of two; in a bigger room the banner says enough. + if (event.content['notification_type'] != 'ring' && + _otherOfTwo(room) == null) { + return; + } + final senderTs = event.content.tryGet('sender_ts'); + final sentAt = senderTs == null + ? event.originServerTs + : DateTime.fromMillisecondsSinceEpoch( + min(senderTs, event.originServerTs.millisecondsSinceEpoch), + ); + final lifetimeMs = min( + event.content.tryGet('lifetime') ?? ringLifetime.inMilliseconds, + maxRing.inMilliseconds, + ); + final expiresAt = sentAt.add(Duration(milliseconds: lifetimeMs)); + if (!expiresAt.isAfter(DateTime.now())) return; + final caller = event.senderFromMemoryOrFallback; + incoming = IncomingCall( + roomId: room.id, + eventId: event.eventId, + callerId: event.senderId, + callerName: caller.calcDisplayname(), + callerAvatar: caller.avatarUrl, + roomName: room.getLocalizedDisplayname(), + video: event.content['m.call.intent'] != 'audio', + expiresAt: expiresAt, + ); + _ringTimer?.cancel(); + _ringTimer = Timer(expiresAt.difference(DateTime.now()), () { + _clearIncoming(); + notifyListeners(); + }); + Notifications.ring( + roomId: room.id, + caller: incoming!.callerName, + video: incoming!.video, + ); + notifyListeners(); + return; + } + + // A decline. + final target = event.relationshipEventId; + if (target == null) return; + if (event.senderId == me && incoming?.eventId == target) { + // Declined on another of our devices. + _clearIncoming(); + notifyListeners(); + } else if (target == _sentNotification && ringingOut && room.id == roomId) { + errors.add( + '${event.senderFromMemoryOrFallback.calcDisplayname()} declined the call.', + ); + leave(); + } + } + + Timer? _ringTimer; + + void _clearIncoming() { + _ringTimer?.cancel(); + if (incoming != null) Notifications.stopRinging(); + incoming = null; + } + + /// Answers the call ringing for us. + Future answer({bool withVideo = true}) async { + final call = incoming; + if (call == null) return; + _clearIncoming(); + notifyListeners(); + await join(call.roomId, video: withVideo && call.video); + } + + /// Declines the call ringing for us. + Future decline() async { + final call = incoming; + if (call == null) return; + _clearIncoming(); + notifyListeners(); + try { + await client.getRoomById(call.roomId)?.sendEvent({ + 'm.relates_to': { + 'rel_type': RelationshipTypes.reference, + 'event_id': call.eventId, + }, + }, type: declineType); + } catch (e) { + Logs().w('Could not decline the call', e); + } + } + + /// The other person in a room of two. + String? _otherOfTwo(Room room) { + if ((room.summary.mJoinedMemberCount ?? room.getParticipants().length) != + 2) { + return null; + } + return room + .getParticipants([Membership.join]) + .map((u) => u.id) + .where((id) => id != client.userID) + .firstOrNull; + } + + // --- Joining ------------------------------------------------------------ + + String? _sentNotification; + String? _delayId; + Timer? _delayTimer; + Timer? _ringOutTimer; + lk.BaseKeyProvider? _keyProvider; + lk.EventsListener? _lkListener; + + /// Our media key, its index, and who has it. + Uint8List? _key; + final _keyIndex = 0; + final _keySentTo = {}; + + /// Keys that came before LiveKit was ready, by identity. + final _pendingKeys = {}; + + String get _deviceId => client.deviceID!; + String get _identity => '${client.userID}:$_deviceId'; + + String _stateKey(Room room) { + final key = '${client.userID}_${_deviceId}_m.call'; + // Room versions with MSC3757/3779 own-state keys take it bare. + final version = room + .getState(EventTypes.RoomCreate) + ?.content + .tryGet('room_version'); + return version != null && + RegExp(r'^org\.matrix\.msc(3757|3779)\b').hasMatch(version) + ? key + : '_$key'; + } + + /// Joins the call in [roomId], or starts one. Starting a call in a room + /// of two rings the other person. + Future join(String roomId, {required bool video}) async { + if (this.roomId == roomId) return; + if (inCall) await leave(); + final room = client.getRoomById(roomId); + if (room == null) return; + if (incoming?.roomId == roomId) _clearIncoming(); + + this.roomId = roomId; + this.video = video; + phase = CallPhase.joining; + encrypted = room.encrypted; + _sentNotification = null; + _keySentTo.clear(); + notifyListeners(); + + final others = memberships(room); + final ringing = others.isEmpty && _otherOfTwo(room) != null; + try { + final transport = await _chooseTransport(others); + final (url, jwt) = await _livekitToken(transport, roomId); + if (this.roomId != roomId) return; + + if (encrypted) { + _keyProvider = await _makeKeyProvider(); + _key = _randomKey(); + await _keyProvider!.setRawKey( + _key!, + participantId: _identity, + keyIndex: _keyIndex, + ); + for (final e in _pendingKeys.entries) { + await _keyProvider!.setRawKey( + e.value.$1, + participantId: e.key, + keyIndex: e.value.$2, + ); + } + _pendingKeys.clear(); + } + + final lkRoom = lk.Room( + roomOptions: lk.RoomOptions( + adaptiveStream: true, + dynacast: true, + e2eeOptions: _keyProvider == null + ? null + : lk.E2EEOptions(keyProvider: _keyProvider!), + ), + ); + livekit = lkRoom; + lkRoom.addListener(_onLivekit); + _lkListener = lkRoom.createListener() + ..on((_) { + phase = CallPhase.reconnecting; + notifyListeners(); + }) + ..on((_) { + phase = CallPhase.connected; + notifyListeners(); + }) + ..on((event) { + if (livekit != lkRoom) return; + if (event.reason != lk.DisconnectReason.clientInitiated) { + errors.add('You were disconnected from the call.'); + } + leave(); + }); + + // The Matrix side: a delayed leave first, then ourselves. + await _scheduleDelayedLeave(room); + final membershipEventId = await client.setRoomStateWithKey( + roomId, + memberType, + _stateKey(room), + _myMembership(transport, video), + ); + if (encrypted) _shareKeyWithNewDevices(room, memberships(room)); + if (ringing || others.isEmpty) { + _sendNotification(room, membershipEventId, ring: ringing, video: video); + } + if (ringing) { + ringingOut = true; + _ringOutTimer = Timer(ringLifetime, () { + if (!ringingOut || this.roomId != roomId) return; + errors.add('No answer.'); + leave(); + }); + } + + await lkRoom.connect(url, jwt); + if (this.roomId != roomId) { + await lkRoom.disconnect(); + return; + } + if (_keyProvider != null) await lkRoom.e2eeManager?.setEnabled(true); + phase = CallPhase.connected; + joinedAt = DateTime.now(); + notifyListeners(); + + try { + await lkRoom.localParticipant?.setMicrophoneEnabled(true); + } catch (e) { + errors.add("Can't use the microphone: $e"); + } + if (video) { + try { + await lkRoom.localParticipant?.setCameraEnabled(true); + } catch (e) { + errors.add("Can't use the camera: $e"); + } + } + notifyListeners(); + } catch (e, s) { + Logs().e('Joining the call failed', e, s); + errors.add(e is CallException ? e.message : 'The call failed: $e'); + await leave(); + } + } + + void _onLivekit() => notifyListeners(); + + /// Leaves the call this device is in. + Future leave() async { + final rid = roomId; + final lkRoom = livekit; + roomId = null; + livekit = null; + phase = null; + joinedAt = null; + ringingOut = false; + _ringOutTimer?.cancel(); + _delayTimer?.cancel(); + _keyProvider = null; + _key = null; + _keySentTo.clear(); + _pendingKeys.clear(); + final delayId = _delayId; + _delayId = null; + notifyListeners(); + + lkRoom?.removeListener(_onLivekit); + await _lkListener?.dispose(); + _lkListener = null; + try { + await lkRoom?.disconnect(); + } catch (_) {} + if (rid != null) { + final room = client.getRoomById(rid); + if (room != null) { + try { + await client.setRoomStateWithKey( + rid, + memberType, + _stateKey(room), + {}, + ); + } catch (e) { + Logs().w('Could not send the leave; the delayed one will', e); + } + } + } + if (delayId != null) { + try { + await _delayedEventAction(delayId, 'cancel'); + } catch (_) {} + } + _refresh(); + } + + Map _myMembership( + Map transport, + bool video, + ) => { + 'application': 'm.call', + 'call_id': '', + 'scope': 'm.room', + 'device_id': _deviceId, + 'membershipID': _identity, + 'expires': membershipExpiry.inMilliseconds, + 'm.call.intent': video ? 'video' : 'audio', + 'focus_active': {'type': 'livekit', 'focus_selection': 'multi_sfu'}, + 'foci_preferred': [transport], + }; + + void _sendNotification( + Room room, + String membershipEventId, { + required bool ring, + required bool video, + }) { + room + .sendEvent({ + 'slot_id': 'm.call#ROOM', + 'm.mentions': {'user_ids': [], 'room': true}, + 'notification_type': ring ? 'ring' : 'notification', + 'm.relates_to': { + 'event_id': membershipEventId, + 'rel_type': RelationshipTypes.reference, + }, + 'sender_ts': DateTime.now().millisecondsSinceEpoch, + 'lifetime': ringLifetime.inMilliseconds, + 'msc4354_sticky_key': 'm.call#ROOM', + 'm.call.intent': video ? 'video' : 'audio', + }, type: notificationType) + .then((id) => _sentNotification = id) + .catchError((Object e) { + Logs().w('Could not send the call notification', e); + return null; + }); + } + + // --- Delayed leave (MSC4140) ---------------------------------------------- + + Future _scheduleDelayedLeave(Room room) async { + try { + final response = await client.request( + RequestType.PUT, + '/client/v3/rooms/${Uri.encodeComponent(room.id)}/state/' + '${Uri.encodeComponent(memberType)}/${Uri.encodeComponent(_stateKey(room))}', + query: {'org.matrix.msc4140.delay': '${_delayedLeave.inMilliseconds}'}, + data: {}, + ); + final id = response['delay_id']; + if (id is! String) return; + _delayId = id; + _delayTimer = Timer.periodic(_delayedLeaveRestart, (_) { + final current = _delayId; + if (current == null) return; + _delayedEventAction(current, 'restart').catchError((Object e) { + Logs().w('Could not push back the delayed leave', e); + }); + }); + } catch (e) { + // Without it, a crash leaves us in the call until our membership + // expires; the call itself still works. + Logs().w('The server has no delayed events', e); + } + } + + Future _delayedEventAction(String id, String action) => client.request( + RequestType.POST, + '/client/unstable/org.matrix.msc4140/delayed_events/' + '${Uri.encodeComponent(id)}/$action', + data: {}, + ); + + // --- LiveKit -------------------------------------------------------------- + + /// The LiveKit JWT service: the one the call already uses, else this + /// server's. + Future> _chooseTransport(List others) async { + for (final m in others) { + final t = m.transport; + if (t != null) return {...t, 'livekit_alias': roomId}; + } + List? foci; + try { + final response = await client.request( + RequestType.GET, + '/client/unstable/org.matrix.msc4143/rtc/transports', + ); + foci = response['rtc_transports'] as List?; + } catch (_) { + try { + final wellKnown = await http.get( + client.homeserver!.resolve('/.well-known/matrix/client'), + ); + final json = jsonDecode(wellKnown.body); + foci = json is Map + ? json['org.matrix.msc4143.rtc_foci'] as List? + : null; + } catch (_) {} + } + final focus = foci + ?.whereType() + .where( + (f) => f['type'] == 'livekit' && f['livekit_service_url'] is String, + ) + .firstOrNull; + if (focus == null) { + throw const CallException("This server doesn't offer calls."); + } + return { + 'type': 'livekit', + 'livekit_service_url': focus['livekit_service_url'], + 'livekit_alias': roomId, + }; + } + + /// A LiveKit URL and token from the JWT service, for an OpenID token. + Future<(String, String)> _livekitToken( + Map transport, + String roomId, + ) async { + final openid = await client.requestOpenIdToken(client.userID!, {}); + final service = (transport['livekit_service_url'] as String).replaceFirst( + RegExp(r'/$'), + '', + ); + final response = await http.post( + Uri.parse('$service/sfu/get'), + headers: {'Content-Type': 'application/json'}, + body: jsonEncode({ + 'room': roomId, + 'openid_token': openid.toJson(), + 'device_id': _deviceId, + }), + ); + if (response.statusCode != 200) { + throw CallException( + 'The call service refused to let you in (${response.statusCode}).', + ); + } + final body = jsonDecode(response.body); + if (body is! Map || body['url'] is! String || body['jwt'] is! String) { + throw const CallException('The call service gave an unusable answer.'); + } + return (body['url'] as String, body['jwt'] as String); + } + + /// Per-participant keys, no ratcheting, keys derived with HKDF: the + /// settings of the web client and Element Call, whose livekit-client + /// imports keys as HKDF material. + Future _makeKeyProvider() async { + final options = rtc.KeyProviderOptions( + sharedKey: false, + ratchetSalt: Uint8List.fromList(utf8.encode('LKFrameEncryptionKey')), + ratchetWindowSize: 0, + uncryptedMagicBytes: Uint8List.fromList(utf8.encode('LK-ROCKS')), + failureTolerance: -1, + keyRingSize: 256, + keyDerivationAlgorithm: rtc.KeyDerivationAlgorithm.kHKDF, + ); + final provider = await frameCryptorFactory.createDefaultKeyProvider( + options, + ); + return lk.BaseKeyProvider(provider, options); + } + + // --- Media keys ----------------------------------------------------------- + + Uint8List _randomKey() { + final random = Random.secure(); + return Uint8List.fromList(List.generate(16, (_) => random.nextInt(256))); + } + + /// Sends our media key to every device in the call that hasn't had it. + Future _shareKeyWithNewDevices( + Room room, + List members, + ) async { + final key = _key; + if (key == null) return; + final targets = members + .where( + (m) => + !(m.userId == client.userID && m.deviceId == _deviceId) && + !_keySentTo.contains('${m.userId}:${m.deviceId}'), + ) + .toList(); + if (targets.isEmpty) return; + for (final m in targets) { + _keySentTo.add('${m.userId}:${m.deviceId}'); + } + try { + final users = targets.map((m) => m.userId).toSet(); + if (users.any((u) => client.userDeviceKeys[u] == null)) { + await client.updateUserDeviceKeys(additionalUsers: users); + } + final devices = [ + for (final m in targets) + ?client.userDeviceKeys[m.userId]?.deviceKeys[m.deviceId], + ]; + await client.sendToDeviceEncrypted(devices, keysType, { + 'keys': {'index': _keyIndex, 'key': base64Encode(key)}, + 'room_id': room.id, + 'member': {'claimed_device_id': _deviceId, 'id': _identity}, + 'session': {'call_id': '', 'application': 'm.call', 'scope': 'm.room'}, + 'sent_ts': DateTime.now().millisecondsSinceEpoch, + }); + } catch (e) { + Logs().w('Could not share the call key', e); + for (final m in targets) { + _keySentTo.remove('${m.userId}:${m.deviceId}'); + } + } + } + + /// Someone's media key, over an encrypted to-device message. + void _onToDevice(ToDeviceEvent event) { + if (event.type != keysType) return; + // Keys sent in the clear are refused, as the web client refuses them. + if (event.encryptedContent == null) return; + final content = event.content; + final keys = content.tryGetMap('keys'); + final member = content.tryGetMap('member'); + final key = keys?.tryGet('key'); + final index = keys?.tryGet('index'); + final device = member?.tryGet('claimed_device_id'); + if (key == null || index == null || device == null) return; + final identity = '${event.senderId}:$device'; + Uint8List bytes; + try { + bytes = base64Decode(base64.normalize(key)); + } catch (_) { + return; + } + final provider = _keyProvider; + if (provider == null || content['room_id'] != roomId) { + // Keys can come before we have joined: keep them for then. + _pendingKeys[identity] = (bytes, index); + return; + } + provider.setRawKey(bytes, participantId: identity, keyIndex: index); + } + + // --- Controls --------------------------------------------------------------- + + bool get micOn => livekit?.localParticipant?.isMicrophoneEnabled() ?? false; + bool get cameraOn => livekit?.localParticipant?.isCameraEnabled() ?? false; + + Future toggleMic() async { + final p = livekit?.localParticipant; + if (p == null) return; + await p.setMicrophoneEnabled(!p.isMicrophoneEnabled()); + notifyListeners(); + } + + Future toggleCamera() async { + final p = livekit?.localParticipant; + if (p == null) return; + await p.setCameraEnabled(!p.isCameraEnabled()); + notifyListeners(); + } + + /// Switches between the front and the back camera. + Future flipCamera() async { + final track = livekit?.localParticipant?.videoTrackPublications + .where((t) => t.source == lk.TrackSource.camera) + .firstOrNull + ?.track; + if (track is! lk.LocalVideoTrack) return; + final options = track.currentOptions; + if (options is! lk.CameraCaptureOptions) return; + await track.setCameraPosition(options.cameraPosition.switched()); + notifyListeners(); + } + + @override + void dispose() { + for (final s in _subs) { + s.cancel(); + } + _expiryTimer?.cancel(); + leave(); + errors.close(); + super.dispose(); + } +} + +enum CallPhase { joining, connected, reconnecting } + +class CallException implements Exception { + const CallException(this.message); + final String message; + @override + String toString() => message; +} + +/// A call ringing for us. +class IncomingCall { + const IncomingCall({ + required this.roomId, + required this.eventId, + required this.callerId, + required this.callerName, + required this.callerAvatar, + required this.roomName, + required this.video, + required this.expiresAt, + }); + + final String roomId; + + /// The notification event, which a decline refers to. + final String eventId; + final String callerId; + final String callerName; + final Uri? callerAvatar; + final String roomName; + final bool video; + final DateTime expiresAt; +} + +/// One device's membership of a room's call. +class CallMember { + CallMember({ + required this.userId, + required this.deviceId, + required this.createdAt, + required this.expiresAt, + required this.transport, + }); + + final String userId; + final String deviceId; + final DateTime createdAt; + final DateTime expiresAt; + + /// The LiveKit transport it publishes to. + final Map? transport; + + /// Reads a call member state event; null for an empty one (a leave) or + /// one that isn't a room call. + static CallMember? fromState(StrippedStateEvent state) { + final content = state.content; + if (content.isEmpty) return null; + if (content['application'] != 'm.call') return null; + final deviceId = content.tryGet('device_id'); + if (deviceId == null) return null; + final origin = state is Event + ? state.originServerTs + : DateTime.fromMillisecondsSinceEpoch(0); + final created = content.tryGet('created_ts'); + final createdAt = created == null + ? origin + : DateTime.fromMillisecondsSinceEpoch(created); + final expires = content.tryGet('expires') ?? 4 * 60 * 60 * 1000; + final foci = content.tryGetList('foci_preferred'); + final transport = foci + ?.whereType() + .where( + (f) => f['type'] == 'livekit' && f['livekit_service_url'] is String, + ) + .map((f) => Map.from(f)) + .firstOrNull; + return CallMember( + userId: state.senderId, + deviceId: deviceId, + createdAt: createdAt, + expiresAt: createdAt.add(Duration(milliseconds: expires)), + transport: transport, + ); + } +} diff --git a/apps/mobile/lib/matrix/notifications.dart b/apps/mobile/lib/matrix/notifications.dart index 1c6def3..c676cbf 100644 --- a/apps/mobile/lib/matrix/notifications.dart +++ b/apps/mobile/lib/matrix/notifications.dart @@ -36,6 +36,60 @@ class Notifications { importance: Importance.high, ); + /// Incoming calls: the ringtone, repeating until answered. + static const _callChannel = AndroidNotificationChannel( + 'calls', + 'Calls', + description: 'Incoming voice and video calls', + importance: Importance.max, + sound: UriAndroidNotificationSound('content://settings/system/ringtone'), + ); + + static const _ringId = 0x7fff0001; + + /// Rings for an incoming call: a call notification with the ringtone, + /// repeating (insistent) until [stopRinging], shown full screen on a + /// locked phone. + static Future ring({ + required String roomId, + required String caller, + required bool video, + }) async { + if (!_ready) await init(); + try { + await _plugin.show( + id: _ringId, + title: caller, + body: video ? 'Incoming video call' : 'Incoming voice call', + payload: roomId, + notificationDetails: NotificationDetails( + android: AndroidNotificationDetails( + _callChannel.id, + _callChannel.name, + channelDescription: _callChannel.description, + importance: Importance.max, + priority: Priority.max, + category: AndroidNotificationCategory.call, + fullScreenIntent: true, + ongoing: true, + autoCancel: false, + sound: _callChannel.sound, + // FLAG_INSISTENT: the sound repeats until the notification goes. + additionalFlags: Int32List.fromList([4]), + color: const Color(0xFF00A389), + ), + ), + ); + } catch (e, s) { + Logs().w('Could not ring', e, s); + } + } + + static Future stopRinging() async { + if (!_ready) return; + await _plugin.cancel(id: _ringId); + } + static Future init() async { if (_ready) return; await _plugin.initialize( @@ -52,6 +106,11 @@ class Notifications { AndroidFlutterLocalNotificationsPlugin >() ?.createNotificationChannel(_channel); + await _plugin + .resolvePlatformSpecificImplementation< + AndroidFlutterLocalNotificationsPlugin + >() + ?.createNotificationChannel(_callChannel); _ready = true; } diff --git a/apps/mobile/lib/screens/call_screen.dart b/apps/mobile/lib/screens/call_screen.dart new file mode 100644 index 0000000..c10ea61 --- /dev/null +++ b/apps/mobile/lib/screens/call_screen.dart @@ -0,0 +1,527 @@ +import 'dart:async'; + +import 'package:flutter/material.dart'; +import 'package:flutter_riverpod/flutter_riverpod.dart'; +import 'package:livekit_client/livekit_client.dart' as lk; +import 'package:matrix/matrix.dart'; + +import '../matrix/calls.dart'; +import '../state.dart'; +import '../theme.dart'; +import '../widgets/avatar.dart'; + +/// The call this device is in: the other person large and you in the +/// corner for a call of two, a grid of tiles for more; mute, camera, flip +/// and leave. "Calling…" while it rings the other person. +class CallScreen extends ConsumerStatefulWidget { + const CallScreen({super.key}); + + static Route route() => MaterialPageRoute( + settings: const RouteSettings(name: 'call'), + builder: (_) => const CallScreen(), + ); + + @override + ConsumerState createState() => _CallScreenState(); +} + +class _CallScreenState extends ConsumerState { + Timer? _clock; + + @override + void initState() { + super.initState(); + _clock = Timer.periodic(const Duration(seconds: 1), (_) { + if (mounted) setState(() {}); + }); + } + + @override + void dispose() { + _clock?.cancel(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final calls = ref.watch(callsProvider); + final client = ref.watch(clientProvider); + return ListenableBuilder( + listenable: calls, + builder: (context, _) { + final roomId = calls.roomId; + if (roomId == null) { + // The call ended: leave the screen. + WidgetsBinding.instance.addPostFrameCallback((_) { + if (mounted && Navigator.of(context).canPop()) { + Navigator.of(context).pop(); + } + }); + return const Scaffold(backgroundColor: Colors.black); + } + final room = client.getRoomById(roomId); + final tiles = _tiles(calls, room); + final title = room?.getLocalizedDisplayname() ?? ''; + return Scaffold( + backgroundColor: const Color(0xFF0F1213), + body: SafeArea( + child: Column( + children: [ + _TopBar(title: title, calls: calls), + Expanded( + child: _Stage(tiles: tiles, calls: calls, room: room), + ), + _Controls(calls: calls), + ], + ), + ), + ); + }, + ); + } + + List<_Tile> _tiles(Calls calls, Room? room) { + final lkRoom = calls.livekit; + if (lkRoom == null) return []; + final speakers = lkRoom.activeSpeakers.map((p) => p.identity).toSet(); + _Tile tile(lk.Participant p, bool local) { + final userId = _identityUser(p.identity); + final user = room?.unsafeGetUserFromMemoryOrFallback(userId); + final camera = p.getTrackPublicationBySource(lk.TrackSource.camera); + final screen = p.getTrackPublicationBySource( + lk.TrackSource.screenShareVideo, + ); + final mic = p.getTrackPublicationBySource(lk.TrackSource.microphone); + lk.VideoTrack? video(lk.TrackPublication? pub) => + pub != null && !pub.muted && pub.track is lk.VideoTrack + ? pub.track as lk.VideoTrack + : null; + return _Tile( + identity: p.identity, + userId: userId, + name: user?.calcDisplayname() ?? userId, + avatar: user?.avatarUrl, + local: local, + video: video(screen) ?? video(camera), + micMuted: mic == null || mic.muted, + speaking: speakers.contains(p.identity), + ); + } + + return [ + if (lkRoom.localParticipant != null) tile(lkRoom.localParticipant!, true), + for (final p in lkRoom.remoteParticipants.values) tile(p, false), + ]; + } + + /// The user of a LiveKit identity, "@alice:example.org:DEVICE": the + /// device is after the last colon (a server name may have a port). + static String _identityUser(String identity) { + final i = identity.lastIndexOf(':'); + return i > 0 ? identity.substring(0, i) : identity; + } +} + +class _Tile { + const _Tile({ + required this.identity, + required this.userId, + required this.name, + required this.avatar, + required this.local, + required this.video, + required this.micMuted, + required this.speaking, + }); + + final String identity; + final String userId; + final String name; + final Uri? avatar; + final bool local; + final lk.VideoTrack? video; + final bool micMuted; + final bool speaking; +} + +class _TopBar extends StatelessWidget { + const _TopBar({required this.title, required this.calls}); + final String title; + final Calls calls; + + @override + Widget build(BuildContext context) { + final since = calls.joinedAt; + final status = switch (calls.phase) { + CallPhase.joining => 'Connecting…', + CallPhase.reconnecting => 'Reconnecting…', + _ when calls.ringingOut => 'Calling…', + _ when since != null => _length(DateTime.now().difference(since)), + _ => '', + }; + return Padding( + padding: const EdgeInsets.fromLTRB(4, 4, 16, 8), + child: Row( + children: [ + IconButton( + tooltip: 'Back to the chat', + icon: const Icon(Icons.keyboard_arrow_down, color: Colors.white), + onPressed: () => Navigator.of(context).pop(), + ), + Expanded( + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text( + title, + maxLines: 1, + overflow: TextOverflow.ellipsis, + style: const TextStyle( + color: Colors.white, + fontFamily: fontDisplay, + fontSize: 18, + fontWeight: FontWeight.w600, + ), + ), + Text( + status, + style: const TextStyle(color: Colors.white70, fontSize: 13), + ), + ], + ), + ), + if (calls.encrypted) + const Tooltip( + message: 'End-to-end encrypted', + child: Icon(Icons.lock_outline, color: Colors.white70, size: 18), + ), + ], + ), + ); + } + + static String _length(Duration d) { + final m = d.inMinutes.remainder(60).toString().padLeft(2, '0'); + final s = d.inSeconds.remainder(60).toString().padLeft(2, '0'); + return d.inHours > 0 ? '${d.inHours}:$m:$s' : '$m:$s'; + } +} + +class _Stage extends StatelessWidget { + const _Stage({required this.tiles, required this.calls, required this.room}); + final List<_Tile> tiles; + final Calls calls; + final Room? room; + + @override + Widget build(BuildContext context) { + if (tiles.isEmpty) { + return const Center(child: CircularProgressIndicator()); + } + final remote = tiles.where((t) => !t.local).toList(); + final local = tiles.firstWhere((t) => t.local, orElse: () => tiles.first); + + // Alone: ringing, or waiting for someone to come. + if (remote.isEmpty) { + return Stack( + children: [ + Positioned.fill(child: _TileView(tile: local, large: true)), + if (calls.ringingOut) + const Align( + alignment: Alignment(0, 0.6), + child: Text( + 'Ringing…', + style: TextStyle(color: Colors.white70, fontSize: 15), + ), + ), + ], + ); + } + // Two: the other person large, you in the corner. + if (remote.length == 1) { + return Stack( + children: [ + Positioned.fill(child: _TileView(tile: remote.single, large: true)), + Positioned( + right: 16, + bottom: 16, + width: 110, + height: 160, + child: _TileView(tile: local), + ), + ], + ); + } + // More: a grid. + final columns = tiles.length <= 4 ? 2 : 3; + return GridView.count( + padding: const EdgeInsets.all(8), + crossAxisCount: columns, + mainAxisSpacing: 8, + crossAxisSpacing: 8, + childAspectRatio: 3 / 4, + children: [for (final t in tiles) _TileView(tile: t)], + ); + } +} + +class _TileView extends StatelessWidget { + const _TileView({required this.tile, this.large = false}); + final _Tile tile; + final bool large; + + @override + Widget build(BuildContext context) { + final video = tile.video; + return ClipRRect( + borderRadius: BorderRadius.circular(large ? 0 : 14), + child: Container( + decoration: BoxDecoration( + color: const Color(0xFF1B2022), + border: tile.speaking && !large + ? Border.all(color: const Color(0xFF22C3A6), width: 3) + : null, + ), + child: Stack( + children: [ + Positioned.fill( + child: video != null + ? lk.VideoTrackRenderer( + video, + fit: large + ? lk.VideoViewFit.cover + : lk.VideoViewFit.cover, + mirrorMode: tile.local + ? lk.VideoViewMirrorMode.mirror + : lk.VideoViewMirrorMode.off, + ) + : Center( + child: Avatar( + name: tile.name, + seed: tile.userId, + mxc: tile.avatar, + size: large ? 112 : 56, + ), + ), + ), + Positioned( + left: 8, + bottom: 8, + right: 8, + child: Row( + children: [ + if (tile.micMuted) + const Padding( + padding: EdgeInsets.only(right: 4), + child: Icon(Icons.mic_off, color: Colors.white, size: 16), + ), + Flexible( + child: Text( + tile.local ? 'You' : tile.name, + maxLines: 1, + overflow: TextOverflow.ellipsis, + style: const TextStyle( + color: Colors.white, + fontSize: 13, + shadows: [Shadow(blurRadius: 4)], + ), + ), + ), + ], + ), + ), + ], + ), + ), + ); + } +} + +class _Controls extends StatelessWidget { + const _Controls({required this.calls}); + final Calls calls; + + @override + Widget build(BuildContext context) { + Widget round( + IconData icon, + String label, + VoidCallback? onTap, { + bool active = true, + Color? colour, + }) => Column( + mainAxisSize: MainAxisSize.min, + children: [ + Material( + color: colour ?? (active ? Colors.white24 : Colors.white), + shape: const CircleBorder(), + child: InkWell( + customBorder: const CircleBorder(), + onTap: onTap, + child: SizedBox.square( + dimension: 60, + child: Icon( + icon, + color: colour != null || active ? Colors.white : Colors.black, + ), + ), + ), + ), + const SizedBox(height: 6), + Text( + label, + style: const TextStyle(color: Colors.white70, fontSize: 12), + ), + ], + ); + + final connected = calls.phase == CallPhase.connected; + return Padding( + padding: const EdgeInsets.fromLTRB(16, 12, 16, 20), + child: Row( + mainAxisAlignment: MainAxisAlignment.spaceEvenly, + children: [ + round( + calls.micOn ? Icons.mic : Icons.mic_off, + calls.micOn ? 'Mute' : 'Unmute', + connected ? calls.toggleMic : null, + active: calls.micOn, + ), + round( + calls.cameraOn ? Icons.videocam : Icons.videocam_off, + 'Camera', + connected ? calls.toggleCamera : null, + active: calls.cameraOn, + ), + if (calls.cameraOn) + round( + Icons.flip_camera_android, + 'Flip', + connected ? calls.flipCamera : null, + ), + round( + Icons.call_end, + 'Leave', + calls.leave, + colour: const Color(0xFFD64545), + ), + ], + ), + ); + } +} + +/// The incoming call card: who, from which room, answer (with or without +/// video) or decline. +Future showIncomingCall( + BuildContext context, + Calls calls, + NavigatorState navigator, +) => showModalBottomSheet( + context: context, + isDismissible: false, + enableDrag: false, + builder: (sheet) => ListenableBuilder( + listenable: calls, + builder: (context, _) { + final call = calls.incoming; + if (call == null) { + // Answered elsewhere, declined, or it stopped ringing. + WidgetsBinding.instance.addPostFrameCallback((_) { + if (Navigator.of(sheet).canPop()) Navigator.of(sheet).pop(); + }); + return const SizedBox(height: 200); + } + final c = context.warp; + final text = Theme.of(context).textTheme; + return SafeArea( + child: Padding( + padding: const EdgeInsets.fromLTRB(24, 0, 24, 24), + child: Column( + mainAxisSize: MainAxisSize.min, + children: [ + Avatar( + name: call.callerName, + seed: call.callerId, + mxc: call.callerAvatar, + size: 72, + ), + const SizedBox(height: 12), + Text(call.callerName, style: text.titleLarge), + Text( + call.video ? 'Video call' : 'Voice call', + style: text.bodySmall, + ), + if (call.roomName != call.callerName) + Text(call.roomName, style: text.bodySmall), + const SizedBox(height: 24), + Row( + mainAxisAlignment: MainAxisAlignment.spaceEvenly, + children: [ + _RoundButton( + icon: Icons.call_end, + label: 'Decline', + colour: c.danger, + onTap: () => calls.decline(), + ), + if (call.video) + _RoundButton( + icon: Icons.call, + label: 'Voice only', + colour: c.tealStrong, + onTap: () { + calls.answer(withVideo: false); + navigator.push(CallScreen.route()); + }, + ), + _RoundButton( + icon: call.video ? Icons.videocam : Icons.call, + label: 'Answer', + colour: c.tealStrong, + onTap: () { + calls.answer(); + navigator.push(CallScreen.route()); + }, + ), + ], + ), + ], + ), + ), + ); + }, + ), +); + +class _RoundButton extends StatelessWidget { + const _RoundButton({ + required this.icon, + required this.label, + required this.colour, + required this.onTap, + }); + final IconData icon; + final String label; + final Color colour; + final VoidCallback onTap; + + @override + Widget build(BuildContext context) => Column( + children: [ + Material( + color: colour, + shape: const CircleBorder(), + child: InkWell( + customBorder: const CircleBorder(), + onTap: onTap, + child: SizedBox.square( + dimension: 64, + child: Icon(icon, color: Colors.white), + ), + ), + ), + const SizedBox(height: 6), + Text(label, style: Theme.of(context).textTheme.labelMedium), + ], + ); +} diff --git a/apps/mobile/lib/screens/home_screen.dart b/apps/mobile/lib/screens/home_screen.dart index 0928e6e..af9f886 100644 --- a/apps/mobile/lib/screens/home_screen.dart +++ b/apps/mobile/lib/screens/home_screen.dart @@ -442,12 +442,13 @@ class _Empty extends StatelessWidget { } } -class RoomTile extends StatelessWidget { +class RoomTile extends ConsumerWidget { const RoomTile({super.key, required this.room}); final Room room; @override - Widget build(BuildContext context) { + Widget build(BuildContext context, WidgetRef ref) { + final inCall = ref.watch(callsProvider).inCallCount(room.id) > 0; final c = context.warp; final text = Theme.of(context).textTheme; final last = room.lastEvent; @@ -513,9 +514,13 @@ class RoomTile extends StatelessWidget { Icon(Icons.lock_outline, size: 13, color: c.inkMuted), const SizedBox(width: 4), ], + if (inCall) ...[ + Icon(Icons.call, size: 14, color: c.tealText), + const SizedBox(width: 4), + ], Expanded( child: Text( - preview, + inCall ? 'Call in progress' : preview, maxLines: 1, overflow: TextOverflow.ellipsis, style: text.bodySmall?.copyWith( diff --git a/apps/mobile/lib/screens/room_screen.dart b/apps/mobile/lib/screens/room_screen.dart index fe58c1a..4aabdc5 100644 --- a/apps/mobile/lib/screens/room_screen.dart +++ b/apps/mobile/lib/screens/room_screen.dart @@ -13,6 +13,7 @@ import '../util/format.dart'; import '../widgets/avatar.dart'; import '../widgets/composer.dart'; import '../widgets/message.dart'; +import 'call_screen.dart'; import 'room_info_screen.dart'; import 'thread_screen.dart'; @@ -150,6 +151,7 @@ class _RoomScreenState extends ConsumerState appBar: _RoomAppBar(room: room), body: Column( children: [ + _CallBanner(room: room), Expanded(child: _buildTimeline(room)), if (room.membership == Membership.invite) _InviteBar(room: room) @@ -298,6 +300,7 @@ class _RoomAppBar extends StatelessWidget implements PreferredSizeWidget { ), ), actions: [ + _CallButtons(room: room), IconButton( tooltip: 'Room info', icon: const Icon(Icons.info_outline), @@ -310,6 +313,98 @@ class _RoomAppBar extends StatelessWidget implements PreferredSizeWidget { } } +/// Voice and video call buttons; "back to the call" while in this room's. +class _CallButtons extends ConsumerWidget { + const _CallButtons({required this.room}); + final Room room; + + @override + Widget build(BuildContext context, WidgetRef ref) { + final calls = ref.watch(callsProvider); + return ListenableBuilder( + listenable: calls, + builder: (context, _) { + if (room.membership != Membership.join) return const SizedBox.shrink(); + if (calls.roomId == room.id) { + return IconButton( + tooltip: 'Back to the call', + icon: Icon(Icons.call, color: context.warp.tealText), + onPressed: () => Navigator.of(context).push(CallScreen.route()), + ); + } + void start(bool video) { + calls.join(room.id, video: video); + Navigator.of(context).push(CallScreen.route()); + } + + return Row( + mainAxisSize: MainAxisSize.min, + children: [ + IconButton( + tooltip: 'Voice call', + icon: const Icon(Icons.call_outlined), + onPressed: () => start(false), + ), + IconButton( + tooltip: 'Video call', + icon: const Icon(Icons.videocam_outlined), + onPressed: () => start(true), + ), + ], + ); + }, + ); + } +} + +/// "A call is going on" with Join, while this device isn't in it. +class _CallBanner extends ConsumerWidget { + const _CallBanner({required this.room}); + final Room room; + + @override + Widget build(BuildContext context, WidgetRef ref) { + final calls = ref.watch(callsProvider); + return ListenableBuilder( + listenable: calls, + builder: (context, _) { + final count = calls.inCallCount(room.id); + if (count == 0 || calls.roomId == room.id) { + return const SizedBox.shrink(); + } + final c = context.warp; + return Material( + color: c.tealSoft, + child: Padding( + padding: const EdgeInsets.fromLTRB(16, 6, 8, 6), + child: Row( + children: [ + Icon(Icons.call, color: c.tealText, size: 20), + const SizedBox(width: 12), + Expanded( + child: Text( + count == 1 + ? '1 person in the call' + : '$count people in the call', + style: TextStyle(color: c.tealText), + ), + ), + FilledButton( + onPressed: () { + calls.join(room.id, video: false); + Navigator.of(context).push(CallScreen.route()); + }, + child: const Text('Join'), + ), + ], + ), + ), + ); + }, + ); + } +} + class _DaySeparator extends StatelessWidget { const _DaySeparator({required this.day}); final DateTime day; diff --git a/apps/mobile/lib/state.dart b/apps/mobile/lib/state.dart index 8cd8a9e..0d79322 100644 --- a/apps/mobile/lib/state.dart +++ b/apps/mobile/lib/state.dart @@ -3,6 +3,8 @@ import 'package:flutter_riverpod/flutter_riverpod.dart'; import 'package:flutter_secure_storage/flutter_secure_storage.dart'; import 'package:matrix/matrix.dart'; +import 'matrix/calls.dart'; + /// The one Matrix client, created in main() before the app starts. final clientProvider = Provider( (ref) => throw UnimplementedError('overridden in main()'), @@ -72,3 +74,11 @@ class SelectedSpaceNotifier extends Notifier { final selectedSpaceProvider = NotifierProvider( SelectedSpaceNotifier.new, ); + +/// Calls: one manager for the signed-in client. Screens listen to it +/// directly (it is a ChangeNotifier). +final callsProvider = Provider((ref) { + final calls = Calls(ref.watch(clientProvider)); + ref.onDispose(calls.dispose); + return calls; +}); diff --git a/apps/mobile/pubspec.lock b/apps/mobile/pubspec.lock index 2d6f0da..c86ee74 100644 --- a/apps/mobile/pubspec.lock +++ b/apps/mobile/pubspec.lock @@ -489,7 +489,7 @@ packages: source: sdk version: "0.0.0" flutter_webrtc: - dependency: transitive + dependency: "direct main" description: name: flutter_webrtc sha256: ebf5ceef2897caae425ed65a38b2961c29d9d9a626159418ed460e1a65038c00 @@ -1334,7 +1334,7 @@ packages: source: hosted version: "1.0.1" webrtc_interface: - dependency: transitive + dependency: "direct main" description: name: webrtc_interface sha256: c6f100eac5057d9a817a60473126f9828c796d42884d498af4f339c97b21014f diff --git a/apps/mobile/pubspec.yaml b/apps/mobile/pubspec.yaml index 114e1db..8cdbce7 100644 --- a/apps/mobile/pubspec.yaml +++ b/apps/mobile/pubspec.yaml @@ -51,6 +51,8 @@ dependencies: livekit_client: ^2.13.1 qr_flutter: ^4.1.0 mobile_scanner: ^7.4.2 + flutter_webrtc: ^1.6.2+hotfix.4 + webrtc_interface: ^1.5.1 dev_dependencies: flutter_test: