Warp/PLAN.md

Warp — a Matrix homeserver and client suite in Go

Project plan. Warp is a from-scratch Matrix homeserver written in Go, taking architectural lessons from Dendrite without copying its code, plus a family of first-party clients (admin panel, web, desktop, mobile).

---

1. Goals and non-goals

Goals

• Spec-conformant homeserver: Client-Server API, Server-Server (federation) API, Application Service API, Push Gateway client side.
• Federation is optional. Warp must run as a fully isolated, private homeserver with federation switched off, and be switchable at runtime.
• Single binary, single process ("monolith"). Dendrite's polylith mode was deprecated for good reason — the internal component boundaries stay, the network hops go.
• Three storage drivers: PostgreSQL (production), SQLite (dev / single-user), PocketBase (all-in-one appliance deployments).
• Pass Complement (the official integration test suite, itself written in Go).
• Federate with Synapse, Dendrite, Conduit/Conduwuit/Tuwunel when enabled.
• First-party clients: admin web panel, web client, desktop client, Android app (iOS later), all sharing as much code as possible.

Non-goals (for now)

• Identity server, SSO/OIDC provider (delegate to MAS later), VoIP/TURN provisioning, legacy Sytest.

---

2. Key decisions

Decision	Choice	Why
Go version	1.26, net/http with pattern routing (GET /_matrix/client/v3/...)	No router dependency needed since Go 1.22.
Event core (canonical JSON, hashing, ed25519 signing, auth rules, state resolution v2)	Own implementation, tested against gomatrixserverlib test vectors and Complement	This is the heart of a homeserver and the point of the project. Fallback: import gomatrixserverlib if it stalls the schedule.
Internal messaging	Go interfaces + channels, one process	Dendrite embeds NATS JetStream; we don't need durable internal queues until we do. Design the producer/consumer interfaces so NATS can be dropped in later.
Storage	database/sql + hand-written SQL behind a per-component Storage interface; drivers: pgx (Postgres), modernc.org/sqlite (pure Go SQLite), PocketBase (embedded core.App, see §4)	Same dual-backend model as Dendrite plus one; no ORM.
Migrations	Embedded SQL files, per component, embed.FS	Simple, reviewable.
Config	Single YAML file + env overrides; a small set of settings (federation on/off, registration on/off) also mutable at runtime via admin API and persisted in DB
Logging / metrics / tracing	log/slog, Prometheus /metrics, OpenTelemetry (later)
Room versions	Implement the latest stable room version first (the one m.room.create defaults to in the current spec), then backfill older versions as needed for federation	Old room versions matter only when joining old rooms elsewhere.
Sync	/sync v3 first; Simplified Sliding Sync (MSC4186) as a later phase	Element X and our own mobile app want sliding sync; classic clients require /sync.
Frontend stack (admin + web client)	Vue 3 + TypeScript + Vite + Pinia; Tailwind CSS with a hand-rolled component set in apps/ui (Headless UI / Radix Vue for accessible primitives: dialogs, menus, listboxes)	Same UI code runs in the browser and inside the Wails desktop shell; Tailwind keeps the bundle small and the design system ours.
Desktop shell	Wails v3 (Go host + webview) reusing the Vue web client	Pure-Go UI toolkits (Fyne/Gio) would mean a second UI codebase.
Mobile	Flutter + matrix_dart_sdk (famedly) with vodozemac for E2EE	Mature SDK; same codebase later ships to iOS.
Client protocol/crypto in Go	mautrix-go (+ goolm pure-Go Olm/Megolm) for the desktop host and any Go-side tooling; matrix-js-sdk + matrix-sdk-crypto-wasm in the browser	Do not write our own client-side crypto.

---

3. Federation switch

federation.enabled is a first-class mode, not an afterthought.

federation:
  enabled: false          # default off for a new install
  allowlist: []           # if non-empty, only these server names may federate
  denylist: []
  allow_runtime_toggle: true

When disabled:

Area	Behaviour
Routes	/_matrix/federation/*, /_matrix/key/*, /.well-known/matrix/server are not registered (404).
Outbound	Transaction queue, key fetcher, remote media fetch, remote keys/query, remote profile/directory lookups are not started.
Client API	/join with server_name for a room not known locally → M_FORBIDDEN with a clear message; inviting @user:otherserver → M_FORBIDDEN; publicRooms?server= → M_FORBIDDEN.
Room IDs / event IDs	Unchanged — still !id:warp.example. Rooms created offline federate normally if federation is later enabled.
Signing	Events are still signed with the server key; only publication/verification of remote keys is skipped.

Runtime toggle: PUT /_warp/admin/v1/federation {"enabled": true} (and the admin panel switch). Implemented as a federation.Module with Start(ctx) / Stop(); the HTTP mux delegates federation routes through a gate that checks the live flag, so no restart is required. The setting is persisted in the server_settings table and overrides the YAML value on boot if allow_runtime_toggle is true.

Allowlist mode is the middle ground for "federate only with my other servers": all remote traffic is filtered by server name at the auth layer and in the outbound queue.

---

4. Storage drivers

All components talk to storage through Go interfaces (e.g. roomserver.Database, userapi.Database). Each driver implements them.

Driver	Engine	Target use	Notes
postgres	PostgreSQL 15+ via pgx	Production, multi-user, large rooms	Primary; Complement runs against this.
sqlite	modernc.org/sqlite (no cgo), WAL mode	Dev, tests, single-user	Serialised writer; fine up to a few hundred users.
pocketbase	Embedded PocketBase (core.App) on its own SQLite file	"Appliance" deployments: one binary that also provides admin auth, file storage (local/S3) for media, and realtime for the admin panel	Records are stored via PocketBase collections; heavier than raw SQLite and its record model is a poor fit for the event DAG, so the roomserver/syncapi tables bypass PocketBase's collection API and use its underlying dbx connection directly. PocketBase adds value for mediaapi (its filesystem abstraction) and for admin-panel auth. Marked experimental until it passes the same test suite as sqlite.

Rules:

• One SQL dialect per driver, kept in internal/storage/{postgres,sqlite,pocketbase}.
• Every storage test runs against all three drivers (go test -tags=storage_all).
• Migrations are per-driver; warp migrate runs them.

---

5. Server architecture

                 +--------------------------------------------------+
 clients ------> |  clientapi   (C-S REST handlers, auth, ratelimit) |
                 +--------------------------------------------------+
 other HS -----> |  federationapi  [gated by federation.enabled]     |
                 +-------------+-------------+----------------------+
                 |  userapi    |  roomserver |  syncapi             |
                 |  accounts   |  event DAG  |  read model / streams|
                 |  devices    |  auth rules |  /sync, /messages    |
                 |  tokens     |  state res  |  filters, tokens     |
                 |  push rules |  redactions |                      |
                 +-------------+-------------+----------------------+
                 |  keyserver  |  mediaapi   |  appservice          |
                 |  device keys|  upload/dl  |  registration, txns  |
                 |  OTKs, x-sig|  thumbnails |                      |
                 +-------------+-------------+----------------------+
                 |  adminapi (/_warp/admin/*) + embedded admin SPA   |
                 +--------------------------------------------------+
                 |  internal/matrix: event, canonicaljson, signing, |
                 |  roomversion, auth, stateres                     |
                 +--------------------------------------------------+
                 |  storage: postgres | sqlite | pocketbase         |
                 +--------------------------------------------------+

Data-flow principle (borrowed from Dendrite): the roomserver is the single writer for room events. Every event — local or federated — goes in through the roomserver, which authorises it, resolves state, persists it, and then emits an "output event" that the syncapi, federationapi (outbound, if enabled), appservice, and push consumers subscribe to. Nothing else writes room state.

---

6. Repository layout (monorepo)

Warp/
├── go.work                       Go workspace tying the Go modules together
├── server/                       Go module: the homeserver
│   ├── cmd/warp/                 serve, generate-keys, create-account, migrate
│   ├── internal/
│   │   ├── config/
│   │   ├── httputil/
│   │   ├── matrix/               canonicaljson, event, signing, roomversion, auth, stateres
│   │   ├── clientapi/  federationapi/  roomserver/  syncapi/  userapi/
│   │   ├── keyserver/  mediaapi/  appservice/  pushserver/  adminapi/
│   │   └── storage/{postgres,sqlite,pocketbase,migrations}/
│   ├── web/admin/                build output of apps/admin, embedded via embed.FS
│   └── test/{complement,vectors}/
├── apps/
│   ├── ui/                       shared Vue 3 component library + Matrix client store (Pinia)
│   ├── admin/                    admin web panel (Vue) — built into server/web/admin
│   ├── web/                      web client (Vue SPA) + Go host (`cmd/warp-web`)
│   ├── desktop/                  Wails v3 app: Go host (mautrix-go) + apps/web frontend
│   └── mobile/                   Flutter app: Android now, iOS later
├── docs/
├── Dockerfile
├── Makefile
└── warp.yaml.example

Frontend packages are managed with pnpm workspaces (apps/ui, apps/admin, apps/web); Go modules with go.work (server, apps/web, apps/desktop).

---

7. Server phases

Each phase ends with something runnable and a Complement subset passing.

Phase 0 — Bootstrap (1 week)

• go mod init, layout above, Makefile (build/test/lint/complement), golangci-lint, GitHub Actions CI.
• Config loading, slog logging, graceful shutdown, /healthz, /metrics.
• HTTP server skeleton with Matrix error envelope (errcode, error), CORS, GET /_matrix/client/versions, /.well-known/matrix/client (and /server only when federation is enabled).
• Storage interfaces + migrations runner; sqlite driver first, postgres second, pocketbase stub compiled but returning ErrNotImplemented.
• warp generate-keys (ed25519 signing key file).
• server_settings table and the runtime-settings service (federation flag, registration flag).

Phase 1 — Accounts & sessions (1–2 weeks)

• userapi: users, passwords (argon2id), devices, access tokens, refresh tokens, account data, admin flag.
• C-S endpoints: register (UIA with m.login.dummy / registration token), login (m.login.password), logout, logout/all, account/whoami, devices/*, profile/*, user/{id}/account_data/*, capabilities.
• Auth middleware, per-endpoint rate limiting (M_LIMIT_EXCEEDED).
• warp create-account [--admin] CLI.
• adminapi v1 skeleton: admin-only auth, GET/PUT /federation, GET/PUT /registration, GET /users, POST /users, GET /stats.
• Complement: TestRegistration, TestLogin, TestDeviceManagement.

Phase 2 — Event core (2–3 weeks, the hard part)

Pure library code under internal/matrix, no HTTP, no DB.

• Canonical JSON, event/PDU model, hashing, redaction algorithm.
• ed25519 signing and verification, key IDs.
• Room version registry.
• Authorisation rules (full spec list).
• State resolution v2.

Phase 3 — Rooms, local only (2–3 weeks)

• roomserver storage and input pipeline (validate → auth → resolve → persist → emit).
• C-S: createRoom, join, leave, invite, kick, ban, send/{type}/{txnId}, state/*, redact, rooms/{id}/event/{id}, members, joined_members, aliases, publicRooms.
• Federation-disabled behaviours from §3 enforced here (remote joins/invites rejected).
• adminapi: room list, room detail, purge room, force-leave.
• Complement: TestRoomCreate, TestRoomState, TestMembership*, TestRedactions.

Phase 4 — Sync (2 weeks)

• syncapi read model, stream tokens, GET /sync, /messages, /context, filters, lazy-loading members.
• EDUs: typing, receipts, presence (config flag), to_device.
• Complement: TestSync*, TestMessagesPagination, TestTyping, TestReceipts.

→ Milestone M1: Element Web works against Warp with federation off. The admin panel and web client (§8) can now be developed against a real server.

Phase 5 — Federation (3–4 weeks)

• federation.Module with Start/Stop, route gate, allow/deny lists.
• Server keys, discovery, X-Matrix auth, /send, joins/leaves/invites, /state, /state_ids, /backfill, /get_missing_events, queries.
• Outbound queue with backoff, blacklist, persistence.
• Complement federation tests.

→ Milestone M2: Warp joins #matrix:matrix.org; toggling federation off at runtime cleanly stops all remote traffic and back on resumes it.

Phase 6 — Media (1 week)

• mediaapi, thumbnails, authenticated media, remote media cache (federation only). PocketBase driver uses PocketBase's filesystem abstraction (local/S3).

Phase 7 — E2EE support (2 weeks)

• keyserver: device keys, OTKs, fallback keys, cross-signing, device list tracking (local always; federated only when enabled), key backup, sendToDevice.

Phase 8 — Push & notifications (1 week)

• Push rules, pushers, notification counts, push gateway client. Needed by the mobile app.

Phase 9 — Application services & admin completion (1–2 weeks)

• AS registration, namespaces, transactions, masquerading.
• adminapi: full user management (deactivate, reset password, devices, admin flag), server notices, media quota, registration tokens, federation allow/deny list editing, audit log.

Phase 10 — Hardening & ops

• Simplified Sliding Sync (MSC4186), spaces, threads, search.
• PocketBase driver to parity with sqlite.
• Tracing, dashboards, Dockerfile, compose/Helm examples, load testing.

---

8. Client sub-projects

All clients speak plain Matrix C-S API, so they also work against Synapse — and Element works against Warp. Nothing proprietary in the wire protocol except the /_warp/admin/* API used by the admin panel.

8.1 Admin web panel — Go + Vue (apps/admin)

• Stack: Vue 3 + TS + Vite + Pinia + Tailwind CSS, components from apps/ui. Built assets embedded into the warp binary and served at /_warp/admin/; no separate process.
• Backend: server/internal/adminapi (Go). Auth = Matrix login of a user with the admin flag; the panel uses the normal access token.
• Features by priority:
  1. Dashboard: users, rooms, events/min, federation status, storage driver, version.
  2. Federation switch, allow/deny list editor, per-server health (queue depth, backoff).
  3. Users: list/search, create, deactivate, reset password, devices, admin flag, registration tokens.
  4. Rooms: list/search, members, state viewer, purge, force-leave, alias management.
  5. Media: usage per user, quotas, purge remote media.
  6. Settings: registration on/off, rate limits, server notices, appservice registrations.
  7. Audit log of admin actions.
• Starts: after server Phase 1 (dashboard + users + federation switch), grows with each server phase.

8.2 Web client — Go + Vue (apps/web)

• Frontend: Vue 3 SPA in apps/web/frontend, built on the shared apps/ui component library and a Pinia-based Matrix store wrapping matrix-js-sdk (sync, rooms, timeline) with matrix-sdk-crypto-wasm for E2EE. Go is not a good fit for in-browser Matrix crypto; the Rust-Wasm crypto is what Element uses and is audited.
• Go host (apps/web/cmd/warp-web): serves the SPA, injects runtime config (homeserver URL, branding), handles /.well-known for client discovery, optional reverse proxy to the homeserver to avoid CORS. Can also be embedded into the warp binary behind web_client.enabled: true.
• Feature order: login/register → room list + timeline → send text/media → read receipts, typing → invites/joins, room settings → E2EE (device verification, key backup) → threads, reactions, spaces → voice messages, calls (much later).
• Starts: after server M1.

8.3 Desktop client — Go (apps/desktop)

• Stack: Wails v3. The UI is the same Vue frontend as apps/web (imported as a workspace package), so web and desktop share every screen.
• Go side: mautrix-go client with goolm (pure Go Olm/Megolm, no cgo) for E2EE, local encrypted SQLite store for sync state and keys, OS keychain for the access token, native notifications, tray icon, auto-update. Rationale: on desktop, doing sync + crypto in Go instead of in the webview gives a persistent local store, background sync, and lets the frontend stay thin — the Vue store talks to Go via Wails bindings instead of matrix-js-sdk. The store therefore has two backends (js-sdk in the browser, wails on desktop) behind one interface.
• Targets: Windows, macOS, Linux.
• Starts: once the web client has a stable room list + timeline.

8.4 Android app — Flutter (apps/mobile)

• Stack: Flutter + matrix_dart_sdk (famedly) + flutter_vodozemac for E2EE, flutter_secure_storage, FCM/UnifiedPush for push (via the server's push gateway support from Phase 8), Riverpod for state.
• Feature order: login → room list → timeline + send → push notifications → E2EE + verification → media → calls (later).
• Uses Simplified Sliding Sync when the server has it (Phase 10) for fast startup; falls back to /sync.
• Starts: after server Phase 8 (push) for a useful MVP; UI work can begin after M1.

8.5 iPhone app — future

• Same Flutter codebase; work items are APNs push (needs a push gateway such as Sygnal, or a Warp-native one), iOS-specific secure storage and keychain, App Store review. Deferred until the Android app is stable.

---

9. Testing strategy

Level	Tool	What
Unit	go test, table-driven	internal/matrix (100% of spec examples); storage against all three drivers
Component	httptest	Each API package with a fake roomserver/userapi; federation gate tests (routes 404 when disabled, resume when enabled)
Integration	Complement (Docker)	Two profiles: federation=off (C-S tests only) and federation=on; track pass counts as the primary progress metric
Interop	Manual + scripted	Federate a dev instance with a Synapse in Docker; join #matrix:matrix.org
Fuzz	go test -fuzz	Canonical JSON, event parsing, X-Matrix header parsing, federation PDU intake
Frontend	Vitest + Playwright	apps/ui components; end-to-end login/send/receive against a Warp container
Mobile	Flutter test + integration_test	Against the same Warp container

---

10. Milestones

#	Milestone	Definition of done
M1	"Hello Matrix"	Element Web can register, log in, create a room, send messages, see them after reload — with federation off (server Phases 0–4). Admin panel shows dashboard, users, federation switch.
M2	"On the network"	Federation toggled on from the admin panel; Warp joins a room on matrix.org; Synapse users join a Warp room; toggling off stops remote traffic cleanly (Phase 5).
M3	"Daily driver"	Warp web client: E2EE rooms with cross-signing and key backup, images/files, push (Phases 6–8). Desktop client runs the same UI via Wails.
M4	"Pocket server"	Android app with push and E2EE; bridges (mautrix) run via the AS API; sliding sync; PocketBase driver at parity (Phases 9–10).
M5	"Everywhere"	iOS build of the Flutter app in TestFlight.

---

11. Reference material

• Spec: https://spec.matrix.org/latest/ — start with Room Versions and the Server-Server API appendices.
• Dendrite (architecture reference; read, don't paste): https://github.com/element-hq/dendrite
• gomatrixserverlib (test vectors): https://github.com/matrix-org/gomatrixserverlib
• Complement: https://github.com/matrix-org/complement
• Federation tester: https://federationtester.matrix.org
• PocketBase as a Go framework: https://pocketbase.io/docs/go-overview/
• mautrix-go + goolm: https://github.com/mautrix/go
• matrix-js-sdk / matrix-sdk-crypto-wasm: https://github.com/matrix-org/matrix-js-sdk
• Wails v3: https://v3alpha.wails.io
• matrix_dart_sdk: https://github.com/famedly/matrix-dart-sdk
• Synapse admin API (feature checklist for our admin panel): https://element-hq.github.io/synapse/latest/usage/administration/admin_api/

---

12. Open questions

1. State storage model: Dendrite's snapshot + block NIDs vs. per-event full-state rows. Decide in Phase 3 after a spike with a 10k-event room.
2. Room version breadth for federation compatibility (v6+ probably enough).
3. Presence default on/off.
4. PocketBase scope: full storage driver as specified, or only media filesystem + admin auth on top of the sqlite driver? The latter is far less work for most of the benefit. Decide after the sqlite driver is done.
5. Desktop sync engine: Go (mautrix-go) as planned, or keep matrix-js-sdk in the webview for maximum code sharing with web? Prototype both in a week and pick.
6. Push gateway for mobile: run Sygnal, or build a small Warp-native FCM/APNs gateway in Go?