Warp/docs/deployment.md

# Deployment and operations

What there is for running Warp somewhere other than a laptop: the container
image, a Compose setup, a Helm chart, a Grafana dashboard, tracing, and a
load tester. All of it is under [`deploy/`](../deploy) except the image,
which is the [`Dockerfile`](../Dockerfile) at the root, and the load
tester, which is `server/cmd/warp-loadtest`.

## Choosing a storage driver

| Driver | For | Notes |
|---|---|---|
| `postgres` | production | Several connections, so reads and writes run side by side. |
| `sqlite` | development, a single user, a small server | One writer. Everything is one file plus `media.base_path`. |
| `pocketbase` | a [PocketBase](https://pocketbase.io) server you want to see and manage the data in | Warp's tables as PocketBase collections on a PocketBase server of its own (0.40 or later), reached only through its REST API. Media goes into PocketBase's storage, which can be an S3 bucket (`database.pocketbase.s3`). |

The pocketbase driver connects to PocketBase the way the postgres driver
connects to PostgreSQL: `database.connection_string` is the server's URL,
and Warp signs in as the superuser in `database.pocketbase`. Every table
the SQL drivers have is a superuser-only collection of the same name and
columns, so the PocketBase admin UI shows Warp's accounts, rooms and events
as records. Warp creates and updates the collections itself (`warp
migrate`), reads them through PocketBase's SQL endpoint with the same
queries the SQL drivers run, and writes them as record operations in batch
requests, each one a transaction. It turns batch requests on in
PocketBase's settings at startup.

Writes go through one lock in the Warp process, so one Warp server per
PocketBase, and every database call is an HTTP request: expect it to be
slower than PostgreSQL. Search has no full-text engine behind it; it
matches the start of words, case-insensitively. The storage tests run
against a real PocketBase. Back up PocketBase's whole `pb_data`.

## The container image

```sh
docker build -t warp .
```

One static binary on Alpine, running as UID 10001, with `/data` as its
working directory and volume. The defaults put the SQLite database, the
signing key and media under `/data`. The image does not create a signing
key on its own; the Compose file and the Helm chart do it on first start,
and a plain `docker run` needs one first:

```sh
docker run --rm -v warp-data:/data warp generate-keys --output /data/signing.key
```

## Compose

[`deploy/compose`](../deploy/compose) runs Warp with PostgreSQL, and with
the `observability` profile, Prometheus, Grafana (with the dashboard
provisioned) and Jaeger:

```sh
cd deploy/compose
WARP_SERVER_NAME=matrix.example.org docker compose up -d
docker compose --profile observability up -d
docker compose exec warp warp create-account --username admin --admin
```

Settings come from the environment, so anything in
[configuration.md](configuration.md) can be added to the `warp` service.
`WARP_TRACING_ENABLED=true` sends traces to the Jaeger container.

## Helm

[`deploy/helm/warp`](../deploy/helm/warp) is a chart for one Warp pod.
Warp is one process, so the deployment always runs exactly one replica and
replaces it with the `Recreate` strategy: two pods would be two
homeservers with the same name.

```sh
helm install warp deploy/helm/warp \
  --set serverName=example.org \
  --set publicBaseURL=https://matrix.example.org \
  --set image.repository=registry.example.org/warp --set image.tag=0.10.0 \
  --set ingress.enabled=true
```

There is no published image; build one and push it to a registry the
cluster can pull from. The chart creates a volume for `/data` that is kept
when the release is deleted, because the signing key is on it unless
`signingKey.existingSecret` names a secret holding one. For PostgreSQL, set
`database.driver=postgres` and either `database.connectionString` or
`database.existingSecret`. `metrics.serviceMonitor.enabled` adds a
prometheus-operator `ServiceMonitor`. The ingress needs a read timeout
above the 30-second long poll; `values.yaml` has the ingress-nginx
annotations.

The admin panel is off by default. `admin.enabled=true` adds a Deployment
and Service for `warp-admin` on port 8009, pointed at the homeserver's
Service. Its image is built separately, also from the repository root:

```sh
docker build -t registry.example.org/warp-admin:0.10.0 -f apps/admin/Dockerfile .
helm upgrade warp deploy/helm/warp --reuse-values \
  --set admin.enabled=true \
  --set admin.image.repository=registry.example.org/warp-admin \
  --set admin.image.tag=0.10.0
```

Without an ingress, reach it with `kubectl port-forward svc/warp-admin
8009`. `admin.ingress.enabled` with `admin.ingress.host` gives it its own
ingress on a separate host, ideally one only admins can reach. warp-admin
reaches `/_warp/admin` inside the cluster, so you can then drop `/_warp`
from `ingress.hosts[].paths`. `admin.resources` and `admin.replicas` are in
`values.yaml`.

## Admin panel

The admin panel is optional and is not part of the `warp` binary. It is a
separate service, `warp-admin`, that serves the panel and proxies the
login endpoints and `/_warp/admin/v1` to the homeserver. Install it only
if you want it:

```sh
cd deploy/compose
docker compose --profile admin up -d    # http://localhost:8009
```

Or build the image (`make admin-docker`) or the binary (`make admin`) and
point it at the homeserver with `WARP_ADMIN_HOMESERVER`. Since only
warp-admin needs `/_warp/admin`, your public proxy can leave that path out
and keep the panel on an internal hostname. See
[apps/admin](../apps/admin/README.md).

## Web client

The web client is optional too, and separate from both: `warp-web` serves
it and, by default, proxies the client's Matrix requests
(`/_matrix/client` and `/_matrix/media`, nothing else) to the homeserver,
so the browser only ever talks to warp-web.

```sh
cd deploy/compose
docker compose --profile web up -d      # http://localhost:8080
```

Or build the image (`make web-docker`) or the binary (`make web`) and point
it at the homeserver with `WARP_WEB_HOMESERVER`. If your homeserver is
already public behind its own proxy, set `WARP_WEB_PUBLIC_HOMESERVER` to
its URL instead: the browser then calls it directly and warp-web serves
only the client. See [apps/web](../apps/web/README.md).

## Calls

Calls need a LiveKit SFU and the LiveKit JWT service next to Warp.
For a local try, `deploy/compose/calls.yml` adds both:

```sh
cd deploy/compose
WARP_REGISTRATION_ENABLED=true docker compose -f docker-compose.yml -f calls.yml --profile web up -d --build
```

The Helm chart can run the JWT service (`calls.jwtService.enabled`) and
sets `client_api.rtc.livekit_service_url` from it; LiveKit itself runs
from its own chart. See [calls.md](calls.md) for how the pieces find each
other and what production needs.

## Reverse proxy

Behind a proxy that terminates TLS:

- Route `/_matrix` and `/.well-known/matrix` to Warp. Route `/_warp` too
  only if something outside reaches the admin API directly; warp-admin
  talks to Warp over its own connection.
- Keep the proxy's read timeout above `sync.long_poll_timeout` (30 s), or
  every idle `/sync` is cut off.
- Allow request bodies of `media.max_upload_bytes` (50 MiB by default).
- Set `client_api.trust_x_forwarded_for: true` only if the proxy
  overwrites `X-Forwarded-For`; the rate limiter keys on it.

## Metrics and the dashboard

`/metrics` is Prometheus text, on by default. Besides the Go runtime and
process collectors:

| Metric | |
|---|---|
| `warp_http_requests_total{route,method,status}` | Requests by route pattern (never by path, so room and user IDs cannot explode the cardinality). |
| `warp_http_request_duration_seconds{route,method}` | Latency histogram. Sync endpoints block by design, so leave them out of latency quantiles. |
| `warp_http_requests_in_flight` | |
| `warp_events_persisted_total{origin,kind}` | Room events persisted: `origin` is `local` or `federation`, `kind` is `state` or `message`. |
| `warp_sync_waiting` | Syncs blocked in a long poll, classic and sliding: roughly the connected clients. |
| `warp_sync_sliding_connections` | Sliding sync connections whose state the server holds. |
| `warp_federation_enabled` | 1 while federation is on. |
| `go_sql_*{db_name}` | The connection pool: open, in use, waits. |
| `warp_build_info{version,commit}` | |

[`deploy/grafana/warp.json`](../deploy/grafana/warp.json) is a Grafana
dashboard over these: overview stats, requests and latency by route,
events, sync connections, the database pool and the Go runtime. Import it
and pick the Prometheus data source; the `job` and `instance` variables
select the server.

## Tracing

With `tracing.enabled`, Warp exports OpenTelemetry traces over OTLP/HTTP to
`tracing.endpoint` (or to `OTEL_EXPORTER_OTLP_ENDPOINT`, or to
`localhost:4318`): an OpenTelemetry Collector, Jaeger, Tempo, Honeycomb.

- Every request is a server span named after its route pattern, with the
  method, path, user agent and status.
- Persisting an event, building a sync or a sliding sync response, and
  running a search are spans of their own inside it.
- Every request Warp sends — to other servers, to push gateways, to
  application services — is a client span, and carries the W3C
  `traceparent` on, so a federation request shows up in the same trace as
  the client request that caused it.
- A request that arrives with a `traceparent` continues that trace.

`tracing.sample_ratio` keeps that fraction of new traces; a request inside
a sampled trace is always kept. Off, tracing costs a function call per
span, and the trace context of incoming requests is still passed on.

## Load testing

`warp-loadtest` drives a server the way a crowd of clients would:

```sh
make loadtest ARGS="-users 50 -rooms 10 -rate 0.5 -duration 2m"
# or
go run ./server/cmd/warp-loadtest -url http://localhost:8008 -users 50 -rooms 10 -rate 0.5 -duration 2m -sliding
```

It registers the users (answering the dummy and registration-token
stages), creates the rooms and joins everybody to all of them, then for the
length of the run each user sends at `-rate` messages per second while
syncing in a long-poll loop — `/sync`, or sliding sync with `-sliding`.
Each message carries the time it was sent, so the report has, beside the
latency of every kind of request, the **delivery** time: from one user
sending to another user's sync handing it over, which is the number a user
of a chat server feels.

Point it at a test server with registration open and rate limiting off
(`WARP_RATE_LIMIT_ENABLED=false`), or it measures the rate limiter. It
creates accounts and rooms and never cleans them up.

A run on a laptop, PocketBase driver, 20 users in 4 rooms at one message a
second each (`-users 20 -rooms 4 -rate 1 -duration 15s`):

| operation | count | p50 | p99 |
|---|---|---|---|
| send | 280 | 24 ms | 52 ms |
| delivery (`/sync`) | 5320 | 147 ms | 280 ms |
| delivery (sliding sync) | 5320 | 205 ms | 368 ms |

Every message reached every other user. Those numbers are one writer
serving twenty clients that each re-sync on every message; PostgreSQL's
parallel reads are what moves them.