Warp/docs/development.md

# Development

## Prerequisites

- Go 1.26+. Nothing else for the default build: the SQLite driver is pure
  Go and `CGO_ENABLED=0` works everywhere.
- A C toolchain only if you want `go test -race`.
- PostgreSQL only to run the storage tests against it.
- `golangci-lint` v2 for `make lint`:
  `go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest`

## Everyday commands

```sh
make build         # bin/warp
make test          # go test ./... in server/
make check         # vet + test
make lint
make cover         # coverage.html
make clean
```

Each one is a two-line shell command if you would rather not use `make`;
look at the [Makefile](../Makefile).

## Running a server locally

```sh
cp warp.yaml.example warp.yaml   # set global.server_name
make build
./bin/warp generate-keys
./bin/warp serve
```

The SQLite file, the signing key and (later) the media store are written
relative to the working directory. They are in `.gitignore`.

For a throwaway server with no config file at all:

```sh
WARP_SERVER_NAME=dev.local WARP_PRIVATE_KEY=./keys/dev.key ./bin/warp generate-keys
WARP_SERVER_NAME=dev.local WARP_PRIVATE_KEY=./keys/dev.key ./bin/warp serve
```

## Adding a database migration

1. Write the same change twice, once per SQL dialect:
   `server/internal/storage/sqlite/migrations/NNNN_name.sql` and
   `server/internal/storage/postgres/migrations/NNNN_name.sql`.
   Use the next free number; `NNNN` is zero-padded to four digits. The
   pocketbase driver uses the SQLite files as they are.
2. The files are embedded with `embed.FS`, so a rebuild picks them up.
3. `./bin/warp migrate --status` lists what would run, `./bin/warp migrate`
   applies it. A server started with `database.auto_migrate: true` (the
   default) runs them itself.

Rules the runner enforces:

- One transaction per migration. A failure rolls that file back entirely
  and leaves the following ones unapplied.
- A migration that has already been applied and whose file has since
  changed is an error. Write a new migration instead of editing an old
  one; the checksum of every applied file is recorded.
- A database that has migrations this binary does not know about refuses
  to start, so an old binary cannot run against a newer schema.

## Long-polling endpoints

`/sync` holds a request open until something happens, which has two
consequences worth knowing about when running a server locally.

`listen.write_timeout` must stay above `sync.long_poll_timeout`, or the
listener cuts off a blocked sync before it can answer. The defaults
(`120s` and `30s`) leave room; a proxy in front of the server needs the same
check.

A blocked sync is woken through `syncapi.Notifier`, not by polling, so a
test that sends an event and then expects a blocked sync to return has to
make sure the sync is actually blocked first — otherwise it finds the event
without ever having waited, and passes without testing anything.
`Notifier.Listeners()` exists for that, and
`internal/clientapi/sync_test.go` shows the pattern.

## Adding an endpoint

Routes are registered in `server/internal/setup/routes.go` on an
`httputil.Router`, which is `http.ServeMux` plus per-route metrics:

```go
r.HandleFunc("GET /_matrix/client/v3/profile/{userID}/displayname", s.handleGetDisplayName)
```

Handlers have the signature `func(http.ResponseWriter, *http.Request) error`
and return Matrix errors directly:

```go
func (s *Server) handleThing(w http.ResponseWriter, r *http.Request) error {
    if somethingWrong {
        return httputil.Forbidden("You are not allowed to do that")
    }
    httputil.WriteJSON(w, http.StatusOK, response{})
    return nil
}
```

Anything returned that is not a `*httputil.MatrixError` is logged in full
and reported to the client as `M_UNKNOWN`, so internal detail never leaks.

Endpoints that belong to the federation surface are registered through the
gate instead, so they disappear when federation is switched off:

```go
r.Handle("GET /_matrix/federation/v1/version", s.federation.ProtectFunc(s.handleFederationVersion))
```

A Client-Server endpoint that would need federation to answer returns
`federation.RejectClientRequest("...")`, which is an `M_FORBIDDEN` that
says why.

## Tests

- Unit tests live next to the code. Each API package holds its own
  HTTP-level tests, built on a throwaway SQLite database with the real
  components behind them rather than fakes; `internal/setup` holds the
  smaller set that goes through the whole wrapped handler, to catch a route
  that was written but never registered.
- Storage tests run against every driver that is available:
  SQLite and PocketBase always, PostgreSQL when `WARP_TEST_POSTGRES_DSN` is
  set. PocketBase runs the SQLite migrations on PocketBase's own database,
  so a SQLite migration is a PocketBase one too. The
  PostgreSQL tests share one database, so each group of them truncates the
  tables it owns first; add a table to `resetRoomTables` or
  `resetSyncTables` when you add one, or the second test to insert the same
  row will collide — and only in CI, where PostgreSQL exists.
- The event core under `internal/matrix` is tested against the spec's own
  cryptographic and canonical JSON test vectors, and has fuzz targets for
  the two functions that read bytes straight off the network. `make test`
  runs the unit tests; the fuzzers are opt-in. See
  [event-core.md](event-core.md).
- CI runs `gofmt -l`, `go mod tidy` (as a diff check), `go vet`, the tests
  on Linux and Windows, the race detector on Linux, `golangci-lint`, and a
  smoke test that boots the real binary and curls it.

## Complement

[Complement](https://github.com/matrix-org/complement) is the official Matrix
integration suite. It runs a container per homeserver per test and drives it
over the real HTTP API, which makes its pass count the honest measure of how
far along the server is — PLAN.md section 9 treats it as the primary progress
metric.

```sh
make complement                                   # the C-S set, federation off
make complement-federation                        # the federation set
server/test/complement/run.sh TestLogin           # one test
COMPLEMENT_DEBUG=1 make complement                # with the container logs
```

It needs Docker. The script builds the image, clones Complement into
`.complement/` (ignored by git) and runs the subset for the phases that are
done. `COMPLEMENT_PROFILE` picks the set: `csapi` (the default) runs the
account, room and stream tests with federation off; `federation` runs the
Server-Server tests with it on. Extend the sets in
[run.sh](../server/test/complement/run.sh) as each phase lands, rather than
switching to the whole suite: a pass count that drops because a later phase
is missing is noise.

The image is [server/test/complement/Dockerfile](../server/test/complement/Dockerfile)
and the deployment is entirely `WARP_*` environment variables — see
[entrypoint.sh](../server/test/complement/entrypoint.sh). Two things there
are worth knowing: rate limiting is off, because the whole suite comes from
one address and would otherwise spend its time reading `M_LIMIT_EXCEEDED`;
and port 8448 is served over TLS with a certificate the entrypoint has the
CA Complement mounts at `/complement/ca` sign, the same CA Warp is told to
trust for its outbound connections.

## Creating the first account

A new install has registration off, so the first account is made from the
command line against the database directly — no server needs to be running:

```sh
./bin/warp create-account --username you --admin
```

It prompts for the password twice. `--password-file` reads it from a file and
`--password` takes it inline, for scripts that have already thought about the
shell history. From there, the admin API can turn registration on:

```sh
curl -X PUT http://localhost:8008/_warp/admin/v1/registration   -H "Authorization: Bearer $TOKEN" -d '{"enabled": true}'
```