Warp/docs/push.md

# Push and notifications

`server/internal/pushrules` and `server/internal/pushserver`: PLAN.md
Phase 8. Push rules decide which events notify a user. Every event that
notifies somebody is recorded, counted per room as unread, and pushed to
the push gateways of that user's devices.

The two packages split along a clear line:

| Package | What |
|---|---|
| `pushrules` | The rule language: the rule model, the spec's default ruleset, conditions, glob matching, evaluation. Pure code, like `internal/matrix`: no storage, no HTTP. |
| `pushserver` | Everything around it: keeping each user's rules, the pushers, the worker that evaluates events, the counts, the push gateway client. |

## Push rules

### Where they are kept

A user's push rules are their `m.push_rules` account data. That is where
the spec has clients read them, and where every client already takes them
from: the account data in its first `/sync`. A separate table would be a
second copy, re-rendered into account data on every change.

The push server is the only writer. `PUT /user/{id}/account_data/m.push_rules`
is refused with `405 M_BAD_JSON`, as the spec requires. Changes go through
`/pushrules`.

The stored document is the whole ruleset as a client sees it. When it is
read back, only the parts the user owns are kept:

- the rules they created, in their order
- the `enabled` flag and `actions` of each default rule

Everything else comes from `pushrules.Defaults`: which default rules exist,
and their conditions. So when a server upgrade adds a default rule (newer
spec versions do), every user gets it on their next read. Nobody's account
data has to be rewritten. A default rule the stored copy still has but
the server has since dropped is ignored.

A user who has never changed a rule has nothing stored. They still have
rules: `/pushrules/`, the account data endpoint and the initial sync all
render the defaults for them.

### Order

Within a kind, user rules come before the defaults of that kind. The
exception is `.m.rule.master`, which stays ahead of everything, so switching
it on still silences everything. This is also where Synapse puts them.

| Kind | Order |
|---|---|
| override | `.m.rule.master`, user rules, the other defaults |
| content | user rules, `.m.rule.contains_user_name` |
| room, sender | user rules |
| underride | user rules, the defaults |

A new rule with no `before` or `after` goes first among the user rules of
its kind. A rule replaced with no `before` or `after` keeps its place and
its `enabled` flag. `before` and `after` must name a user rule of the same
kind; a default rule, or one that does not exist, is `404 M_NOT_FOUND`.

### The defaults

The spec's predefined rules, in the spec's order:

- override: `master` (off), `suppress_notices`, `invite_for_me`,
  `member_event`, `is_user_mention`, `contains_display_name`,
  `is_room_mention`, `roomnotif`, `tombstone`, `reaction`, `server_acl`,
  `suppress_edits`
- content: `contains_user_name`
- underride: `call`, `encrypted_room_one_to_one`, `room_one_to_one`,
  `message`, `encrypted`

An event with an `m.mentions` property says outright whom it mentions.
For such an event the three legacy rules that guess from the body —
`contains_display_name`, `contains_user_name` and `roomnotif` — are skipped,
as the spec requires.

### Evaluation

| Condition | Matches when |
|---|---|
| `event_match` | the string at `key` matches the glob, case-insensitively. `*` is any run of characters and `?` one. The whole value has to match, except for `content.body`, where the pattern has to match whole words somewhere in it. A number or boolean at the key never matches. |
| `event_property_is` | the value at `key` is exactly `value`: a string, an integer, a boolean or `null` |
| `event_property_contains` | the array at `key` contains `value` |
| `contains_display_name` | the body contains the user's display name in the room as whole words. The name is a literal, not a glob, so a user called `*` is not mentioned by everything. |
| `room_member_count` | the joined member count compares with `is` (`2`, `==2`, `<2`, `>2`, `<=2`, `>=2`) |
| `sender_notification_permission` | the sender's power level is at least `notifications.{key}` in the room's power levels (default 50) |
| anything else | never: a rule with a condition the server does not know never matches |

Dotted keys follow the spec's escaping: `content.m\.mentions.user_ids`
reaches into a property whose name has a dot in it.

A word boundary is any character that is not a letter, a digit or an
underscore, in any script. That is Python's `\w`, so it matches what
Synapse does.

Compiled patterns are cached, up to 4096 of them. Patterns come from users,
so the cache is bounded and simply starts over when full.

## Notifications

### The worker

The push server is a listener on the roomserver's emit stage, like the
federation queue, and like the queue it does no work there. The emit stage
runs while the room's write lock is held. Evaluating an event needs the
room's members, their display names, the power levels, and a ruleset for
each member, so the event goes on a list and a worker of the push server's
own takes it from there, in order.

For each event the worker:

1. If it is a local user leaving or being banned, marks everything in the
   room read for them. A count left in a room they are no longer in would
   stay on their badge for good.
2. Reads the room (`roomserver.PushRoomFor`): joined members and their
   display names, the name and canonical alias, the power levels.
3. Evaluates the event for every local member except the sender, and for
   the target of an invite, who is not a member yet.
4. Records every event that notifies somebody, in one transaction that also
   moves each (user, room) count.
5. Wakes the syncs of the users it notified.
6. Pushes to their pushers' gateways, if `push.enabled` is on.

The room is read as it is now, not as it was at the event. The worker runs
moments after the event lands, so the two only differ if somebody joined or
renamed themselves in between. Reading the state at each event would cost a
state resolution per message per member.

Each user's merged ruleset is cached in memory. The push server makes every
write to the rules, which is what makes the cache safe in one process.

### Counts

| Table (migration `0010_push.sql`) | What |
|---|---|
| `push_notifications` | one row per event that notified a user: the event, its position in the room event stream, the actions, highlight, read |
| `push_room_counts` | each user's unread and highlight counts in each room, kept current by the writes, stamped with a notification stream position |
| `push_pushers` | pushers |
| `push_sequences` | the notification stream's counter |

The counts are a stream of their own: the seventh position of a sync token.
Every write that changes a user's counts in a room stamps the room with a
new position in the same transaction. An incremental sync then asks which
rooms' counts moved since its token, and reports them even when nothing
else happened in them.

That stream is what makes an asynchronous worker safe. A sync can show a
message before the worker has counted it. When the worker's write lands, it
moves the stream and wakes the sync, and the next sync carries the
corrected count. It also covers the case where counts move without any
event: a receipt sent from another device.

`unread_notifications` is reported for every joined room in a response, not
only the rooms whose counts moved. A client showing a new message wants the
count next to it, whichever sync moved it.

### Reading

A read receipt — `m.read`, or `m.read.private`, which is private from other
users but not from the user's own badge — marks the user's notifications
in the room read, up to and including the event it names. The counts are
then recounted from the rows that are still unread rather than decremented,
so if anything ever put them out of step, the next receipt puts them right.
`/read_markers` goes through the same path.

Notifications are not tracked per thread (threads are Phase 10). So a
receipt counts when it is for the whole room or for its `main` timeline. A
receipt inside a thread changes nothing; thread notifications are cleared
when the main receipt moves past them.

Read notifications are kept for `push.notification_retention` (30 days by
default) for `GET /notifications`, then pruned hourly. Unread ones are kept
until they are read.

## Pushers and the gateway

`POST /pushers/set` accepts the `http` kind only. An `email` pusher would
need a mail server and templates, and a client asking for one is told
`400 M_INVALID_PARAM` rather than given a pusher that never fires.

The gateway URL must be absolute, must have the path
`/_matrix/push/v1/notify`, and must not carry credentials. The spec asks for
HTTPS; plain HTTP is also accepted, because the usual deployment runs the
gateway (Sygnal, or a Warp-native one later — PLAN.md open question 6) next
to the homeserver on a private network.

A pusher remembers the session that registered it. Logging that session out
or deleting the device removes the pusher: the userapi's device observer
now fans out to the keyserver and the push server. Registering the same app
ID and push key for another user removes it from the previous user, unless
`append` is set.

A push carries the event's IDs, type, sender, sender display name, room
name and alias, the content (unless `push.include_content` is off), the
user's total unread count, the matched rule's tweaks, and the pusher's
`data` minus its `url`. `prio` is `high` for a highlight or anything that
makes a sound, and `low` otherwise. A pusher with `"format": "event_id_only"`
gets only the IDs, the priority and the counts; the app fetches the event
itself.

Pushes go out from four senders fed by a queue of 1024, so a slow gateway
delays only its own pushes. A server error, a 429 or a network failure is
retried twice more, after 1 s and then 4 s. A 4xx is not retried. When the
queue is full, a push is dropped and logged. A push is a best-effort
wake-up: the notification is already counted, and the device sees it on
its next sync whether or not the push arrived.

A push key the gateway lists in `rejected` is removed, so it is not pushed
to again.

## Endpoints

| Endpoint | Notes |
|---|---|
| `GET /pushrules/` · `GET /pushrules/global/` | The ruleset, under `global` or bare. The forms without the trailing slash are served too. |
| `GET` / `PUT` / `DELETE /pushrules/global/{kind}/{ruleId}` | `PUT` takes `?before=` / `?after=`. IDs starting with `.` are reserved (`400`). Deleting a default rule is `404`. |
| `GET` / `PUT /pushrules/global/{kind}/{ruleId}/enabled` | default rules included |
| `GET` / `PUT /pushrules/global/{kind}/{ruleId}/actions` | default rules included |
| `GET /pushers` · `POST /pushers/set` | `kind: null` removes a pusher |
| `GET /notifications` | `?from=`, `?limit=` (20 by default, at most 100), `?only=highlight`. Newest first. A notification whose event has since been purged, or that the user can no longer see, is left out. |
| `GET /user/{id}/account_data/m.push_rules` | the rendered ruleset |
| `PUT /user/{id}/account_data/m.push_rules` | `405 M_BAD_JSON` |

## What is deliberately missing

| Thing | Why |
|---|---|
| `unread_thread_notifications` | needs threads (Phase 10) |
| Skipping senders in `m.ignored_user_list` | ignored users' events still notify; a client hides them, but they count |
| Email pushers | needs a mail server; see above |
| Durable evaluation | the emit stage is in memory, so events persisted just before a crash are never evaluated. Their messages are still there; only the count and the push are missed. |
| A gateway URL deny-list | push gateways commonly sit on private networks, so private addresses cannot simply be refused the way remote media is. An operator who needs it should firewall outbound traffic. |
| Retries that survive a restart | a push waiting for its retry is lost on shutdown, for the same reason as above |