Warp/docs/push.md

# Push and notifications

`server/internal/pushrules` and `server/internal/pushserver`: PLAN.md
Phase 8. Push rules decide which events notify a user. Every event that
notifies somebody is recorded, counted per room as unread, and pushed to
the push gateways of that user's devices.

The two packages split along a clear line:

| Package | What |
|---|---|
| `pushrules` | The rule language: the rule model, the spec's default ruleset, conditions, glob matching, evaluation. Pure code, like `internal/matrix`: no storage, no HTTP. |
| `pushserver` | Everything around it: keeping each user's rules, the pushers, the worker that evaluates events, the counts, the push gateway client. |

## Push rules

### Where they are kept

A user's push rules are their `m.push_rules` account data. That is where
the spec has clients read them, and where every client already takes them
from: the account data in its first `/sync`. A separate table would be a
second copy, re-rendered into account data on every change.

The push server is the only writer. `PUT /user/{id}/account_data/m.push_rules`
is refused with `405 M_BAD_JSON`, as the spec requires. Changes go through
`/pushrules`.

The stored document is the whole ruleset as a client sees it. When it is
read back, only the parts the user owns are kept:

- the rules they created, in their order
- the `enabled` flag and `actions` of each default rule

Everything else comes from `pushrules.Defaults`: which default rules exist,
and their conditions. So when a server upgrade adds a default rule (newer
spec versions do), every user gets it on their next read. Nobody's account
data has to be rewritten. A default rule the stored copy still has but
the server has since dropped is ignored.

A user who has never changed a rule has nothing stored. They still have
rules: `/pushrules/`, the account data endpoint and the initial sync all
render the defaults for them.

### Order

Within a kind, user rules come before the defaults of that kind. The
exception is `.m.rule.master`, which stays ahead of everything, so switching
it on still silences everything. This is also where Synapse puts them.

| Kind | Order |
|---|---|
| override | `.m.rule.master`, user rules, the other defaults |
| content | user rules, `.m.rule.contains_user_name` |
| room, sender | user rules |
| underride | user rules, the defaults |

A new rule with no `before` or `after` goes first among the user rules of
its kind. A rule replaced with no `before` or `after` keeps its place and
its `enabled` flag. `before` and `after` must name a user rule of the same
kind; a default rule, or one that does not exist, is `404 M_NOT_FOUND`.

### The defaults

The spec's predefined rules, in the spec's order:

- override: `master` (off), `suppress_notices`, `invite_for_me`,
  `member_event`, `is_user_mention`, `contains_display_name`,
  `is_room_mention`, `roomnotif`, `tombstone`, `reaction`, `server_acl`,
  `suppress_edits`
- content: `contains_user_name`
- underride: `call`, `encrypted_room_one_to_one`, `room_one_to_one`,
  `message`, `encrypted`

An event with an `m.mentions` property says outright whom it mentions.
For such an event the three legacy rules that guess from the body —
`contains_display_name`, `contains_user_name` and `roomnotif` — are skipped,
as the spec requires.

### Evaluation

| Condition | Matches when |
|---|---|
| `event_match` | the string at `key` matches the glob, case-insensitively. `*` is any run of characters and `?` one. The whole value has to match, except for `content.body`, where the pattern has to match whole words somewhere in it. A number or boolean at the key never matches. |
| `event_property_is` | the value at `key` is exactly `value`: a string, an integer, a boolean or `null` |
| `event_property_contains` | the array at `key` contains `value` |
| `contains_display_name` | the body contains the user's display name in the room as whole words. The name is a literal, not a glob, so a user called `*` is not mentioned by everything. |
| `room_member_count` | the joined member count compares with `is` (`2`, `==2`, `<2`, `>2`, `<=2`, `>=2`) |
| `sender_notification_permission` | the sender's power level is at least `notifications.{key}` in the room's power levels (default 50) |
| anything else | never: a rule with a condition the server does not know never matches |

Dotted keys follow the spec's escaping: `content.m\.mentions.user_ids`
reaches into a property whose name has a dot in it.

A word boundary is any character that is not a letter, a digit or an
underscore, in any script. That is Python's `\w`, so it matches what
Synapse does.

Compiled patterns are cached, up to 4096 of them. Patterns come from users,
so the cache is bounded and simply starts over when full.

## Notifications

### The worker

The push server is a listener on the roomserver's emit stage, like the
federation queue, and like the queue it does no work there. The emit stage
runs while the room's write lock is held. Evaluating an event needs the
room's members, their display names, the power levels, and a ruleset for
each member, so the event goes on a list and a worker of the push server's
own takes it from there, in order.

For each event the worker:

1. If it is a local user leaving or being banned, marks everything in the
   room read for them. A count left in a room they are no longer in would
   stay on their badge for good.
2. Reads the room (`roomserver.PushRoomFor`): joined members and their
   display names, the name and canonical alias, the power levels.
3. Evaluates the event for every local member except the sender, and for
   the target of an invite, who is not a member yet.
4. Records every event that notifies somebody, in one transaction that also
   moves each (user, room) count.
5. Wakes the syncs of the users it notified.
6. Pushes to their pushers' gateways, if `push.enabled` is on.

The room is read as it is now, not as it was at the event. The worker runs
moments after the event lands, so the two only differ if somebody joined or
renamed themselves in between. Reading the state at each event would cost a
state resolution per message per member.

Each user's merged ruleset is cached in memory. The push server makes every
write to the rules, which is what makes the cache safe in one process.

### Counts

| Table (migration `0010_push.sql`) | What |
|---|---|
| `push_notifications` | one row per event that notified a user: the event, its position in the room event stream, its thread (`0013_thread_notifications.sql`), the actions, highlight, read |
| `push_room_counts` | each user's unread and highlight counts in each room, kept current by the writes, stamped with a notification stream position |
| `push_pushers` | pushers |
| `push_sequences` | the notification stream's counter |

The counts are a stream of their own: the seventh position of a sync token.
Every write that changes a user's counts in a room stamps the room with a
new position in the same transaction. An incremental sync then asks which
rooms' counts moved since its token, and reports them even when nothing
else happened in them.

That stream is what makes an asynchronous worker safe. A sync can show a
message before the worker has counted it. When the worker's write lands, it
moves the stream and wakes the sync, and the next sync carries the
corrected count. It also covers the case where counts move without any
event: a receipt sent from another device.

`unread_notifications` is reported for every joined room in a response, not
only the rooms whose counts moved. A client showing a new message wants the
count next to it, whichever sync moved it.

### Reading

A read receipt — `m.read`, or `m.read.private`, which is private from other
users but not from the user's own badge — marks the user's notifications
in the room read, up to and including the event it names. The counts are
then recounted from the rows that are still unread rather than decremented,
so if anything ever put them out of step, the next receipt puts them right.
`/read_markers` goes through the same path.

Each notification is filed under the thread its event is in: the root
that its `m.relates_to` names when `rel_type` is `m.thread`, or the main
timeline for anything else (a thread root itself included). A receipt
reads the part of the room its `thread_id` names:

| `thread_id` | Reads, up to the receipt's event |
|---|---|
| none | the whole room, every thread included |
| `main` | the main timeline only |
| a thread root's event ID | that thread only |

So reading one thread leaves the main timeline and the other threads
unread, and a client that knows nothing of threads sends unthreaded
receipts and clears the room as it always did. `/read_markers` sends
unthreaded receipts. A local user leaving or being banned reads the whole
room.

### Per-thread counts

`push_room_counts` stays the room's total across every thread, which is
what `unread_notifications` reports by default, what sliding sync reports,
and what a push's badge counts. A sync whose filter sets
`room.timeline.unread_thread_notifications` gets the counts split instead:
`unread_notifications` counts the main timeline only, and
`unread_thread_notifications` maps each thread root with something unread
to its `notification_count` and `highlight_count`. A thread with nothing
unread is left out of the map, and a room with no unread threads has no
map, both of which the spec allows.

The per-thread counts are counted from the unread rows of
`push_notifications` when a sync asks, rather than kept in a table like
the room totals: a room has few unread notifications at a time, they are a
range of an index, and a second table of counts would be one more thing
every write and receipt had to keep in step. They need no stream of their
own either: any change to a thread's counts is a change to its room's,
which stamps the room and brings it down the next sync.

Notifications stored before migration 0013 are filed by the relation the
roomserver projected onto their events in 0012; one whose event is gone
stays on the main timeline.

Read notifications are kept for `push.notification_retention` (30 days by
default) for `GET /notifications`, then pruned hourly. Unread ones are kept
until they are read.

## Pushers and the gateway

`POST /pushers/set` accepts the `http` kind only. An `email` pusher would
need a mail server and templates, and a client asking for one is told
`400 M_INVALID_PARAM` rather than given a pusher that never fires.

The gateway URL must be absolute, must have the path
`/_matrix/push/v1/notify`, and must not carry credentials. The spec asks for
HTTPS; plain HTTP is also accepted, because the usual deployment runs the
gateway (Sygnal, or a Warp-native one later — PLAN.md open question 6) next
to the homeserver on a private network.

A pusher remembers the session that registered it. Logging that session out
or deleting the device removes the pusher: the userapi's device observer
now fans out to the keyserver and the push server. Registering the same app
ID and push key for another user removes it from the previous user, unless
`append` is set.

A push carries the event's IDs, type, sender, sender display name, room
name and alias, the content (unless `push.include_content` is off), the
user's total unread count, the matched rule's tweaks, and the pusher's
`data` minus its `url`. `prio` is `high` for a highlight or anything that
makes a sound, and `low` otherwise. A pusher with `"format": "event_id_only"`
gets only the IDs, the priority and the counts; the app fetches the event
itself.

Pushes go out from four senders fed by a queue of 1024, so a slow gateway
delays only its own pushes. A server error, a 429 or a network failure is
retried twice more, after 1 s and then 4 s. A 4xx is not retried. When the
queue is full, a push is dropped and logged. A push is a best-effort
wake-up: the notification is already counted, and the device sees it on
its next sync whether or not the push arrived.

A push key the gateway lists in `rejected` is removed, so it is not pushed
to again.

## Endpoints

| Endpoint | Notes |
|---|---|
| `GET /pushrules/` · `GET /pushrules/global/` | The ruleset, under `global` or bare. The forms without the trailing slash are served too. |
| `GET` / `PUT` / `DELETE /pushrules/global/{kind}/{ruleId}` | `PUT` takes `?before=` / `?after=`. IDs starting with `.` are reserved (`400`). Deleting a default rule is `404`. |
| `GET` / `PUT /pushrules/global/{kind}/{ruleId}/enabled` | default rules included |
| `GET` / `PUT /pushrules/global/{kind}/{ruleId}/actions` | default rules included |
| `GET /pushers` · `POST /pushers/set` | `kind: null` removes a pusher |
| `GET /notifications` | `?from=`, `?limit=` (20 by default, at most 100), `?only=highlight`. Newest first. A notification whose event has since been purged, or that the user can no longer see, is left out. |
| `GET /user/{id}/account_data/m.push_rules` | the rendered ruleset |
| `PUT /user/{id}/account_data/m.push_rules` | `405 M_BAD_JSON` |

## What is deliberately missing

| Thing | Why |
|---|---|
| Per-thread counts in sliding sync and in pushes | sliding sync reports each room's total, and a push's badge is the user's total across rooms; neither has anywhere the spec puts a per-thread count |
| Skipping senders in `m.ignored_user_list` | ignored users' events still notify; a client hides them, but they count |
| Email pushers | needs a mail server; see above |
| Durable evaluation | the emit stage is in memory, so events persisted just before a crash are never evaluated. Their messages are still there; only the count and the push are missed. |
| A gateway URL deny-list | push gateways commonly sit on private networks, so private addresses cannot simply be refused the way remote media is. An operator who needs it should firewall outbound traffic. |
| Retries that survive a restart | a push waiting for its retry is lost on shutdown, for the same reason as above |