tajniak81andClaude Opus 5.5
487e0f410b
CI / Test (ubuntu-latest) (push) Canceled after 0s
CI / Test (windows-latest) (push) Canceled after 0s
CI / Storage tests on PostgreSQL (push) Canceled after 0s
CI / Complement (federation off) (push) Canceled after 0s
CI / golangci-lint (push) Canceled after 0s
CI / Smoke test the binary (push) Canceled after 0s
Implements PLAN.md Phase 10. Simplified Sliding Sync (MSC4186) is served at the unstable org.matrix.simplified_msc3575 endpoint and advertised in /versions. Lists are sorted by activity and filtered by DM, invite, encryption, room type, space and tag; room subscriptions, required_state with wildcards, $ME and $LAZY, and the to_device, e2ee, account_data, receipts and typing extensions are supported. Per-connection state is kept in memory and an unknown pos answers M_UNKNOWN_POS. Threads and relations: m.relates_to is indexed on roomserver_events (migration 0012), /relations (with recursion) and /threads are served, and every event a timeline read returns carries its m.thread, m.replace and m.reference summaries in unsigned.m.relations. Notifications carry a thread ID (migration 0013): threaded receipts clear only their thread, and sync reports unread_thread_notifications when the filter asks. Spaces: /hierarchy walks a space breadth first, asking other servers about rooms this one is not in through the new federation hierarchy endpoint; room_summary describes one room. /search runs on FTS5 (sqlite, pocketbase) or tsvector (postgres), bounded to what each user may see, with context, profiles, state, grouping and paging. The PocketBase driver is real: the sqlite schema and store on an embedded PocketBase app's database, with media in its filesystem, local or S3 (database.pocketbase.s3). It runs the full storage test suite. Ops: OpenTelemetry tracing over OTLP/HTTP (tracing.*), with server spans per route, component spans, and trace propagation on federation, push and application service calls. New metrics for persisted events, waiting syncs, sliding sync connections and the database pool, with a Grafana dashboard. deploy/ adds a Compose setup (PostgreSQL, Prometheus, Grafana, Jaeger) and a Helm chart; cmd/warp-loadtest loads a server and reports request and delivery latencies. The load tester found a long-standing bug, fixed here: on a server where no account data had been written, every incremental /sync re-sent the push rules and returned at once, and an all-zero token was taken for an initial sync, so long polls became busy loops. The PostgreSQL migrations and search dialect compile but were not run against a live PostgreSQL in this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
80 lines
3.6 KiB
Markdown
80 lines
3.6 KiB
Markdown
# Search
|
|
|
|
`POST /_matrix/client/v3/search` searches the rooms a user is in, or was
|
|
in, for words in messages, room names and topics.
|
|
|
|
The roomserver decides where a user may look (`internal/roomserver/search.go`),
|
|
the store runs the query against a full-text index
|
|
(`internal/storage/sqlstore/relations.go`), and the sync API shapes the
|
|
response (`internal/syncapi/search.go`).
|
|
|
|
## The index
|
|
|
|
Each storage engine uses its own full-text search, and the drivers hand the
|
|
shared store the statements for it:
|
|
|
|
| Driver | Index | Ranking |
|
|
|---|---|---|
|
|
| sqlite, pocketbase | an FTS5 table, tokenised with `unicode61 remove_diacritics 2`, so `cafe` finds `café` | `bm25` |
|
|
| postgres | a `tsvector` column with the `simple` configuration and a GIN index | `ts_rank` |
|
|
|
|
`simple` lower-cases and splits words without stemming for any one
|
|
language, which suits a server whose rooms are in every language at once.
|
|
It keeps diacritics, so on PostgreSQL `cafe` does not find `café`.
|
|
|
|
An event joins the index when it joins a timeline: `content.body` of an
|
|
`m.room.message`, `content.name` of `m.room.name`, `content.topic` of
|
|
`m.room.topic`. Outliers are not indexed; an outlier promoted into the
|
|
timeline is, unless it had already been redacted. A redaction removes the
|
|
event from the index, and purging a room removes the room's rows.
|
|
Migration 0012 indexes the events already stored.
|
|
|
|
An encrypted message has no body the server can read and is never found.
|
|
That is what end-to-end encryption means; clients search encrypted rooms
|
|
themselves.
|
|
|
|
## What a user may search
|
|
|
|
Every room the user is joined to, or has left or been banned from, each
|
|
bounded by the same visible range as a timeline read
|
|
([rooms.md](rooms.md)). A user who left a room finds what was said while
|
|
they were in it and nothing since; a user who joined a room with `joined`
|
|
history visibility finds nothing from before they came. The bounds go into
|
|
the query itself, one clause per room, so a page is never short because
|
|
results it could not show were filtered out afterwards.
|
|
|
|
## The query
|
|
|
|
What the user typed is reduced to words — runs of letters and digits —
|
|
and each is matched as a prefix, all of them required. Both engines'
|
|
query languages have operators (FTS5's quotes, `NEAR` and column filters;
|
|
tsquery's `&`, `|`, `!`), and a search box is not the place to expose
|
|
them: a user searching for `C++ & Go` means the words. A term with no words
|
|
in it finds nothing.
|
|
|
|
## Request and response
|
|
|
|
| Request field | |
|
|
|---|---|
|
|
| `search_term` | Required. |
|
|
| `keys` | Any of `content.body`, `content.name`, `content.topic`; all three by default. |
|
|
| `filter` | A room event filter: `rooms`, `not_rooms`, `types`, `not_types`, `senders`, `not_senders`, and `limit` (10 by default, 100 at most). |
|
|
| `order_by` | `rank` (default) or `recent`. |
|
|
| `event_context` | `before_limit` and `after_limit` (5 each by default, 20 at most), and `include_profile` for the senders' names and avatars. |
|
|
| `include_state` | The current state of every room with a result. |
|
|
| `groupings.group_by` | `room_id` and/or `sender`. |
|
|
|
|
The response carries `count` (every match, not just this page),
|
|
`highlights` (the words searched for), the `results` with their `rank`, and
|
|
`next_batch` when there is more. A ranked search pages by offset; a search
|
|
by recency pages by stream position, which keeps a page from repeating a
|
|
result when new messages arrive between two requests.
|
|
|
|
## What is deliberately missing
|
|
|
|
| Thing | Why |
|
|
|---|---|
|
|
| Ranking across engines | `rank` is only comparable within one search on one server. |
|
|
| Highlighting stemmed forms | Highlights are the words as typed. |
|
|
| Searching rooms by their contents over federation | Search covers what this server holds. |
|