# Search

`POST /_matrix/client/v3/search` searches the rooms a user is in, or was
in, for words in messages, room names and topics.

The roomserver decides where a user may look (`internal/roomserver/search.go`),
the store runs the query against a full-text index
(`internal/storage/sqlstore/relations.go`), and the sync API shapes the
response (`internal/syncapi/search.go`).

## The index

Each storage engine uses its own full-text search, and the drivers hand the
shared store the statements for it:

| Driver | Index | Ranking |
|---|---|---|
| sqlite, pocketbase | an FTS5 table, tokenised with `unicode61 remove_diacritics 2`, so `cafe` finds `café` | `bm25` |
| postgres | a `tsvector` column with the `simple` configuration and a GIN index | `ts_rank` |

`simple` lower-cases and splits words without stemming for any one
language, which suits a server whose rooms are in every language at once.
It keeps diacritics, so on PostgreSQL `cafe` does not find `café`.

An event joins the index when it joins a timeline: `content.body` of an
`m.room.message`, `content.name` of `m.room.name`, `content.topic` of
`m.room.topic`. Outliers are not indexed; an outlier promoted into the
timeline is, unless it had already been redacted. A redaction removes the
event from the index, and purging a room removes the room's rows.
Migration 0012 indexes the events already stored.

An encrypted message has no body the server can read and is never found.
That is what end-to-end encryption means; clients search encrypted rooms
themselves.

## What a user may search

Every room the user is joined to, or has left or been banned from, each
bounded by the same visible range as a timeline read
([rooms.md](rooms.md)). A user who left a room finds what was said while
they were in it and nothing since; a user who joined a room with `joined`
history visibility finds nothing from before they came. The bounds go into
the query itself, one clause per room, so a page is never short because
results it could not show were filtered out afterwards.

## The query

What the user typed is reduced to words — runs of letters and digits —
and each is matched as a prefix, all of them required. Both engines'
query languages have operators (FTS5's quotes, `NEAR` and column filters;
tsquery's `&`, `|`, `!`), and a search box is not the place to expose
them: a user searching for `C++ & Go` means the words. A term with no words
in it finds nothing.

## Request and response

| Request field | |
|---|---|
| `search_term` | Required. |
| `keys` | Any of `content.body`, `content.name`, `content.topic`; all three by default. |
| `filter` | A room event filter: `rooms`, `not_rooms`, `types`, `not_types`, `senders`, `not_senders`, and `limit` (10 by default, 100 at most). |
| `order_by` | `rank` (default) or `recent`. |
| `event_context` | `before_limit` and `after_limit` (5 each by default, 20 at most), and `include_profile` for the senders' names and avatars. |
| `include_state` | The current state of every room with a result. |
| `groupings.group_by` | `room_id` and/or `sender`. |

The response carries `count` (every match, not just this page),
`highlights` (the words searched for), the `results` with their `rank`, and
`next_batch` when there is more. A ranked search pages by offset; a search
by recency pages by stream position, which keeps a page from repeating a
result when new messages arrive between two requests.

## What is deliberately missing

| Thing | Why |
|---|---|
| Ranking across engines | `rank` is only comparable within one search on one server. |
| Highlighting stemmed forms | Highlights are the words as typed. |
| Searching rooms by their contents over federation | Search covers what this server holds. |