Reference and internals

How search works

The technical side of search: saved content, indexes, word forms and ranking.

Understand why a search finds a result or misses it: what gets indexed, how matches are ranked and how to return to the source. Search covers messages, notes and saved text from files or meeting transcripts. The examples below focus on the shared message-search engine behind tg and max. For everyday requests and commands, start with Searching.

Choose the right path

What can be searched and connected

Messages, email, notes, files and meeting transcripts are searchable, and each result keeps a reference to its source. Which records connect to which, and who creates each link, is on How things are linked.

search all queries messages and notes in tg and max, and messages, email and notes in memo, then merges their ranked lists. It reports sources it skipped; it does not turn every content type into one message index. Files need text extraction, and recordings need transcription before their contents can be searched.

Results can be connected through stored references and links. Message search joins messages to chats, senders and attachments for filtering. Direct reply links connect an answer to its parent message. A separate conversation build groups messages within a chat; it does not automatically group notes, files and meetings into one discussion.

Indexing overview

An index is a prepared lookup that avoids reading every stored text for each query. The store keeps original text and source references alongside searchable copies.

From saved content to searchable indexes
Saved content
messages, notes, extracted file text and transcripts
Local storage
original text, source references and metadata; indexes differ by content type
Words and stems
word indexes plus Snowball word forms
Similar spelling
message vocabulary trigrams → typo candidates
Parts of text
message substring index; separate from word matching

For messages, the word index supports words and phrases, and stemming groups word forms such as “invoice” and “invoices”. A vocabulary index uses trigrams (three-character pieces) to propose similarly spelled words for typo correction, then checks their edit distance. A separate raw-text trigram index supports substring matching in legacy search. Strict body: patterns can also express substring conditions, through bounded text checks. These are different matching paths; strict search does not silently fall back to the legacy substring index.

BM25 ranks word matches by how distinctive the words are and how they occur in a message. It is a ranking method, not a model download or an indexing step. Notes have their own text indexes; optional vectors support search by meaning for prepared notes and conversations.

Three paths through the message archive
Message archive
saved messages, metadata and direct reply links
Strict search
words + stems + filters → BM25 or newest first
Discovery
partial word matches + direct replies → coverage and rank fusion
Topic search
built conversations → words + optional meaning vectors
Original sources
message references and context; ranking is not proof of an answer

Strict message search applies the whole query. Discovery (--discover) retrieves partial word matches and direct replies, then ranks them by word coverage and lexical ranks. Topic search uses built conversations, with an optional meaning branch. Strict search and discovery need no conversation build or embedding model. Open original sources and check archive coverage before treating a result as an answer.

1. Archive and identity

Reads, explicit history fetches and background capture write messages into one shared SQLite store. Search reads this local copy and never downloads a chat's history on its own; word search can also ask the messenger's own search for candidates (below). Provider, account, chat and message identifiers keep equally numbered messages from different sources apart, and every result carries a qualified locator, so an agent can return to the original.

The store keeps each message's original text and, separately, its normalized text for search. It also records which ranges of each chat's history it holds. An index, a build or a vector does not prove that a whole chat was downloaded.

Good search needs downloaded chats. messages sync --all --background starts downloading the last 90 days of every chat; add --since-time 365d to go further back. Each run fetches at most 1,000 messages per chat in tg or 1,200 in max by default; repeat it to continue busy histories. Counts, topic search and filters read only the archive.

Message search starts in the active account. --source all or in:all widens it to every account in the store; a provider narrows that. Names resolve inside the chosen scope, and an ambiguous author or chat asks for a more precise identifier. Topic search also starts in the active account: its built conversations, optionally one chat and a time boundary; --source widens it explicitly.

The pipeline:

  1. Parse the text into a versioned Boolean syntax tree that keeps source positions for errors. The grammar follows a tested profile of Apache Lucene query syntax. Lucene is the query language here, not a search server or a Java index.
  2. Validate fields, values and operators against one registry. CLI text, MCP text and MCP syntax trees reach the same validator and service.
  3. Resolve the account scope, stored names and calendar boundaries in the chosen IANA time zone. topic: needs exactly one chat, because topic numbers are local to a chat.
  4. Compile the tree into parameterized SQLite. Words and phrases use an FTS5 word index over the normalized text; metadata predicates narrow the candidates.
  5. Expand wildcards and text: regular expressions through the word vocabulary. Full-text patterns and presets run bounded checks over candidates. Regular expressions run as a non-backtracking automaton.
  6. Rank or order only messages that match, then optionally fetch neighbours in each hit's own account and chat. Ranking never widens the result.
Strict message search, step by step
Query and scope
parse and validate; resolve accounts, chats, people and dates
Words and word forms
FTS5 words and Snowball stems; exact: keeps literal forms
Other conditions
metadata, attachment content and requested text patterns
Rank matching messages
BM25, exact forms first; newest for queries without required words or --newest
--context
optional neighbors in the same account and chat

The fields are text, exact, body, from, chat, date, kind, has, topic, in, preset, content, filename, mime, size and tag. File fields match a message when at least one of its attachments matches, by the name, size and type the messenger reported; MAX reports no type, so mime: finds nothing there. content: searches text extracted from attachments or supplied by an agent. tag: reads the owner's local tags on the message, its chat or its sender. date: also takes today, yesterday and offsets such as 7d, counted from the moment the query runs. Fuzzy, proximity, boost and interval operators are refused with an explicit error.

tg personal search messages 'invoice AND has:file' --source all --context 2 --json
max personal search messages 'chat:"Project team" AND date:[2026-10-01 TO 2026-10-31]' --timezone Europe/Madrid --json
tg personal search messages 'filename:*.pdf size>1MB date:7d' --json

text matches normalized words; body addresses the original full text, case-sensitive. body:/.*invoice.*/ expresses a substring condition. A date without a time is a calendar day in the chosen zone; an inclusive upper day includes the whole day, and days when clocks change are not assumed to last 24 hours.

When every branch requires words, results are ranked by BM25 over the word index, exact forms first; otherwise they come newest first. --newest always orders by time. Ties break on account-qualified identifiers, so pages are stable.

If the word index is not ready, a query with words fails with index_not_ready, the share already built and the command that finishes it (store migrate); it never falls back to substring matching. A query without words still runs. Query length, tree depth, automaton size, vocabulary expansion (10,000 words), candidates, scanned bytes, work and time are bounded. Exceeding a budget is an error (query_limit), never a silently shorter result; in tg, max, wa and zm, a search that runs past its time limit stops with timeout (reason query.time-limit).

The JSON answer includes the language, field and preset versions, the time zone, the order, wordsReady, per-chat completeness, and account and chat coverage, even with no hits. lastSyncedAt is the oldest time a chat in scope was last fetched by messages sync, or null if one never was. inventoryComplete says every account in scope has at some point listed all its chats; it is knowledge about the past, not a promise about now. coverage also reports the stored messages and chats in scope, counts never-fetched and behind chats, lists up to ten attention chats, and names next, the one command that would improve the answer. When no message is found and next is set, an agent runs it or asks the owner before concluding that the message does not exist.

stats messages show runs the same query and counts each matching message once, grouped by chat, sender, calendar day or hour. It counts the archive only. A successful search or count writes its query and options, never its results, to a local history of the newest 1,000 runs; saved searches are named entries that are parsed again on every run.

Strict Lucene is the only query language. For a regular expression, write a Lucene one such as body:/.*invoice.*/.

Questions and partial evidence

Use message search with --discover (MCP discover: true) when you remember a question rather than the exact words. It searches the local archive without downloading or loading a neural model. It does not need a conversation build or embeddings.

A candidate is a stored message selected for possible relevance before the final results are ordered. For ordinary English and Russian questions, discovery removes recognized question scaffolding, keeps the content words and retrieves partial word matches. Unknown qualifiers stay in the query. Chat, sender, date and account filters remain hard restrictions. Explicit Boolean syntax, phrases, wildcards, syntax trees, --exact and --newest keep strict behavior.

Scoring assigns candidates a relative position. Word coverage is the fraction of retained query terms found in the message and, for a direct reply, its matching parent. Reciprocal rank fusion adds a small bonus for appearing near the front of the lexical result lists; SQLite's word index supplies their BM25 ranking. Coverage carries most of the final score. This can retrieve replies that omit the parent's topic words. The final pool is capped at 300 messages, with at most 100 proposed replies and bounded text work. There is no cross-encoder or language-model reranking step.

query.discovery reports the actual retrieval work and truncation. Each hit names matched and missing terms; a reply's parent links to its matching message. These are evidence clues, not answer probabilities. A question, proposal, rejected option or old decision can rank first. Inspect the messages, surrounding context and archive coverage before answering. Discovery always uses the archive; --backend server is refused.

See the question planner, retrieval and ranking service and synthetic evaluation for implementation and measured limits.

Word forms

Words and quoted phrases in text, the default field, match every form of their words. квартира finds квартиру and квартиры, and piso finds pisos. Each word is compared by its Snowball stem, taken before the word is folded (ё→е, accents removed). Quotes keep words together but still allow every form: "оплатил счёт" finds оплатила счета. Messages that contain the exact form you typed come first. For the exact form only, write exact:квартира or exact:"оплатил счёт", or add --exact (MCP: exact: true) to treat every word without a field that way. -квартира excludes every form; -exact:квартира only the exact one. A wildcard such as квартир* and a text: regular expression still read whole words, without stems.

Word forms are not typo correction. Strict search, the default, does not correct misspellings in Telegram or MAX: presentaion does not find presentation. Try a wildcard on the part you are sure of (present*), or topic search by meaning. Automatic typo correction is available only in discovery (--discover) for ordinary keyword queries. Natural-question discovery preserves unknown terms. When you ask an agent, describe what you remember; it can choose these approaches for you.

Snowball stems recover more word forms than exact matching, but can also join unrelated words. See the word-form benchmark for the comparison with spelling-based trigram neighbors. Use exact: when that distinction matters.

The stemmers are a store-wide setting, shared by every profile, tg and MAX: search_stemmers.cyrillic (russian or none) and search_stemmers.latin (english, spanish, both — the default — or none). store rebuild applies your own change. While the stem index is being built, a search matches exact word forms and says so; serve finishes the stems in the background and store migrate at once. Some different words share a stem — часть/часто, caso/casa — and exact: or --exact separates them. The JSON answer says whether stems were used: query.stemming names the stemmer and each word's stem, stemsReady sits beside wordsReady, and each hit carries exact: false when only another form matched.

Server search beside the archive

search messages can also ask the messenger's own search, which can find messages that were never fetched. tg asks Telegram by default (--backend both). max asks MAX's server when the query has words: within the named chat (--chat or chat:), or without a chat across all chats, one best match from each, as the search bar on web.max.ru does; the archive supplies the other matches it holds. --backend server shows only the server's results, and --backend archive searches only the archive.

The server's matching is its own. Telegram does not document its rules; MAX's server finds a word by its beginning but not its other forms. So the server's results are only candidates: they are saved into the archive, and the same strict query then runs once over the archive and the new rows. exact:, -word, phrases, filters and the ranking keep their meaning, and a message is never listed twice. A candidate the strict query rejects is kept but not shown.

Only what narrows the candidates is sent: the required words and phrases, one chat, a sender with that chat, and dates; OR makes up to three server searches. Negations, wildcards, regular expressions, has:, tag: and presets stay local, and a query without words does not ask the server. The tool waits at most 5 seconds (--server-time, up to 60); a later answer is dropped, and nothing is marked read. Each hit says where it came from (source: archive, server or both), and a server block says what the server returned and why it was skipped or stopped. With --backend both, an unreachable server, a profile without the messages.server-search permission or a query without words leaves an archive answer; an explicit --backend server refuses instead. stats never asks the server, since the server counts by its own rules.

3. Reconstruct discussions with a message graph

Conversation search, from build to evidence
conversations rebuild
per-chat graph: reply → agent answer → rules
chunks
about 1,200 characters at message boundaries, with a content hash

search conversations

meaning
optional: embed the question and compare prepared chunk vectors
words
OR word matches → built conversations
rank fusion
each list adds 1 / (60 + rank) to a conversation
evidence
open the original messages; a candidate is not proof

conversations links rebuild walks one chat's stored messages in time order. Each message can have candidate links to earlier messages. A chosen parent joins it to that discussion; no parent starts a new one.

Precedence:

  1. The messenger's reply: an explicit reply to an earlier message present in the build wins.
  2. The agent's answer: a valid stored answer from the owner's own agent can choose an earlier parent or start a new conversation.
  3. Rules: the highest-confidence candidate from mentions (by @username or by name) or from the same sender continuing.

Rules look back at most 50 messages: in a measured group, 97% of reply parents were within 50. Same-sender continuation looks within the last 10 messages and five minutes. Both respect native thread identifiers. Rule weights are heuristics, not calibrated probabilities. An explicit reply can point farther back when its parent is in the build.

This is a per-chat graph of message relationships, not a knowledge graph of people, projects and chats. Missing reply parents, incomplete history and interleaved discussions can produce imperfect grouping.

Building is explicit. A new build publishes new memberships and conversation identifiers; the previous build stays readable until the new one is complete, and a failed build never exposes half a result. Each build records its rules version, so conversations status and store check can name the chats built with older rules.

Agent-assisted linking: skill show link-conversations gives the owner's agent its instructions; conversations batches status estimates the messages, batches and text; batches next hands out a window of 10–200 messages (50 by default) with the messages before it; conversations links add stores the agent's JSON answer, all or nothing, and conversations links clear drops the agent's answers for a chat. Storing answers needs the profile permission conversations.links. In this workflow the CLI never calls a model itself. An answer that refers to a changed or deleted message becomes stale and is left out of later builds.

conversations links rebuild --chat <chat> --analyze is the explicit alternative: the CLI sends bounded batches to a configured OpenAI-compatible or Anthropic endpoint. It needs one named chat, reports the volume, endpoint and token limit, and asks for consent, which is remembered for that account, chat and provider until revoked (conversations consents list, conversations consents remove). Defaults are 50 messages per batch and at most 100,000 reserved tokens per run; --yes grants consent in scripts.

tg personal conversations links rebuild --chat "Project team"
tg personal conversations list "Project team" --json
max personal conversations batches status --chat "Project team" --json

4. Pieces and vectors

The build cuts each conversation at message boundaries into pieces of about 1,200 characters, sender names included. A single message longer than that is split into overlapping pieces, so meaning search reads all of it. A conversation with no text makes no piece.

Each piece keeps its first and last message and a hash of its text. Vectors are stored by model and text hash, so identical text reuses a vector and an interrupted run resumes with what is left. Messages remain the evidence; vectors are a derived representation for retrieval.

Local models run through ONNX Runtime (WebAssembly) in Node or Bun. The default is e5-small (multilingual-e5-small, 8-bit, 384 dimensions, MIT). embeddinggemma (EmbeddingGemma 300M, 4-bit, 768 dimensions) finds more but runs about seven times slower and downloads only with --accept-terms, since it comes under Google's Gemma terms. Model files are pinned and checksum-verified, and two models' vectors are never compared with each other.

One local session uses min(8, cores) threads. --workers starts several sessions with the threads split between them, each with its own copy of the model, and --threads sets the total.

tg personal models download e5-small
tg personal conversations embed --chat "Project team"
tg personal conversations embeddings status --chat "Project team" --json

A remote provider computes vectors with the owner's key: OpenAI, or any server with OpenAI's /v1/embeddings shape through --base-url with --model and --dims, such as Gemini, Jina, Ollama or LM Studio. That sends the pieces' text, and each search sends the question. Before sending anything, embed states the pieces, the maximum tokens and the maximum price and waits for a yes (--yes in scripts, --max-tokens as a cap).

Before embedding, each piece's text is rebuilt from the current messages and checked against its stored hash. A changed piece is skipped until the next build.

Measurements

A synthetic benchmark with 100,000 messages measured building, embedding and searching separately. Embedding was the expensive preparation step; extra workers increased memory use and gave little speedup. Keeping the model open reduced search startup time. Hardware, archive size and model choice affect these results; see testing for how we measure performance.

5. Hybrid conversation retrieval

search conversations takes a question in natural language, not a Lucene query. It runs two branches:

  • Meaning: embed the question with the selected model, scan the vectors of current builds in scope and keep each conversation's best piece. With the local e5-small, a meaning match needs cosine similarity above 0.80.
  • Words: take the question's words, join them with OR over the word index and map the matching messages to built conversations. Exact word matches stay eligible below the meaning threshold.

The two ranked lists are combined by reciprocal rank fusion: each list adds 1 / (60 + rank) to a conversation, ranks starting at one. A result reports by: ["meaning"], ["words"] or both. Its score is the best piece's cosine similarity, not the fusion score and not a probability that it answers the question; a result found only by words has score: null. Its summary is metadata — identifiers, dates, message and sender counts — not generated prose.

--filter 'from:me date:7d' limits conversations before ranking: a single message in the conversation must match the whole strict query. --source personal|bots|all|<provider> widens the scope beyond the active account. Neither can accompany --catch-up.

With no local model downloaded, search answers by words alone and reports "meaning": "unavailable"; it never downloads a model or switches to a remote one. A chat embedded only with another model is left out of the meaning branch and named in embeddedOnlyElsewhere. A chat that was never built returns nothing here.

Vectors are SQLite blobs scanned in pages and scored in JavaScript. There is no approximate-nearest-neighbour index and no separate vector database: total work grows with the pieces in scope. Nearest neighbours appear even when the archive holds no answer, so verify the original messages.

conversations related uses the mean of a conversation's current vectors as the query and searches every built chat. No model runs, so it needs no download.

tg personal search conversations "Where did we discuss renting a flat?" --chat "Project team" --json
max personal conversations related "Project team" 4521 --json

6. Freshness

conversations status reports, for each built chat, a state — ready, stale, partial, words-only or not-built — with the messages the build has not seen (new, edited, deleted), whether the build used older rules, and the pieces whose vector is current, stale or missing. Without --chat it also counts the group chats never built.

  • Edits. A result whose piece changed after it was embedded is marked stale: true; its score is for the old text until the next build and embedding.
  • Deletions. A deleted message keeps its row, so a later sync cannot bring it back, but loses its text: in the row, the search copy, the edit history, transcripts and every vector of a piece that held it, unless a current piece without a deleted message still uses the same text.
  • Catch-up. conversations links rebuild and embed without --chat, and search conversations --catch-up, build and embed on this machine the chats that changed or were never built: at most 20 chats and 2,000 pieces a run by default (--max-chats, --max-chunks). They never download a model; a remote model is only ever used per chat, with the owner's yes. MCP offers the same catch-up as the conversations refresh write command.

7. Evidence and context for an agent

After retrieval, open the original message or the reconstructed conversation. An answer should keep four things apart: an exact message match, a candidate by meaning, a relationship from the graph and the agent's own conclusion. Keep the locator, chat, author, date and the relevant surrounding text with each claim.

--context follows time, not the graph. --thread follows the stored reply graph instead (by default 8 hops, 50 messages, 65,536 bytes and one day around each hit) and falls back to time order when a chat has no graph. messages links explains graph relations and the chosen chain of parents; conversations show reads the grouped messages. Evidence packets add source identities, content fingerprints, bounds and truncation metadata; even so, a page of evidence cannot prove a chat's full history. The report example in group administration shows why complete relationships between saved messages do not imply complete history.

Reading, searching and context never mark messages read or send anything. Fetching history, building, embedding and remote providers are separate, explicit actions.

8. What the playground demonstrates

The interactive playground runs the shared query parser over 50 fictional messages in six chats. It shows strict matching, field and value suggestions, removable filters, dates, file and link predicates, and context around a hit. An invalid edit keeps the last valid results and marks the syntax problem.

The browser opens no account and runs no conversation build, embedding or live AI. Its short summaries are prepared examples shown only when their evidence is retrieved. The interface is translated; the sample messages stay in English. Use it to learn how to inspect a source, then use the messenger guides for a real archive.

See testing for functional checks, ranking metrics, performance measurements and their limits.

Source map

Engine source in cli-messaging, the shared library behind both tg and max. These references describe an inspected implementation snapshot, not the installed version or a performance promise.

ResponsibilityImplementation
Cross-source search and typo matchingSearch all, notes search, typo correction and legacy matching
Archive preparation and coverageArchive service, fetch command and coverage summary
Parse, validate, scope and coverageQuery parser, message search service and Lucene SQLite executor
Word index, vocabulary and stemsWord matching, index filling and stemming
Server search candidatesServer search
Graph links and build lifecycleLink rules, conversation service and build persistence
Pieces and vector storageChunking and vector storage and scoring
Models, fusion and freshnessEmbedding service, model catalog and worker pool
Evidence packet contractEvidence service

For the exact grammar, fields, presets and limits, read the Telegram or MAX query-language reference. For preparation and options, read the Telegram archive and MAX archive guides.

Ready to try? Continue with search examples for Telegram or MAX.

The search research account connects the experiments and decisions. Model-selected evidence supplements remain experimental; they are not part of the released discovery path.