Skip to content

Architecture

Every stage below is configured from config.yaml and can be swapped, tuned, or turned off. Solid paths are always on; the dotted paths (HyDE, the scope lanes, the routed lane) are off by default; the code lane opens only when a query calls for it.

The query path

flowchart TD
    Q["Question"] --> R{Intent routing}
    R -->|prose, HyDE off| HY["Hybrid retrieval"]
    R -.->|prose, HyDE on| H["Optional HyDE query expansion<br/>LLM drafts a hypothetical answer to embed"]
    R -->|code intent| C0["Skip HyDE · widen pool · open code lane"]
    H -.-> HY
    C0 --> HY
    HY --> D["Dense · ChromaDB<br/>bge-small-en-v1.5"]
    HY --> S["Sparse · bm25s"]
    HY -.-> SL["Scope lanes<br/>domain / path / file-type filters"]
    HY --> CL["Code lane<br/>.ipynb / .py / .R / .sql / …"]
    HY -.-> RT["Routed lane · generated_mode route<br/>primary pages a generated hit cites"]
    D --> GF["Generated filter · inside every lane<br/>drop or route: generated chunks stay out"]
    S --> GF
    SL -.-> GF
    CL --> GF
    GF --> RRF["Reciprocal Rank Fusion (k=60)<br/>+ optional metadata boosts"]
    RT -.-> RRF
    RRF --> RR["Configured rerank policy<br/>cross-encoder / HTTP / lexical / none<br/>(experimental: Laya)"]
    RR --> GT["Optional relevance gate<br/>drops weak chunks · abstains if none pass"]
    GT --> EX["Optional small-to-big<br/>context expansion"]
    EX --> G["Grounded generation<br/>answer + [n] citations + confidence"]
    G --> V["Optional second-pass<br/>citation verification"]

Ingestion

The index is built from JSONL chunk files, one per source family. Each loader normalises its inputs into the same chunk schema (text + metadata: source_file, domain, file_type, has_code, wikilinks, and a configurable grouping label — taxonomy.label, which defaults to course but is just as happily project, client, or topic):

Loader Handles
obsidian_parser Markdown notes; heading-aware sectioning; group/domain tagging; wikilink capture.
pdf_loader Lecture PDFs and textbooks; OCR-capable for scanned pages.
ipynb_loader Jupyter and R notebooks (.ipynb, .R, .Rmd, .py).
code_loader Other source languages (.js/.ts/.sql/.go/.java/.c/.cpp/.rs/.sh/…) into a dedicated code lane.
canvas_loader Obsidian .canvas files: one chunk per text node, its id keyed on the canvas path and node id (so an edited node keeps its id), with the node's labelled edges flattened into its text and metadata, so the graph needs no retrieval-time code. With canvas.label_tags on, #tags written in an edge label also become tags of the chunks of both nodes it joins.
ocr_vlm Vision-model OCR path for hard scanned material.

Selectable chunking

Oversized sections are split by one of four strategies: heading (the default — packs paragraphs and sentences, so it respects document structure and wins on well-structured PDFs), fixed (a strict sliding window, better for OCR walls of text with no paragraph breaks), document (element-aware packing that keeps each fenced code block, table, list and paragraph whole, falling back to the sliding window only for one element that alone exceeds the size) or none (no splitting: each section stays one chunk, which an embedding model truncates when it is long). Small sections chunk the same under every strategy. Choose per run with --chunking heading|fixed|document|none.

Hybrid retrieval

  • Dense — sentence embeddings (bge-small-en-v1.5) in ChromaDB capture meaning and paraphrase.
  • Sparse — BM25 (bm25s) captures exact terminology, symbols, rare names, and code tokens that embeddings blur.

  • Fusion — the two ranked lists are combined by Reciprocal Rank Fusion:

    $$\text{RRF}(d) = \sum_{\ell \in \text{lanes}} \frac{1}{k + r_\ell(d)}, \quad k = 60$$

    where $r_\ell(d)$ is document $d$'s rank in lane $\ell$. RRF needs no score normalisation across incompatible scales, and $k=60$ damps the influence of any one lane's top ranks so a single list can't dominate.

Lane weights are opt-in

Each lane's term can be scaled — w_ℓ / (k + r) — with retrieval.lane_weights, per preset or per call. Every weight defaults to 1.0, so an untouched config is plain RRF. See Lane weights.

Generated content never grounds an answer

Chunks an LLM wrote (knowledge maps, summaries, agent-instruction files) are kept out of every lane by retrieval.generated_mode, whose shipped default is drop: the dense lanes over-fetch and check each row's stored provenance, the sparse lanes remove those rows before the top-k cut so each lane refills. Under route one more lane runs: it searches the generated rows, reads the pointer sidecar (retrieval.route.pointers_file) for the primary pages each hit cites, and brings those pages into the pool as ordinary candidates. They fuse (the routed lane weight) and rerank like any other, and each carries found_via. Route needs retrieval.sparse_backend: bm25s. A call can also restrict every lane to one file's primary chunks, or a page range of it, with source_file and pages. See Route mode.

Scope routing

Queries that name a domain or content type are routed toward where they point:

  • retrieval.domain_signals maps aliases (e.g. "BI", "DataViz", "pytorch") to domain metadata values.
  • retrieval.content_signals maps phrases ("the handbooks", "meeting notes", "the reference books", "cheat sheet") to path substrings or file types.

With retrieval.scope_lanes on (it ships off; a call can also set scope_lanes), a detected scope adds filtered dense + sparse lanes to the fusion. With it off, the scope is still detected and reported in the retrieval echo, but no scope lane runs. Routing is soft: in-scope chunks are guaranteed seats in the candidate pool, but the reranker still makes the final call — so a wrong hint degrades gracefully instead of returning nothing. Both dictionaries are config-only; extend them without touching code.

The code lane

Code and notebook chunks are a small slice of the corpus. For a query like "show me a complex ggplot from my code", a prose-oriented pipeline with HyDE on writes a prose draft that lands near lecture notes, and BM25's code hits get outvoted in fusion by textbook pages that merely repeat the keyword — so the reranker never even sees the user's own code.

The fix, triggered by a configurable code-intent signal list:

  1. Skip HyDE if it is on (a prose hypothetical hurts code retrieval).
  2. Widen the dense/sparse candidate pool.
  3. Reserve a filtered lane for script/notebook chunks so they always reach the reranker.

Concept queries are untouched; code queries get their own material back at the top.

Reranking and context expansion

The configured rerank policy is one of several deliberately different lanes:

  • cross_encoder reads each query/candidate pair in process.
  • http delegates the same candidates to a configured OpenAI-style /v1/rerank service.
  • lexical measures model-free query-term coverage.
  • none preserves the fused RRF order for a no-rerank baseline.
  • laya (experimental, off by default) scores each passage with a fine-tuned Laya checkpoint. It is refused unless retrieval.laya.enabled, and never replaced by another mode.

Known in-process models carry their context limits in the reranker registry. BAAI/bge-reranker-base, for example, accepts at most 512 input tokens; an over-length configuration is rejected explicitly instead of failing later with an opaque tensor-index error. Runtime model failures are returned as reranking errors, distinct from generation-provider failures.

After reranking, an optional relevance gate (off by default) drops chunks scoring below a threshold and, when none pass, makes the answer abstain with a fixed "nothing relevant" reply and no LLM call. It judges the original question, reads either the active reranker's own score or the experimental Laya scorer's probability, and runs before the next step; see Relevance gate.

After that, an optional small-to-big step expands each survivor with its surrounding parent context before generation, trading prompt length for completeness.

Grounded generation

The generator answers from the retrieved excerpts only, emits inline [n] citations and a confidence line, and can run a second pass that checks each citation actually supports the sentence it's attached to. A provider registry resolves the wire protocol, endpoint, default model, and secret environment variable; a query may override only the configured provider/model pair, never inject an endpoint or key. If generation is unavailable, the API returns a provider-aware error object and retrieval-only continues to work.

Serving

Two FastAPI services, designed to run side by side:

  • serve_api (:8051) — the warm query endpoint. /query and /search return stable evidence ids plus their retrieval origins; /chunks/{id} dereferences evidence when lookup_available is true; /providers reports backend readiness without secrets; and /compare runs bounded preset, reranker, or generation-provider branches. It loads the indexes and models once at startup and stays hot.
  • manage_api (:8052) — the Corpus Ledger console backend: browse, retag, and delete documents; run ingest and maintenance jobs; and an in-app Info tab that draws this pipeline with a query/ingestion toggle.