Asking the documents (RAG)#

An assistant over the project’s own documents, with three properties that a usual RAG does not have:

  1. every answer cites the section it came from, and the citation is checked — an answer whose citation is not supported by the chunk becomes a refusal, not a plausible sentence;

  2. the corpus is fixed by a hash, so a receipt made today stops verifying the day a document changes;

  3. the generation is greedy, and Vision’s tensor library gives the same tokens on every device, so the same question over the same chunks gives the same answer — and a third party can re-run it.

Quick start#

vision rag index docs README.md --out vision-rag.json          # 729 chunks from 57 documents here
vision rag ask "como provo que o resultado é o mesmo em qualquer GPU?" \
    --index vision-rag.json --model /path/to/qwen2.5-coder-3b-instruct --format mx8 \
    --json answer.json

The answer goes to stdout; the sources and their support to stderr; --json writes the answer and its receipt (corpus hash, chunk hashes, prompt hash, answer hash, and the support of each citation).

How it works#

Step

What it does

index

splits the documents by heading — a chunk never crosses a section — and hashes each chunk

retrieve

ranks the chunks with BM25, ties broken by chunk id, so the ranking is the same everywhere

prompt

the rules, one worked example, and the numbered context

generate

greedy decoding: the same answer on every device

verify

the citation must be there, and the chunk it names must support the answer

receipt

the corpus hash, the chunk hashes, the prompt hash, the answer and the support of each citation

The refusal rules, in order: no chunk matches → NAO CONSTA; the answer cites nothing → NAO CONSTA; a citation whose chunk does not support the answer (less than half of the answer’s content words) → NAO CONSTA.

Measured (RTX 4050 Laptop, 6 GB)#

model

answer

why

Qwen2.5-Coder-3B, mx8

cites correctly

29.7 tokens/s, 3.2 GB, 15 s to load

Qwen2.5-0.5B, mx8

refuses (NAO CONSTA) even when the corpus answers

too small to follow the citation rule: it answers from its own knowledge and cites nothing, which the verifier correctly rejects

So the model is part of the contract: a 3B is the smallest that works here, and the verifier is what stops a smaller one from looking useful.

From an AI agent (MCP)#

VISION_RAG_INDEX=vision-rag.json VISION_RAG_MODEL=/path/to/3b vision mcp

The tool vision_rag takes {"question": "...", "k": 5} and returns the answer with its sources and the corpus hash. The index and the model are checked at the first call; the model is loaded once and kept for the session.

From LangGraph#

pip install -r integrations/langgraph/requirements.txt      # optional
python -m vision_rag "o que faz o verify_rate auto?"         # after installing the integration

build_graph(index, model) returns a compiled graph with three nodes — retrieve, generate, verify — and as_tool(index, model) returns a LangChain tool. Without LangGraph installed the same module returns an object with the same invoke, so the integration never becomes a hard dependency.

Limits, honestly#

  • No embeddings. Retrieval is BM25: it does not know that “prova” and “verificação” are related. It is deterministic, has no model and no network, and is good enough for a technical corpus with precise terms. Embeddings would be a second mode, with a second non-determinism to declare.

  • The corpus hash invalidates receipts on any edit. That is the design: documentation drift must fail verification loudly, not answer wrongly in silence.

  • The model must fit next to the rest: 3B in mx8 is 3.2 GB of a 6 GB card.

  • Greedy only. A sampled answer would not be reproducible; sampling would need a declared seed and a declared “not reproducible across devices” label.

  • The support check is lexical. It catches an answer from the model’s own knowledge; it does not prove a paraphrase right. A stronger check (an entailment model) would break the determinism story.

Tests#

The behaviour is covered by the project’s test suite: the index digest (same corpus, same hash; one word changed, different hash), the ranking order, the refusal on a question outside the corpus, the refusal when the citation is not supported, and the receipt fields. Two defects those tests caught in the first version: the chunk id used the absolute path (so the same corpus in another folder gave another digest), and the retriever matched function words (so “qual é a capital da Mongólia” found three chunks).