Three stages, one probable cause
A RAG answer can go wrong in three places: the retriever fetched the wrong documents, the model said something the documents don't support, or it answered a different question. TraceroAI checks all three for every trace and names the first that failed, so a bad answer becomes a lookup rather than a guess.
1RetrievalProbable cause
Did the retrieved context match the question?
Retrieval miss
The documents that answer the question weren't retrieved, so the model had nothing right to work from.
For example: Asked about annual-plan refunds; only the monthly billing pages came back.
Usual fix: Retrieve more, or fix the index.
2GroundingNot the cause
Is every claim supported by that context?
3AnswerNot the cause
Does the answer address the question?
- Healthy answer
- All three checks passed.
- Correct refusal
- The documents don't cover it, and the model said so.
- Needs review
- The checks couldn't decide; a person should look.
Every verdict says who made it
Checks run in three tiers. The cheap ones run immediately; the LLM judge runs in the background when you give the server a provider key. The dashboard draws a judge's verdict solid and calls it Final, and draws an estimate hatched and calls it Preliminary, so you always know how much to trust what you're reading.
| Check | When it runs | Shown as |
|---|---|---|
| Term overlap | On ingest, in the request. No network calls. | Preliminary |
| Embedding similarity | In the background, when a provider key is set. | Preliminary |
| LLM judge | In the background, when a provider key is set. Overrules the estimates. | Final |
Without a judge configured, every diagnosis is Preliminary and the dashboard says so. When the judge disagrees with an estimate, the estimate stays visible, struck through.
When a stage fails, retry that stage
The optional RecoveryAgent wraps your own retrieve and generate functions. After a retrieval miss it retrieves more; after an unsupported claim it regenerates with a stricter prompt. Every attempt is traced and grouped into one run, so you can see what failed, what the agent changed, and how it ended. By default it only retries on the judge's verdicts, not on estimates.
from traceroai.recovery import RecoveryAgent
agent = RecoveryAgent(
client,
retrieve=retrieve, # (question, top_k) -> chunks
generate=generate, # (question, context) -> answer
max_attempts=3,
)
result = agent.run("Is money refunded within 10 days?")
result["succeeded"], result["attempts"], result["trace_ids"]Show that a change helps before it ships
Run the same labelled questions through each version of your pipeline, a different top_k, prompt or model, and let the judge grade every answer. The dashboard shows each variant's score and the questions where they disagree, which is where the change actually made a difference.
from traceroai.eval import Case, Variant, run_experiment
result = run_experiment(
client=client,
dataset=[Case("refunds", "How long does a refund take?", "5 to 7 business days")],
retrieve=retrieve,
generate=generate,
variants=[Variant("k2", "top_k=2", top_k=2), Variant("k5", "top_k=5", top_k=5)],
)
print(result.recommendation)How often the diagnosis is right
We measured each tier against 117 labelled traces. The numbers below are the whole result, not a selection; the full report lists every verdict and how to reproduce it.
| Tier | Right diagnosis |
|---|---|
| Term overlap | 58.1% (68 of 117) |
| Embedding similarity | 76.1% (89 of 117) |
| LLM judge | 94.0–95.7% over 3 runs |
The cases come from one small support-style knowledge base, and most of their labels follow from how they were constructed. Treat the figures as a comparison between tiers, not as accuracy on your production traffic.
What runs where
Everything runs on your machine or your servers: one Docker Compose file starts the API, the queue, the worker and the dashboard. Trace data leaves only if you configure an LLM provider for the judge, and then only to that provider.
- Your RAG app sends traces with the SDK to the API.
- The API runs quick checks, stores the trace in Postgres and queues it on a Redis stream.
- A worker runs embeddings and the LLM judge, calling your LLM provider only if you set a key, and writes verdicts back.
- The dashboard reads everything from the API.
Run it yourself
Start the stack, point the SDK at it, and send your first trace.
1. Start TraceroAI
git clone https://github.com/chinmai-sd-123/TraceroAI.git
cd TraceroAI
docker compose -f infra/docker-compose.yml up --build
# API on http://localhost:8000, dashboard on http://localhost:3000Add TRACEROAI_OPENAI_API_KEY to the environment to turn on the LLM judge.
2. Send a trace from your pipeline
from traceroai import TraceroClient
client = TraceroClient(base_url="http://localhost:8000")
with client.trace(question) as t:
chunks = retrieve(question, top_k=3)
t.log_retrieval(chunks, strategy="vector")
prompt = build_prompt(question, chunks)
t.log_prompt(prompt, version="grounded_v1")
answer = llm(prompt)
t.log_generation(answer, model="gpt-4o-mini")
# Each step is timed, and the trace is sent in the background.Read the documentation for the full SDK reference, recovery options and API keys.


