Skip to content
TraceroAI

Find the step that broke.

TraceroAI reads every run of your RAG pipeline, names the step that failed, and shows the evidence, including whether a judge or an estimate decided.

$ pip install traceroai
Read the quickstart

See how it decides

Case 5f7bf901gpt-4o-mini · 1.68 s

Question

3 chunks · refund-policy · 223 msShow
  1. [1]Refunds are processed within 5 to 7 business days after the request is approved. The approval review itself typically completes within 1 business day of submission.
  2. [2]Refunds are always issued to the original payment method used for the purchase.
  3. [3]Subscriptions can be refunded within 14 days of renewal.

Answer

  1. Retrieval

     

  2. Grounding

     

  3. Answer

     

 

A real run from the example app, judged by an LLM.

  1. Term overlapOverlap58%
  2. EmbeddingsEmbeddings76%
  3. LLM judgeJudge94%
The 117 labelled cases behind the accuracy figures, as each tier diagnosed them: red where it put the cause, amber where it couldn't decide. Hatched is an estimate, solid is the judge, and a ring is a wrong diagnosis. Point at a mark to see its case.

A trace isn't an answer

A trace tells you everything that happened. You still have to read it to find what went wrong. TraceroAI reads it and gives you three lines.

Show the trace these three lines came from (18 lines)

What the trace shows you18 lines

  1. claim_groundednessnot_grounded · 1 of 4 claims
  2. claim"Money is typically refunded within 10 days." unsupported
  3. diagnosisunsupported_claim · source judge

What TraceroAI tells you

Step
Grounding
Cause
Unsupported claim. “Money is typically refunded within 10 days” isn't in the retrieved context.
Decided by
The LLM judge, so it's final. The quick estimate had said “unclear”.

The run, taken apart

One real run from the example app, separated into what went in, what came out, and what the judge found. The red layer is where it went wrong.

  1. Question1 / 5

    is money within 10 days refunded?

    trace 5f7bf901 · project recovery-agent · dev

  2. Retrieved context2 / 5

    1. [1]Refunds are processed within 5 to 7 business days after the request is approved. The approval review itself typically completes within 1 business day of submission.
    2. [2]Refunds are always issued to the original payment method used for the purchase.
    3. [3]Subscriptions can be refunded within 14 days of renewal.
    4. vector · top_k 3 · 223 ms
  3. Answer3 / 5

    Yes, money is typically refunded within 10 days, as refunds are processed within 5 to 7 business days after approval, which usually occurs within 1 business day of submission.

    gpt-4o-mini · 410 tokens · 1,451 ms

  4. The judge's checks4 / 5

    • Money is typically refunded within 10 days.
    • Refunds are processed within 5 to 7 business days after the request is approved.
    • Approval review typically completes within 1 business day of submission.
    • Most customers receive their full refund within a week and a half at most.
  5. Verdict5 / 5

    Probable cause · Grounding

    Unsupported claim

    1 of 4 claims are not supported by the context.

    Final · LLM judgePreliminary · term overlap: unclear

Every verdict says who made it

Checks run in three tiers. The cheap ones run at once; the LLM judge runs in the background when you give the server a provider key. A judge's verdict is drawn solid and called Final. An estimate is drawn hatched and called Preliminary.

CheckWhen it runsShown as
Term overlapOn ingest, in the request. No network calls.Preliminary
Embedding similarityIn the background, when a provider key is set.Preliminary
LLM judgeIn the background, when a provider key is set. Overrules the estimates.Final

Without a judge configured, every diagnosis is Preliminary and the dashboard says so. When the judge disagrees with an estimate, the estimate stays visible, struck through.

How often it's right

We measured each tier against 117 labelled traces. These are the whole result, not a selection.1

Diagnosis accuracy by evaluation tier
TierRight diagnosis
Term overlap
58.1% (68 of 117)
Embedding similarity
76.1% (89 of 117)
LLM judge
94.0–95.7% over 3 runs
Precision and recall per label
Precision is how often a label is right when it's given; recall is how many of the cases with that label it finds. Judge figures are from the first of three runs.
LabelCasesTerm overlap P / REmbedding P / RJudge P / R
Healthy answer221.00 / 0.140.93 / 0.640.95 / 0.95
Correct refusal180.56 / 1.000.94 / 0.941.00 / 0.94
Retrieval miss200.53 / 1.001.00 / 0.851.00 / 0.95
Unsupported claim210.92 / 0.520.77 / 0.620.81 / 1.00
Wrong answer360.73 / 0.440.97 / 0.780.97 / 0.89

The cases come from one small support-style knowledge base. Treat the figures as a comparison between tiers, not as accuracy on your traffic.

1Full report, with every verdict and how to reproduce it: docs/evaluation-benchmark.md. ↩

What it can diagnose

Today TraceroAI diagnoses RAG pipelines: it checks three stages and names the first that failed. Agent runs come next, and they're marked as planned until they ship with measured accuracy.

RAG pipelines

Available
  1. 1Retrieval

    Retrieval miss

    The retrieved context doesn't cover the question.

  2. 2Grounding

    Unsupported claim

    The answer says something the context doesn't.

  3. 3Answer

    Wrong answer

    It answers a different question, or refuses one it could answer.

Healthy answer
All three checks passed.
Correct refusal
The context didn't have it, and the model said so.
Needs review
The checks couldn't decide; a person should look.

Agent runs

Planned
  • Wrong tool chosen
  • Arguments that break the tool's schema
  • A tool error the agent ignored
  • The same call repeated
  • Step limit reached

trace f82d0284

What is your adress and refund policy?

I don't know.

Wrong answer. It refused, but the context held the refund policy.

trace b59e4ce1

How long does a refund take?

A refund typically takes 5 to 7 business days to process after the request is approved…

Healthy answer. Retrieval, grounding and answer checks all passed.

From one failure to a fix you can prove

  1. 01

    Failures group into issues

    Five hundred failed runs are usually a dozen causes. Failures with the same cause become one issue, with a count, first and last seen, and a state: open, resolved, or regressed if it comes back.

    IssueGrouped byTracesState
    Needs reviewplans-and-limitstop source4Open
    Needs reviewrefund-policytop source4Open
    Retrieval missrefund-policytop source2Open
    Needs reviewregionstop source2Open
    Wrong answerrecovery_v1prompt version1Open
    Retrieval missplans-and-limitstop source1Open
    Figure 1. The bundled demo data: of 30 recorded runs, the 14 that failed become six issues. It was seeded without a judge, so these diagnoses are estimates (hatched).
  2. 02

    Retry the step that failed

    The optional RecoveryAgent wraps your own retrieve and generate functions. After a retrieval miss it retrieves more; after an unsupported claim it regenerates with a stricter prompt. Every attempt is traced. By default it only retries on the judge's verdicts.

    1. Original attemptbfe5cdc6

      Unsupported claim

      Finaltop_k 3 · 1.76 s

    2. The agent regenerated with a stricter prompt

      Attempt 25f7bf901

      Unsupported claimThe case at the top of this page

      Finaltop_k 3 · 1.68 s

    3. The agent regenerated with a stricter prompt

      Attempt 3

      Healthy answer

      Finaltop_k 3 · 1.94 s

    Figure 2. One recovery run, attempt by attempt. The third answer was healthy.
    pip install "traceroai[recovery]"
    from traceroai.recovery import RecoveryAgent
    
    agent = RecoveryAgent(
        client,
        retrieve=retrieve,    # (question, top_k) -> chunks
        generate=generate,    # (question, context) -> answer
        max_attempts=3,
    )
    result = agent.run("Is money refunded within 10 days?")
    result["succeeded"], result["trace_ids"]
  3. 03

    Show the change helps before it ships

    Run the same labelled questions through each version of your pipeline and let the judge grade every answer. You get each variant's score, tokens and cost, and the questions where they disagree.

    An experiment's question grid: ten questions graded for top_k 2, 3 and 5. The variants disagree on one question, which top_k 3 got wrong.
    Figure 3. Ten questions, three variants: they disagree on one, and it's listed first.
    run_experiment
    from traceroai.eval import Case, Variant, run_experiment
    
    result = run_experiment(
        client=client,
        dataset=[
            Case("refunds", "How long does a refund take?",
                 "5 to 7 business days"),
        ],
        retrieve=retrieve,
        generate=generate,
        variants=[
            Variant("k2", "top_k=2", top_k=2),
            Variant("k5", "top_k=5", top_k=5),
        ],
    )
    print(result.recommendation)

Works with what you run

Send traces from Python, over HTTP, or from any app already instrumented with OpenTelemetry. Routes still planned are marked that way until they ship.

RouteHowStatus
Python SDK, sync and asyncpip install traceroaiAvailable
HTTP API, any languagePOST /v1/tracesAvailable
Recovery agentpip install "traceroai[recovery]"Available
OpenTelemetry (OTLP/HTTP)POST /otlp/v1/tracesAvailable
LangChain and LangGraph, via OpenInferenceLangChainInstrumentorAvailable
LlamaIndexThrough OpenInference; not yet testedPlanned
TypeScript SDKnpmPlanned
from traceroai import TraceroClient

client = TraceroClient(base_url="http://localhost:8000")

with client.trace(question) as t:
    chunks = retrieve(question, top_k=3)
    t.log_retrieval(chunks, strategy="vector")
    prompt = build_prompt(question, chunks)
    t.log_prompt(prompt, version="grounded_v1")
    answer = llm(prompt)
    t.log_generation(answer, model="gpt-4o-mini")
# Each step is timed, and the trace is sent in the background.

What runs where

Everything runs on your machine or your servers. Postgres is the only thing it has to have; Redis and a separate worker are there for more traffic. Trace data leaves only if you configure an LLM provider for the judge, and then only to that provider.

  1. Your RAG app sends traces with the SDK to the API.
  2. The API runs quick checks, stores the trace in Postgres and queues it on a Redis stream.
  3. A worker runs embeddings and the LLM judge, calling your LLM provider only if you set a key, and writes verdicts back.
  4. The dashboard reads everything from the API.

Find the step that broke, on your own servers.

One command starts the API, the worker, Postgres and the dashboard. Add the demo profile and the dashboard opens with 30 recorded runs already in it.

Terminal

git clone https://github.com/chinmai-sd-123/TraceroAI.git && cd TraceroAI. docker compose -f infra/docker-compose.yml --profile demo up --build. # API on http://localhost:8000, dashboard on http://localhost:3000. # --profile demo adds 30 recorded runs to a project called demo

Add TRACEROAI_OPENAI_API_KEY to turn on the LLM judge.

Licence
Apache-2.0
Hosting
Yours. No hosted service.
Telemetry
None. Your copy sends nothing.