TraceroAI

Investigate failures

From a pile of failing traces to a fix you can show works: issues, the trace page, an experiment, and resolving.

1. Start from issues#

Open Issues in the dashboard. Each issue is a group of failing traces with one cause; regressed issues, ones that failed again after you resolved them, come first. The overview's Issues panel shows the five that most need attention.

Pick the issue with the most traces, or the one that regressed. Its link opens the trace list narrowed to its traces.

2. Read a few traces#

Open two or three of the issue's traces and read what the checks found (Read your first diagnosis). Prefer traces marked Final: the judge's reason names what went wrong, while an estimate can flag answers that are fine.

What to look for, by diagnosis:

DiagnosisLook atCommon fixes
Retrieval missThe chunks: was the right passage retrieved at all?Chunking, top_k, a better query rewrite, hybrid search, missing documents
Unsupported claimThe quoted claim against the chunksA prompt that keeps answers to the context; a smaller, more relevant context
Wrong answerThe question against the answerPrompt instructions, the model
Correct refusalWhether the documents should cover the questionAdd the missing documents, or accept the refusal

3. Change one thing and measure it#

Make one change, then compare old and new on labelled questions with an experiment. Keep some questions back from the ones you tuned on and measure on those; a change that only helps the questions you looked at isn't a fix. Experiments report accuracy, latency, tokens and cost per variant.

For a fix you want applied automatically at run time, the recovery agent retries a failed stage.

4. Resolve the issue#

Once the fix ships, mark the issue resolved. If the same failure comes back, the issue returns as regressed and goes to the top of the list.