TraceroAI

Issues

Failing traces with the same cause, grouped, so a hundred failures become a few things to fix.

An issue groups failing traces that share a cause. The Issues page lists them, the ones that need attention first, and each links to its traces.

How traces are grouped#

The grouping depends on what failed, because different failures have different causes:

DiagnosisGrouped byWhy
Retrieval miss, low context relevance, needs reviewThe source of the top retrieved chunkThe problem is in what was retrieved
Unsupported claim, wrong answerThe prompt version (or template name)The problem is in how the answer was produced

Log a prompt version with t.log_prompt(..., version="grounded_v2") and grounding failures from one prompt become one issue. Without one, they share an issue named "no prompt version".

When the judge changes a trace's diagnosis, the trace moves to the right issue by itself. Healthy traces and correct refusals belong to no issue.

Open, resolved, regressed#

  • Open: failing traces, not marked resolved.
  • Resolved: you marked it resolved, and nothing has failed since.
  • Regressed: it failed again after you marked it resolved. Regressed issues are listed first.

Resolve an issue once its cause is fixed. If it comes back, you'll see it.

In the API#

Terminal
curl "http://localhost:8000/v1/issues?project_id=my-app"
curl "http://localhost:8000/v1/traces?project_id=my-app&issue_key=retrieval_miss%7Crefunds.md"

See HTTP API for resolving and reopening.