TraceroAI

Estimates and judge verdicts

Diagnoses come from three tiers of checks. The dashboard marks each one Preliminary or Final, and they mean different things.

The three tiers#

TierWhen it runsNeedsMarked
Term overlapThe moment a trace arrives, no networkNothingPreliminary
Embedding similarityIn the backgroundA provider keyPreliminary
LLM judgeIn the backgroundA provider keyFinal

The quick check gives every trace a diagnosis straight away. With a provider key, embeddings and then the judge run in the background, and the judge's verdict replaces the estimate. Without a key, every diagnosis stays an estimate, and the dashboard says so.

Why the difference matters#

Estimates are often wrong in a particular way: term overlap rarely recognises a correct answer worded differently from its source. On the labelled set it gets 58% of diagnoses right; the judge gets 94–96%. See Accuracy.

So act on Final diagnoses. The recovery agent retries only on judge verdicts by default, for the same reason: retrying on an estimate often "fixes" an answer that was fine.

Judge status#

Each trace also records where the judge got to:

StatusMeaning
pendingWaiting for the judge
completeAll three checks done
partialSome checks failed to run; the diagnosis uses the ones that did
failedThe judge couldn't run; the estimate stands
skippedNo judge configured

When the worker is running (with Redis), a check that never completes, after a worker restart for example, is picked up again by a periodic sweep.

Bring your own model#

The judge works with any OpenAI-compatible endpoint: set TRACEROAI_JUDGE_BASE_URL and TRACEROAI_JUDGE_MODEL to use another provider or a model you host. See Configuration.