Evidence and confidence
An automated diagnosis is only useful if you can check it faster than you could have found it yourself. Evidence and confidence exist to make that check take about ten seconds.
What counts as evidence
Every diagnosis carries two to five evidence entries, and each one must be a verbatim quotation from something the agent actually read — a line from the failing step's log, or a file:line reference with the source text at that line.
E AttributeError: 'dict' object has no attribute 'expires_at'
tests/unit/test_auth.py:47: AttributeError
app/auth.py:18: return {'value': fresh.value, 'user': fresh.user, 'expires_at': fresh.expires_at}
1 failed, 402 passed in 12.41sRead those four lines in order and the argument is complete without the prose: the error, where it surfaced, the line that caused it, and the scope. That is what evidence is for. If you have to read the root-cause paragraph to understand why the evidence was chosen, the evidence is weak.
What does not count
- Paraphrase. “The test failed with an attribute error” is a summary, not a quotation. You cannot check it against the log by eye.
- Unattributed source. A code fragment without a path and line number cannot be located, so it cannot be verified.
- Restating the conclusion. Evidence that says the same thing as the root cause in different words is circular.
Check the evidence supports the specific claim
What confidence is claiming
Confidence is a 1–10 integer, and it is a statement about sourcing, not about how likely the answer is to be right in some general sense. The question it answers is: how much of this root cause is present in the quoted lines, and how much did the agent infer?
| Score | What it means and what to do |
|---|---|
| 8–10 | Nothing is inferred. Every part of the root cause appears in a quoted line. Read the evidence, confirm it says what the summary says, and act. |
| 5–7 | The mechanism is supported but at least one link is reasoning rather than quotation — typically the agent saw the effect and inferred the cause. Verify the inferred link yourself before changing code. |
| 1–4 | The agent is reasoning past its evidence and is telling you so. Treat this as a lead, not an answer. Usually means the log window was thin, the failure is non-deterministic, or the tool budget ran out. |
Rendering the score as ten discrete ticks rather than a percentage is deliberate. It is a coarse judgement, and a bar labelled “90%” would imply a precision that does not exist.
A low score is a feature
A system that always sounds certain is a system you eventually stop reading, because you cannot tell its good answers from its bad ones. TraceCI is prompted to score honestly and to reserve 8+ for cases where nothing was inferred, which means a 4 is real information: it tells you where to spend your own attention.
If low scores are the norm on your repository rather than the exception, that is usually a context problem, not a model problem — see best practices.
When you disagree with the evidence
Open the full record of the investigation. It contains the exact log window and diff summary the agent was given. Nine times out of ten a wrong diagnosis is not a reasoning failure but an input failure: the log window was anchored on a warning rather than the real error, or the baseline commit was further back than expected and the diff contained unrelated work. Both are visible immediately from the record and neither is guesswork.