Limitations

Hard boundaries

• GitHub Actions only. There is no support for other CI providers. The log retrieval, job model and run history are all GitHub-specific.
• One repository at a time. A failure caused by a change in a different repository — a shared library, a reusable workflow, a container image built elsewhere — is visible only through its effects.
• One failure at a time. TraceCI diagnoses the first failing step. A run with two genuinely independent failures produces one diagnosis, not a list.
• Nothing is executed. It cannot run the test, reproduce the failure locally or bisect. Everything is inferred from logs, diffs and source.
• Read-only, permanently. No commits, comments, pull requests or re-runs. This is a design decision, not a missing feature.

Where accuracy degrades

Long-red branches

The baseline is the last successful run. A branch that has not been green for thirty commits gives the agent a thirty-commit diff, and the signal that would normally point at one small change is spread across everything. Accuracy falls roughly with the size of the diff.

Failures the log does not describe

If a job is killed by the runner — out of memory, out of disk, a hard timeout — the log frequently ends mid-sentence with no error at all. TraceCI will usually categorise this correctly as infrastructure, but it cannot tell you which test allocated the memory, because nothing in the log says.

Suppressed or restructured output

Pipelines that redirect test output to a file and print only a summary have removed the traceback. Custom reporters that reformat failures into a shape unlike anything in the training distribution have the same effect. The window anchors on what looks like the first real error, and if nothing looks like an error, the anchor is arbitrary.

Non-determinism

A single failed run cannot establish flakiness. TraceCI can often identify the shared fixture or ordering assumption that would explain it, but the claim “this is flaky” is not provable from one run and any diagnosis asserting it at high confidence is overreaching.

Very large files

File reads are clipped at a character budget, tighter still on free-tier models. A relevant definition several thousand lines into a large module may fall outside the excerpt. The clip is marked in the text the agent sees, so it knows it is working from a fragment, but it cannot always recover the missing part inside its tool budget.

Small models on subtle failures

The judgement that matters is deciding to open a source file rather than guessing from the traceback. Smaller models systematically under-call tools here and produce a fluent diagnosis of the symptom. See models and keys.

Things it is not designed to do

• Fix the build. The suggested patch is illustrative and minimal, not a production change.
• Review code. It looks at a failure, not at quality, security or design.
• Track failures over time. There is no aggregation, no flake-rate tracking and no trend analysis. Each investigation is independent.
• Replace reading the log. When you already know what broke, opening the log is faster.

Data boundaries

• The log window, diff summary and any source the agent reads are sent to the model provider you selected. If your CI logs contain secrets, those secrets are in the request.
• Checkpoints on the server contain the failure context and the diagnosis. They do not contain your key.
• Your investigation list, preferences and theme are stored in your browser only. There are no accounts and nothing is synced.