Can a decision model find where an agent failed?

6 October 2026 · Jev on 257 human-labelled failed runs, against published results and a frontier model

See the results in Colab · Code on GitHub

When an AI agent fails a task, a person must read the run and find the step where it went wrong. That work is slow. Jev is a decision model: it does not write text, and it answers typed questions with probabilities for a fraction of a cent. I did a test of Jev on 257 failed agent runs that people annotated by hand. On runs that fit in its window, it finds the failing step approximately as often as a frontier model. It cannot name the kind of failure, but its confidence tells you which answers to trust.

This post is the practical half of a pair. Confident and wrong uses a toy world to examine when you can trust the confidence of a decision model. This post does a test of a real decision model on a real job.

The claim

Decision models are sold as a cheap layer for routing and classification. You give the model some text and a closed set of options, and it returns a choice with a probability, in milliseconds. The pitch is that this covers the many small decisions in a software system that do not need a text-generating model.

Reading a failed agent run is not a small decision. A run has dozens of steps, tool calls and tool outputs. To find the step where it went wrong, a reader must read the plan. The reader must compare each action with the policy and find the point of no return. This is the kind of task that usually goes to a large reasoning model, or to a person. The claim under test is that a decision model can do a useful part of it.

There is a practical reason for this test. A team with an agent in production has hundreds of failed runs a week and nobody to read them. A model that costs cents could point at the right step in a good share of runs, and say when it is unsure. Then the team gets triage for free.

Hypothesis

I expected four things.

• Jev finds the failing step more often than chance, but less often than a frontier model that reads the full run.
• Its confidence is informative: the answers it gives with high probability are right more often.
• It does worse on long runs and on multi-agent runs. Its own documentation says that accuracy falls with long states full of unrelated detail, and with questions that need several steps of reasoning.
• It does worse at naming the kind of failure than at pointing at the step. A category needs a judgement of why. A step needs only where.

Setup

The task

A failed run comes in. The reader gets the task, the policy if there is one, and every step in order. The reader answers three questions:

• Step: which step contains the decisive mistake?
• Agent: in a multi-agent system, which agent made it?
• Category: which kind of failure is it, from a fixed list?

An answer counts as correct when it is the same as the human annotation. For the step I also report “within 1 step”.

The human-labelled runs

In two public datasets, people marked the answers on failed runs:

• AgentRx (Microsoft, 2026): 29 runs of a tau-bench retail customer-service agent and 44 runs of Magentic-One, a multi-agent web system. Each run has the failing step, the responsible agent, and a category from a list of 9.
• Who&When (Zhang et al., 2025): 58 runs of Magentic-One and 126 runs of generated multi-agent teams. Each run has the failing step and the responsible agent.

That is 257 runs. The runs are short in the customer-service and generated-team sets, and long in the Magentic-One sets: the longest has approximately 300,000 characters. I did not use MAST, because an LLM judge made the labels of its 1,642 traces, not people.

Jev

Jev is a decision model from TypeSafe. A request has a state and a map of typed questions. A choice question returns the chosen option and a probability for each option. A noul question returns the probability of yes. The model reads at most 32,000 tokens per request.

I called it through its API, model jev-1.13.0. The list price is $0.042 per million input tokens. Everything in this post cost $0.44.

For each run, the state held the task, the policy if any, and the steps as a list of {step, who, content} objects. The step question was a choice over all step ids. The agent question was a choice over the agent names. The category question was a choice over the 9 categories. When a run did not fit, I cut every step to the same maximum length, the largest that fit. This affected 33 of the 257 runs.

The comparators

Baselines. Chance, and “always answer the most common step in this dataset”, which is a strong baseline when runs are short and failures come early.

Published results. The AgentRx paper (2026) reports GPT-5 and o3 as judges on its runs, and its own method. The Who&When paper (2025) reports GPT-4o with three methods. I quote their numbers with the caveats they need.

Claude. To get a like-for-like competitor, Claude (Fable 5.1) read 130 of the runs in full and answered the same questions, with the same category definitions. It did not see the human labels. It read the runs in batches, several per session.

The tuning protocol

I cannot train Jev's weights. I can change how I ask the questions. To keep that honest, I split the 257 runs in half by a hash of the run id. The tuning half has 127 runs and the test half has 130. I tried changes on the tuning half, chose by the mean accuracy across the four datasets, and scored the test half once. All Claude comparisons are on the test half.

Experiment

1. Jev, untuned, on all 257 runs

I wrote each question once and did not change it.

Four things stand out.

• Short runs go well, long runs do not. Jev named the right step in 94 of the 224 runs that fit in its window. It named the right step in 1 of the 33 runs that I had to cut.
• The category does not work. On the tau-bench runs, Jev matched the human category in 2 of 29. It answered “instruction/plan adherence failure” for 24 of them. Always answering the most common category would give 28 percent.
• The agent is mixed. On the generated teams, with many agents to choose from, Jev reached 61 percent. On Magentic-One it over-blamed the orchestrator, and on the Who&When set it did worse than always naming the most common agent (38 against 57 percent).
• The confidence is informative. When Jev gave its step answer a probability of 0.7 or more, it was right in 58 percent of those runs. Below 0.3, in 15 percent.

2. Published results from stronger models

The AgentRx numbers are from 2026: the paper's first version is from February 2026, and I read the version of August 2026. The Who&When numbers are from 2025: the paper is from April 2025, revised in June 2025. I ran Jev (version 1.13.0) in October 2026.

The Who&When paper (2025) reports GPT-4o with three methods, without the correct answer. The best method gets 16.6 percent on the step and 51.1 percent on the agent for the generated teams. For Magentic-One it gets 8.8 and 53.4 percent. Jev, untuned, gets 50 and 61 percent on the generated teams, and 21 and 38 percent on Magentic-One.

These comparisons are loose, for three reasons. The AgentRx paper's tau-bench set has 39 runs and the public set has 29, so they are not the same runs. The Who&When results are from 2025 and use GPT-4o, an older model than the GPT-5 and o3 in the AgentRx paper. And both datasets are public. It is possible that Jev saw them in training. I cannot check that.

3. Tuning

I tried four changes on the tuning half.

• Wording. Four versions of the step question, with and without a note of who acted at each step. None was reliably better than the first, so the first stayed.
• Windows. A scan of long runs in overlapping windows, with an “elsewhere” option, instead of a cut. On the 22 long runs in the tuning half it found 1 step instead of 0. It did not help.
• Agent from step. Instead of asking for the agent, read it off the step that Jev chose. This helped.
• Category by yes/no. One noul question per category, with the paper's own definitions, and a correction for each category's average yes-rate on the tuning half. Then take the highest. This helped.

On the 130 held-out runs, the step stayed at 54 correct of 130. The agent went from 59 to 66 correct of 109. The category went from 6 to 13 correct of 42. The tuning cost $0.22. Changing the questions has little left to give.

4. Jev against Claude on the held-out runs

On the step, the two results are almost the same. The difference is 6 runs in 130, which chance can produce at this sample size. Both were right on 43 runs, only Claude on 17, only Jev on 11, and neither on 59. They chose the same step in 64 runs. Claude is clearly better where reading matters: on long runs, on the multi-agent runs, and on the category. On the short generated-team runs, Jev was slightly ahead (56 against 52 percent).

Even the frontier model matches the human step in less than half of the runs. Part of that is real error. Part of it is probably disagreement on which step is the decisive one. Each run has one annotation, so I cannot separate the two.

Cost and speed are the other difference. Jev scored the 130 runs for approximately $0.05, in less than 1 second each. Claude used approximately 2.5 million tokens, in batches that took minutes.

5. A second use: patterns in 159 failed runs of GPT-5

The human-labelled sets measure whether Jev can point at a step. A second question is whether it can detect known failure patterns across many runs. For that I used a set without human labels: the 159 failed runs of GPT-5 on tau2-bench, airline and retail. They come from Sierra's own leaderboard run of August 2025: 656 runs, 4 trials per task. In that run, 73 of the 164 tasks had at least one failure.

Claude read the 159 runs first, with a written brief, and labelled each with what went wrong, whose fault it was, and the cheapest fix. I merged its 48 free-text names into 8 patterns. These are model labels in one pass, with no human check.

By that reading, 128 of the 159 failed runs were the benchmark's fault, not the agent's. The task expected an action that the policy forbids (49 runs). The simulated customer left its script (39). The script did not specify a detail that the grader checked (14). The expected action contradicted the data (13). The agent's own mistakes were 19 runs, with 12 more shared.

One independent check exists. After this run, Amazon published tau2-bench-verified and Sierra revised its own tasks. Of the 67 runs the reader marked as “task definition is wrong”, 60 are on a task that one of the two later changed. Of the 19 runs it marked as the agent's fault, 10 are. The reader did not see the corrections.

Then Jev scored each run against the 8 patterns, with the policy, the customer's script and the grader's expectation in the state. It chose the same pattern as the reader in 92 of 159 runs (58 percent; the most common pattern alone gives 31 percent). For the largest pattern, a yes/no question found 44 of the 49 runs, with 64 flags in total.

When Jev gave its answer a probability of 0.9 or more, it agreed with the reader in 9 of 9 runs. Below 0.5, it agreed in 24 of 63. For the question “whose fault”, Jev agreed with the reader in 130 of 159 runs. But “always answer benchmark” would agree in 128. That single verdict adds nothing. This part compares Jev with a model's labels, not with people; it shows the shape of a use, not a measured accuracy.

Conclusion

Three of the four expectations held, and one did not.

• Jev finds the failing step far above chance and the most-common-step baseline. On runs that fit in its window, it is approximately level with a frontier model: 54 against 60 correct of 130. I expected a larger gap.
• Its confidence is informative. The confident third of its answers is right approximately 3 times in 4. The least confident answers are almost always wrong.
• Long runs and multi-agent runs are its weak point. It found no step in 11 long held-out runs; Claude found 4. Windows did not fix this.
• It cannot name the kind of failure. After tuning it reaches 31 percent on 42 runs, level with a plain GPT-5 judge in the AgentRx paper and well below that paper's method.

So for this task Jev is a triage tool, not a judge. You run it over every failed run for cents. It says “look at step 17” with a probability. The confident answers are mostly right, and the rest go to a person or a stronger model. It does not tell you what kind of failure you have, and it does not handle runs longer than its window.

Limitations

The human-labelled sets are small. With 21 to 61 runs per dataset on the test half, a few answers move a percentage by several points. Each set has one annotation per run. Both datasets are public, so it is possible that both models saw them in training.

The comparison between Jev and Claude is not equal on input. Claude read the full runs. For Jev, I cut the long runs or scanned them in windows. Each model ran once, so there is no measure of run-to-run variation. Claude judged several runs per session, not one fresh call per run.

The published numbers come from different run sets, methods and models, and I quote them as context, not as a direct comparison.

The tuning was small: four wordings, one window scheme, two structural changes, chosen on 127 runs. A larger tuning set, or a fitted combiner on Jev's probabilities, could move the category result. It cannot move the long-run result, which is a limit of the context window.

The 159 tau2-bench runs have model labels, not human labels. The pattern names are mine. A person who reads the same runs would draw some lines elsewhere. The published sheet has a one-line reason and the decisive evidence for each run, so you can check each label against the run.

On prior work: the AgentRx and Who&When papers define and measure this task for text-generating models, and taxonomies of agent failures exist (MAST, TRAIL). I did not find a test of a decision model on these datasets. My search was limited.

Check it yourself

The quickest way is the notebook. Open it in Colab and it shows the saved results as tables and charts in your browser. It does not call Jev, so you do not need a key.

Jev's answers on the human-labelled runs are in ground_truth_scores.csv (untuned, all 257) and tuned_test_scores.csv (tuned, the 130 held-out). Claude's are in judge_scores.csv. The reference labels for the 159 tau2-bench runs are in reference_labels.csv. The scripts fetch the datasets:

python fetch_annotated.py      # 11 MB: AgentRx and Who&When
python jev_ground_truth.py     # Jev, untuned, on the 257 runs (needs a TypeSafe key)
python jev_tune.py             # tuning on one half, test on the other
python judge_score.py          # Claude's answers against the same labels
python fetch.py                # 27 MB: the tau2-bench traces and task versions
python jev_score.py            # Jev against the 8 patterns on the 159 runs
python figures.py              # the four figures

The code is in the repository on GitHub, with the exact questions in jev_ground_truth.py and patterns.json, and the brief that the reader used.

Sources

• The decision model: TypeSafe Jev documentation, including its page on known weak points.
• Human-labelled failures: AgentRx (Microsoft, February 2026, revised August 2026), data at github.com/microsoft/AgentRx; Who&When (Zhang et al., April 2025, revised June 2025), data at Hugging Face.
• The tau2-bench traces: sierra-research/tau2-bench and the leaderboard at taubench.com; the corrected tasks at amazon-agi/tau2-bench-verified.
• Taxonomies: Cemri et al., MAST (2025); Deshpande et al., TRAIL (2025).

Reading a system's failed runs, and building the tools that read them at scale, is usually the first thing I do with a client's agent.

Back to writing