Confident and wrong
Some AI models do not write text. You give them a text and a list of possible answers, and they give back a probability for each answer. The promise is that you can rely on that probability: let the model act when it says 95 %, send the case to a person when it says 60 %. I built a small model of this kind for sorting security alerts and tested where that reliability comes from. The way the model was trained mattered far less than its makers suggest. What mattered was how much data it had seen and whether the input looked like that data. On logs it could not read, it was right 26 % of the time and still 52 % sure.
The claim
Many steps in an AI system are decisions, not writing: which team gets this support ticket, whether this message is an attack, what kind of alert this is. The usual way to make them is to ask a chatbot-style model and read its reply. That is slow and costly for what is really a multiple-choice question.
A decision model is built for the multiple-choice question. It takes a text and a list of options and returns one probability per option, in a single step. In September 2026 the company TypeSafe launched one called Jev, available only through its paid service. Shortly afterwards Nandakishor M released Laya, a free and open one, which collected about 29,500 stars on GitHub in two weeks. This post examines the open one in a toy world. I did a test of Jev on real agent failures in a second post, Can a decision model find where an agent failed?
Two words carry the rest of this post.
- Confidence is the probability the model gives to the answer it picked.
- A model is calibrated when its confidence matches how often it is right. Of all the answers it gives with 80 % confidence, about 80 % should be correct. A model that says 99 % and is right 80 % of the time is over-confident.
Calibration is what makes a rule like “act automatically above 90 %” safe. Without it, the number is decoration.
Laya's author says its calibration comes from the way it is trained. Training works like this: the model answers many examples, a formula gives each answer a score, and the model is adjusted to score higher. I will call that formula the reward. The standard reward is called cross-entropy. The author argues that cross-entropy makes a model over-confident, and that Laya avoids this with a different training method, called RLCD. Laya's documentation draws the practical conclusion: because of that training, its “confidence scores are statistically meaningful”.
That is a claim about a reward, and a reward can be tested on a small model.
Hypothesis
First, what makes a reward good for calibration. Suppose an answer is truly 70 % likely to be right, and the model has to report a probability.
- If the reward is one point for a right answer, the model does best by always claiming 100 %. Claiming more never costs anything.
- A better reward punishes a wrong answer more the surer the model was. With the right formula, the model's average reward is highest when it reports exactly 70 %. Statisticians call such a formula a proper scoring rule. Put simply, it is a reward that pays for honesty.
The middle panel, the log score, is cross-entropy under another name. Laya's reward is the log score plus half of the spherical score in the right panel. Both pay for honesty, so I doubted that the choice between them decides anything. My two predictions:
- A model trained with cross-entropy and a model trained with Laya's method end up equally calibrated. A model trained with one point per right answer ends up over-confident.
- Whichever reward is used, the model is calibrated only on input that resembles its training data. A reward can teach honesty about the examples it sees and says nothing about anything else.
If Laya's documentation is right, the first prediction should fail: Laya's method should give a clearly better-calibrated model than cross-entropy.
Setup
How a decision model works
A chatbot-style model writes its answer in small pieces, each a word or part of a word. Every piece needs one full run of the network, and each new piece is fed back in to produce the next. If you ask it how sure it is, that number is written text as well, and nothing ties it to how often the model is right. A decision model runs the network once and reads the probabilities directly.
Inside Laya the trick is in how the input is laid out. For each question it builds one sequence: the question, then every option with a placeholder in front of it, written [MASK], then the text to judge.
Reading the diagram from top to bottom:
[CLS]and[SEP]are plain markers: one for the start, the others for where one part ends and the next begins.- The network reads the question, the options and the text together, so each option is judged against the whole text.
- At each
[MASK]placeholder it produces one raw score for that option. A higher score means a better fit. - A standard formula called softmax turns the raw scores into probabilities that add up to 1.
- Before that, the scores are divided by one number, the temperature. It never changes which option wins. A higher temperature makes the probabilities more even, so the model sounds less sure; a lower one makes it sound more sure. It is chosen after training, and it comes back later in this post.
The size of a network is counted in parameters, the numbers that training adjusts. Laya has 421 million. My model has 76,353. It is the same design, small enough to train on a laptop in a couple of minutes.
A world where the right confidence is known
Calibration is normally checked by comparing confidence with accuracy on examples that have a known answer. I wanted something stricter: for every single case, the probability that an ideal model should report. To have that, I had to generate the data myself.
The setting is a security team. Monitoring software raises alerts, and someone has to decide what each one is. This sorting is called alert triage. In my toy version:
- An alert is a session: between 4 and 10 log events, such as
login_fail,port_probe,sql_errororlarge_upload. There are 30 kinds of event. - Each session belongs to one of 12 classes: normal activity, called benign, or one of 11 kinds of attack.
- Every class produces ordinary events most of the time. About 30 % of the time it produces an event typical of that class: failed logins for password guessing, probes for a port scan.
- Some attacks look alike on purpose. Brute force, guessing passwords, and credential stuffing, trying stolen passwords, both produce failed logins.
Because I wrote the program that generates the sessions, I can calculate the exact probability of each class for any session. A short, ambiguous session has an honest answer that is well below 100 %.
Each question offers the true class and a few wrong ones, in random order. With 4 options, even an ideal model is right only 84.7 % of the time, because the remaining sessions are truly ambiguous. That is the best possible accuracy in this world.
The sessions are invented. Nothing here comes from real logs. The security setting is there because it makes the stakes easy to see: a confident wrong answer is either an attack filed as harmless or an analyst's morning spent on nothing.
Three ways to train it
The same small model is trained with three rewards.
- Cross-entropy. The standard reward.
- One point if right. The reward from the left panel of the figure above, included as the one that should fail.
- Laya's method. RLCD, as its author describes it: make small random changes to the model's raw scores, see which changes earn more of the honesty-paying reward, and adjust the model in that direction.
Each is trained on 1,000, 4,000, 16,000 or 64,000 sessions, for the same number of training steps. Every run is repeated three times with different random starting points, and I report the average. That makes 36 training runs.
What is measured
Each trained model answers 5,000 new sessions, and I record three numbers:
- Accuracy: how often its answer is right.
- Confidence: how sure it says it is, on average.
- Calibration error: roughly, the average gap between the two. It is computed by grouping answers of similar confidence and comparing each group's confidence with its accuracy. Zero is perfect, and 0.20 means the confidence is off by about 20 percentage points. By chance alone, even an ideal model scores about 0.01 on 5,000 sessions, so 0.01 is the best achievable.
The models are tested in five situations:
- Like training: 4 options, the same kind of sessions.
- 8 options and 12 options: more than the model ever saw in training, where it had 2 to 6.
- Unreadable log format: the same sessions, but every event is written with a symbol the model has never seen, as if the logs came from an unfamiliar system. The model cannot read them, so the honest answer is 25 % for each of 4 options.
- 90 % benign traffic: in training, all 12 classes were equally common. Here nine sessions in ten are benign, which is closer to what a real queue looks like.
Experiment
On sessions like the training data
Calibration error with 4 options. Each cell is the average of 3 runs, with the spread between runs after the ±. Lower is better and 0.010 is the best achievable.
| training sessions | cross-entropy | Laya's method | one point if right |
|---|---|---|---|
| 1,000 | 0.198 ± 0.005 | 0.201 ± 0.003 | 0.209 ± 0.004 |
| 4,000 | 0.152 ± 0.015 | 0.108 ± 0.010 | 0.163 ± 0.002 |
| 16,000 | 0.016 ± 0.002 | 0.019 ± 0.004 | 0.153 ± 0.003 |
| 64,000 | 0.009 ± 0.002 | 0.011 ± 0.002 | 0.148 ± 0.003 |
Cross-entropy and Laya's method give the same result. With 16,000 sessions or more, both reach the best achievable value: their confidence is the true probability, give or take chance. With 1,000 sessions both are badly over-confident, 97 % sure and 77 % right. A small model shown the same few examples many times ends up memorising them instead of learning the pattern, and a model that has memorised its examples is sure of everything.
So the amount of data changes the calibration error twenty-fold. The choice between these two rewards changes nothing at three of the four sizes. At 4,000 sessions Laya's method is somewhat better, 0.108 against 0.152. My guess is that its random changes slow the memorising down a little. I did not test that, and 0.108 is still ten times the best achievable.
One point if right never becomes calibrated. That model is 99 % confident at every size, and its error stays at 0.15 however much data it sees. Its accuracy is as good as the others', 84.2 %. It knows the answer as well as they do and has learned to overclaim, as the figure in the hypothesis predicted.
The first prediction holds. A reward that pays for honesty is needed. Laya's method is one way to get it, and ordinary cross-entropy is another.
On sessions unlike the training data
This is the best case for every model, 64,000 training sessions. Each cell shows accuracy, then confidence. When the two numbers are close, the model is calibrated.
| situation | cross-entropy | Laya's method | one point if right | best possible accuracy |
|---|---|---|---|---|
| like training | 84 % / 85 % | 84 % / 85 % | 84 % / 99 % | 85 % |
| 8 options | 78 % / 76 % | 78 % / 77 % | 77 % / 99 % | 78 % |
| 12 options | 73 % / 71 % | 73 % / 70 % | 73 % / 98 % | 73 % |
| unreadable format | 26 % / 52 % | 27 % / 50 % | 26 % / 89 % | 25 % |
| 90 % benign | 86 % / 69 % | 86 % / 70 % | 86 % / 99 % | 98 % |
More options did no harm. I expected this to break calibration, and it did not. With 8 and 12 options, cross-entropy and Laya's method stay within three points of the truth. This design treats the options as an unordered list, so a longer list is nothing new to it.
Input it cannot read breaks it completely. On the unreadable format, accuracy drops to 26 %, which is what guessing gives, and confidence is 52 %. The honest answer was 25 %. The reward taught the model to be honest about sessions like the ones it saw. It has no way of noticing that these are different. The model trained with one point if right is 89 % confident on the same input.
A different mix of traffic breaks it the other way. When 90 % of sessions are benign, a model that knew this would be right 98 % of the time. These models are right 86 % of the time and too unsure, at 69 %. They learned from data where every class was equally common, so they keep suspecting attacks that are not there.
The second prediction holds, with one correction. Not every change matters. The ones that do are the ones that change what the evidence means.
What this does to an automatic rule
Take the rule “handle the session automatically when confidence is 90 % or more, otherwise pass it to a person”. The 90 % is the threshold. For the model trained with cross-entropy:
| situation | handled automatically | of those, right |
|---|---|---|
| like training, 64,000 training sessions | 62 % | 98.6 % |
| like training, 1,000 training sessions | 90 % | 81 % |
| unreadable format | 3 % | 40 % |
| 90 % benign | 9 % | 84 % |
The first row is the promise kept: an ideal model would handle 62 % at 98.7 %, and this one matches it. The second row is the same design and the same reward with less data. It handles nine sessions in ten on its own and gets one in five of them wrong. In the last row the rule fails quietly in the opposite direction. It passes 91 % of an almost entirely harmless queue to people, where an ideal model would have handled 95 % of it automatically.
Can it be repaired afterwards?
The usual repair uses the temperature from the diagram. Take some examples with known answers that the model was not trained on, try different temperatures, and keep the one that makes confidence match accuracy. It is one number, and it cannot change which option the model picks.
For each model and each situation I chose the temperature on 5,000 separate sessions from that same situation, then measured again on the test sessions.
| case | calibration error before | after | temperature chosen |
|---|---|---|---|
| cross-entropy, 1,000 training sessions, like training | 0.198 | 0.017 | 7.8 |
| one point if right, 64,000 training sessions, like training | 0.148 | 0.020 | 12.4 |
| cross-entropy, 64,000 training sessions, unreadable format | 0.259 | 0.031 | 12.6 |
| cross-entropy, 64,000 training sessions, 90 % benign | 0.195 | 0.061 | 0.6 |
One number repairs both over-confident models almost completely, including the one trained with the wrong reward. That model was ranking the options correctly all along and only its scale was off. On the unreadable format the repair turns the temperature up until every answer is close to 25 %, which is honest and of no use. For the benign-heavy traffic it helps least, and accuracy stays at 86 % where 98 % was possible. A temperature cannot tell the model that most traffic is harmless.
In every row, the repair needed examples with known answers from the very situation being repaired.
The same pattern in the real model
I did not run Laya itself. Its repository publishes its own test results, and they show the same pattern.
My unreadable-format test has a direct counterpart. Laya comes in an English version and a multilingual one, and the project tested both on the same 20-option question in 51 languages, 100 cases each.
On Khmer, the English version gets 0 of 100 right while being 71 % sure on average. This is why Laya ships with a language check that runs before the model and sends non-English text to the multilingual version. The safeguard had to sit outside the model, because the model's own confidence gave no warning.
The repair has a counterpart too. Choosing a new temperature on separate examples, across 49 test sets, takes the English version's average calibration error from 0.466 to 0.081.
A model whose error falls by four fifths when one number is changed was not made calibrated by its training. It behaves like my models trained on too little data: the ranking is sound and the scale is wrong.
One result differs. My model stayed calibrated with more options, and Laya's does not: a report in its repository shows a 0.85 threshold working with 3 options and failing with 20. The same report finds the cause in the temperature that Laya is released with for long option lists, 0.10, which makes every answer sound about ten times surer. So this is the repair step going wrong, not the design of the model.
It is to the project's credit that all of this is in its own repository, much of it under a heading called “Honest limits”.
Conclusion
The claim was that the training method is what makes a decision model's confidence trustworthy. In a world where the right confidence is known, the result was narrower. A reward that pays for honesty is needed, and without one the model overclaims however much data it sees. But Laya's method and plain cross-entropy were indistinguishable. What decided whether either was calibrated was how much data it had, and whether the input resembled that data.
So the confidence of a decision model is a raw signal. It ranks answers well. It becomes a probability you can act on only after it has been checked, and usually adjusted, on examples from the place where it will be used. That checking is the job of whoever puts the model to work.
If I were putting one in front of a security team, I would:
- Check that the input is something the model can read before it runs. Its confidence will not say so.
- Choose the temperature on examples with the same log sources, the same question and the same mix of harmless and malicious traffic as in real use.
- Pick the threshold by measuring how often the model is right above it, not by copying a number from the documentation.
- Tell the model how common attacks really are, or correct for it afterwards.
- Run it alongside the analysts first, recording what it would have done, and automate a small part that is easy to undo.
- Measure again whenever the log sources, the list of options or the mix of traffic changes.
Limitations
This is a toy. Sessions are a handful of independent events from a program I wrote, and my model is several thousand times smaller than Laya. The toy can show whether a reward produces calibration when everything else is kept the same. It cannot say how large these effects are in the real model reading real text.
My version of Laya's method follows the author's written description, because the code that trained the released model is not public. The fine-tuning code that is public uses different settings and, notably, adds ordinary cross-entropy to the reward. If anything that makes the two methods more alike than I made them.
All three rewards used the same training settings, chosen once and not tuned for each. With a fixed number of steps, the small training sets are seen many times over. Stopping training earlier would have reduced the over-confidence at 1,000 and 4,000 sessions, and it would have done so for every reward.
My unreadable format is harsher than a real unknown language, where some characters and word pieces are shared. Laya's numbers suggest the real case is no kinder: 0 % on Khmer at 71 % confidence.
The Laya figures are the project's own measurements, taken from its repository on 1 October 2026 at version 0.3.22. Some samples are small, 100 cases per language, and the project changes quickly. I have not run the real model, and that is the obvious next step.
Run it yourself
The quickest way is the notebook. Open it in Colab and it runs in your browser, with nothing to install. It shows one session with its exact probabilities, trains the model in the three ways in a short run of about a minute, and draws the two main charts from the saved results of the full run. All the code and the results are in the repository on GitHub.
To run the full experiment on your own machine:
The experiment needs PyTorch and NumPy and runs on an ordinary processor; the full set of runs took about 45 minutes on an M1 laptop. The figure script needs only Python.
pip install torch numpy
python alert_triage.py smoke # under a minute, checks the setup
python alert_triage.py run # all 36 runs -> results.json
python alert_triage.py example # the worked session -> example.json
python make_figs.py # the figuresThe code is here: alert_triage.py and make_figs.py, with the raw results. The two Laya result files that the figure script reads, the 51-language test and the benchmark results, are unchanged copies from its repository, which is published under the Apache 2.0 licence.
Two things worth changing: SIGNAL, which sets how often a session shows its class's typical events and therefore how ambiguous the world is; and the share of benign traffic in the benign90 situation, to see how quickly the automatic rule degrades.
Sources
- The Laya repository: its documentation including “Honest limits”, its benchmark report, its model and fine-tuning code, and its result files.
- Report #394 in that repository, on a confidence threshold that works with 3 options and fails with 20.
- The author's launch article, for the design, the argument against cross-entropy and the description of RLCD.