Executive Summary

Key Numbers

Sources: ZDNet Korea, AI Times, and eDaily (October 26, 2026), plus SelectStar's published experiment repository selectstar-ai/EVAL.Davi. The z-values and the JevBench item conversions were computed by Pebblous from the published figures.

SelectStar was founded in 2018 as an AI trustworthiness evaluation company. It started by collecting and labeling AI training data, and has since widened into verifying whether finished models can be relied on. DAVI, unveiled on October 26, sits on that trajectory. Outlets introduce it slightly differently, mixing judgment-specialized AI, decision AI, and decision-making AI, but they are all describing one job. Instead of writing answers, it judges.

The company lists three uses. The first is gauging a question's difficulty before deciding which model to send it to. The idea is to stop wasting the most expensive model on easy questions, and the industry calls it model routing. The second is scoring whether another AI's answer is adequate; the third is predicting in advance whether an AI agent can finish the task assigned to it. SelectStar released a plugin implementing the first use on GitHub under Apache License 2.0, and said it will add DAVI to its own evaluation platform, Datumo Eval, during November.

What sits in the published repository selectstar-ai/EVAL.Davi is not only plugin code. A measurement harness that reproduces the experiment, the complete results table, and the limitations the company wrote down itself are all up there. The two lines the reporting carried were pulled from that table, and most of the numbers counted later in this article come from the lines that were not pulled.

What this release quantifies is the first use. The design is simple. The company prepared K-HLE, an in-house set of 174 hard Korean question-answering items, and repeated each item five times to make 870 trials. On one side DAVI chose one of Anthropic's Haiku, Sonnet, or Opus for each item; on the other, every item went to Opus, the largest and most expensive of the three. Same items, same number of repeats, different allocation. DAVI never sees the answer. It reads only the question, assigns a probability across five difficulty levels, and sends the item to Haiku if the expected value falls below 0.48, to Opus at 0.78 and above, and to Sonnet in between.

The result was announced as two lines. The side where DAVI picked the model got 619 of 870 right, or 71.1%, and the Opus-only side got 612, or 70.3%. DAVI's side also ran at about 28% less cost than Opus alone. Most of the coverage set those two lines next to each other and reported them as an achievement: accuracy up, cost down.

The results table in the repository, though, is not two lines but six. Three baselines that sent every item to a single model, and three runs of the same three-tier routing with only the judge swapped, are posted together. All six came from the same 870 trials.

Source: selectstar-ai/EVAL.Davi README. Costs are list-price estimates rather than billed amounts, and the cost of calling the judge is in none of the rows.

What the reporting carried is the fifth row and the sixth. Set the six side by side and one more reading appears. The row that handed the same three-tier routing to Jev produced 69.4% at $9.66. That is 15 fewer correct answers than DAVI's side, at $1.12 less. Against Opus alone it works out to 35.6% saved rather than 28%. Which one to choose is a decision that needs both numbers on the table, and the coverage offered one option.

Back to the two numbers the coverage did choose. They came from the same 870 trials, but they are made in different ways. The 28% is the ratio of two sums: every one of the 870 calls was priced, and all of them were added up. Items routed to Haiku count cheaply, items routed to Opus count dearly, and every trial contributes to the total without exception. Allocating one or two items differently does not move the ratio of the totals much.

The 71.1% and the 70.3% are not like that. Each trial is recorded as either right or wrong, and the gap between the two scores arises only from the difference in how many came out right. The phrase 0.8 percentage points is that count difference rendered as a percentage. Turn it back and it is 619 against 612, which is to say seven.

How easily seven can move is something you can check by hand. Suppose four items that DAVI got right and Opus got wrong had fallen the other way. The 619 becomes 615 and the 612 becomes 616. The order changes. Four flips across another 870 trials are not hard to imagine.

A statistical ruler returns the same answer. Put the difference between the two proportions through a standard two-proportion test and the z-value is 0.37. That means a gap this size would turn up with probability 0.71 if the two methods were equally capable, and it does not come close to 1.96, the threshold the 5% significance level demands. To separate 0.8 percentage points reliably under these conditions would take something like 50,000 trials per arm. The 870 here is about one-sixtieth of that.

It should be added that this calculation is tilted in DAVI's favor. The two-proportion test assumes the 870 trials are independent of one another. But the 870 were made by repeating 174 items five times each, so the five trials from one item are bound to resemble each other. Reflect the clustering and the effective sample falls below 870, and the z-value drops further below 0.37. Measuring this properly calls for a paired test on how the two methods diverged item by item. The repository contains code that would run that calculation, but K-HLE itself stays inside the company, so the table cannot be rebuilt from outside.

One line should be made explicit to head off a misreading. Not significant does not mean DAVI is worse than Opus, and it does not mean the two methods are the same. It means this one experiment is not enough grounds to say a difference exists. The difference may well be real, and a larger experiment may make it plain.

SelectStar did not hide the conditions either. The ZDNet report notes alongside the result that this was the company's own evaluation, that it came from a single domain of hard Korean question-answering, and that the cost saving does not include the cost of calling DAVI. That last condition puts an asterisk on the cost number as well. The 28% counts only what was paid to the three models; the share belonging to DAVI, which did the judging, has not been added yet.

The limitations section of the repository goes a step beyond the coverage. Three items are listed. First, the cut-offs of 0.48 and 0.78 that separate Haiku from Sonnet from Opus were fitted on these same 870 trials, which makes the table's numbers optimistic. Second, costs are list-price estimates rather than billed amounts and exclude the judge call, and they are excluded because DAVI's price has not been set. Third, what was measured is one language and one task type only, with no coding and no long-horizon agent work.

To strip out the first condition the company measured once more. There is a separate table where the cut-offs were chosen on half the items and the scores were taken on the other half with the budget held inside 80% of Opus alone. There DAVI gets 69.9%, Jev 66.9%, and Claude Haiku 5.5 63.4%. That is the side the company calls the fairer comparison. Yet this table has no Opus-only row, because it pits judges against judges. The 0.8 percentage points the coverage carried has no counterpart at all on the fairer side.

The round-by-round record points to the same place. The document states that DAVI matched or beat Opus in three of the five rounds, which means it trailed in the other two. Cost stayed between 70 and 75% of Opus in all five. Add the five together and DAVI is seven ahead, but look at the rounds one at a time and wins and losses alternate; the wobble this article counted by hand is already there in the record.

The same release holds one more scorecard besides the routing experiment. SelectStar said it put DAVI head to head with Jev, the judgment model from TypeSafe AI. eDaily's headline said outright that DAVI had beaten the global model Jev. Pebblous covered Jev once before, in September. That piece took a classification-only model, one that returns multiple-choice answers and probabilities instead of composing sentences, and split apart what its claimed several-hundred-fold price drop was measured against.

The comparison was announced as covering five areas including JevBench, Laya Bench, and Ko-Feedback Bench. Figures appeared in the reporting for three of them. On the 231 items of JevBench, which measures judgment ability, DAVI led 89.2% to 86.6%, a gap of 2.6 percentage points; on hard problems it led 77.5% to 73.0%, or 4.5 points; and on Korean feedback evaluation it led 72.5% to 66.8%, or 5.7 points.

One thing needs correcting first. The name makes it easy to read JevBench as Jev's own benchmark, and it is not. JevBench is a public benchmark run separately by an outfit called Benchmark Heaven, whose documentation states flatly that it is maintained independently of TypeSafe AI. Jev is one of several judgment models listed there. The framing of winning on the opponent's home ground does not hold.

Looked at as gaps alone, 2.6 to 5.7 percentage points is three to seven times wider than the routing experiment's 0.8, which makes this side easy to read as firmer evidence. Apply the ruler from the previous section, though, and the picture changes. The percentages have to be turned back into item counts, and for that a denominator is needed.

The reporting gives a denominator for JevBench alone. Apply 89.2% and 86.6% to 231 items and they land on 206 and 200 exactly. The identity of that 2.6-point gap is six items. That is one fewer than the seven from 870 trials. Fewer items does not make the evidence thinner, though. The denominator is a quarter the size, so those same six items carry more weight than seven out of 870, which is why the z-value rises from 0.37 to 0.86. Raised, it still falls short of 1.96. Separating 2.6 percentage points on 231 items would take close to 2,500 items per arm.

For the hard-problem figure, a denominator follows once you open the benchmark maintainer's documentation. JevBench's 231 public items split into 72 original, 48 easy, and 111 hard. Apply 77.5% and 73.0% to the 111-item hard set and they land on 86 and 81, accurate to the decimal. No other denominator fits this cleanly. If the release's hard problems are that set of 111, then the most convincing-looking gap, 4.5 percentage points, comes to five items, with a z-value of 0.78. Thinner than the six-item gap. The remaining area, Korean feedback evaluation, stops here because its item count turns up nowhere. The possibility that the widest gap, 5.7 points, rests on thicker evidence stays open.

The number 231 carries a condition of its own. JevBench has 534 items in total, of which 303 are not published. Some were withheld to prevent the whole benchmark from being trained on, and some are items sourced elsewhere, for which there is no right to redistribute the text. The 231 is the public portion anyone can download. The benchmark's documentation states that items and scoring change together with each version, so scores from different versions should not be compared as if they were the same experiment. Which version's 231 items were used appears neither in the release nor in the reporting.

Inside the same release sits one more line, thicker than any gap measured so far. A router is useful only if it keeps sending the same task to the same model. Counting the items whose destination changed across the five repeats gives DAVI 2 out of 174, Jev 14, and Claude Haiku 5.5 40. Put the gap between 2 and 14 on the ruler used above and the z-value is 3.07; because the counts are small, a Fisher exact test gives p = 0.003. This is the only number in the release that clears 1.96. The claim with the thickest evidence behind it is not that DAVI was more accurate but that its judgment held steady, and no outlet carried that row.

What this article has pointed at is not a fault of SelectStar's. The company disclosed its item count and repeat count, and it wrote down that this was its own evaluation and what the cost calculation leaves out. That the cut-offs were fitted on the same data and the numbers are therefore optimistic, and that Opus was matched in only three of five rounds, are lines the company wrote itself. Few releases write down that much. Because the denominators 231 and 870 were there and the results table was open in full, converting the gaps back into counts was possible from outside. The problem lies not with the release but with a way of reading that lifts two lines out of a six-row table, erases the denominator, and passes on the percentages alone.

That is where this case stops being somebody else's problem for anyone working on data quality. Occasions for producing and reporting evaluation numbers are everywhere inside a company too. We switched models and accuracy rose a point. We revised the labeling guidelines and agreement improved. Option A beats option B. When a report like that lands on your desk, four questions are worth asking.

• • Is the denominator written next to the percentage? Without how many out of how many, there is no way to gauge the size of the gap. A table of ratios alone looks precise and cannot be checked
• • How many does that gap come to as a count? Here 0.8 percentage points was seven answers, and the far wider-looking 4.5 points was five items. The conversion takes one piece of mental arithmetic, and few people do it
• • Was the sample inflated by repeated measurement? The 870 trials from running 174 items five times are not 870 independent pieces of evidence. Calling an evaluation set repeatedly to grow n makes the number look steadier without growing the evidence to match
• • Who built the evaluation set used for the comparison, and which version is it? It is easy to go by the name and assume it belongs to the rival, and JevBench showed that assumption failing. Benchmarks change items and scoring together across versions, and their maintainers write that versions should not be mixed in a comparison. Recording the name, the version, and the item count together leaves somewhere to return to when a score later shifts

A fifth question can be added. Is what you are measuring really this technology's strength? With DAVI the accuracy gap wobbled, while the record of judgment holding steady was solid. Given what a router is asked to be, the second is the more important score, yet both the release and the coverage led with the first. Change the axis you measure on and the thickness of the evidence changes with it.

AI-Ready Data usually brings training data to mind, but evaluation data is held to the same demand, because decisions to pick a model and approve a deployment rest on evaluation numbers. If there is no record of how many items an evaluation set holds, how they were drawn, and how many times they were repeated, then the decisions made on top of it become decisions that cannot be verified. It is also worth noting that the market DAVI is aiming at is itself this problem turned into a product. Handing judgment to a machine requires someone to grade that judgment first.

Thank you for reading this far. The facts here were checked against three news reports dated October 26 and the experiment repository SelectStar published, and JevBench's item composition against the documentation of the group that maintains the benchmark. The count conversions and test statistics were computed directly from the published percentages and item counts. If SelectStar opens K-HLE and the per-item right-and-wrong record, or discloses the item count for Korean feedback evaluation, we intend to measure again with a paired test. If you report evaluation numbers inside your own company, we would be glad to hear how you record denominators and repeat counts.

Pebblous Data Communication Team
October 27, 2026

News Coverage

• 1.ZDNet Korea. (2026). "SelectStar unveils judgment-specialized AI 'DAVI'." October 26, 2026.
• 2.AI Times. (2026). "SelectStar unveils judgment AI 'DAVI'... compares against Jev across five areas." October 26, 2026.
• 3.eDaily. (2026). "SelectStar's DAVI beats global model 'Jev'." October 26, 2026.

Primary Sources

• 4.SelectStar. (2026). "EVAL.Davi — DAVI model-routing experiment repository." GitHub, Apache-2.0. Source for the full six-row results table, the company's own overfitting disclosure, the split-half validation table, and the per-round consistency figures.
• 5.Benchmark Heaven. "JevBench." GitHub. A public benchmark maintained independently of TypeSafe AI — source for the 231-item public split (72 original, 48 easy, 111 hard).