Executive Summary

Key Figures

The first three are records from five months on live websites. The last one is the size of the material that produced them.

Source: Benjamin Tannenbaum, "Selling in the Wild", arXiv:2610.25461 (submitted October 18, 2026)

Javo is a sales agent that sits in the chat window on a website, answers visitor questions, takes down the details it needs, and carries the exchange through to an order or a booking. It is built by AISO Boost, an Israeli company, and sold at getjavo.com. The model underneath it was not scaled up to compete on size; it was fine-tuned on 4.6 million commercial conversations. What this paper measures is how much an agent built that way actually sells on the screens where selling happens.

The evaluation comes in four parts. First, Javo was scored on two public benchmarks against frontier models and ten open-weight models. The heart of it is what comes next. The field trial ran from May 1 to September 30, 2026, with 100 live websites across 14 industries taking part, among them e-commerce, travel and lodging, and high-consideration services. The 14.8 million visitors who came to those sites were randomly assigned across the arms: one with no chat agent at all, one with Javo, and one that kept the same interface, the same product-lookup tools and the same guardrails while swapping the model for a frontier model. Because everything but the model was held the same, the difference between the two agent arms reads as a difference the model made.

Results were reported in two directions. Against the arm with no agent, revenue per visitor was 10.0% higher, with a 95% confidence interval from +5.1 to +14.9. Against the frontier-model agent on the same interface, conversion among engaged visitors was 21.6% higher, with an interval from +16.9 to +26.3. An engaged visitor is someone who sent at least one message into the chat window, and the denominator of that conversion rate is those people, not every visitor. The frontier-model comparison was pinned to April 24, 2026. Under a rule set in advance, the model picked was the one available through an API that day with the highest sales-benchmark score, GPT-5.2.

The comparison with human reps was carved out separately. On the 20 sites that already kept staffed chat reps, visitors were split four ways: 10% were left with no agent, and the rest went to human chat, Javo and the frontier agent at 30% each. Conversion among engaged visitors was 11.26% for Javo and 11.20% for the human reps. Handling one conversation cost Javo 1.8% of what it cost a human. Return rates, the rate of unsupported product claims and complaint rates all stayed inside safety thresholds set in advance, the paper reports.

Those safety numbers are printed in full. Returns per order ran 4.0% on the Javo arm against 4.1% on the arm with no agent, cancellations 11.9% against 12.2%, and complaints 0.18 per 1,000 visitors against 0.20. Among audited conversations, the share carrying an unsupported product claim was 0.15% for Javo and 0.56% for the frontier model on the same interface. Discounts or terms offered unprompted and outside site policy numbered zero for Javo and three for the frontier model. The raw cost figures sit in the same table: $0.06 per engaged conversation for Javo, $0.14 for the frontier model, and $3.40 for a human rep. The 1.8% above is 0.06 against 3.40.

What produced this record is not a parameter count. The difference the paper puts forward is the material. Where a general-purpose model reads internet documents and reasons its way toward a sale, Javo read conversations in which something was actually sold and learned the next line from them. Those 4.6 million conversations are records of someone asking about a product on some website, weighing a price, hesitating over a payment. Looking at where that material came from before judging the numbers is the right order.

How the material was used is laid out in a table. The base is Qwen3-32B, an open-weight model. On top of it, 1.65 million examples drawn from the 4.6 million conversations were run through supervised fine-tuning for two epochs, followed by 190,000 preference pairs that rank a seller's line leading to a purchase, a booking or a lead above one that did not. Last came 34,000 constraint examples, teaching the model not to offer discounts outside site policy, not to make claims absent from the catalog, to disclose that it is an AI when asked, and to hand off to a person. Training took 2,240 hours on 32 H100s, and the version put into the field is javo-v1, frozen on April 24, 2026.

The claim that the material made the record gets tested once inside the paper itself. The author put the untouched base model and the intermediate version on the same benchmark. On SalesLLM, which scores a sales conversation out of 10 over several turns against a model playing the buyer, the base model alone came in at 5.73. Supervised fine-tuning alone brought it to 6.51, and the finished version with preference learning reached 7.01. The best frontier model in the same table scored 6.21. The paper calls this contrast the key control: a gain over frontier models could come from the base model, while a gain over its own base model can only come from training. The 1.28 points from 5.73 to 7.01 are what reading those conversations added. On SmartSalesBench, which covers smartphone purchases, the order holds: 0.57 for the base model, 0.70 after supervised fine-tuning, 0.79 for the finished version, and 0.69 for the best frontier model.

The same table carries a row showing a loss. The version trained without the constraint data scored 7.13 on selling, higher than the finished version. In exchange, on τ-bench retail, carried in that same table and covering retail support where a model has to use tools while holding to set rules, the score dropped from 73.0 to 58.0. It reads as teaching the rules costing a little on selling and gaining a lot on following them, though the paper does not comment on the contrast.

What the paper writes about where the material came from is one line. It was drawn from a governed collection of commercial conversations assembled by Aiso, gathered between February 2024 and March 2026. Which companies' websites those visitors were on does not appear. Whether Aiso is a collection operation inside AISO Boost or a separate supplier cannot be pinned down from what is public either.

The shape of the pile is published, at least. The 4.6 million conversations hold 58.4 million turns, with a median of 10 turns per conversation. The languages run 68% English, 16% French and 8% Spanish, with every remaining language together making up 8%. By industry it splits 46% retail, 28% travel and 26% services. And the conversations where it is known whether a purchase followed, the ones carrying an outcome label, are only 17.4% of the set. The industry mix is knowable; which sites sit behind that mix still is not.

The de-identification side is written out concretely. Personal information was stripped automatically, identifiers were hashed, and people reviewed the results by hand. An audit of 2,500 sampled conversations found personal information remaining in 0.16% of them. That is a passage written so a reader can check it against a number.

Consent is a different matter. For the training material, what the paper writes is that it was restricted to conversations collected with user consent and de-identified before use. On which screen, in what wording, that consent was taken does not appear. The field trial side is fuller: site owners gave written consent to randomization and data sharing under a data processing agreement, visitors were covered by each site's privacy notice and consent banner, and the reps who took part in Study 4 consented to the use of de-identified transcripts. Consent is stacked in three layers here, and only the bottom layer, the visitor, may have passed a one-line banner without registering that their conversation would become material for training one company's sales model.

A companion white paper the same author posted ten days earlier covers a corpus audit. It tallies a table of 725,219 conversations supplied by Aiso, of which 41,800 (5.76%) were classified as commercial and 2,387 (0.33%) as transactional. The white paper then narrows its own scope: it describes itself as a metadata audit and a technical design, not a validation of label accuracy, model training volume or sales conversion, and it releases neither the source conversations nor personally identifying information. The audited table holds roughly 725,000 conversations while the training set is said to hold 4.6 million, so the published audit does not cover the training material itself.

Printed to two decimal places, 11.26% against 11.20% reads as though Javo came out ahead. That is not the paper's verdict. This comparison was designed as a non-inferiority trial, a design that shows only that the new arrival is not worse than the incumbent by more than a line drawn in advance, and here that line was a relative difference of −7%. The one-sided p-value came in at 0.004, so the line was not crossed, but the paper states outright that superiority was not statistically significant. The right way to read 0.06 percentage points is as the same.

One more condition is tilted to one side. The Javo widget tells the visitor it is an AI before the conversation starts, while the human chat window puts up a rep's name as usual. Conversations held by someone who knew they were talking to a machine and conversations held by someone who thought they were talking to a person were set in the same place and compared on conversion. The author flags that mismatch too. The two windows differed in more than who was answering.

The character of the human sample is disclosed too. The comparison group is whichever staff each site already employed, not people brought in for being good at selling. That goes down as a limitation in the paper: typical humans, not the best ones. The cost figure of 1.8% under those same conditions stands as large as ever. What the comparison tells us, though, is not that AI sells better than people. It is that in a spot where a staffed rep used to sit, an option with a different cost structure now exists.

There is some record of a match against people who are good at it. The offline benchmarks carry scores for human salespeople as well. Scoring sales conversations out of 10, salespeople with one to three years of experience came in at 6.13 and those with more than five years at 6.88, against 7.01 for Javo. Those scores came against a model playing the buyer, not a human buyer. The author grants the point and says field evidence against expert salespeople remains an open question.

The paper sets down what went unmeasured as well. Five months covers purchases, returns and cancellations, but not customer lifetime value, repeat purchase or whatever is left with the brand. A record taken over a short window and one taken over a long one can differ. On lodging and rental sites, where rooms and vehicles are finite, a booking in one arm takes supply away from another. The assumption behind randomization that arms do not affect each other is incomplete on those sites, and the author writes that a sell-out sensitivity analysis narrowed the range without removing the problem.

The paper has one author. That author is Benjamin Tannenbaum, founder and CEO of AISO Boost, which operates Javo. The conflict-of-interest statement in the paper reads: "Benjamin Tannenbaum is Founder/CEO of AISO Boost Ltd., which operates Javo (getjavo.com). The author is company-affiliated. No independent statistical review, study registration or external ethics approval is claimed." The chain that runs from designing the evaluation to running it, analyzing it and reporting it closes inside one person.

How the sample was gathered sits on the same line. The limitations say the 100 sites consented to the study and that many came through the author's client and agency network, so they are not a random sample of the web. Randomization split visitors within each site; the sites themselves were not drawn at random. If those sites were already using this company's tools, the result is the record of that environment.

Not everything here is closed. Twenty-one sites agreed to be named, and that list is published as a file in the appendix. Confidence intervals and the pre-specified margin are in the text, safety metrics were set in advance and reported, and the paper states that conversations gathered in the field trial were not used to train any model during the study. It concedes that the frontier-model comparison is pinned to a single version from April 24, 2026, and says the evaluation harness has been released so that comparison can be rerun against later models. This is not the shape of a paper trying to hide.

So the question is not whether the numbers are true or false. It is how far they were verified before they went out into the world. No third party has measured the same result again, and there is no record of a hypothesis and analysis plan registered in advance. This is why drug trials require pre-registration and outside review for a non-inferiority finding. Drawing the line after seeing the result is easy. This paper went out having stated for itself that it does not meet those requirements, and the reader has to read it with that condition attached.

What held our attention in this paper is not the conversion rate but the character of that number, 4.6 million. If the same interface and the same tools, with only the model changed, moved engaged conversion by 21.6%, what made the difference is not the design but the material the model read. The proposition that conversations in which something was actually sold become an asset has now been propped up once by a field measurement. And that asset is already piling up on some company's servers.

Any organization holding customer support records finds the question turning back on itself. Are our support logs the same material? How far does the right to train a model on them belong to us? If you run your chatbot or support tooling as an outside service, the data clauses in that contract have probably already settled the answer. Three things are worth checking. Whether the consent taken from customers reaches service delivery only or model training as well; whether you can obtain the level of de-identification applied and the results of a residual-PII audit; and whether those conversations remain material for the other side's retraining after the contract ends.

The last item is one this paper also leaves open. It states that conversations gathered during the field trial were not used in training during the study, and says nothing about what happens to them afterward. From the side handing data over, a promise written with a window on it and a promise without one are entirely different documents.

For anyone working in Korean, one more condition attaches. English makes up 68% of the conversations used in training, and every language other than French and Spanish adds up to 8% together. The author writes that results may not carry over to languages underrepresented in the corpus, giving as the reason that consistency across languages is weak in selling. Whether the same record appears on a Korean-language site is not a question this paper answers. Turned around, that gap is an open seat for anyone who has been stacking up conversations in which something was sold in Korean.

Pebblous works on getting data into a state AI can use directly, and the thing we watch alongside that is always provenance and rights. Data with clean labels and matched formats still becomes a risk the moment it goes into training if there is no record of where it came from and under what consent. The reverse holds as well: a conversation log whose source and consent scope are on the record is material that changes performance without changing the model. What Javo's report card proves is not the size of a model but the character of the material.

Thanks for reading this far. The figures and quotations here were checked directly against the paper and the companion white paper as published on arXiv, and where something cannot be confirmed outside the paper we have said so. We would be glad to hear what standard you are using to settle the question of putting customer support records into model training.

Pebblous Data Communication Team
October 21, 2026

• 1.Tannenbaum, B. (2026). "Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations." arXiv:2610.25461.
• 2.Tannenbaum, B. (2026). "Commercial Intent in Human–AI Conversations: A Corpus Audit and Architecture for Website Sales Agents." arXiv:2610.06205.