Executive Summary

Key Numbers

Source: Rezaeimanesh et al., the INCONVENIENT paper (arXiv:2610.21038, October 15, 2026).

The full name is "INCONVENIENT: A Benchmark to Assess LLM Generalization Capabilities from Convenience Samples." Sara Rezaeimanesh and Mohammad Ghassemi of Michigan State University wrote it with Chad Klochko and Kundan Thind of Henry Ford Health, and it went up on arXiv on October 15. The header on page one says it is published as a conference paper at ICLR 2027.

The data the benchmark works on is tabular. One row is one person or one event, one column is one variable, and into that table go the records that decisions rest on directly, such as a hospital's clinical notes or a company's underwriting history. Two of the four authors are based at Henry Ford Health, a hospital system.

The setup the benchmark builds is simple. It treats data as two things, a population and a convenience sample. The population is the whole world the model will actually work in, and the convenience sample is the share of that world that arrived systematically tilted to one side. Training happens on the convenience sample, and evaluation always happens on the original population. That is the order of events in the field. You build a model out of the data you can collect, and then you apply the model to everyone.

Earlier distribution-shift benchmarks do not use this setup. Collections like WILDS and TableShift split data along naturally occurring domains such as hospital, region or time period, then train on one side and evaluate on the other. When the score drops, the cause then forks in two. You cannot separate a failure to adapt to an unseen domain from a failure to recover the population's relationships out of a skewed sample. INCONVENIENT pins the evaluation side to the population and twists only the training sample, which fixes the cause of the drop on the collection method alone.

The LLM in the title follows from the same setup. A language model carries population-level common sense from pretraining, such as the fact that smoking raises the risk of lung cancer. Even when the training data blurs or inverts that relationship, the model has room to flag the oddity using knowledge it brought from outside. Whether that room amounts to anything in practice is measured in an experiment near the end of the paper, which section 4 takes up.

The raw material is 200 tabular datasets: 80 regression, 39 binary classification and 81 multiclass, pulled from OpenML, Kaggle and TALENT. Observation counts run from the hundreds to the tens of thousands, and feature counts from under ten to over a hundred. Each dataset is cut in half into a training pool and an evaluation set, and a reference linear model is fit on the whole training pool, logistic regression for classification and least squares for regression. The coefficients that come out become the baseline: in the population, the features and the target stand in this relationship.

Skew is then defined as pushing that baseline by a target amount. When collection tilts to one side, the coefficients fit on that sample drift away from the population's coefficients. Fix the direction and size of that drift first, then find the sample that matches it, and you get training data whose degree of tilt is written as a number. So one feature is picked and its coefficient is moved in three directions: amplification, which raises the magnitude by 1.15× up to 5.0×, attenuation, which shrinks it to 0.2× down to 0.9×, and a sign flip that reverses it outright. The example the paper gives is a head and neck cancer dataset in which the relationship between smoking and risk comes out inverted.

Here the benchmark parts ways with the alternatives. No numbers are edited and no noise is injected to hit the target coefficient. Every sample in the training pool is scored for how far including it would push the coefficient toward the target, and candidate sets are then widened in that order until a subset satisfies the condition. What comes out is a sample containing only observations that genuinely exist. It is not a hypothetical built out of forged data, but the same shape of sample a real collection process produces when it screens certain people in and others out.

Not every variant built this way gets used. A variant whose performance differs from unskewed training by less than 0.1 is too weak and is dropped, and extremes where the difference exceeds 1 or falls below −1 are dropped too. Variants already subsumed by a stronger one, which add no new information, are deleted as well. What survives the filter is 6,489 classification variants and 8,970 regression variants, about 15,500 in all.

Three difficulty dials were prepared, and the authors state themselves that one of them is not usable. The smaller the minimum subset size, and the more important the feature being aimed at, the more performance falls, and it falls monotonically. Those two can be trusted. The amplification and attenuation multipliers cannot. In regression, larger amplification multipliers tend to bring larger drops while attenuation never settles into a pattern, and in classification even amplification shows no trend. Raising the multiplier drags along coefficients that were never the target, and the next section puts a number on how far they move.

On the test bench are seven standard models of the kind commonly used on tabular data. The sentence the paper's conclusion uses to sum up the result reads this way. Across 200 datasets and roughly 15,500 variants, the seven baselines lose between 20% and 45% of their performance on average.

Split by task, the band changes. The point of comparison is the score the same model reaches when it trains and is evaluated on the original distribution, which the paper calls the in-distribution score. In classification, AUC comes in 0.18 to 0.24 below that score, a relative drop of 20% to 29%. Regression is worse, with R² drops of 0.35 to 0.45 and relative drops reaching 53% to 82%. The 20% to 45% in the conclusion averages all seven models together, so regression taken on its own lands above that band.

One result in the ranking cuts against intuition. Ridge regression and the multilayer neural network, which sat at the low end on in-distribution score, took the largest drops. On unskewed data their R² was 0.524 and 0.472, and they fell by 0.425 and 0.386 respectively, landing under 0.1. The expectation that a model with low performance to begin with has less to lose gets it backwards. The authors explain it by saying that the more heavily regularized a model is, the more faithfully it copies down the relationships written in the sample, and if those relationships are skewed, it learns the skew along with them. The validation split comes out of the same sample, so nothing inside the model registers as abnormal.

The manipulation also does not stay where it was aimed, and that was measured. Pushing one feature's coefficient to a target moves other coefficients that nobody touched. The median relative change was 26% under amplification, 61% under attenuation and 55% under the sign flip in regression, and 104% to 109% in classification. Distortion travels along the correlations between features. That is where the multiplier dial in the previous section lost its grip.

Whether existing methods can repair the situation is settled in an appendix first. Domain generalization techniques such as IRM, DANN and VREx work by finding common structure across several training domains, so they have no foothold in a convenience sample, which has only one domain. Importance reweighting works when sample selection depends on the input features alone, but once selection depends on the target or on the relationship between features and target, identification itself becomes impossible. The situation INCONVENIENT manufactures is the latter. That is where the benchmark's question turns inconvenient.

That leaves one thread: knowledge from outside the data. The authors gave Claude Haiku 4.5 nothing but the original codebook, showed it the list of coefficients fit on the skewed sample, and asked it to name the suspect features. A pick was scored as correct if the feature it named fell inside the top five that had actually moved the most. Across 300 variants, 100 sampled per manipulation type, the overall hit rate was 37.4%. That is 2.2× the 16.8% hit rate of random guessing.

By type, amplification was caught best at 42.7%, followed by the sign flip at 36.7% and attenuation at 32.7%. Relationships that have been exaggerated stand out, and relationships that have been blurred are harder to recognize. Still, the overall picture should not be inflated. The 2.2× gap is meaningful, but the hit rate is still under four in ten, and the number came out of a setting where the model was handed a codebook, which is a description of the domain.

Go out on a data quality engagement and you generally look at the same items. Are the labels right, how many cells are empty, are there duplicates and outliers. Things that can be counted and fixed. What this benchmark touches is not on that list. Even a clean table with accurate labels and no missing values can, if it was assembled from one particular set of people, carry a relationship between features and target that differs from the world's. And that gap stays invisible no matter how long you stare inside the table.

Carried over into everyday data work, the question becomes this. Did our training data arrive because it resembles the population, or because it was easy to gather? Answering it takes four things written down somewhere.

• • Where this data came from. Record the institution, the period and the channel it was collected through. If the collection route is never written down, there is no ground for even suspecting skew later
• • Whether the validation set came from the same route as the training set. Split one sample and the internal metrics stay good to the end. That structure is why models in this test broke without raising a single warning
• • Whether any sample from the population side exists at all. It can be small, but some share has to be collected close to randomly and kept separately. With nothing to compare against, skew never becomes something you can measure
• • Who checks the features whose relationships contradict common sense. A fit in which smoking lowers risk takes domain knowledge to see. The paper's experiment shows a language model can serve as a first filter in that check, but a hit rate of 37.4% is not enough to replace human review

That said, this benchmark should not be stretched too far. What gets twisted here are the coefficients of a reference linear model. Not every collection bias in the real world is expressible as the amplification or attenuation of a linear coefficient. The design is still useful in spite of that. Make the collection method itself a variable, and a risk that appears under no item of data quality comes out as a number. Which means quality metrics, the next time they are drawn up, need one more item on the list.

Thank you for reading this far. The facts here were checked against the paper published on arXiv, and once the benchmark's data and code are released we intend to look again at how the variants are actually composed. If you keep a record of the collection route behind your training data, or have a case where you held a separate sample to compare against the population, we would be glad to hear about it.

Pebblous Data Communication Team
October 19, 2026