Executive Summary

Key Numbers

Put the origin of the added compounds next to the R² change in each setting, and the places where the numbers moved cluster on one side of the drug axis.

Source: arXiv:2608.11444, Tables 1, 3 and 4

Drug response prediction is the problem of having a model guess in advance how many cancer cells will die when a given compound is applied to them. Drug discovery has held long-standing hopes for it, since candidates can be filtered before they reach the bench, but comparing results across papers has been hard. Each study cut its data differently, normalised it differently, and reported a different metric. IMPROVE is a framework that attacks that problem by unifying the data schema and the evaluation protocol first.

This study enlarged that standard benchmark. The training data in the original benchmark came from four families of cell-line screens: CCLE, GDSC v1 and v2, gCSI, and CTRPv2. The expanded version adds NCI60, PRISM, and FIMM as training sources, and it also brings in four studies of organoids grown from tissue taken from patients, organised under the same schema. Raw measurements and compound annotations were pulled from PharmacoDB through the PharmacoGx package and pushed through a single preprocessing flow.

What unification actually means shows up in that list of preprocessing steps. On the compound side, broken entries were stripped out of the SMILES strings, salts and counterions were removed, the strings were converted to canonical SMILES, and tautomers that differ only in notation were merged into one. On the cell-line side, names written differently in each source were mapped to Cellosaurus accessions so that the various names pointing at the same cell came together, and gene identifiers were aligned against the NCBI gene information table. Response values were not discarded as biologically unexplainable outliers but truncated at 200% viability, then fitted to a four-parameter logistic curve and summarised as the area under that curve.

Because the data went through this cleanup, the 55,282 compounds you get by adding up the rows of Table 1 shrink to 53,949 unique compounds. Identical compounds were counted once on the basis of structure rather than name. Splitting the added volume by source makes the picture sharper still.

Excerpted from Table 1 of arXiv:2608.11444. The compound total is a deduplicated count of unique structures, so it is smaller than the plain sum of the rows (55,282), while the experiment total equals the sum of the rows. The four organoid studies are grouped here from their individual rows.

Of the 53,949 compounds in the expanded set, 53,323 sit inside NCI60 alone, and 4.27 million of the 5.45 million experiments come from the same source. That is close to 99% by compound count and 78% by experiment count. The real content of the sentence "more than 50,000 compounds were added" is less that they were gathered evenly from many places, and more that the NCI60 screen the US National Cancer Institute has been running for decades was folded into the IMPROVE standard.

Add up the experiments in the five rows the original used for training, meaning CCLE, CTRPv2, GDSC v1 and v2, and gCSI, and you get roughly 690,000. The paper does not state that original total separately, so this is a sum over the rows of Table 1; set against the 5.45 million of the expanded set, the experiments available for training grew nearly eightfold.

Diversity on the cancer-model side is a different story. The deduplicated number of cancer models is 1,362, and NCI60, which single-handedly drove up the compound count, in fact carries only 85 cell lines. What actually holds up the cell-line count is GDSC (986 and 808) and CTRPv2 (845), both present from the original. Six omics types are in place: 1,907 gene-expression samples, 1,764 copy-number variation, 1,769 mutation, 894 protein, 946 miRNA, and 839 DNA methylation. Fewer of those samples actually connect to a drug-response measurement, which brings gene expression down to 1,354 and DNA methylation to 803. What the models actually saw in this evaluation, though, was only transcriptomics and chemical features.

Had the authors reported the effect of adding data as a single number, this paper would have little to talk about. Instead they split the test four ways, along two questions: was the drug on the exam seen during training, and was that cell line seen during training?

• Mixed: both the drug and the cell line were seen in training, and only their pairing is new. The easiest setting.
• Drug-blind: the drug on the exam was never seen during training.
• Cancer-blind: the cancer cell line on the exam was never seen during training.
• Disjoint: the drug and the cell line are both new. The strictest setting.

The settings that carry value for drug discovery are the second and the fourth. Predicting the response of a drug you already know well can be substituted, to a large degree, by looking up an experimental record; predicting the response of a compound nobody has ever tested cannot. Whether a model can push the relationship between chemical structure and activity beyond its training data shows up here.

For the comparison to hold, the two benchmarks have to sit the same exam. The authors built the evaluation folds of their ten-fold cross-validation from GDSC, CCLE, and CTRPv2 only. Those three sources are in both the original and the expanded set, so the exam can be held fixed while only the training data changes. NCI60, PRISM, and FIMM, newly brought in by the expansion, were used for training alone and never to define a test fold.

The four settings are not separately built exams; they fall out of one split. The authors randomly divided the unique drug identifiers and the unique cell-line identifiers appearing in those three sources into ten pieces each. In a given fold, experiments involving a held-out drug become the drug-blind exam, experiments involving a held-out cell line become the cancer-blind exam, and experiments involving both naturally remain as the disjoint exam. After that holding-out was done, 10% of the remaining experiments became the mixed exam and another 10% the validation set used for early stopping.

The handling of leakage is worth a look too. Once a fold decided to hold out a particular drug and cell line, every experiment in which that drug or cell line appeared was erased from training regardless of source. If the same compound survives in NCI60 under a different name, the expanded model would effectively be sitting the exam with the answers in hand, so without this step the improvements above would be hard to trust. Both models were trained on Argonne National Laboratory's 8-GPU cluster with the same splits and the same protocol.

Two models of very different character were put on the benchmark. UNO encodes the gene expression of a cell line and the physicochemical descriptors of a drug through separate fully connected subnetworks. GraphDRP treats a drug as a molecular graph of atoms and bonds and reads it with graph convolution. One side sees a drug as a list of numbers, the other as a picture.

Mean R² over the ten cross-validation folds came out as follows. Comparing the original and expanded columns within a row gives the effect of adding data in that setting.

Table 3 of arXiv:2608.11444. Mean values over ten-fold cross-validation.

A negative R² means the model did worse in that setting than answering with the mean of the observed values. GraphDRP trained on the original benchmark scored −0.11 on unseen drugs and −0.26 when both were new. It was less a matter of weak predictive power than of a state you could not call prediction. Trained on the expanded set, those became 0.12 and 0.08, flipping sign, and UNO climbed from 0.03 to 0.22 and from −0.10 to 0.20. The Pearson correlation points the same way: UNO's drug-blind figure rose from 0.39 to 0.51 and its disjoint figure from 0.34 to 0.49.

The other two axes, however, did not move. On unseen cell lines UNO went from 0.60 to 0.58 and GraphDRP from 0.60 to 0.57, slightly down rather than up. In the mixed setting, which asks about drugs and cell lines the model has already seen and only swaps the pairing, the drop was larger: GraphDRP fell from 0.75 to 0.66. Growing the data to 5.45 million measurements bought a large gain on the hard problems at the cost of losing a little on the easy ones.

The authors offer one cause for that decline: heterogeneous data mixing in and raising variability. While generalisation to unseen targets improves, performance on familiar targets can fall.

The four settings form a 2×2 grid made by two axes, whether the drug was seen and whether the cell line was seen. Below, UNO's R² change is laid onto that grid, and the cells that rose and the cells that stalled divide along an axis.

The wobble between folds deserves attention as well. For UNO, the standard deviation of R² in the disjoint setting fell from 0.39 to 0.18, and in the drug-blind setting from 0.31 to 0.18. Not only did the mean rise, the result also bounces less when the fold changes. GraphDRP tells a different story: in those same two settings its variability grew, from 0.27 to 0.28 and from 0.19 to 0.28. Only one of the two models gained stability along with accuracy.

Open the folds one by one and the improvement did not happen in every piece either. Values in the drug-blind and disjoint settings rose in most folds, but some folds came out lower with the expanded set, and one fold worsened sharply for GraphDRP while moving the other way for UNO. So the authors treat this result not as an improvement manufactured by a few favourable splits, but as a robust trend that is not perfectly uniform.

Up to here it reads as a story in which chemical diversity won. The authors shake that reading themselves. Because most of the experiments in the expanded set come from NCI60, they write, the balance between sources became worse than in the original, and the models may have tilted toward patterns specific to NCI60. The shared test set was built from CCLE, CTRPv2, and GDSC, and those three sources use a different cell-viability assay from NCI60. The drop in the mixed setting, they note, may also come from that heterogeneity.

Two explanations therefore sit on top of the same result. One is that a wider chemical space let the models push the relationship between structure and activity beyond their training data. The other is that the training distribution shifted toward scale and chemical diversity while giving up balance between sources, and that the price showed up as the decline on the mixed and cell-line axes. This experimental design cannot separate the two. The place where the authors left this caveat instead of closing the conclusion as a victory for diversity is the most honest passage in the paper.

For anyone who works with data, there are three things to take from this result.

First, the word "grew" has to be broken down by axis. What this study grew was the compound axis, and what rose were only the two settings in which the compound is unseen. The cell-line axis was not much different from the original, and performance on that axis stayed where it was. A report that the data grew eightfold says nothing about where performance will rise unless it also says which axis grew.

Second, before and after an expansion have to be measured with the same exam. This paper's comparison holds because the test folds were built only from the three sources common to both benchmarks, and because every experiment involving a held-out drug or cell line was erased from training regardless of source. If the test set grows along with the data, the place where you could trace the cause of a performance change disappears.

Third, count whether one source dominates the data. When 78% of the experiments come from a single source, that source's way of measuring becomes the model's standard. Here the type of assay made that difference; elsewhere the same slot is filled by labelling guidelines, collection equipment, or the data of one particular customer. Without a count of the weight of each source, there is nowhere to look for the cause when performance later wobbles.

The limitations the authors wrote down should be read alongside all this. As the data grew, the resources training requires grew sharply, making it hard to handle without multiple GPUs and shrinking the room for hyperparameter search. Because the resource pools heterogeneous sources, residual noise and systematic bias are hard to remove entirely. Six omics types are in place, yet this evaluation used only transcriptomics and chemical features, so which axis moves when mutation, copy-number variation, and protein data are added remains an open question. The authors also see this scale as a size suited to pretraining. Learning first from broad, diverse data and fine-tuning per task, they suggest, could soften some of the trade-off seen above. The expanded dataset, the curation code, and both model implementations are all public, so anyone can carry that question forward.

Academic Papers

• 1.Lavelle, V., Zhu, Y., Marlor, K., Brettin, T., & Stevens, R. (2026). Large-scale AI-Ready Data for Anti-Cancer Drug Response Modeling. arXiv:2608.11444, Argonne National Laboratory.
• 2.Weil, M., Partin, A., Kousa, N., et al. (2024). The IMPROVE framework for automating and standardizing tumor drug response prediction modeling. European Journal of Cancer, abstract PB003.
• 3.Xia, F., Allen, J., Balaprakash, P., Brettin, T., et al. (2021). A cross-study analysis of drug response prediction in cancer cell lines. Briefings in Bioinformatics (the UNO model).
• 4.Nguyen, T., Nguyen, G. T. T., et al. (2021). Graph convolutional networks for drug response prediction. IEEE/ACM Transactions on Computational Biology and Bioinformatics (the GraphDRP model).
• 5.Smirnov, P., Kofia, V., Maru, A., et al. (2017). PharmacoDB: an integrative database for mining in vitro anticancer drug screening studies. Nucleic Acids Research.

Data & Code

• 6.AI-Ready Drug Response Dataset, open data on figshare (the full expanded benchmark).
• 7.vincevelle/CSA-and-PharmacoGx-Integration, GitHub repository (data curation and processing code).
• 8.JDACS4C-IMPROVE, project repository (the UNO and GraphDRP implementations and the standard workflow).