Executive Summary

Key figures

The first two numbers describe what this pipeline was built out of. The second two describe what came out of it, and how far that output has been checked.

Source: Prunty et al. (2026), arXiv:2608.25623

The problem the report takes up has a name: the scoping problem. When an organisation brings AI in, it has to decide which tasks can be automated, which should stay with people, and which are best shared between the two. The authors argue that both of the usual grounds for that decision are weak.

The first is the aggregate benchmark score. A figure like 85% on MMLU says something about overall performance, but it says little about why a system failed particular instances, or how it will do on ones that differ meaningfully from the evaluation distribution. The analogy the authors use is the high jump. Knowing that an athlete clears two metres tells you what will happen at a three-metre bar, whereas knowing that they cleared 85% of their attempts tells you almost nothing unless you also know how the bar heights were distributed.

The second is human intuition. Here the authors reach for the word jagged: AI systems are strong and weak in non-human-like patterns, which makes our sense of where they will struggle unreliable. Anthropic's survey of 81,000 users, which the report cites, likewise found reliability to be respondents' top concern.

The target, then, is not the elimination of failure but its predictability. A less capable system whose failures are reliably tied to identifiable task demands, the authors write, is substantially more deployable than one with higher benchmark accuracy and unpredictable failure modes. Once failure is predictable, where to place human oversight and how to design guardrails and handoff protocols become answerable questions.

Existing work that estimates AI exposure at the level of occupations falls short of this decision for the same reason. Estimates inferred from observed usage look backwards, capturing only how today's systems are already deployed. Asking experts whether AI could do a given task looks forwards, but rests on forecasts about opaque, non-human-like systems and so goes wrong easily. Both are anchored to the current generation of models, and both go stale with each new release.

The pipeline has three stages. Cognitive capability profiling estimates what an AI system can do. Task requirements weighting gathers how heavily an activity leans on each capability. Suitability mapping multiplies the two. The authors flag up front that the two bodies of data are different in kind: the first is a level of capability, the second a weight of importance.

No new exam was written for the first stage; the raw material is benchmarks that already exist. The team drew 18 core cognitive capabilities relevant across workplace activities from the psychometric and cognitive science literature, and wrote a six-point scoring rubric for each, running from level 0, where the capability is not required, to level 5, where a very high level of it is. The spacing of the levels was set intuitively by how large a share of a human population would meet each one: level 1 a demand most adults would meet, level 5 one that only a small minority would. The authors are careful to say they do not claim these levels correspond to actual population proportions.

Items were then pulled from existing benchmarks into a battery aimed at roughly 20,000 items, and each item was annotated for the level at which it demands each capability. The annotator is not a person but an LLM. The method follows the rubric-based annotation of Zhou et al., in which the model is not the subject under evaluation but a classifier applying a rubric that human experts wrote. After datasets whose accuracy could not be scored automatically were removed, 22 benchmarks and 19,576 items remained.

Annotation was done twice, independently, by models from different families: GPT-4o and Gemini 3 Flash. Agreement between them fell below threshold on two dimensions, Attention and Inhibitory Control and Prospective Memory, which were dropped; the remaining 16 sat between rank correlations of 0.38 and 0.81. Even on the lower-correlation dimensions the two raters agreed to within one demand level on at least 74% of items. Averaging the two annotations into a single demand matrix, and dropping the 41 items where either rater returned an invalid value, left a final battery of 19,535 items.

What remained was that the capabilities are entangled. Hard items demand many abilities at once, so the demand profiles correlate strongly, and the inference model loses any basis for deciding which capability an outcome belongs to. The team ran hierarchical clustering on the correlation matrix of the demand profiles and merged them into eight composite dimensions. These eight are the axes on which AI capability is estimated. The questionnaire elicits importance at the level of the original, pre-cluster capabilities, so getting the two rulers onto one another is a step that returns at the end.

The eight dimensions appear across the battery at very different rates. Language is required by every item, but 97% of those demands sit at levels 1 to 2, while Object Permanence is active on only 36% of items and carries a tail of high demands where it is active. No item in the whole battery reaches level 5 on any dimension and only a small proportion reach level 4, which the authors read as the upper bound of demand present in the selected benchmarks.

▲ Battery coverage of the eight composite cognitive dimensions, and mean posterior log-capability across the six systems. Capability is on the same log scale as the demand levels, so a higher value means more headroom against higher demands. Note that demand levels are defined within each capability, so two values from different dimensions should not be read as absolutely equivalent | Source: Prunty et al. (2026), arXiv:2608.25623, Tables 2 and 6

Finally, each system's item-by-item record of correct and incorrect answers goes into a Bayesian item-response model that estimates its capabilities. The output is a posterior distribution, not a single score. Dimensions with thin evidence show up as wide uncertainty, and that uncertainty carries straight through into the suitability scores downstream.

Six systems from two developer families were profiled. On the Google side, Gemini 2.5 Flash, Gemini 3 Flash and Gemini 3.1 Pro; on the OpenAI side, GPT-4o mini, GPT-5-nano and o4-mini. By overall mean capability, Gemini 3.1 Pro comes in at 3.70 and Gemini 3 Flash at 3.38, while the remaining four cluster between roughly 2.6 and 2.9.

This is where the report's first finding appears. Despite those differences in overall level, all six systems share a remarkably similar profile shape, and variation between capability dimensions substantially exceeds variation between systems. Which cognitive dimension a task leans on matters more than which model you picked.

The location of the strengths and weaknesses is common to all six. Semantic Memory, Language and Social Cognition sit at the top; Action Planning and Simulation, Instrumental Reasoning and Object Permanence sit at the bottom. Semantic Memory was the highest dimension for five of the six, the exception being Gemini 2.5 Flash, which peaked on Social Cognition. That is the same shape as the work today's chatbots are good at.

Within that common shape, the leading systems pull away at exactly two points. On Information Integration and Control, Gemini 3 Flash at 5.06 and Gemini 3.1 Pro at 4.90 separate clearly from the rest of the catalogue, which sits at roughly 2 to 4. On Action Planning and Simulation the gap is wider still: Gemini 3.1 Pro reaches 4.25 against 2.37 for the next-highest system, Gemini 3 Flash.

The job-side data comes from a questionnaire, and one choice made here shapes the character of the whole pipeline. The straightest route would be to apply the same rubric used on benchmark items to real work, but annotating the difficulty of every instance of work at organisational scale is not feasible. So instead of difficulty, the questionnaire asks about importance.

It runs in four stages. Respondents give their domain and experience, pick the five activities most important to their role from a list of 18 adapted from O*NET categories, then rank them and report weekly hours. They are next familiarised with the 18 cognitive capabilities one at a time through definitions, cross-domain examples and an illustrative image, and take a matching quiz. Finally, for each activity they selected, they choose the five most essential capabilities and distribute 100 points across them. The whole thing takes about 30 minutes.

The sample combined 125 people recruited through collaborating companies with 414 recruited via the online panel Prolific, for 539 responses in total, of which 410 passed three quality filters: completion, at least 50% on the capability quiz, and at least 10 minutes spent. Respondents averaged 40.3 years of age, 5.7 years in their current role and 11.8 years in their wider field. Whether two samples of such different character could be pooled was checked separately. Across activities, their capability-importance profiles agreed at a mean cosine similarity of 0.91 and a Pearson correlation of 0.80. Against a split-half noise ceiling, the observed between-source correlation reaches the expected ceiling, so the company and online samples agree about as closely as two random halves of the same sample would.

4.1A Common Core with Secondary Specialisation

Averaged across the six domains, the highest-ranked activities were Problem solving (2.04) and Decision making (1.78), followed by Checking (1.44), Researching (1.35) and Computer use (0.96). Problem solving and Decision making rank at or near the top in every domain, which the authors call a domain-general core of judgement-intensive work.

The core is not what separates the domains; the activities that attach to it are. Warehouse and Logistics and Manufacture, Maintenance and Repair put much more weight on Checking and Tool use than the overall average, while the numerical-digital domain lifts Computer use and Analysing data together to 1.89. In Hospitality, Sales and Client Care, Building rapport reaches 1.83, its highest value across the six domains.

▲ Frequency-adjusted importance scores. Respondents assign 5 points down to 1 point across their five selected activities, averaged within each domain | Source: Prunty et al. (2026), arXiv:2608.25623, §4.3.3

The questionnaire also picks up the difference between importance and time. Checking ranks high on importance but occupies only 10.2 hours a week, while Computer use takes the largest share at 15.7 hours despite only moderate importance. Which of the two you treat as the target changes the candidate list.

One level down, at capabilities, the report's second finding appears. In the activity-by-capability importance matrix, Planning, Semantic Memory, Working Memory, Language and Procedural Memory receive the greatest weight across nearly every activity. What differentiates activities is the secondary capabilities layered on top: interpersonal work leans on social cognition, analytical work on pattern recognition, creative work on mental simulation. In the authors' phrasing, the capability profile of an activity is best understood as a common cognitive core tuned by task-specific secondary demands.

Splitting by domain leaves that structure intact. The domain-specific matrices correlate cell for cell between 0.53 and 0.77, with a mean of 0.63. The differences again sit in the secondary capabilities: Warehouse and Logistics is 2.1 points above the pooled average on Spatial Reasoning and Navigation, Manufacture, Maintenance and Repair is 2.5 points above on Planning, and Hospitality, Sales and Client Care is 1.7 above on Theory of Mind and 1.2 above on Emotion Perception and Empathy.

The last stage multiplies the two bodies of data, and before multiplying it has to align the rulers. Capability was inferred over the eight clustered dimensions while importance was elicited over the original capabilities, so each inferred cluster is first expanded onto its constituent capabilities, every constituent taking the level inferred for its cluster. Suitability is then a weighted mean of the system's capabilities under the activity's importance weights. Two dials in that formula are not estimated from data: the compensatory power, which sets how far strengths may offset weaknesses, and the sharpness, which sets how tightly the importance weights concentrate. Those come from deployment policy, the authors insist, not from the data.

Under neutral settings, Gemini 3.1 Pro is the most suitable system for every one of the 18 activities (log suitability of roughly 4.1 to 4.6), followed consistently by Gemini 3 Flash at 3.3 to 4.1. The other four form a lower, closely overlapping group between 1.7 and 3.4. What stands out is how little the ranking changes from one activity to the next.

That happens because the two findings land on top of each other. Almost every activity places substantial weight on knowledge, language, planning and control. The systems differ relatively little on knowledge and language, and much more on Information Integration and Control and Action Planning and Simulation. With the work converging on a shared core and the model differences concentrated at two points inside that core, any activity you substitute in returns the same order.

The exceptions are the socially oriented activities. Building rapport, Listening and Communicating place more weight on Social Cognition and Language than most others, and partially break the ordering. GPT-4o mini in particular rises to the top of the mid-pack on Building rapport. Uncertainty also varies by activity: Coding, whose importance profile came from only around 20 respondents, had the widest credible intervals, while a well-sampled activity like Problem solving had narrow ones.

Swinging the two policy dials across a broad range does not move the conclusions much. On average nearly 12 of the 15 pairwise system orderings remain unchanged, and Gemini 3.1 Pro holds first place on 17 of the 18 activities. The single activity that reorders is Admin, and only under the most extreme compensatory setting.

Suitability alone is half an answer, because the tasks AI can do well and the tasks worth automating are not the same set. So the authors multiply suitability by activity importance into a separate deployment-priority score. Researching, Admin and Analysing data rise to the top of that ranking.

The same calculation was run for a single organisation, anonymised as Company X, whose 35 quality-controlled responses came predominantly from the administration and customer-facing domains. Recomputed with that company's own weights, Communicating and Researching emerged as the clearest deployment opportunities, while Problem solving and Computer use remained high-priority tasks where much of the current AI catalogue still falls short. The pipeline also runs at finer grain. The team interviewed a senior customer service employee at Company X and elicited capability weights separately for the role overall and for three core duties within it, such as informing customers about their rights.

The part of this pipeline with the longest practical shelf life is not the ranking of six models. That table changes with the next release. What lasts is the fact that the two streams update independently of one another, which the abstract states outright: as both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty.

A new model means running the battery once more on that model. A reorganised company means running the questionnaire again. Neither one going stale forces you to rebuild the other. The very thing that made the authors sceptical of usage-based estimates and expert forecasts flips into an advantage here. Those two methods measure the current generation of AI, so their records age out with the model, whereas cognitive demand is a property of the work and survives the swap.

There is a question worth putting to your own organisation here. On what axis is your internal discussion of which work to give AI actually recorded? If it is recorded as model names and demo results, the shelf life of that record runs until the next launch. The judgement that was genuinely useful in those discussions, the judgement about what a given piece of work demands, is worth checking for in the record.

6.1The Boxes Still Empty

The first limitation to write down is the validation. Nobody knows the true capabilities of a real model, so the team drew 20 synthetic agents with predetermined capability profiles from the prior, simulated their responses to the battery, and checked whether the inference recovered those true values. Profile shape was recovered well, at a correlation of 0.92, and with a shared intercept across systems the overall level was recovered at 0.98. But because the same modelling parameters were used for both simulation and inference, the authors write that the analysis establishes internal recoverability and does not establish the empirical validity of those modelling assumptions. Whether performance on real tasks combines the same way is left to future work.

The sharpness of the estimates also varies by dimension. Table 6, the source of the capability figures above, also reports for each dimension the maximum posterior correlation with any other, a measure of how entangled it is. The closer to 1, the harder that dimension is to separate out. The three highest are Object Permanence at 0.73, Instrumental Reasoning at 0.66 and Action Planning and Simulation at 0.61. Those are the three dimensions on which the systems are jointly weakest, and one of them is the dimension this article leaned on for the model gap.

A reservation of the same kind attaches to the requirements side. What the pipeline measured is importance, not demand, so the suitability scores are comparative rather than calibrated. They indicate which system is better matched to an activity, not the probability that it will successfully perform it.

Reliance on self-report is a weakness the authors name themselves. Many cognitive processes are automatic and hard to report accurately. Metacognition, spatial reasoning and object permanence were selected relatively infrequently, plausibly because they are less salient to conscious reflection rather than because they contribute little. Restricting respondents to five capabilities per activity, they add, compounds that tendency.

On the capability side the constraint is in the material. The battery is assembled primarily from text-based evaluations and so under-represents multimodal perception, long-horizon planning and interactive tool use, which are precisely the capabilities on which contemporary systems appear weakest and most differentiated. The true spread between systems may therefore be larger than these profiles show. That the profiling targets foundation models in isolation matters too. Real deployments run scaffolded agents whose external memory, planning modules and tools compensate for the base model: a monitor that tracks background state effectively supplies object permanence, and well-specified tooling makes the available actions explicit and so lowers the perceptual demand.

The last limitation cannot be resolved from within the framework. A requirements profile ultimately describes what a task demands of a human, while an AI system may arrive at the same outcome by another route. Coding is the authors' example: human programmers rely heavily on planning and procedural memory, whereas contemporary AI systems often succeed through statistical pattern matching. That mismatch can only be found by validating against observed deployment outcomes.

• 1.Prunty, J., Tešić, M., Quinn, P., Hernández-Orallo, J., & Cheke, L. (2026). "Using profiles of cognitive capability to assess AI suitability for workplace tasks." arXiv:2608.25623.
• 2.Kinds of Intelligence (CFI), University of Cambridge. "Task-Suitability-Profiles" (code and rubric repository).