Executive Summary

Key Numbers

Sources: the Kolab paper (arXiv:2610.21109, October 15, 2026) and the project page.

What Kolab collects is not a transcript but a screen. Professionals interacted with an agentic AI, Claude, for two one-hour recording sessions while handling their own work, and they were free to use other AI agents when appropriate. Fifty participants at two sessions each produced 100 workflow sessions, and the researchers divided them into 508 steps. More than 90 software applications turn up in the footage. Not only the tools everyone shares, such as Google Docs and Office, but tools used only in one trade, such as Revit and DaVinci Resolve.

The 50 participants span 10 sectors and 40 occupations under the O*NET classification. Even the examples the paper lists run from video editors, graphic designers and audio engineers to aerospace engineers, GIS specialists, lawyers and bioinformaticians. This is not a sample scooped out of one industry.

Not just any work was recorded. The researchers set four conditions. The problem had to originate from the professional's actual work, require the application of professional knowledge, skills or judgment from the participant's field, require a sequence of interdependent steps to reach its final goal, and result in a concrete final deliverable. Tasks lifted from existing datasets and invented scenarios were turned away. The sessions that came through that filter run to 5.2 steps on average, and a step breaks into 8.7 nodes on average.

A node stands for a single action by the person or the agent, and carries a start time and an end time. String those nodes together and one piece of work becomes a graph. Running several things at once makes the graph branch, and repeating the same action makes it loop, so work that does not travel in a straight line keeps its shape in the record.

The five-step job of an architect that the paper walks through shows the structure well. The architect uploaded an existing structural drawing and asked for a full room layout, between 24:10 and 25:28 in the recording. Claude returned a dimensioned floor plan. What comes next is the part that matters. From 58:40 to 72:59 the architect kept the zoning they liked, fixed the lobby and storage circulation, and rebuilt the walls and partitions in Revit. The paper describes that moment as an action not directly observable by the agent.

To borrow the abstract's phrasing, the judgment on an agent's contribution lands once that contribution leaves the chat window. The model cannot see how its answer was cut and pasted inside a design tool, and when the next question arrives, it arrives without that knowledge. A screen recording is the only record that holds the invisible stretch whole.

This is also where the design parts ways with existing benchmarks. Widely used agent evaluations such as WebArena and OSWorld count whether a model finished a given task on its own. What the person did with the result never reaches the scoresheet. The paper puts it plainly: strong performance on such benchmarks does not imply effective real-world collaboration.

The paper gathers the difference into a single table. It lines Agents' Last Exam, Workflow-GYM, WeaveBench, TheAgentCompany, GDPVal and DeskCraft up beside Kolab and asks five things of each. Does it capture a real human workflow, does it contain human-AI collaboration, is the software diverse, how many domains does it cover, and what modalities does it carry. Kolab is the only one with a mark in both of the first two columns. DeskCraft also carries a mark in the collaboration column, but its real human workflow column is empty. Kolab's 40 domains come third behind Agents' Last Exam at 55 and GDPVal at 44, and Kolab is the only one holding video as a modality. What Kolab carries beyond the rest is, in the end, the part that opens after the answer goes out.

Annotators attached one of three labels to every agent response. The definitions are short. Adopt covers a response the user incorporated into the workflow without changing it, Modify covers one incorporated after revision, and Reject covers one that was not incorporated at all. Reject carries one further condition: the label is applied only when the user explicitly states that they are rejecting the response.

That extra condition on Reject is hard to walk past. A response the user passed over without saying anything never enters the 15.0%. Given that people at work usually meet a suggestion they dislike by closing the window rather than saying so, a figure that counts only spoken refusals reads more like a floor on real refusal. This reading is not the paper's claim but this article's interpretation, following from the definition.

Split the data by occupation and the numbers wobble. Most fields lean toward taking the agent's output as it comes, but the adoption rate runs from 57% in legal work up to 79% in the sciences and in computer and mathematical fields. Legal work stands out on the modify side in particular, where 41% of agent output was used only after revision. The same model and the same level of difficulty land differently depending on whose desk the answer reaches.

Collaboration quality was not put as a single question. Professionals rated seven statements from 1 to 5. Three of them asked whether the agent understood the task, context, and instructions provided; whether it responded appropriately to requests and adapted when given feedback or corrections; and whether it reduced effort, time, or cognitive load during the task. Four more asked whether it proactively suggested useful ideas, alternatives, or next steps when appropriate; whether it produced accurate, relevant, and trustworthy outputs that required minimal correction; whether the interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work; and whether they were able to guide, redirect, and override the agent's actions when desired. A separate single question asked them to rate the overall collaboration experience.

The researchers also measured the shape the work traveled in. How much the graph branches, and how much the work bunches into one stretch, became the indicators. Burstier work came with a longer opening stretch in which the professional planned alone before calling the agent (ρ=0.35), and the pattern grew sharper among participants who said their plan was clear before they turned to AI (ρ=0.53). Work that branched heavily, on the other hand, drew lower collaboration ratings (ρ=−0.24). This is the evidence behind the abstract's line on the importance of when and how professionals coordinate with agents.

The professionals' own verdicts ran generous. Asked after each session to rate the collaboration experience out of five, 54.9% gave a 5 and 29.7% gave a 4, leaving 15.3% at 3 or below. Output quality clustered the same way, at 53.5% for 5 and 31.0% for 4. That the scores pile up at the top turns into a problem again when the model results arrive, because the job of separating good collaboration from bad has to be done on data in which most of the collaboration is already written down as good.

The researchers turned this data into two kinds of task. One is recognition. The model is shown how the professional actually reacted, then asked whether that reaction was an adoption, a modification or a rejection. The other is forecasting. The screen is cut at the moment the agent generated its response, the human reaction is hidden, and the model has to call what will happen next. Four frontier models and two open-weight models, six in all, sat both tasks.

The gap between those two numbers is the point of the benchmark. Show a model what became of the answer and the best of them climbs to 0.713; withhold it and ask for a forecast and the same model drops to 0.513. Even that figure only means something next to the 0.265 of a baseline that keeps picking the majority class. On what a person will do with the answer in hand, models are still guessing roughly.

The way they get it wrong has habits too. With video inputs, Gemini 3.8 Flash forecast Adopt for 83.8% of examples, against a ground-truth adoption rate of 65.9%. It reads its own chances of being taken up as systematically better than they are. Qwen3-VL-32B tipped the other way, calling Modify in 87.3% of cases where the true share is 19.1%. Both models look less like something that reads the input and changes its mind, and more like something that fixes on one label and repeats it.

The hope that more context would help is not carried by the data either. The researchers ran separate settings that showed a few preceding steps or the entire workflow, and macro-F1 moved by less than 0.07 for most video and graph settings. Longer context seems to provide limited improvements, in the paper's own words.

The second task, predicting the professionals' ratings, went much the same way. On collaboration satisfaction, the strongest correlation belonged to GPT-6 Astra at a Spearman ρ of 0.379, while Gemini 3.1 Pro managed 0.078. On output quality it was GPT-6 Astra at 0.352, Claude Opus 5.5 at 0.208 and Gemini 3.1 Pro at 0.046. The abstract reads these results as frontier models overestimating collaboration with experts and struggling to distinguish between high- and low-quality collaboration.

One sentence the paper attaches to the recognition task sits close to where this article started. Once agents can work out what happened to their output after it went out, naturalistic screen recordings could be turned into feedback signals for training and for auditing deployed agents. And all of it without asking the expert to label anything.

That labeling cost holds evaluation back is familiar to anyone working on data quality. Checking whether a deployed agent is any good in the field means someone has to look at the output and write down good or bad, and the person able to do that is the very expert who is busy using the agent. So evaluation usually halts at the pre-release benchmark, and post-deployment quality gets approximated through surveys and usage metrics. What Kolab points at is the seat that was empty in between. Use the behavior people already leave behind as the label.

The scores as they stand, though, do not yet support that use. A recognition F1 of 0.713 means that even with the person's actual behavior in full view, the three labels cannot be told apart reliably. Before this can serve as an audit signal, that number has to rise. It looks like the reason the paper positions Kolab as a dataset for studying collaboration rather than as an evaluation instrument.

Read from an organization's side, the questions come down to three.

• • Where does your organization's use of AI leave a trace? If all that remains is the chat log, the place where the judgment happens goes unrecorded. Those fourteen minutes the architect spent rebuilding walls in Revit are missing
• • Does an outcome get attached to that trace? Requests and responses piling up do not make quality data. Only when each response carries whether it was used as-is, revised or thrown away can any of it serve training or auditing
• • Is the attaching being pushed onto people? Ask experts to score every exchange and the record dries up quickly. Pulling the signal out of what they are already doing lasts longer

The limits worth holding alongside the reading are clear enough. Data and code are both pending release as of the paper. The agent under observation was mainly Claude, and 692 labeled instances is small measured against automation benchmarks. With the ratings bunched at the top, how far the data separates the lower band is also hard to say yet. And two one-hour sessions cannot hold the full rhythm of real work.

Thank you for reading this far. The original is at arXiv:2610.21109, and the figures and quotations here were checked directly against that paper and the University of Minnesota project page. If your team is already turning its record of AI use back into quality data, we would be glad to hear which traces you keep and what you attach to them.

Pebblous Data Communication Team
October 17, 2026