Executive Summary
EU AI Act Article 10 is the first law to name, clause by clause, how a high-risk AI system prepared the data it learned from. Paragraph 2(c) enumerates data-prep work as "annotation, labelling, cleaning, updating, enrichment and aggregation." Paragraph 3 demands that training, validation, and testing data be "relevant, sufficiently representative, and to the best extent possible free of errors and complete." And paragraphs 2(f)–(h) require that bias examination, mitigation, and the identification of data gaps all be put on paper. For years, data quality was an engineering problem — the work of pushing model performance higher. Under Article 10, it becomes a matter of evidence you hand to an auditor.
The crux is not the deadline but the nature of the evidence. The Digital Omnibus pushed application of this obligation to 2 December 2027, yet the artifacts the regulation asks for are unchanged. Version history for labeling guidelines, inter-annotator agreement, label error rates, bias-examination records, data-gap logs — none of these can be produced retroactively once the model has been trained. The fact that even leading benchmark test sets carry an average 3.3% label error tells us up front that "free of errors" is a matter of measurement, not perfection.
For a prepared organization, the 16-month deferral is not a break but a window to instrument the pipeline. The moment the work of diagnosing a dataset's representativeness, gaps, and bias — what Pebblous has long called "diagnosing the data" — translates one-to-one into the individual requirements of Article 10, compliance stops being a cost and becomes a barrier only the prepared can cross. This report follows where in the labeling workflow that translation happens.
Four numbers say first where this translation lands: that "free of errors" is a measurement rather than a declaration of perfection, the annotator agreement that gauges representativeness, the cost of standing up a new quality system, and the gap left by organizations that have yet to begin.
3.3%
Average benchmark label error rate
Why "free of errors" is a measurement · Northcutt 2021
α ≥ 0.80
Reliable annotator agreement
Quantitative evidence of representativeness · Krippendorff
€193K–330K
Cost of a new QMS
Plus €71.4K/yr upkeep · CEPS estimate
26.2%
Organizations that had begun
One month after the AI Act took force · Deloitte
The Law Names the Labeling Process
An annotator draws a box over an image, picks "pedestrian" from a dropdown, and moves to the next frame. Repeated thousands of times a day, that click was for a long time an internal company affair. Whether the label was correct, whether the guidelines were consistent, who judged by what standard — all of it was left to the diligence of the team. EU AI Act Article 10 pulls that very workbench inside the statute.
We have covered the classification criteria for high-risk AI and the course of the deferral negotiations several times already, so we will not repeat them here. Readers who want the background can turn to the high-risk deferral and data governance and the practical impact of the Annex III deferral. What this report digs into is one point none of that background touches: that Article 10 calls the labeling process itself by name.
Six Verbs
Read Article 10(2)(c) as written and data-prep work is enumerated in six verbs: annotation, labelling, cleaning, updating, enrichment, and aggregation. No regulatory text has ever named the individual stages of data preparation this specifically. The list is not a set of examples but a set of audit coordinates. The moment a regulation calls a task by name, the artifacts that task leaves behind become something an auditor can demand.
Paragraph 3 lays the language of quality on top of this. Training, validation, and testing datasets must be "relevant, sufficiently representative, and to the best extent possible free of errors and complete." It further requires that the data have "the appropriate statistical properties" in light of the intended purpose. As we will see, each of these four adjectives translates into a value measurable at the label level.
Finally, paragraphs 2(f)–(h) nail the obligation down in three strands. (f) Examine the data in view of possible biases; (g) take appropriate measures to prevent and mitigate those biases; and (h) identify data gaps or shortcomings and decide how they are to be addressed. What the three share is that each presupposes not "we did something" but "we documented something." A claim that bias was examined does not hold up in an audit without a record of that examination.
The fact that the deadline moved does not change this structure. The Digital Omnibus pushed Article 10's application date to 2 December 2027, but it did not touch a single word of the article's text. What was deferred is the date, not the requirement. From the moment a regulation calls the six verbs of data preparation by name, the artifacts those verbs leave behind become evidence that will one day have to be produced. The next question follows naturally: where exactly in the labeling workflow are those artifacts made?
Where the Workflow Becomes Evidence
Labeling is not a job that ends with a single click but a process of chained stages. You define the label schema, write the guidelines, have annotators annotate, sample-check agreement and quality, and re-label where needed. The evidence Article 10 asks for is already contained in the byproducts these stages shed. The question is whether you keep them as evidence or let them slip past.
The table below sets out, for each stage, which statutory requirement it maps to, what it leaves behind as evidence, and which tools already generate that record automatically. The far-right column matters most. "Regulation is coming" is not a forecast — the evidence infrastructure is already a product, in the present tense.
| Labeling stage | Article 10 requirement | Audit artifact | Example vendor feature |
|---|---|---|---|
| Schema & guideline definition | 10(2)(c) labelling, 10(2)(g) mitigation | Guideline version history, change log | Label Studio "Iteration History" |
| Annotation | 10(2)(c) annotation & labelling | Per-label lineage (who, when, what), skip/reject events | Encord "label lineage", Snorkel lineage |
| IAA & consensus | 10(3) representativeness, 10(2)(f) bias examination | Inter-annotator agreement (α/κ), consensus metrics | Encord automated IoU, V7 "Consensus Stage" |
| QA & re-labeling | 10(3) free of errors, 10(2)(h) gap identification | Per-field pass/fail + rationale, label error rate, gap log | Appen "Quality Audit", per-contributor audit trail |
| Dataset snapshot | 10(2)(c) updating & aggregation, Art. 18 retention | Versioned snapshots, provenance map | Label Studio "Snapshots" |
Source: Pebblous reconstruction based on each vendor's official documentation (Label Studio, Encord, V7, Appen, Snorkel) and the text of EU AI Act Article 10. Feature names verified as of the publication date (July 2026).
The right-hand column makes one thing clear: labeling vendors did not wait for regulation. Label Studio already offers "Iteration History," which tracks every create, edit, skip, and reject event, and "Snapshots," which pin a dataset to a point in time. Encord records the lineage of each individual label and configurable consensus as automated IoU metrics. Appen's "Quality Audit" logs pass/fail and the reason for it, field by field. Rendered in regulatory language, these features are, as they stand, tools for generating Article 10 evidence.
But a tool leaving logs does not make the evidence complete. What an auditor asks for is not a single document but a living lineage. A one-line declaration that "we ran a bias-examination procedure" will not clear a market-surveillance audit unless it is backed by records showing how that procedure actually ran in production (Bird & Bird). Traces of provenance reconstructed retroactively after training are, in themselves, a red flag. Evidence is not written after the fact — it must accumulate on its own while the work flows.
Proving "Free of Errors" at the Label Level
Of the four adjectives in Article 10(3), the one most easily misread in an audit is "free of errors." Taken at face value it reads as a demand for a 0% error rate, yet the article clearly attaches the qualifier "to the best extent possible." What that qualifier means in practice becomes clear once you actually measure the data.
Northcutt et al. (2021) used confident learning to measure label errors across ten widely used ML benchmark test sets. The average error rate was at least 3.3%, ranging from 0.15% to 10.12% across datasets. For the ImageNet validation set alone it was about 6%. Even flagship benchmarks vetted in multiple human passes carry this much error. The low-quality baseline for labels gathered hastily via crowdsourcing reaches 18%. Read "free of errors" as 0% and no dataset on earth would clear the requirement.
So what a labeling team can hand to an audit is not perfection but three measured values. The first is the label error rate we just saw. Compute and document it via confident learning or QA sampling, and that figure itself becomes a quantitative answer to "free of errors" in Article 10(3). The second is inter-annotator agreement (IAA). By academic convention, Krippendorff's α of 0.80 or above is treated as reliable and 0.667–0.80 as tentative. There is no statutory threshold, so what counts as direct evidence for Article 10(3) representativeness and 2(f) bias is less the number itself than the fact that you measured and reported α or κ per task, class, and subgroup. The third is a record of data gaps and class imbalance. Identifying which subgroups are underrepresented and how they were handled maps to 2(h).
Standards Turn Regulatory Language into Numbers
The frames for documenting these measurements already exist. The fields of Datasheets for Datasets (Gebru 2018), Data Statements for NLP (Bender & Friedman 2018), and Model Cards (Mitchell 2019) map almost one-to-one to Article 10's requirements. Items asking for data provenance, collection method, annotator composition, and known biases and limitations are precisely the documentation structure Article 10 demands. Croissant-RAI, a machine-readable successor spec, cites the EU AI Act outright as an explicit design basis. On top of this, the ISO/IEC 5259 series — especially -4, which specifies labeling as a governance process, and -2, which defines measurable data-quality characteristics — together with their foundation in ISO/IEC 25012 and 25024, supplies metrics whose vocabulary of "accuracy, completeness, consistency" nearly matches Article 10(3) word for word.
One caveat is worth attaching. The EU harmonized standard being prepared by CEN-CENELEC JTC 21 (prEN 18284, "dataset quality and governance") had, as of mid-2026, not yet been cited in the Official Journal of the EU (OJEU). Following a harmonized standard confers the legal benefit of a presumption of conformity, but until it attains that status, today's de facto templates are the ISO standards and the documentation frameworks above. There is no reason to wait for the regulator to finalize a harmonized standard. The text of the requirements and the ways to measure them are already in hand.
There is one tension worth surfacing honestly. At the frontier of labeling research sits a position called perspectivism: disagreement between annotators is not noise to be eliminated but signal to be preserved, and some labels have no single correct answer to begin with. For subjective tasks such as judging hate speech or offensiveness, this view carries real force. Yet the phrase "free of errors" in Article 10(3) reads as if it presupposes a single ground truth. This friction between regulatory text and the research frontier is not something to paper over with optimism. The practical answer is not to erase disagreement but to measure it (with α), document its reasons, and preserve it as soft labels. "Free of errors," in the end, is not a declaration of perfection but a measured record of what you count as an error and what you count as legitimate dissent.
Bias Enters Through Labeling
The bias examination and mitigation required by Article 10(2)(f)–(g) is often mistaken for a model-stage problem — the work of measuring and correcting bias in the outputs of a trained model. But a large share of bias has already entered at the earlier labeling stage. And this causation is not intuition but an academically established fact.
Sap et al. (2019) showed that in hate-speech-detection datasets, tweets written in African American English (AAE) were labeled "offensive" up to twice as often as tweets in Standard American English. The follow-up study "Annotators with Attitudes" (2022) goes a step further: annotators' beliefs, identities, and racial attitudes systematically shift their labels. Given the same sentence, the outcome diverges depending on who does the labeling. Bias does not seep into the data in secret — it enters through the person applying the label and the instructions that person was given.
Translated into regulatory language, this causation turns Article 10(2)(f)–(g) into an obligation to document two controls. One is "who labels" — recording the demographic composition and training history of the annotators. The other is "how the task is defined" — when guidelines are vague, an annotator's personal judgment fills the gap, and that judgment becomes a channel for bias. Dialect and context priming, annotator-demographic documentation, and disagreement preservation are all auditable controls. Bias mitigation is a matter of paper trail, not intent. A claim that mitigation measures were taken does not hold up in an audit without a record of what those measures were and what effect they had.
Here we should draw a clear boundary with a companion piece. Our earlier article on the sensitive-data exception for bias detection dealt with the exception clause that permits processing special-category data (race, gender, and the like) in order to detect bias. What this report deals with is not that exception but the obligation itself to examine and mitigate bias. If the exception is a permission — "you may use sensitive data to see bias" — then Article 10(2)(f)–(g) is a command: "see the bias, reduce it, and leave a record of the process." The Digital Omnibus reintroducing a "strict necessity" test for processing special-category data for bias detection also shows that this obligation and exception operate as a pair.
For the Prepared, Regulation Is a Moat
Bind the requirements we have seen so far together and compliance turns from a matter of writing documents into a matter of instrumenting the pipeline. Computing a label error rate, measuring annotator agreement, identifying representativeness gaps, and recording bias controls are all the work of diagnosing a dataset. And diagnosis is not something you do once after the fact but something that must keep running while the pipeline turns. The table below sets out how these diagnostic metrics map to the individual clauses of Article 10.
| Data diagnostic metric | Individual Article 10 requirement | Form left for audit |
|---|---|---|
| Label error rate & QA sampling | 10(3) "to the best extent possible, free of errors" | Measured error rate + method documentation |
| Inter-annotator agreement (α/κ) | 10(3) representativeness, 10(2)(f) bias examination | α/κ table by task and subgroup |
| Representativeness & class distribution | 10(3) representativeness & completeness | Subgroup distribution table, datasheet fields |
| Data-gap identification | 10(2)(h) gap & shortcoming identification | Gap log + record of handling policy |
| Bias-control & mitigation history | 10(2)(f)–(g) examination & mitigation | Control list + before/after metric change |
Source: Pebblous reconstruction based on the text of EU AI Act Article 10. The left column lists the practical metrics for diagnosing a dataset's representativeness, gaps, and bias — an area that overlaps with what Pebblous DataClinic targets.
Standing up this diagnosis costs money. According to the EU impact assessment as reconstructed by CEPS, building a new quality management system (QMS) for a high-risk system runs €193K–330K, with €71.4K in annual upkeep (the labeling process is a sub-component of this total cost, not something that costs this amount on its own). The preparation gap, on the other side, is wide. In a Deloitte survey, only 26.2% of organizations had begun one month after the AI Act took force, and more than half — 53.8% — did not even have a dedicated team (this figure is overall AI Act readiness, not a value specific to labeling). The cost is certain and the preparation is slow. That gap is exactly where a barrier to entry is made.
The weight of the penalties is worth pinning down precisely too. A violation of Article 10 falls under Tier 2 of Article 99, carrying the higher of up to €15M or 3% of global turnover. The frequently cited €35M/7% is Tier 1, reserved for prohibited practices, and does not apply to Article 10. And since penalties only bite for violations after the obligation applies, a "fines from August" framing does not hold. Article 10's application date is 2 December 2027, and the penalties are for violations after that.
To sum up: the evidence infrastructure is already a product, the ways to measure the requirements are in hand, and the cost has been calculated. What is missing is not time but a start. For an organization that uses the 16-month deferral as a window for instrumentation, December 2027 is not a deadline but the point at which the gap to competitors opens. Because audit evidence cannot be produced retroactively, the difference between those who instrumented the pipeline first and those who scramble to reconstruct it at the deadline is, by that point, already irreversible. This is what it means for regulation to become a barrier to entry. For the prepared, regulation is not a cost but a moat.
Why Pebblous Is Watching
The standing of the phrase "AI-Ready Data" shifts right here. Whether data was fit for training, representative, and checked for bias used to be the voluntary concern of a team trying to push performance higher. Article 10 promotes that concern to a legal requirement. What was a marketing slogan — "data ready for training" — becomes a state you have to prove in front of an auditor.
From a data-quality standpoint, this shift matters especially because quality is decided upstream. Label noise, class imbalance, and data gaps carry over into bias in a model's internal representations. The label-to-model amplification causation that Sap et al. demonstrated evidences that path, and the disaggregated subgroup evaluation of Model Cards confirms the result after the fact. Article 10 nails down exactly the origin of this causation — examine, mitigate, and document bias at the training-data stage. In effect, the regulation redefines data quality from a problem of performance optimization into a problem of compliance, legislating the proposition that "quality is decided upstream."
For customers releasing high-risk AI in the EU and for labeling-vendor partners, this story is an operational burden. What an auditor asks for is not a document but evidence that reflects what actually ran in production. Table 1 and Table 2 of this report move that burden onto two checklists. Confirming which evidence each stage of the labeling workflow must leave behind, and which clause of Article 10 each data diagnostic metric maps to, is the first step of preparation. Tools that auto-generate diagnosis and documentation as byproducts of the pipeline — rather than reconstructing them in a scramble at the deadline — become a direct source of demand here.
Editor's Note. If a labeling vendor leaves its work history as logs, those logs answer "what did we do." The question that remains is "does the result meet the requirements." Diagnosing the quality, representativeness, and bias of the result to build the quantitative evidence for Article 10(3) and 2(f)–(h) is the answer to that question. Pebblous is watching this subject because the question DataClinic has long targeted — how to prove, through diagnosis, that a dataset is representative and has been checked for bias — overlaps precisely with the individual requirements of Article 10. This does not replace labeling itself; it holds in the adjacent position of diagnosing what labeling produces.
References
This report was written by cross-checking the primary regulatory texts, academic papers, standards, and industry documents below. For application timing, the Digital Omnibus (final Council approval on 29 June 2026) was taken as the primary reference.
Legislation & Primary Regulation
- 1.EU AI Act (Reg. (EU) 2024/1689), Article 10 — Data and data governance. Link
- 2.EU AI Act, Article 99 — Penalties. Link
- 3.European Commission, "Regulatory framework for AI" (timeline reflecting the Digital Omnibus). Link
- 4.Council of the EU press release (2026-05-07), "Council and Parliament agree to simplify and streamline rules." Link
Academic — Documentation, Label Quality, Bias
- 5.Gebru et al., "Datasheets for Datasets," arXiv:1803.09010 (2018; CACM 2021). Link
- 6.Bender & Friedman, "Data Statements for NLP," TACL 6:587–604 (2018). Link
- 7.Mitchell et al., "Model Cards for Model Reporting," FAccT 2019, arXiv:1810.03993. Link
- 8.Northcutt, Athalye & Mueller, "Pervasive Label Errors in Test Sets," NeurIPS 2021, arXiv:2103.14749. Link
- 9.Sap et al., "The Risk of Racial Bias in Hate Speech Detection," ACL 2019, P19-1163. Link
- 10.Sap et al., "Annotators with Attitudes," NAACL 2022. Link
Standards
- 11.ISO/IEC 5259-1..5 (2024–2025), JTC 1/SC 42 — Data quality for analytics and ML (in particular -4 labeling process, -2 data quality characteristics).
- 12.ISO/IEC 25012:2008 & 25024:2015 — SQuaRE data quality model and measurement.
Industry & Practice
- 13.Label Studio, "Operationalizing Compliance with the EU AI Act's High-Risk Requirements." Link
- 14.Encord, "What the European AI Act Means for You." Link
- 15.Bird & Bird, "European Union Artificial Intelligence Act Guide" (2026). Link
- 16.Holistic AI, "AI Regulation in 2026: Navigating an Uncertain Landscape." Link
※ Statistics on market size, compliance cost, and readiness (TBRC, CEPS, Deloitte) differ in scope of definition and survey timing across institutions, so sources and caveats are noted alongside them in the body.