Executive Summary

Starting July 30, 2026, Amazon will no longer accept new customers for Mechanical Turk (MTurk). On the same day, SageMaker Ground Truth and Amazon Augmented AI will also close to new users. The original crowdsourced labeling service, launched in 2005 under the slogan "Artificial Artificial Intelligence," is effectively retreating into maintenance mode after 20 years. And this exit is more than one service closing its doors: it strikes directly at the provenance of AI training data.

The point is not the shutdown itself but what sits behind it. In 2023, one study estimated that 33–46% of Mechanical Turk workers completed a writing task using an LLM. Researchers believed they were entrusting judgment to people and collecting human signal, when in fact they were re-collecting a machine's guesses. The seat where humans once pretended to be AI has now been quietly taken over by real AI.

So the question this event raises is not about data quality but about data provenance. Once you can no longer prove who or what produced a label, the accuracy and agreement metrics you would use to judge that label lose their meaning before you even reach them. Provenance breaks before quality. Mechanical Turk's exit shows that order in plain sight.

33–46%

Estimated LLM use by workers

Abstract summarization task · Veselovsky et al. (2023)

20 years

Mechanical Turk's lifespan

Launched in 2005 as 'Artificial Artificial Intelligence'

Jul 30

New-customer cutoff date

Ground Truth · A2I close the same day

Nov 2022

The provenance inflection point

Data gathered after ChatGPT carries a question mark

1

A 20-Year Exit

Amazon said it will stop accepting new customers (requesters) for Mechanical Turk starting July 30, 2026. Existing customers can keep using it, but there are no plans to add new features — effectively, maintenance mode. On the same day, the data-annotation service SageMaker Ground Truth and the human-in-the-loop review pipeline Amazon Augmented AI (A2I) will also stop onboarding new customers. This is not a problem confined to Mechanical Turk alone; it is Amazon winding down its entire "human labeling" business line.

Mechanical Turk arrived in 2005. Its slogan was "Artificial Artificial Intelligence." Jeff Bezos coined the phrase himself, and it compressed the nature of the service into a single line. The model outsourced small, hard-to-automate tasks to people for tiny payments, and it called each unit of work a HIT (Human Intelligence Task). On the surface it looked like an automation service, but behind it real people were doing the answering — a twist baked into the name that was, itself, the business model.

18th-century engraving revealing the human chess player hidden inside the automaton known as The Turk
▲ The original "Turk" that Amazon borrowed its name from — a chess-playing automaton that dazzled Europe in the 1770s, with a person hidden inside the cabinet | Source: Wikimedia Commons (Public Domain)
Twenty Years of Mechanical Turk: From Slogan to Sunset 2005 Launched as 'Artificial AI' 2018 SageMaker annotation shift to AI-training labels 2026 Jul 30: stops new customer signups Source: TechCrunch, the-decoder (2026) | Pebblous original diagram
▲ A platform that began with CAPTCHAs and surveys shifted its center of gravity to AI-training labels in 2018, and closes to new customers in 2026 | Pebblous original diagram

The decisive turn came in 2018. As Mechanical Turk was repackaged as part of SageMaker's data-annotation pipeline, its primary use shifted from CAPTCHAs and survey responses to "labeling for AI training." By then the early community was already suffering from bots and low-quality responses. One user described Amazon's decision as finally letting an old friend rest — someone who had passed their expiration date long ago. The substantive decline, in other words, began years before the announcement.

2

What 33–46% Revealed

Why the platform could not hold its ground was pinned down by a 2023 paper. Veselovsky and colleagues published "Artificial Artificial Artificial Intelligence" (arXiv:2306.07899). The fact that "Artificial" gains one more repetition in the title is itself the metaphor for this event: on top of a structure where humans pretended to be AI, those humans now quietly send in real AI as their stand-in.

The researchers estimated whether workers used an LLM on a real Mechanical Turk abstract summarization task using two methods. One was keystroke detection, which watches typing patterns; the other was a classifier that flags LLM-generated text. Combining the two signals, they estimated that 33–46% of workers used an LLM to complete the task.

Circular contamination: reaching for human signal, collecting machine guesses Researchers · AI labs "We collect human judgment" Crowdworkers Up to 46% answer via LLM Task assigned (HIT) Machine guess returned LLM output is recorded as human signal in a 'human-label' dataset → fed back into the next model's training, so the contamination loops Source: Veselovsky et al., arXiv:2306.07899 (2023) | Pebblous original diagram
▲ When a task outsourced to humans comes back from an LLM, that output is recorded as a "human label" and loops into the next round of training | Pebblous original diagram

This number is alarming not because it is a rate of cheating, but because it creates a loop. Researchers believe that by assigning tasks to people they are collecting human judgment. But when a large share of those people generate answers with an LLM and send them back, the dataset fills up with machine guesses wearing a "human label" tag. That data then trains the next model, and that model becomes the answer to yet another task. A pipeline meant to gather human signal quietly turns into a device for recycling machine output.

The authors were clear about the limits. It is not certain that this estimate generalizes to other tasks where using an LLM is harder. Yet the direction of the conclusion was unmistakable: platforms, researchers, and workers alike need a new way to keep human data genuinely human.

3

Provenance Breaks Before Quality

When we talk about data quality, we usually separate two things. One is quality, which asks whether this label is accurate. The other is provenance, which asks who or what produced this label, and through what process. Most data management focuses on the former. Accuracy, inter-annotator agreement, review pass rates — these are all the language of quality.

What do you ask first: quality or provenance? Quality "Is this label accurate?" Accuracy · agreement · pass rate Loses meaning when provenance falls Provenance "Who or what made it?" Origin · verification history · traceability More fundamental, and breaks first Pebblous original diagram | When provenance is shaken, quality metrics lose their footing
▲ Quality asks about a label's accuracy; provenance asks about its origin. What the Mechanical Turk event shook is the latter | Pebblous original diagram

What the Mechanical Turk event shook is the latter — provenance. The moment an answer a worker made with an LLM is recorded as a "human label," it no longer matters how accurate that data is. If what we bought believing it was human judgment was in fact a machine's guess, then the agreement or pass rate we compute on top of it becomes a number whose referent we can no longer name. When provenance is shaken, quality metrics lose their footing. That is what it means to say provenance is more fundamental than quality, and that it breaks first.

Timing makes the problem worse. Any data gathered on Mechanical Turk after ChatGPT's public release in November 2022 — whether an academic NLP benchmark or an RLHF preference dataset — carries an unresolved question: "Is this really a human signal?" There is no ready way to retroactively determine when, and on which task, a worker used an LLM.

Quality can be fixed later. A wrong label can be reviewed again. Provenance is different. If you never recorded, in the first place, whether an answer came from a person or a machine, no amount of careful review can reconstruct it after the fact. That is exactly why, in AI-Ready Data, provenance has to stand before quality.

4

What Cannot Be Undone, and What Is Now Being Born

The contamination of data already collected is, for all practical purposes, hard to undo. Retroactively deciding which responses came from humans and which from LLMs cannot be done at a trustworthy level of accuracy unless the origin was recorded in the source. Many datasets that leaned on Mechanical Turk after November 2022 will have to live with this question mark. That does not mean the models trained on them perform badly; it means we can no longer fully explain the basis of that performance.

On the other hand, something new is being born too: the market value of verified human feedback. AI labs are already moving away from anonymous crowdsourcing toward specialized labeling vendors such as Scale AI and Surge AI. Their differentiator is not that they are cheap, but that they employ people whose identity, expertise, and track record are verified, and hand them the difficult edge cases and quality judgments. The very condition of being a verified human has become the product.

So reading Mechanical Turk's exit as "human labeling is over" is only half right. More precisely, it is a redefinition: the era of anonymous, cheap, unverified human labeling is over. Human judgment is still needed. What is now required alongside it is the ability to prove that the judgment really came from a human.

The question AI-Ready Data has to ask moves up by the same measure. Before "Is this data accurate?" comes "Can we prove who or what produced this data?" Recording provenance, tracing the producing agent, and attaching a verification history to the data become the next standard. At the bottom of the accuracy race, proof of provenance has to take its place first.

Editor's note. The concern that unverified data provenance destabilizes the quality metrics built on top of it is a consistent lens through which Pebblous has approached AI-Ready Data. Because the argument we have made about labels in training data resurfaces here in the form of a crowdsourcing platform's shutdown, we record it as a standalone piece. If you are interested in a perspective that diagnoses data provenance and label quality, we suggest looking at DataClinic.

Frequently Asked Questions

Exactly when and how is Amazon Mechanical Turk shutting down?

Starting July 30, 2026, it will stop accepting new customers (requesters). Existing customers can keep using it, but there are no plans for new features, so it effectively moves into maintenance mode. On the same day, SageMaker Ground Truth and Amazon Augmented AI (A2I) also close to new customers, which reads as Amazon winding down its whole human-labeling business line.

What does "Artificial Artificial Intelligence" mean?

It was Mechanical Turk's 2005 launch slogan. On the surface the service looked automated, but the twist was that real people were doing the work behind it. The model broke hard-to-automate tasks into small pieces and outsourced them to people for tiny payments, calling each unit of work a HIT (Human Intelligence Task).

Where does the figure that 33–46% of workers used an LLM come from?

It is an estimate from the 2023 paper "Artificial Artificial Artificial Intelligence" (arXiv:2306.07899) by Veselovsky and colleagues. On a real Mechanical Turk abstract summarization task, they combined keystroke detection, which watches typing patterns, with a classifier that flags LLM-generated text, and estimated that 33–46% of workers used an LLM.

What is "circular contamination"?

You assign a task to people believing you are collecting human judgment, but if those answers were in fact produced by an LLM, machine guesses pile up in the dataset under the name "human label." That data trains the next model, and that model becomes the answer to another task, so the contamination loops. A pipeline meant to gather human signal turns into a device for recycling machine output.

How are data "quality" and "provenance" different?

Quality asks "Is this label accurate?"; provenance asks "Who or what produced this label, and through what process?" Accuracy and agreement are the language of quality. Once provenance breaks — meaning data you thought was human judgment was actually machine output — the quality metrics computed on top of it no longer tell you what they measured.

Why is data from after November 2022 a problem?

That is when ChatGPT was released publicly. Data gathered on Mechanical Turk afterward, whether an academic NLP benchmark or an RLHF preference dataset, carries the unresolved question of whether it is really human signal. Because there is no ready way to retroactively determine when a worker used an LLM, it is hard to cleanly remove the contamination after the fact.

Does this mean human labeling is no longer needed?

No. What has ended is anonymous, cheap, unverified human labeling. Human judgment is still needed, and AI labs are moving toward specialized vendors such as Scale AI and Surge AI that use people with verified identity and expertise. The very condition of being a verified human is becoming a scarce resource and a product.

What is the lesson of this event from an AI-Ready Data perspective?

That before "Is this data accurate?" you have to ask "Can we prove who or what produced this data?" Recording provenance, tracing the producing agent, and attaching a verification history to the data become the next standard. Quality can be fixed after the fact, but provenance that was never recorded cannot be reconstructed later.

R

References

Academic Paper

Industry & Press