What You’ll Find This Week

Amazon is shutting down Mechanical Turk on September 30. For 21 years, Mechanical Turk was where you went to buy human judgment in bulk at a few cents a task, and the people building modern AI were among its heaviest users. The image dataset that kicked off the deep learning boom was labeled by roughly 49,000 Mechanical Turk workers.

Paying humans to train AI has never been a bigger business than it is now. Mercor, a marketplace that recruits doctors, lawyers, engineers and PhDs to produce training data for the big AI labs, pays its specialists upwards of $100 an hour and was in talks in July to raise money at a $20 billion valuation. The median wage on Mechanical Turk was about $2 an hour.

This week: what Amazon was actually selling for 21 years, why researchers called Mechanical Turk a market for lemons two years before ChatGPT arrived, and why the only grade the platform kept ended up rewarding the thing that killed it.

Here’s what you’ll find:

Don’t Miss Our Latest Podcast

This Week’s Article

In 2005, Amazon launched a marketplace where you could rent a person by the task. Jeff Bezos called it "artificial artificial intelligence," and Amazon named the service after the Mechanical Turk, Wolfgang von Kempelen's 1770 chess machine that beat its opponents because a man was folded inside the cabinet working the levers.

Mechanical Turk was built for work computers couldn't do and no company would staff a department to handle. Look at one photograph, say whether there's a cat in it, then do that ten thousand more times.

In 2007, a Princeton researcher named Fei-Fei Li needed exactly that. She was assembling ImageNet, a library of millions of labeled photographs that became the training material for the first generation of image-recognition models, and her team calculated that having undergraduates verify the images would take nineteen years. Roughly 49,000 Mechanical Turk workers across 167 countries did the job instead, and the first version of ImageNet was published two years later.

Seventeen years after launch, the same marketplace was still supplying human judgment to the companies building the most advanced models. When Anthropic assembled the public dataset behind its red teaming work, the practice of paying people to try to provoke bad behavior out of a model, some of those people came from Mechanical Turk.

Amazon is closing the marketplace on September 30. The company's entire public explanation: "We regularly evaluate our programs, tools, and services and make adjustments based on those assessments. Following an assessment, we've made the decision to close AWS Mechanical Turk, effective September 30, 2026."

Paying humans to train AI has never been a bigger business. In June 2025, Meta paid $14.3 billion for 49% of Scale AI and hired its founder. Surge AI, which supplies human contractors to train machine learning models, passed $1 billion in revenue without taking a dollar of venture money, selling to Google and OpenAI. Mercor, which recruits doctors, lawyers, engineers and PhDs to produce training data and pays its specialists upwards of $100 an hour, raised a $350 million round at a $10 billion valuation last October and was in talks by July to raise again at $20 billion.

All three broker access to people who supply judgment a model can't supply on its own, which is what Amazon brokered for 21 years. Amazon created that market and is walking out of it while the companies that followed are worth tens of billions.

Here is how AWS describes its cloud servers today:

And here is how AWS describes Mechanical Turk today:

One noun changes. The rest of the sentence, and the rest of the product, works the same way. A task is "a single, self-contained task." The requester "pays the Workers for satisfactory work (only)." IEEE Spectrum's history of the platform describes communication between the two sides as "entirely depersonalized," on a service "designed to make human labor invisible."

Fei-Fei Li didn't have to hire, interview, train, or manage 49,000 people to build ImageNet. She had to write a task and fund it.

The work paid accordingly. A 2018 study at the ACM CHI conference tracked 2,676 workers across 3.8 million tasks and found that the median worker earned about $2 for an hour of work, counting the unpaid time spent hunting for tasks and redoing rejected ones. 96% of workers earned less per hour than the federal minimum wage of $7.25. Pew found that 61% of tasks on the site paid ten cents or less.

Price was the only dial anyone could turn.

A requester picking a worker had one number to go on, which was approval rate: the share of that worker's past submissions requesters had accepted. Not how good the work was. How often somebody had clicked approve. A worker who did the job twice as well as the next one had nothing to show for it and no way to charge more, and a requester who wanted better work had nothing to buy. The marketplace had no way to grade anybody’s output.

The grading problem showed up in the very first big job. Fei-Fei Li's team had to write statistical models of worker behavior to catch people clicking through the ImageNet tasks without looking at the images.

By 2018 the cheating was organized. Ryan Kennedy and five co-authors audited 38 studies covering 24,930 respondents and found that 16.7% of the respondents in the later studies were fraudulent, mostly people outside the country running virtual private servers to qualify for work restricted to Americans. Three years after that, Douglas Ahler, Carolyn Roush and Gaurav Sood put the diagnosis in a title, "The Micro-task Market for Lemons," after George Akerlof's model of what happens to a market when buyers can't check what they're buying.

Requesters built their own defenses. They asked three workers the same question and took the majority answer. They planted questions with known answers. They restricted work to accounts with high approval rates.

Every one of those defenses assumes the workers are answering independently.

In June 2023, three researchers at EPFL ran keystroke detection on an MTurk summarization task and estimated that 33 to 46% of the workers were using large language models to produce the work, six months after ChatGPT opened to the public. They titled the paper "Artificial Artificial Artificial Intelligence."

Three workers who paste the same task into the same chatbot agree with each other perfectly. They pass the planted question. Their writing is cleaner than the humans'. And their approval rate climbs, because approval measures whether a requester accepted the submission, not whether a person wrote it. The only grade Mechanical Turk kept was now rewarding the substitution that made its product worthless.

Follow that to the buyer. An AI lab pays for human judgment because a model can't produce it. If a third to a half of the answers were sourced from a model, the lab is effectively paying a premium for its own technology's output and can't tell which answers are which. Nature published research in 2024 finding that models trained on recursively generated data degrade.

When a buyer can't tell good work from bad, the buyer stops paying for good. The labs didn't offer Mechanical Turk more money. They went to suppliers who could show them, worker by worker, what they were buying.

Amazon saw the market moving. On November 28, 2018, at re:Invent, it launched SageMaker Ground Truth, a managed labeling service for machine learning teams, and Andy Jassy called it "a total game changer in being able to label your data." Customers could send the work to their own employees, to third-party vendors, or to Mechanical Turk.

Amazon rebuilt the buying experience, but they left the labor market underneath it exactly as it was.

A better grade was available the whole time. Ahler, Roush and Sood found that restricting a study to workers with 1,000 or more completed tasks cut low-quality responses from 35% of a sample to 10%. Worker track record predicted quality, and Amazon was already counting every worker's completed tasks. They simply never turned that count into a product.

Building that product meant doing the work Mechanical Turk existed to avoid. Someone has to test applicants, rank them, keep score across projects, and charge more for the ones who score higher. That is a staffing business, with recruiters, account managers, and headcount that grows every time revenue does. Mercor runs exactly that business. Amazon built the opposite on purpose: a marketplace that carried more than 500,000 workers without Amazon hiring anyone to manage a single one.

Ground Truth shuts down on September 30 alongside Mechanical Turk.

Anthropic kept a grade on every record. The red-teaming data it released publicly carries a field called is_upworker, a binary flag marking whether that person came from Upwork or from Mechanical Turk. Which pool someone came out of was worth storing next to their answer, because it indicated something about the answer.

OpenAI graded before hiring anyone. For the human-feedback work behind InstructGPT, published in March 2022, it "hired a team of about 40 contractors on Upwork and through ScaleAI," chosen with a screening test built to measure the specific judgment the job required. Forty people, all tested first.

Both labs were buying the same thing, which is a worker whose quality is known before the work starts. That is what Scale, Surge and Mercor sell. Mercor can tell a customer that the person answering a clinical question is a physician, and can tell them how that physician's earlier work scored. Mechanical Turk could tell you how often a stranger's submissions had been accepted.

A marketplace like this one does more than supply labor. It produces a record of which tasks people keep paying for, how often, and at what price, which is the first document an automation project needs. The same thing is running now in the marketplaces where AI systems hire humans for physical work, where every completed job is a specification for the machine that eventually does it.

Mechanical Turk ran that arc end to end over 21 years. No robot arrived. The workers opened a browser tab, pasted the task into a chatbot, submitted what came back, and watched their approval ratings hold.

Amazon spent 21 years selling work that nobody could grade. That was survivable while buyers wanted volume at the lowest price and could absorb garbage in the results. It stopped being survivable when the richest buyers in the category started paying for graded work, because approval rate was the only grade Mechanical Turk kept, and by 2023 a worker using a chatbot scored well on it.

Scale, Surge and Mercor are worth what they're worth because they sell the grade. Mercor's specialists earn upwards of $100 an hour. Mechanical Turk's median was about $2. Buyers are paying that gap to stop guessing.

So look at your own pricing. Can you charge more for better work, and can you show a customer which of your work is better? If you can't do both, price is the only thing you're competing on, whether or not that was the plan.

The market Amazon created is worth tens of billions of dollars, and every company serving it can grade its own workers. But Mechanical Turk closes on September 30.