A Brief History of AI Victories: Where There Is a Scoring System, There Is an AI Invasion

Author: Robonaissance

Translated by: Deep Tide TechFlow

Deep Tide Quick Read: Go, chess, protein folding—why does AI crack these fields one after another, even though they seem to require human intuition and creativity? This article reveals a neglected rule: what determines whether AI can surpass humans is not the difficulty of the task, but whether you can score success. Once a domain establishes scoring criteria, it sets a countdown for its own takeover.

Selected from the “Whatever You Ask For” series, about the one job that machine can’t do for us.

One ten-thousandth

On March 10, 2016, in a hotel in Seoul, a machine played a move that everyone in the room couldn’t understand, while its opponent was not present.

Lee Sedol went out to have a cigarette. He was 33 then, holding 18 world champion titles, widely regarded as the strongest Go player of his generation. He had lost the first game the day before; by the time he entered the second game, his confidence before the match had shrunk considerably, and his mindset became more cautious. While he was outside, AlphaGo chose move 37. DeepMind researcher Hwang Sijie substituted for the machine, quietly placing the stones on the board.

The stone landed on the fifth line.

To those who don’t know Go, this means nothing. To those who do, it was almost absurd. In the opening and middle game, positions that far from the edge were considered inefficient: you neither gain territory nor defend effectively, which is essentially handing free points to your opponent. It’s the kind of move a teacher corrects beginners with. During the live broadcast, top pro Michael Redmond initially thought there was a signal error on the board. He picked up a stone and put it back down.

Lee Sedol returned and sat down, staring at the board for a long time without moving. Explanations ranged from about 12 minutes to 15 minutes. Five months earlier, the European champion Fan Hui—watching from inside the building—after watching for that long judged: “This isn’t a human move. I’ve never seen anyone play it like this.”

He was right—and the rightness could be quantified. Inside DeepMind’s system was an estimation mechanism: by learning from huge numbers of human games, it estimated the probability that a human would play a given move in any position. For move 37, that estimated value was about one ten-thousandth.

The machine still played it—and that move won the whole game.

There’s a comforting interpretation of the story: the machine worked very hard to learn human masters, and eventually caught up to the strongest among them. That interpretation is wrong, and the number one ten-thousandth is exactly why it’s wrong. A learning-and-imitation system has a ceiling, and the ceiling is human moves. What happened in Seoul was that a system was no longer constrained by that ceiling, because the goal it was given wasn’t human approval—it was winning.

That distinction points somewhere more useful than admiration. If you want to know which human activities have already been lost purely in terms of capability, and what’s next, the questions aren’t how difficult the activity is, how much creativity it requires, or how much intuition. The question is much less boring.

The question is whether success can be scored.

One game after another

Chess was already beaten 19 years ago, but in a completely different way.

Deep Blue beat Kasparov in 1997 using an architecture that was, in essence, a huge act of human expression. Its evaluation function—components that look at a position and output how good it is as a number—was assembled and tuned with help from grandmaster advisors. Human chess knowledge was painstakingly extracted from human players and written into the machine. What the machine adds is search capability: faster lookahead over more moves, tireless, never dropping stones due to attention drifting.

That was a real victory, and it should count as a victory. But pay attention to its form. Deep Blue’s understanding of chess was human understanding. Its advantage was speed. If you asked why it evaluated a position that way, the answer ultimately traced back to some person who told it to do so.

Twenty years later, DeepMind released a system called AlphaZero—its form changed.

AlphaZero was given the rules of chess. It was not given an opening book—an initial-move directory that every serious program depends on. It was not given endgame tables. It was not given a single human game to play through. Instead it played against itself, starting from random moves, adjusting itself based on what could win.

After about four hours, DeepMind estimated its Elo rating surpassed the strongest traditional program of the time, Stockfish 8. After about nine hours, under time limits, it played 100 games against Stockfish and won 28, losing none; the remaining 72 were draws. The paper reported that within 24 hours, it reached superhuman levels in chess, shogi, and Go.

When these numbers are cited, they often omit the other half of the story, and that other half matters. The games generated from self-play were produced on 5,000 first-generation tensor processing units, and another 64 second-generation units trained the networks, with everything running in parallel. Four hours of wall-clock time equaled four hours of computation—an amount no individual, and even very few institutions, could assemble. The achievement isn’t that learning chess is easy. The achievement is that when you have that much compute, human knowledge is no longer something you need.

Kasparov had every reason to treat this result as a personal attack more than any living person could, and yet he wrote a generous commentary in Science. His observation was that AlphaZero didn’t play the kind of chess that everyone would assume a perfect machine—boring, cautious, preference for draws—plays. It preferred activity over material, sacrificing pieces for positions that, in his view, looked risky. He pointed out that traditional programs carry the priorities and biases of the people who wrote them. AlphaZero wrote itself. He concluded that its style therefore reflected something more real than programmer taste, and observed that it checked far fewer positions per second than the best traditional engines in the world—yet it won.

Finally, remember this detail. Traditional engines can search so many more possibilities per second and yet they lost. AlphaZero’s advantage wasn’t looking harder; it was knowing where to look. And this knowledge was assembled from millions of self-play games and a rule about who wins—nothing else.

The shogi result is stranger to those qualified to read it. Shogi is a Japanese board game where captured pieces return to the capturer’s side, making the volatility of positions higher than in chess. It also has centuries of accumulated theory on how to protect the king. After watching the machine’s games, strong players reported that in opening moments it violated known theory, and then—at the moment every book says the king should tuck into the corner—the king ran to the center of the board. These games are almost impossible to interpret for trained eyes, but they won.

Lay the two systems side by side and the pattern becomes so clear it’s unsettling. Deep Blue is human knowledge plus machine speed. AlphaZero is the definition of machine speed plus winning, with human knowledge deliberately removed. The version with human knowledge removed is better.

None of this means the 2016 machine was flawless—evidence from that same Seoul match remains.

In Game 4, after going three behind and fighting for dignity, Lee Sedol wedged a stone between two AlphaGo stones at the center of the board. It later became known as the Move of God. AlphaGo’s own estimate of the probability that a human would play that move was also about one ten-thousandth—marvelously symmetrical. The system couldn’t respond. Its evaluation of its win probability collapsed in the next few moves: its playing strength deteriorated and it lost that game.

So in March 2016, there was still a hole in the machine—one that a human found under the world’s gaze and maximum pressure. It’s worth stating clearly, not quietly skipping over. Worth stating too is what happened later: these kinds of holes were steadily patched; the system’s successor no longer lost games like this; in serious settings, top engines had not lost to humans at all for a long time. The 2016 result was a snapshot of a transition, not a permanent balance of power.

From these chessboards to everything else, what transferred wasn’t victory itself, but the mechanism. Chess and Go were destined to fall first for an awkwardly simple reason: in games, success is defined with perfect precision by the rules. You win or you don’t; scoring is free, instant, and uncontroversial. A system can play 40 million games against itself over a weekend and receive 40 million unequivocal verdicts.

This points to the next place to look. Not those simple tasks, but tasks that come with an embedded scoreboard.

A fifty-year problem

Proteins are chains of amino acids that fold into complex three-dimensional shapes the moment they’re made. The shape determines what the protein does. The sequence determines the shape. Deriving the latter from the former has been an open problem in biology since roughly the early 1970s—and it has remained open in a way that resists every attack: from first-principles physical simulations, to statistical analyses of evolutionary relatedness, to decades of accumulated structural intuition—none work.

In 1994, a team led by John Moult did something about this fact: everyone in the field was claiming progress, yet no one could verify it.

They created an exam. Every two years since then, CASP—Critical Assessment of Structure Prediction—releases protein structures that have just been experimentally determined, and in some cases are even still being determined, publishing the sequences to the world. Any team can submit predictions. No one can access the answers, because for some targets the answers don’t exist yet. When the experimental structures come out, the predictions are scored against them.

Scoring uses a metric called the Global Distance Test, from 0 to 100, which can be roughly understood as the percentage of how close the chain’s final position is to the actual position. Moult once said that around 90 points is informally viewed as being equivalent to structures determined in the laboratory. In most of the history of the evaluations, the best predictions hovered around 60.

At CASP14 in 2020, a contestant registered as Group 427 had a median score of 92.4 across all targets. In the hardest category—targets for which no useful structural homology could be relied on—the median score was 87.0. The average error was about 1.6 angstroms, roughly the width of an atom.

Moult had served as chair from the start of the competition. Before showing the graphs, he walked the audience through the history of the contest. The graphs tell the whole story with a single picture. Lines crawled around 60 for twenty years, and then one line stood somewhere the others had never been.

Group 427 was AlphaFold2. Moult announced that for a single protein chain, the problem is solved.

The caveat should be there, in every honest narrative. A single chain isn’t the whole of protein science. How proteins assemble into complexes, how they move, how they behave inside living cells—these are still open. The grand challenge that’s “closed” is a challenge stated in a specific, precise way.

And it’s precisely that precision that matters for the story. What knocked down structure prediction wasn’t that it was originally simple—half a century of failure says otherwise. What knocked it down was that in 1994, the field built a scoreboard for itself.

What you get when you view CASP as engineering rather than science is clear. It provides an unambiguous measure of success, applicable to any prediction, computable in seconds. It provides a benchmark fact that can’t be gamed, because the answers are physically determined by other people in laboratories. It provides a stream of new problems generated according to a fixed schedule. It provides three decades of scoring attempts—meaning the grading record distinguishes what counts as better and what counts as worse in that field.

A field that does all of this, inadvertently, prepares itself for automated attacks. A scoreboard is the prerequisite. Everything else is engineering and computation—and both have, for a long time, been getting cheaper every year.

And the unsettlingly casual generalization follows. Whenever a discipline reaches a baseline consensus, it publishes a target. Machine translation has a scoring benchmark. Speech recognition has a scoring benchmark. Image classification has a labeled dataset of a million photos and an annual competition, and anyone who has watched that competition knows how the story ends. The pattern isn’t that the hard things fall first, or that the simple things fall first. The pattern is that what gets measured falls first.

This raises an obvious question for everything not yet measured.

Score what can’t be scored

The last line of defense was supposed to be the things that couldn’t be scored.

Writing is a standard example. No program can take a paragraph and return a number saying how good it is. Two competent editors disagree. The same editor disagrees with themselves on different days. Language quality is entangled with context, audience, purpose, and taste—none of which can be reduced to a measurement. If a machine needs a scoreboard and language doesn’t have one, then language is safe.

That argument holds. What happens to it is the single most important cause in this entire narrative.

The field never found an objective metric for good writing. Still doesn’t have one. What it did was manufacture a score from the only available material: human judgment itself.

The history of this approach is longer than most people think. In 2008, Knox and Stone described a system called TAMER that let humans observe agents’ behavior and give evaluations, which were then used to train a model to predict what humans would say. Then the model, rather than humans, produced the training signal. In 2017, Christiano and colleagues published work that most people today see as a direct origin of current practice, applying the idea to agents in Atari games. In 2019, Ziegler and colleagues applied it to language models, and most terminology used today was established in that paper. In 2022, Ouyang and colleagues demonstrated the approach at industrial scale on instruction-following tasks, and shortly after that most people on Earth encountered these results without knowing it.

The core mechanism is worth understanding specifically, because it’s simpler than its reputation makes it sound.

A person sits in front of two pieces of text, and both texts are generated by the model from the same prompt. The task isn’t to score them. Scoring is exactly what humans do very badly, because a seven for one person is a five for another, and the same person is inconsistent within an afternoon. The task is simply to choose which one is better. This is the kind of judgment humans can make reliably, and the mathematics method that turns a pile of such comparisons into a consistent scale is older than the field using it, tracing back to Bradley and Terry’s 1952 work on paired comparisons.

Consider the working conditions under which such judgments are produced, because they end up mattering. This person has to make lots of such judgments per hour, gets paid for them, works from a written guideline comparing what the client thinks is the better answer. Some pairs are almost identical. Some involve topics the judge knows nothing about—in those cases, the more confident and better-organized of the two answers is often chosen, regardless of whether it’s more accurate. This isn’t a critique of those people. It’s a description of what anyone would do under those conditions, and the resulting choices are the raw material.

Collect enough of those choices, and you can train a second model whose job is to predict which of two text samples humans will prefer. This second model outputs a number. And a number is exactly what had been missing all along.

Look carefully at what was built here, because it’s easy to overlook. The optimized score isn’t about facts in the world. It’s about our models. It’s a compressed, learned imitation of the judgments of a particular group of people who were hired at a particular time, working under specific instructions, just like everyone else getting tired on an afternoon. CASP scores predictions against physical reality determined by laboratories; this system scores text according to a statistical profile of what humans accept.

That profile is useful. But it must also be an approximation, with differences between it and the object being depicted, and no one fully understands those differences. Anything about our real preferences that the preference model fails to capture isn’t in the target, so it isn’t optimized—and thus it can be affected by any circumstances, because a powerful optimizer has no reason to protect that quantity.

This is about construction facts, and making any claims about how bad the consequences are would be missing the point. The point here is narrower and harder to refute. The category of things that can’t be scored is smaller than it seems. If a field resists measurement, you can build a measurement from human preferences and move forward. This defense works only when nobody has thought to manufacture a score.

What will fall next

This leaves a real-world question, and a usable answer.

Pick any activity you like: a job, a craft, a profession, a task you spent years learning to do well. Ask three questions.

First: Can success and failure be reliably separated? Not perfectly, and not every time for everyone, but consistently enough that capable evaluators usually agree. Games sail through this. Protein structure prediction passes too, because laboratories will eventually produce the answer. Writing fails in an absolute sense, but passes in a relative sense, because people can usually say which of two attempts is better.

Second: Can such judgments be produced at low cost and at scale? This determines timing more than possibility. A free, immediate judgment—like in games—means the system can generate tens of millions of training data by itself. A judgment that requires laboratory work is slower but still feasible, with a 30-year archive of scored attempts. A judgment that requires paying humans to read text is expensive—hence all the effort to train models that imitate these humans and remove them from the loop.

Third: What is decided isn’t when a field falls, but what happens after: is this score really what you want? Game scores are what you want, because in games winning—by definition—is the whole point. Measured experimental protein structures are very close to what you want. A model trained on a preference label isn’t what you want. It’s an image of what you want, and there’s a gap between the image and the face.

Test your own work with these three questions. Most people find the first answer comes quickly and is unsettling. Most things we call skill—at least in a relative sense—are scoreable when capable judges view two attempts side by side. The second question is where the true uncertainty lies, because cost and scale are moving targets, and they’ve been moving in one direction for a long time. The third question is one that almost nobody asks, and it decides what your field looks like on the other side.

It’s worth doing this exercise slowly on some ordinary things. Take radiology. Can success and failure be separated? Yes—and precisely—because diagnoses are eventually confirmed or refuted by patients’ conditions. Can judgments be produced at low cost and at scale? Mostly yes, because hospitals have spent decades accumulating score samples in the form of images plus outcome confirmation. Is this score what you want? Here the answer gets complicated, because what’s being scored is consistency with the recorded diagnosis, while what you want is that patients do well—these align most of the time, but they diverge in exactly the most important cases.

Now take something that looks safer. Management. The first question is already hard, because capable people disagree about whether a given manager is good, and disagreements don’t get resolved quickly. The second is harder, because the results of management decisions arrive years later, entangled with everything else that happens. The third is hardest, because any alternative metric for good management proposed by anyone—from retention rates to engagement scores to output per person—clearly isn’t the thing itself. Based on this diagnosis, management isn’t safe because it’s deep and sound. It’s unmeasured protection of a different, less pleasant kind.

Machines have occupied the optimization axis. With enough compute for any explicitly specified target, finding good moves stops being a competition. The results are often not only stronger than ours, but stranger—reaching places our traditional teaching told us not to look. This isn’t prediction. It’s a description of chess, Go, shogi, protein structures, and an ever-growing list of things that used to be exclusively human.

What remains is the target itself. Someone decides what the score means. In each of the cases above, that decision took an afternoon, and every extraordinary result originates from it, is faithful to it, and is indifferent to anything it omits.

So the final question is worth thinking about. In your work, is there already a score? If so, who wrote it, and did they think about you when they wrote it?

This is Part Two of the “Whatever You Ask For” series, about what machines ultimately need from us—and how badly we’ve done.