ARC-AGI-3 announces the largest human test in history: all levels have been conquered by humans, AI still has gaps

ME News, April 15 (UTC+8), according to Beating Monitoring, ARC Prize Foundation announced the human performance dataset for ARC-AGI-3, which is the largest human testing study in the ARC-AGI series to date, with 458 participants. The dataset includes 342 complete human operation playback records, covering 25 public environments, all open-sourced. ARC-AGI-3 contains 135 abstract reasoning environments, where testers receive no instructions, and must explore, infer rules, and develop strategies on their own. The tests are conducted at an offline testing center in San Francisco, each lasting 90 minutes, with participants earning about $130 base pay plus $5 for each environment they pass. All tests are “first-time pass” conditions, meaning each person only sees the environment once and attempts it once, measuring learning and adaptation abilities when facing entirely new problems. Humans and AI are given exactly the same information, with no information gap.
Core conclusion: All environments in ARC-AGI-3 are passed by humans, with at least two independent participants completing each environment, and most environments being passed by more than five people. The ARC Prize Foundation states, “We have not yet achieved AGI; this dataset is evidence.”
Since the preview of ARC-AGI-3, nearly one million AI evaluation submissions have been received for the open environments. Based on this data, the foundation also announced two scoring rule adjustments: first, changing the human benchmark per level from the “second-best player” to the “median player” to reduce luck’s influence on scores; second, increasing the maximum score per level from 100% to 115% to prevent poor performance on one level from dragging down overall results. The net effect of these adjustments is a slight increase in scores for both humans and AI, about 0.5 percentage points. (Source: BlockBeats)