Augment Code actual test of AGENTS.md's impact on code generation: the best is equivalent to one level of model upgrade, the worst is worse than not writing.

ME News, April 23 (UTC+8): According to Dongcha Beating monitoring, AI programming tool company Augment Code extracted dozens of AGENTS.md files from its own monorepo, and used its internal evaluation suite AuggieBench to measure their actual impact on the output of coding agents.

The method was to take already merged high-quality PRs as a benchmark, have the agent redo the same task under two conditions—with and without AGENTS.md—and then compare the scores. The gap was much larger than expected.

The best-written AGENTS.md delivered a quality improvement equivalent to switching the model from Haiku to Opus. The worst-written ones were even worse than having no AGENTS.md at all. Moreover, the same file could have opposite effects on different tasks: it increased specification-compliance for a bug fix by 25%, yet reduced completion of a complex feature in the same module by 30%.

There are a few effective ways to write them. First, keep the main file to 100–150 lines, and pair it with a few focused reference documents—this can bring a comprehensive improvement of 10% to 15% in medium-sized modules with around a hundred core files.

Writing the process as numbered steps works best: a 6-step deployment process cut PRs with missing files from 40% to 10% and increased accuracy by 25%. Using a decision table to help the agent choose the right approach before taking action also increased specification-compliance by 25%.

When writing prohibitions, you must provide alternative solutions. Simply writing “don’t” leaves the agent indecisive, and having more than 15 consecutive warnings noticeably worsens the outcome.

The easiest way to run into trouble is having too many documents. Once the agent is pulled into a large number of architecture documents, after loading hundreds of thousands of tokens, the output actually gets worse. In one module, 226 documents totaled over 2MB—no matter how good the AGENTS.md was, it was useless.

In addition, AGENTS.md is the only document location the agent will read 100% of the time. The discovery rate for documents not referenced under _docs/ is below 10%.

(Source: BlockBeats)