how do you know your agent solved the task, and did not just find the answer key?


on a custom SWE-Bench the intended fix already lives in the repo history, and the grading tests ship with the task itself.
a capable model takes that shortcut every single time unless you remove it first. that is the real work in building an eval.
post-image
This page may contain third-party content, which is provided for information purposes only (not representations/warranties) and should not be considered as an endorsement of its views by Gate, nor as financial or professional advice. See Disclaimer for details.
  • Reward
  • Comment
  • Repost
  • Share
Comment
Add a comment
Add a comment
No comments
  • Pinned