pass@k

Model outputs vary from run to run, so a single attempt can be misleading. pass@k runs each task several times and asks whether any of k attempts passed. It became widely known from code generation benchmarks, where a solution passes if it makes the tests succeed. pass@1 is simply the chance a single attempt succeeds.

pass@k is the right metric when you can cheaply check results and keep the best one, for example generating several candidate fixes and running tests on each. It tells you what is achievable with retries.

For agents that act once in front of a user, you usually care about the opposite: whether the agent succeeds every time. Some benchmarks report a stricter metric, sometimes written pass^k, that counts a task as solved only if all k attempts succeed. A big gap between the two signals an inconsistent agent.