Do AI Agent Optimization Gains Compound? Terminal-Bench 2.0 Study

A two-phase Terminal-Bench 2.0 evaluation tests whether agent-optimization gains survive new tasks and a second optimization round.

TL;DR — A two-phase Terminal-Bench 2.0 evaluation finds that one-shot agent-optimization gains can be misleading. GEPA, Meta Harness, and RELAI-VCL all improved a baseline agent in a static setting, but only RELAI-VCL both transferred positively to new tasks and kept improving after a second optimization round.

Background: why one-shot agent gains may be misleading

AI agents use a language model to take actions toward a goal: running commands, editing files, checking outputs, and retrying after failures. In terminal or software environments, the model is often wrapped in a harness: prompts, tools, instructions, verification steps, and control logic that shape how the agent behaves.

A common way to improve these systems is to optimize the harness against a benchmark. The usual evaluation is straightforward: measure a baseline, tune the setup, measure again, and report the gain. If the optimized agent solves more tasks, the optimization method looks effective.

The paper “Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0” argues that this one-shot setup misses the harder deployment question. Real agents do not face one fixed benchmark forever. They encounter new failures and new tasks, and their optimization process may need to run repeatedly.

Full analysis is available via x402 micropayment.