TRACE: Dense Credit Assignment for Long-Horizon Tool Agents
TRACE assigns turn-level rewards to tool-using agents by measuring how each tool-call transition changes the likelihood of the gold answer.
TL;DR — TRACE is a dense reward-assignment method for long-horizon multi-turn agents. It represents rollouts as state transitions at tool-call boundaries, uses a frozen reference model to score gold-answer log-probabilities, converts those scores into log-ratio state values, and assigns per-action rewards from Temporal-Difference value changes. The abstract reports large closed-web BrowseComp-Plus gains for Qwen3-4B and Qwen3-30B-A3B using pure RL, without an added critic, process labels, cold-start SFT, agentic mid-training, or live-web training data.
Problem: final rewards are too coarse for long tool-use trajectories
TRACE addresses a post-training problem for multi-turn agents: final outcomes do not reveal which earlier tool interactions helped or hurt. Evidence: the abstract says multi-turn agents “solve complex tasks through extended sequences of tool interactions before producing a final answer,” which makes “credit assignment a fundamental challenge during post-training.”
The abstract’s central criticism is that outcome rewards work better for short horizons than long ones. Evidence: outcome rewards “provide reliable supervision for short-horizon reasoning” but “become sparse and high-variance as trajectories grow to tens or hundreds of tool calls.”
The failure mode is not just sparsity. Outcome-only training can actively misassign credit: a failed rollout may include many useful actions that move the agent closer to the goal, but those actions can receive the same negative advantage as the later mistake that caused failure.
Full analysis, extracted claims, numerical results, entity graph, FAQ, related work, applications, and BibTeX are available via x402 micropayment.