Scaling Automatic Research Agents via World Models

Overview

An AutoResearch agent writes code, runs it, and learns from what comes back. RL is the natural way to train one, but the two halves of a trajectory do not scale alike. Generation batches, so extra trajectories are nearly free. Execution does not: every candidate solution needs its own sandbox on real machine time. Past a certain scale, grading is the training cost.

WMRL hands the grading to a world model, which reads the task and the submitted solution and predicts the outcome in a few forward passes. Now grading batches like generation, and the bottleneck is gone. The predicted reward is imperfect, though, and the imperfection splits cleanly into a systematic bias and a zero-mean noise. Both show up in the convergence bound as their own error term.

To pay that bill, WMRL keeps about a tenth of groups graded by real execution as an anchor signal, and spends it on one mechanism per term: Online Debiasing for the bias, Inverse-Variance Denoising for the noise. We prove both strictly improve the convergence guarantee, and show they hold up in practice at 4B and 9B, on two held-out benchmarks, and again on embodied VLA post-training.

Method

The world model takes the same input the sandbox would and returns a predicted score, so the RL pipeline runs unchanged with cheap rewards in place of measured ones.

What that costs is visible once the predicted score is written as r̂(τ) = r(τ) + b(τ) + ξ(τ), splitting the error into a systematic bias b and a zero-mean noise ξ. They enter the convergence bound as two separate terms, O(B²) and O(σ²). Estimating either one needs ground truth, so WMRL keeps a thin stream of it. About 10% of groups are graded by both the world model and real execution, and those score pairs drive the two mechanisms below.

Removes the O(B²) term

Online Debiasing

The anchor pairs show how far the world model has drifted from the truth. We fit a monotone map to them by isotonic regression and push every predicted score through it before advantages are formed. Refitting each step keeps the map current as the drift moves.

Suppresses the O(σ²) term

Inverse-Variance Denoising

Noise cannot be estimated pointwise, so we suppress it instead. The anchor stream is clean but scarce; the world model stream is noisy but abundant. Both estimate the same gradient, so we fuse them by inverse variance, which lands strictly below either stream alone.

Theory

Neither correction is a heuristic. Each one divides a term of the convergence bound by a factor strictly greater than one — and the factor on the bias term grows without limit as training proceeds.

Assume the usual regularity: J is smooth and gradient-dominated, and the log-likelihood gradient is bounded. Run T ascent steps at step size γ. Training on real execution rewards converges at the familiar rate epsilon(T) = (1 - gamma mu / 4)^T Delta_0 , which both statements below carry unchanged. Everything the world model contributes enters through two numbers: the bias magnitude B and the noise standard deviation σ.

Theorem 1

Training on world model rewards (baseline)

The bias arrives at full size, the noise arrives damped by the step size. Neither term carries T: no amount of training and no choice of step size removes them. The achievable score stays capped by how wrong the world model is.

Theorem 2

Training with WMRL (ours)

The same bound with the bias term divided by 1 + T over T naught and the variance term divided by 1 + V_WM over V_E

The leading term is unchanged and the two error terms align one for one with Theorem 1, each divided by a denominator strictly greater than one. Both are therefore strictly smaller the moment the world model is biased or noisy at all.

The bias factor

Every step feeds fresh anchor pairs to the isotonic fit, so the residual bias keeps shrinking and T₀ sets the rate. The permanent floor of Theorem 1 contracts to zero as T → ∞: in the limit WMRL reaches the same optimum as training on real execution, while paying for real execution on a tenth of the groups.

The variance factor

Inverse-variance fusion puts the update below either reward stream alone, and the factor grows as the anchor stream becomes comparatively more reliable. Nothing is tuned here: the ratio is read off the anchor residual at each step.

Both theorems are stated for the GRPO estimator of Section 2; the constants, the conditions on T₀, and the full proofs are in the appendix of the paper.

ML research tasks

WMRL cuts the training compute of real-execution GRPO by 3.1× and 3.4×, and still scores higher on every benchmark average. The post-trained agents also beat much larger off-the-shelf ones: our 4B passes the 48B agent, our 9B passes the 120B agent.

Cheaper and better at the same time

Each arrow is one agent scale moving off the real-environment run it replaces: left is compute saved, up is score gained. Both land in the band above the strongest off-the-shelf agent.

RL in real environment (GRPO)RL with pure world modelWMRL (ours)

Every trained point shares the data split, the scaffold, and the hyperparameters, so the horizontal distance is the price of the reward signal and nothing else. The rule in each panel is the strongest off-the-shelf agent in the comparison, 13 times larger than our 9B agent and post-trained on nothing.

Held-out benchmarks, by agent scale

The world model alone gives back most of the gain. Adding the two corrections carries WMRL past the real-environment run on both benchmarks and both scales.

Untrained baseRL in real environment (GRPO)RL with pure world modelWMRL (ours)

DSBench is disjoint from the training set, and the MLE-Dojo test split holds out tasks never trained on. Bars start at zero.

Main results. Leaderboard percentile (%), higher is better. Both benchmarks hold out tasks never trained on. GPU-hours count total training compute.

Method	GPU-hours	MLE-Dojo (test)				DSBench
Method	GPU-hours	Tab	Text	Img	Avg	Bin-Cls	Multi-Cls	Regress	Other	Avg
Baseline Models
Qwen3.5-4B	—	13.7	8.0	0.2	7.3	21.9	11.8	25.8	8.8	17.1
Qwen3.5-9B	—	17.4	8.1	3.4	9.6	29.1	16.0	36.3	14.1	23.9
Large Agents
Kimi-48B-A3B	—	12.3	10.1	2.0	8.1	20.3	10.0	29.0	9.8	17.3
Nemotron-120B-A12B	—	35.1	18.8	7.6	20.5	38.7	26.6	40.7	20.6	31.7
RL in Real Environment
Qwen3.5-4B-GRPO	883	24.8	16.6	4.3	15.2	34.3	17.8	28.3	22.4	25.7
Qwen3.5-9B-GRPO	1174	33.3	17.1	5.9	18.8	40.9	26.8	40.4	16.7	31.2
RL with Pure World Model
Qwen3.5-4B-WM	269	21.2	14.6	3.0	12.9	27.1	16.9	32.5	16.0	23.1
Qwen3.5-9B-WM	330	28.0	15.9	4.5	16.1	35.4	21.8	37.9	16.9	28.0
WMRL (ours)
Qwen3.5-4B-Ours	286	28.0↑3.2	16.1↓0.5	5.2↑0.9	16.4↑1.2	35.8↑1.5	22.3↑4.5	32.3↑4.0	24.7↑2.3	28.8↑3.1
Qwen3.5-9B-Ours	349	38.0↑4.7	19.4↑2.3	7.4↑1.5	21.6↑2.8	45.4↑4.5	26.9↑0.1	39.2↓1.2	19.8↑3.1	32.8↑1.6

Scores are per-task avg@8, averaged within each category. Avg is the mean over categories. Bold marks the best value per column; green and red arrows mark the gain and drop of WMRL over same-scale GRPO.

Ablation: the two corrections are complementary

Both corrections earn their place

Every run here consumes the same two reward streams at the same ratio and differs only in which correction is switched on. The leftmost bar of each group is the uncorrected mixture.

No correctionInverse-Variance Denoising onlyOnline Debiasing onlyBoth (WMRL)

Denoising alone adds 0.9 to 1.7 points and debiasing alone adds 2.2 to 2.8, which matches the analysis: bias enters the bound at full size while noise enters damped by the step size. Together they add 2.9 to 4.8, more than either alone, so the two mechanisms are complementary rather than redundant.

Online Debiasing	Inverse-Variance Denoising	4B Agent		9B Agent
Online Debiasing	Inverse-Variance Denoising	MLE	DS	MLE	DS
○	○	13.5	25.3	16.8	29.5
○	●	14.9	26.2	18.0	31.2
●	○	15.7	28.1	19.4	31.7
●	●	16.4↑2.9	28.8↑3.5	21.6↑4.8	32.8↑3.3

A filled circle marks the correction as active. The last row recovers WMRL, and its green arrows give the gain over the uncorrected first row.

Embodied manipulation

Neither mechanism is specific to AutoResearch, so we apply the same recipe to VLA post-training on LIBERO-Long. Robometer scores eight frames sampled from each rollout as the dense reward, and the single sparse success the environment returns at the end serves as the anchor.

Either signal alone barely moves the policy. RL on the sparse outcome adds 0.9 points over the SFT baseline, RL on the raw world model signal adds 1.8. WMRL combines the two and lifts overall success by 3.8 points, with the largest margin on unseen initial states.

Embodied manipulation on LIBERO-Long

Each signal on its own barely moves the SFT policy. Combining them under the same two corrections gives the largest gain, and the widest margin is on unseen initial states.

MiniVLA-1B-SFTRL in real environment (GRPO)RL with pure world modelWMRL (ours)

All reinforcement-learning rows start from the same SFT checkpoint, so the spread within a row is attributable to the reward signal alone. The axis starts at 37 to resolve that spread; absolute values including the untrained policy are in the table below.

VLA post-training on LIBERO-Long. Success rate (%), higher is better. All RL rows share the SFT initialization.

Method	In-Domain			Out-of-Distribution			Overall
Method	Avg	Best@8	All@8	Avg	Best@8	All@8	Overall
Baseline Models
MiniVLA-1B	5.6	5.6	5.6	2.9	2.9	2.9	3.8
MiniVLA-1B-SFT	37.3	41.3	33.1	37.5	44.7	32.1	37.4
RL in Real Environment
MiniVLA-1B-GRPO	39.3	48.8	34.4	37.8	45.3	31.2	38.3
RL with Pure World Model
MiniVLA-1B-WM	39.1	45.6	34.4	39.3	45.6	34.1	39.2
WMRL (ours)
MiniVLA-1B-Ours	41.2↑1.9	47.5↓1.3	37.5↑3.1	41.2↑3.4	48.8↑3.5	37.1↑5.9	41.2↑2.9

Overall averages over every initial state, seen and unseen. Best@8 counts at least one success of the eight, All@8 counts all eight. Bold marks the best value per column; arrows mark the gain and drop of WMRL over MiniVLA-1B-GRPO.