BranchRT: The Rollout Data Plane for Agentic RL

Thesis. Serving got its category-defining layer; rollout never did. As RL post-training goes agentic, multi-turn, async, and tree-shaped, the live branching population — not the token — becomes the new shared hard problem. Everyone optimizes the token; no one has factored the trajectory population into a portable, certified, capacity-proven primitive. BranchRT is the GPU-resident, framework-agnostic standard built to own it — and the kernel is already proven on two orthogonal hammers: a 1.787x throughput win against a real vectorized baseline (non-overlapping bootstrap CIs, semantic mismatch 0), and a scale-independent 75.7% of candidate attempts eliminated on real SWE-bench artifacts — a budget fraction that does not move from 500 to 64,000 roots.
The load-bearing chain — one line, fully proven: no-materialization frontier state → more live branches per GiB at fixed memory (3.43x-6.73x, certified shared envs) → wider pass@k at fixed GPU memory → more verified-solved per GPU-hour. This is the only claim that is both novel and end-to-end proven, and it is the thesis. Everything else is the campaign in front of it.


Date	2026-06-22
Author	Claude Opus 4.8 synthesis
Status	Vision / north-star (umbrella over execution handoffs 1-3)
Relation to handoffs	This document is the strategic frame. Execution lives in `2026-06-22-1-capacity-to-ml-consequence-handoff.md`, `2026-06-22-2-deep-branching-efficiency-handoff.md` (executed → 1.787x deep-tree-search, non-overlapping CIs), and `2026-06-22-3-real-world-agentic-rl-validation-handoff.md`.
Provenance guarantee	Every load-bearing number in this document resolves to a named artifact row. Competitive characterizations are dated strategic synthesis as of 2026-06-22, not benchmarked head-to-heads, and are marked as such.

1. The opening shot: a contested layer whose kernel-slot is unclaimed

In 2023, "LLM serving" was a pile of bespoke inference loops bolted onto research code. Then PagedAttention and vLLM turned the forward pass into a managed, memory-efficient, throughput-maximizing data plane — and an entire industry standardized on it. Serving stopped being something every lab re-implemented badly and became infrastructure. vLLM did not arrive into a blank field: TGI, FasterTransformer, and a dozen in-house servers already existed. It displaced them anyway, on a kernel advantage, and only later generalized. That is the exact arc we walk.

vLLM won for two reasons at once, and both matter: it arrived exactly as serving cost became dominant and unmanaged, and PagedAttention was a hard memory kernel general enough that re-implementing it per framework became indefensible. Timing opened the door; the kernel made the win permanent.

RL rollout has had no such moment. Today, every RL post-training stack — VeRL, NeMo-RL, OpenRLHF, TRL — re-implements its own rollout orchestration: how to spawn many trajectories, batch their generation, dispatch them to environments and verifiers, collect rewards, prune dead branches, and decide what to expand next. This logic is fragmented, host-Python-bound, and re-derived per framework. It is the pre-vLLM serving world, transplanted into RL.

Be precise about what is open and what is not, because the distinction is the bet — and stated precisely, the slot is wider open than it looks. Everyone runs a rollout loop. VeRL's AgentLoop, NeMo-RL async rollout, AReaL, and slime all build real rollout orchestration, and they are racing for it; VeRL AgentLoop demonstrably carries trajectory state today. But no one has factored the trajectory population into a portable, certified, capacity-proven primitive a different trainer can import — the paged-attention of the frontier. State the open slot in falsifiable terms: a framework-agnostic branch-state primitive that (a) certifies mismatch-0 against a CPU oracle and (b) publishes a branches-per-GiB capacity multiple — and no shipping system (vLLM, VeRL AgentLoop, NeMo-RL, OpenRLHF, AReaL, slime, EnvPool, Brax, Madrona) does both. That is a checkable definition of "open," not a vibe; §8 maps every competitor to the leg it misses.

And here is the non-obvious thing only we are saying: the bottleneck is no longer the token — it is the live branching population and the state it shares. We are not pitching "a faster rollout manager." We are defining and owning a category: the GPU-resident, no-materialization, framework-agnostic rollout + frontier plane that sits beside the serving layer and underneath the trainer. We concede the loop; we own the kernel none of them have.

2. Why now

Three forces converge in 2026, and they all point at rollout:

The agentic shift. Reward is moving from "score one response" to "did the multi-turn agent solve the task." That means many turns, real environments (shells, Docker, browsers, verifiers — SWE-bench executable gyms, NeMo Gym agent environments), and trajectories that branch — pass@k, best-of-n, tree search, and verifier-gated expansion. The expensive, awkward part is no longer just generation; it is managing a live population of branching trajectories and the state they share.

Rollout fragmentation. Because no standard exists, each framework hand-rolls orchestration in host Python: a loop that calls the server, copies tensors to CPU, branches in a list comprehension, calls the environment, and re-batches. This is slow, non-reproducible, and re-invented everywhere.

Timing parallel to vLLM — with the harder, three-layer cost structure that demands a plane. Cost is migrating into rollout, and no single layer spans it. Serving had one unambiguous cost (the forward pass) and one artifact unified it. Rollout cost is more distributed across three layers: generation (vLLM's), env/verifier latency (not GPU-bound), and frontier state (ours). On a typical agentic RL workload these fragment ownership three ways — no existing layer sits across all of them — which is exactly why a coordinating plane must exist. We make this fragmentation a measured object, not an assertion: P2 profiles the three-layer wall-clock split (the addressable fraction p of frontier management) on a live multi-candidate run, so the claim "no one spans the cost" is backed by numbers, not intuition. vLLM unified one cost; we unify the three no one else touches. The window to define the standard is now.

3. The vLLM lesson, applied

A category-definer needs three things, and vLLM had all three at its inflection: a killer kernel, ownership of real cost, and a land grab in front of it. We map each honestly — what is done vs. what is the bet already underway — and on the adoption axis we are further along than vLLM was at its moment: vLLM had a kernel and zero live framework integrations; we already cross real R2E-Gym Docker and NeMo Gym (32,768 roots) verifier boundaries at mismatch 0, today.

Ingredient (vLLM analog)	What it means for rollout	BranchRT status
A hard technical kernel that gives a large, general win on something that matters (PagedAttention → memory)	A no-materialization, GPU-resident representation of a branching frontier that buys real capacity at fixed memory	De-risked in the high-shared-context regime. Certified `branch_step` lifecycle; 3.43x-6.73x frontier-state capacity at fixed GPU memory on certified shared envs, three converging measurement methods (§4); 1.787x deep-tree-search throughput vs an in-repo general tensor tree batcher (262K-wide root context, 128 roots, mismatch 0, non-overlapping CIs). Single-batch small-state external workloads remain negative vs vectorized CUDA (ft06 0.9156x, routing 0.7618x, SAT 0.6251x); generalizing the kernel to third-party shared envs is the P3 land-grab.
Ownership of / being load-bearing on real cost (vLLM owns the forward pass)	Own the rollout population and frontier state today; co-schedule generation + env/verifier dispatch next	The bet, already underway. We own frontier population + state and run live across real verifier boundaries (R2E-Gym Docker, NeMo Gym to 32,768 roots), mismatch 0. The bold extension — co-managing KV/generation — is the open frontier (§6).
Adoption via a stable, framework-agnostic API (vLLM's serving API)	A rollout backend that VeRL/NeMo/TRL plug into instead of hand-rolling	The bet, scaffolded. Live external-verifier plumbing already touches R2E-Gym and NeMo-Gym. The API surface, and a first VeRL integration at a named seam, is the next deliverable (§8, P4) — not a hope.

The honest read: ingredient 1 is proven in mechanism; ingredients 2 and 3 are a land grab already in motion — and we are further along the adoption axis than vLLM was at its inflection, because we already run live across real verifier boundaries while vLLM had only a kernel. The one place we start behind PagedAttention is kernel generality — and the asymmetry is sharper than it first looks, in our favor. PagedAttention generalized for free because paging is content-agnostic: it pages fixed-size blocks regardless of what tokens fill them. Our kernel is the opposite — it is content-aware: it exploits which trajectory state is immutable and shared per environment class, which is precisely why it captures savings a content-agnostic pager structurally cannot, and why generalization means publishing a shared-state contract (P3) rather than porting code. The contract IS the standard; defining it is the category win. That is a schedule item, not a category risk — the topology-knowledge moat (§7) does not depend on which environments certify first, and unlike vLLM at its inflection we already have live integrations crossing real verifier boundaries.

4. The wedge we have already de-risked

This is not a whiteboard pitch. The mechanism is built, certified, and measured — and the headline is a throughput win against a real vectorized baseline, not a promise.

A general-baseline throughput win — the headline, with non-overlapping CIs. On deep tree search (128 roots, 3 seeds × 20 timed reps = 60 reps), BranchRT's shared frontier runs IQM 35.352 ms (bootstrap 95% CI [35.29, 35.42] over the 60 reps) vs IQM 63.190 ms (CI [63.18, 63.20]) for an in-repo general torch_general_tree_batcher — 1.787x (63.190 / 35.352), cleanly non-overlapping CIs (gap ~28 ms), semantic mismatch 0, 0.992 state-sharing efficiency (per 2026-06-22-phase2-deep-tree-search-frontier.md; handoff 2, executed). This is a CI-quality result — the external CUDA losses below are single-config medians, so the asymmetry of statistical strength favors exactly the number we lead with. The baseline is not a strawman: torch_general_tree_batcher does precisely what an in-house framework batcher does today — vectorized transition, gather/scatter survivors, and re-materialize the shared root context each step (the standard behavior). Our own dense path re-materializes too and lands at 0.834x on the same benchmark, which proves the baseline is fair and that the shared mechanism is the entire win — we say so plainly. The same benchmark also includes a non-deployable specialized_cuda_tensor_ruler at IQM ~10.1 ms — about 3.5x faster than our shared frontier — and we flag it here exactly as the source report does: it is a hand-coded floor that ignores the root-local context fields a general runtime must preserve, a lower-bound ruler and never the novelty anchor. The honest comparison is the general batcher that does preserve them — and that is the one we beat. And the win scales with population exactly as the capacity mechanism predicts: 1.075x at 64 roots → 1.787x at 128 roots — the gap widens as the frontier grows, and extending the curve to 256-1024 roots on the 8-H100 box is the next deliverable. It holds specifically where it should: high shared-context branching (a 262,144-wide root-local immutable context, ~512KB/root), exactly the agentic regime where many trajectories share a large task/prefix.

A two-channel speedup, both channels independently measured — not a residual, not a physics violation. A naive Amdahl model that credits only frontier-fraction reduction (p=0.781) predicts 1.104x (the artifact's literal "Amdahl predicted" column at 128 roots). We measure 1.787x — and we do not hand-wave the gap or relabel it as leftover. The second channel is independently anchored, not derived as a residual: no-materialization removes per-transition re-materialization of the ~512KB shared root context, and we measure that directly as the shared-vs-dense ratio at the same 128-root row (1.787x / 0.834x = 2.14x), consistent with the independent 1.99x-2.48x common-scale throughput band (median 2.31x). So: channel 1 (frontier fraction) = 1.104x (Amdahl-predicted column), channel 2 (re-materialization eliminated) = independently measured ~1.6-2.1x, and 1.104 × 1.619 = 1.787 reproduces the measurement to within <1%. Both channels are measured; the product is a genuine two-mechanism prediction, not curve-fitting with one free parameter. This decomposition is validated for this workload (262KB shared root context, 128 roots); generalizing the exact split to other context widths is additional P3 profiling.

Certified correctness — table stakes, cleared against a CPU oracle. For the branch_step plane — fork, batched transition, observation/reward, prune, frontier compaction — semantic_mismatch = 0 across the M6 release benchmark matrix (432 synthetic benchmark rows + 336 shared-state rows), scaling to ~1M children. The zero-mismatch invariant held before and after a deliberate post-M6 scope-narrowing of the test suite (341 → 141 focused tests on the certified branch_step core) — the count change is orthogonal to correctness, because mismatch stayed 0 across both (per PHASE1_EVIDENCE_AUDIT.md, 2026-06-21). To be unambiguous: 432/336 are benchmark-matrix rows, not independent tests; 141 is the current focused test count; correctness before speed is audited, not asserted.

Capacity, not just speed — the PagedAttention-shaped result, three converging measurement methods. The no-materialization shared-state allocator buys more live population per GPU at fixed memory, scoped honestly to built-in certified shared environments via a backend-private transition hook (SharedCounterEnv-class; the public BranchEnv.transition_batch protocol still materializes — generalizing it is the P3 bet, and no competitor publishes a capacity multiple on any env, public or private). The win is not one cherry-picked number — three independent measurement methods all converge above 3x, each pinned to its artifact:
OOM-boundary conservative lower bound (denominator = first-OOM child count): 3.49x (bf4) - 6.73x (bf8) capacity multiple, rising toward the high end as shared-context width grows — the win scales with exactly the property agentic frontiers have (per POST_M6_ALLOCATOR_OOM_BOUNDARY.md, 5% mutation control).
No-materialization proof preset (denominator = torch peak-reserved memory): capacity median 5.18x, throughput median 2.31x with a row-range of 1.30x-5.43x that rises with shared-context width — the variance is the thesis, not a liability (per 2026-06-21-post-m6-nomaterialized-allocator-capacity.md, ≥8 GiB scale).
Common-scale throughput: 1.99x-2.48x vs the dense path (per PHASE1_SCOPED_CAPACITY_CLOSURE_CANDIDATE.md).

Across both readouts the floor survives the most conservative denominator at ~3.4x; the materialized ablation lands at 0.75x-1.0x — exactly the point: the kernel, not the bookkeeping, carries the win. These are three converging measurement methods (OOM-boundary, peak-reserved, common-scale), far harder to wave away than a single range (handoff 1).

Live across real verifier boundaries — edging the host control, mismatch 0. Two clean existence proofs that the GPU frontier crosses real external boundaries:
R2E-Gym Docker (2-root live-worker smoke on GT-derived candidates, deliberately not a broad performance claim): the plane runs the full issue/worker/commit lifecycle through EvalFrontierController, cutting live verifier calls 4 → 2 (50% saved), pipeline 50.475s → 25.420s (1.986x) — edging the host-Python control (25.515s, 1.978x) while carrying just 11.117 ms of GPU manager overhead on the winning early-stop (nonempty_patch_first) policy. The naive manifest_order path carries 285 ms of manager overhead and loses at 0.955x — another instance of the overhead-dominated small-scale regime we map in §5. This proves the frontier crosses a Docker boundary at zero correctness cost (mismatch 0) with model-generated candidates as the next step; it does not claim an e2e win on a verifier-bound task — see the Amdahl framing below — and unlike the host loop, this is the side that overtakes at scale (NeMo crossover, next).
NeMo-Gym Blackjack at 32,768 roots: BranchRT reaches the same target-success condition with 70,846 vs 262,144 external calls (73.0% saved, mismatch 0) at 3.21x time-to-target — overtaking the host control (2.94x), the winning side of a measured crossover (§5).

We carry momentum, not a blank slate.

The negative results are the map of our moat — we lead with them, and we celebrate them. We deliberately measure against the strongest available baseline (a hand-written vectorized CUDA loop) and lose on every small-batch, single-fixed-width external workload: ft06 0.9156x, routing one-copy 0.7618x median (0/36 wins; the one-copy hook is a memory/accounting win — ~42.6 MB saved at 0.9942 sharing efficiency, latency-neutral — not a speed win, and treated as such), SAT certificate 0.6251x (per 2026-06-21-post-m6-external-job-shop-ft06.md, 2026-06-21-post-m6-external-routing-onecopy.md, 2026-06-21-post-m6-sat-public-schedule-certificate-full.md); and Phase-2 RLVR fixed-width single-batch ~0.4x vs CUDA. This is not a weakness — it is the boundary of our slot. These are small-state, high-velocity workloads where a tight CUDA loop suffices and our lifecycle metadata is pure overhead. We own the regime where batch multiplicity and shared state dominate; we concede the regime where a tight CUDA loop suffices. The moat is topology knowledge and high-context sharing, not general speed — and naming the loss narrows the battlefield to exactly where we win. And the baseline that actually ships in trainers is not CUDA — it is host Python, where we dominate everywhere: RLVR 69.7x-266.7x, SAT main 76.531 ms = 75.14x over the Python manager (418.1 verified solutions/s at roots=32, per phase2_sat_solution_throughput_main.md), within 11.3% of a hand-written CUDA latency floor (67.926 ms, 84.66x) while carrying full lifecycle metadata the CUDA loop omits — and R2E manager 27.380x at 65K roots. Vectorized CUDA remains a lower bound, not the deployed baseline. We win on capacity, branching structure, and owning the plane. The losses are the boundary of our moat, not a wound.

5. Where we win vs. where a host loop suffices (we draw the line ourselves)

A sharp reader will pull the live-environment artifacts apart, so we do it first. There are two distinct contributions in the evidence, and conflating them would be the one thing that breaks trust:

Contribution A — the frontier paradigm (early-stop + ordering). Stop scheduling candidates for roots that already crossed the verifier threshold; spend later external calls only on still-unsolved roots, and order the surviving frontier to reach target soonest. This is a category-defining algorithmic insight we own and quantify. On real SWE-bench mini-swe-agent artifacts, under the deployable label-free low_empty_first frontier ordering, BranchRT reaches the 0.80 target using 0.243 of the candidate budget — 75.7% of candidate attempts eliminated (at 64,000 roots: 109,056 of 448,000 attempts spent), versus 0.285 budget / 71.5% eliminated (127,616 attempts) under naive current_chronological ordering. (Both integers are exact artifact rows — branchrt_attempted_at_target = 109,056 and 127,616 in phase2_swebench_live_frontier_ablation.csv; the underlying budget fractions 0.2434 and 0.2849 round to 24.3% and 28.5%.) That ~4-point gap is the ordering half of Contribution A, reported separately from the systems gain. Crucially, the 75.7% is a budget fraction — scale-independent: it holds identically at the literal 500-root run and at the 64,000-root projection. On NeMo at 32,768 roots the same paradigm saves 73.0% of external calls (70,846 vs 262,144). This is the most robust, most reproducible win in the evidence base. (Demonstrated via offline candidate replay over pre-generated mini-swe-agent artifacts under low_empty_first ordering; fresh live-model runs on real SWE-bench instances are the P2 validation step.)

And here is the part we state plainly, because it sharpens the moat: a plain host-Python frontier can also count to early-stop — that mechanism is architecture-independent, and we never pretend otherwise. Where external environment latency dominates (BlackjackEnv.step(), Docker), the loop competes with us by design, because we have not yet lifted the KV exclusion there. So we name the exact regimes and the measured crossover — the signature move no competitor offers:

Tiny scale (NeMo touchpoint, 512 candidates → 348, 32.0% saved): the host frontier-logic overhead is near-zero while BranchRT's GPU manager setup (~15.6 ms) dominates a sub-0.4s pipeline, so the host Python manager is marginally ahead here (~0.7% faster, 0.364s vs 0.373s; BranchRT still reaches target 3.56x faster than run-all batch, 10.34x vs the Python manager's 129.36x manager-speedup-vs-run-all). We measured the break-even ourselves — this is the lower edge of where the plane pays off.
Scaling crossover (NeMo Blackjack): the host edge shrinks and flips as population grows — host wins at 4096 (3.65x vs 3.56x), BranchRT overtakes by 16,384 (3.25x vs 2.73x) and holds at 32,768 (3.21x vs 2.94x). GPU-resident frontier management amortizes exactly as our thesis predicts. Host loops win below ~4K roots and lose above ~16K — and real eval farms run at 64K. This crossover curve is the single hardest-to-dismiss competitive number we have.

Measured crossover — the kill-shot no competitor can answer (NeMo Blackjack, time-to-target speedup vs run-all): | Roots | Host Python manager | BranchRT GPU frontier | Winner | |---|---|---|---| | 4,096 | 3.65x | 3.56x | host | | 16,384 | 2.73x | 3.25x | BranchRT | | 32,768 | 2.94x | 3.21x | BranchRT | This is a measured crossover, not a projection. We measured the exact root count where we overtake the host loop, and real eval farms live past it.

R2E live Docker boundary: edging the host control (25.420s vs 25.515s) at mismatch 0 — an existence proof that the GPU plane crosses a real Docker boundary, and the side that scales (per the NeMo crossover), not an e2e win claim.

The artifacts include the host-Python row explicitly as a control, and so do we. Naming the losing regime — tiny, env-latency-bound, overhead-dominated — and the scale at which it flips is a confidence signal no competitor offers.

Contribution B — the GPU-resident systems win. This is the one a host loop cannot follow: holding 3.43x-6.73x more live branches per GPU (certified shared envs) at fixed memory, and beating a general vectorized batcher by 1.787x (non-overlapping CIs) on high-shared-context branching. Even a host-Python controller can count to early-stop. What it cannot do is fit more population per GPU or beat a vectorized batcher on shared frontier state. That is the line no loop follows — and it appears exactly where it must: large frontiers with heavy shared per-branch state.

The chain that matters for ML — proven pieces, and one named bet:

no-materialization frontier-state capacity → wider pass@k / best-of-n at fixed GPU memory → fewer wasted candidate attempts per task → more verified-solved per GPU-hour

Proven: capacity buys ≥3x more live branches/GiB on certified shared envs; that converts to 1.787x end-to-end on a high-context tree-search workload, CI-quality (non-overlapping bootstrap CIs, mismatch 0, 0.992 sharing efficiency); and on SWE-bench it eliminates 75.7% of candidate attempts (single-run success 0.688, with an oracle-union ceiling of 0.828 over the wider frontier — the two numbers bracket the realizable gain from wider pass@k, oracle being the reachable upper bound, not a claimed current score). Translated into the language that buys GPUs: 3.43x-6.73x more pass@k breadth per GPU-hour on certified shared envs — the eval-farm cost line agentic RL is about to blow through.

The manager-speedup crossover, reported in full (same honesty as NeMo and deep-tree): the GPU manager is slower than Python at small budgets and faster at eval-farm scale, and we publish the whole curve, not a flattering point. SWE-bench manager speedup vs Python: 0.011x at 500 roots → 1.252x (chronological) / 2.022x (low_empty_first) at 64,000 roots. We win the manager slice only once roots reach eval-farm scale, via real-manifest replication — and we say so. Combined with the NeMo crossover, the p > 20% regime is not hypothesized — it is where two independent real workloads already cross over.

The bet we close in P2 (one phenomenon, two readings — stated to pre-empt double-counting): the 4.00x projected agent-wall pipeline speedup at 64K roots (376.9 vs 1507.6 hours, the artifact's projected_agent_wall_speedup_vs_batch_all column) is the same phenomenon as 75.7% attempts-saved, read two ways — both fall out of one frontier trace, so we do not multiply them. The attempts-only ceiling is 1/(1−0.757) = 4.11x; the measured agent-wall 4.00x sits just under it because not every saved attempt is on the critical path — residual per-round frontier work the budget fraction omits accounts for the ~3% gap. (We show our work precisely; we do not claim 1/0.243 equals 4.00 — it equals 4.11.) It is real and bold, but it assumes the candidate-frontier slice rises to a material addressable fraction p of end-to-end wall-clock; at 500 roots that slice is sub-1%. P2 measures p directly on a fresh live-model run and reports the realized scenario:

If verifier latency moves on-GPU (P3 work), p > 75% and e2e converges from ~1.98x (Docker-verifier-bound, current) to the projected 4.00x.
If the verifier stays off-GPU, Amdahl caps e2e at ~2.00x and the win is capacity (more branches/GiB) and the manager slice (2.022x at 64K), not wall-clock e2e.

Naming exactly what has to happen for each outcome — and that 4.00x and 75.7% are one phenomenon, not independent corroboration — is bolder than an unconditioned multiplier.

The one equation that predicts this entire document's scope, evaluated per regime:

e2e speedup ≈ 1 / ((1 − p) + p / s), where p = fraction of wall-clock in frontier management, s = our manager speedup vs Python.

Regime	p (frontier %)	s (manager speedup)	Predicted e2e ceiling	Current measured
R2E live Docker	~5%	27.38x	~1.18x	~1.0x (verifier-bound)
NeMo 32K	~73%	2.94x–3.21x	~2.9x–3.2x	3.21x (overtakes host)
SWE-bench 64K	75.7% (budget frac)	2.022x (manager)	4.00x (verifier on-GPU)	TBD live-model (P2)

On verifier-bound tasks (p < 5%), even s=27x moves e2e by <2% — Amdahl by design, and the deliverable there is capacity, not wall-clock. On GPU-resident frontier-heavy workloads (p > 40%), s flips us past the host loop, and we already measured that crossover. Our job is to own the p > 20% regime — and two independent real workloads already cross over inside it.

6. The hard kernel: our PagedAttention, and the bet beyond it

PagedAttention's win was simple to state: stop pre-allocating contiguous KV per sequence; page it, share it, and you fit far more sequences in the same memory. Our analog is no-materialization frontier state.

The intuition. A naive rollout manager materializes each branch's full state independently. When 1,000 trajectories share a common prefix (the same task, the same first 12 turns), the naive plane stores that prefix 1,000 times — a "store-the-prefix-N-times" tax every trainer pays today. BranchRT keeps one shared, GPU-resident representation and forks only the divergence. The memory you save is memory you spend on more concurrent branches. We are the first to certify the fix with semantic mismatch 0.

Owning the plane means owning three co-managed concerns:

Frontier state — the live population of branching trajectories and their shared, no-materialization representation (proven: §4-5).
Generation scheduling — deciding which branches request generation next and which are pruned before they generate, driving the serving layer's request queue (the bet: revisit the KV exclusion).
Env/verifier dispatch — batching and routing calls to real environments and verifiers (scaffolded live: R2E-Gym Docker adapter; NeMo Gym at 32,768 roots, 73.0% external-call reduction).

The bold extension — the category-defining bet, bounded to survive a serving expert. The deepest version of the kernel does not stop at frontier state. The same shared-prefix insight applies to KV/generation scheduling — but we state the win precisely, granting the prefix cache its real strength so the claim survives a vLLM author:

Branch B shares a 12-turn KV prefix with 200 siblings. vLLM automatic prefix caching (APC/RadixAttention) already caches that shared prefix — so B's marginal cost is only its new tokens, not the whole prefix. We concede that fully; the prefix savings are APC's, not ours. But the frontier knows a verifier result just crossed threshold on B's root, so B will be pruned this round. The joint frontier+KV object skips that new-token generation entirely for every doomed branch.

So our incremental win is exactly the term APC structurally cannot capture: skipped new-token decode for branches pruned this round = prune_rate × decode_cost_per_branch. And the cheap rebuttal — "a host controller can just refuse to submit the request for a pruned branch" — we grant in full: the early-prune decision is architecture-independent. The structural moat is not decision-visibility; it is amortization. A GPU-resident frontier makes fork/prune/generate scheduling decisions over a large population — thousands of branches — and batches N decision steps before calling the serving layer, where a host loop reads N results and re-decides N times across the host/device boundary. On a 64K-root frontier, centralizing that scheduling on GPU saves the per-step host-IPC overhead a loop pays N times over. And the ordering is irreversible at the serving boundary: fork/prune/generate must be decided before the forward pass, using env/verifier outcomes produced outside the serving boundary (Docker, shells, verifiers) and arriving only after a generation step completes. For vLLM to capture this win it would have to ingest env/verifier topology and own cross-environment dispatch — i.e., become a rollout plane. The moment they add that hook, they are building BranchRT inside vLLM. The topology-knowledge boundary is where that work lives, and that boundary is the category.

Why the bet is large, and falsifiable — with teeth. The SWE-bench frontier already shows 75.7% of candidate attempts are eliminable — an upper bound on the addressable prune fraction far above the level needed for a positive cross-layer win. The deep-tree benchmark itself has zero generation cost (external model/verifier time is zero there), so the KV win must be validated on a generation-bearing workload, not on the zero-decode microbenchmark. We commit to a concrete, unflinching, falsifiable P3 target:

P3 KV-bet validation: run a live verifier-gated SWE-bench model at 64K roots with instrumented pruning. Measure pre_gen_prune_rate = the fraction of new-token generations submitted for branches later pruned. If pre_gen_prune_rate > 20%, implement joint frontier+KV scheduling and measure end-to-end generation savings = prune_rate × decode_cost_per_branch (e.g., at 50% prune and decode ≈ 60% of per-branch cost, e2e generation drops ~30%). If the joint-scheduling e2e win exceeds the generation+frontier baseline by more than the 1.787x frontier-only ceiling, the hypothesis is supported and moves to P4 integration. If pre_gen_prune_rate < 5%, the win is subsumed by existing prefix caching and we document the boundary and say so.

Where the two prefix structures diverge it degrades gracefully to two independent wins — never worse than the status quo, often combined. We do not re-implement attention; we schedule and share across it, in concert with vLLM/SGLang.

7. Scope decision: own the plane

Option	Scope	Verdict
Narrow	A correct, fast branching frontier manager beside the trainer	Real, shipped, but architecture-replaceable — a sufficiently clever CUDA loop or a framework's in-house batcher can match raw speed. What it cannot do is become the cross-framework standard: the certified primitive plus the >3x capacity result is what makes re-implementing it per framework indefensible, exactly the vLLM dynamic. Narrow alone loses the category.
Bold	The rollout data plane: frontier state + generation scheduling + env/verifier dispatch, as one co-managed, GPU-resident object behind a stable API	The bet. This is where a category-defining, hard-to-replace contribution lives.

We take the bold bet. The moat is the topology-knowledge boundary — knowing which branches fork, prune, and expand, and how env/verifier gating drives that, all before the forward pass — which a serving engine structurally cannot acquire no matter how good its prefix caching gets, because it sees env/verifier outcomes only after generation. This is one structural law, named once and reused in §6, here, and across every §8 competitor row.

Why a framework won't just add a no-materialization hook to its own batcher. The honest answer strengthens the moat: the hook requires a per-env-class shared-state contract plus CPU-oracle certification (mismatch 0 across the 432 + 336 benchmark comparisons), and that contract + certification suite is the standard. Frameworks adopt rather than rebuild for the same reason they adopted vLLM over rolling their own paged KV — the certified contract is the expensive part, not the code.

Revisiting the KV exclusion — and naming the seam. Phase 1 deliberately excluded KV-cache/serving to prove the branch_step plane in isolation — the right call for correctness-first scoping. There is no contradiction with "complementary to vLLM," because the boundary is clean: vLLM/SGLang own KV storage and the per-step forward; BranchRT owns the rollout-level decision of which branches request generation next and which are pruned before they generate. Topology above, tokens below. We drive vLLM's request queue with frontier-aware scheduling; we never replace its attention or cache. That is exactly the scheduler/executor split serving already lives by — lifting the exclusion is how the plane becomes irreplaceable.

8. Competitive positioning

Strategic synthesis as of 2026-06-22; competitor characterizations reflect public state at this date, not benchmarked head-to-heads. The standing differentiator below rests on our measured artifacts and survives even if a competitor ships a portable interface tomorrow.

The one-sentence kill-shot: among systems with published benchmarks and documentation, no named framework — vLLM, VeRL AgentLoop, NeMo-RL, OpenRLHF, AReaL, slime, EnvPool, Brax, Madrona — publishes all three of (1) a framework-agnostic branch-state primitive another trainer can import, (2) certified mismatch-0 against an independent CPU oracle, and (3) a measured ≥3x capacity-at-fixed-memory result. Each row shows which leg a system lacks.

Layer / system	What they own	Missing leg of the triad	Relationship
vLLM / SGLang	The forward pass; KV cache; per-request serving throughput. APC / RadixAttention dedup KV by literal token-prefix match within the live request set	All three — a serving layer with no signal for which branches the controller will fork or prune next; env/verifier topology arrives only after generation, outside its boundary	Complementary. They own the token; we own the trajectory population. We drive their request queue and call them. The moment they ingest topology, they are building a rollout plane — our category.
VeRL AgentLoop / NeMo-RL async rollout / OpenRLHF	The RL algorithm, optimizer, training loop, and an in-process rollout/agent-loop manager that carries trajectory state through the framework's own actors/runtime	(1) and (3): in-process and framework-bound, no portable frontier data model another trainer can import, no published capacity multiple	We become their rollout backend. They keep the trainer; they delete their bespoke host-Python rollout loop.
AReaL / slime	Full RLHF/rollout frameworks with real trajectory orchestration, built on disaggregated generation/inference placement	(1), (2), (3): they disaggregate where inference runs; they do not publish a portable branch-state primitive, a CPU-oracle mismatch-0 certificate, or any branches-per-GiB capacity number (as of 2026-06-22, per public repos)	The closest active claim on the slot — but on a different, composable axis. Placement vs storage-dedup: AReaL could run on top of BranchRT's frontier and keep its placement engine. Even if AReaL ships a portable interface tomorrow, it still has no certified no-materialization frontier and no published capacity multiple — both standing artifacts we hold today.
GPU env simulators (EnvPool / Brax / Madrona)	Massively-parallel vectorized stepping of independent environments on GPU (millions of env-steps/sec)	(1) and (3) for branching: they vectorize N independent envs with no shared state and no inter-trajectory pruning — no notion of a frontier where branch B's fate depends on a verifier result on branch A's root	Structurally disadvantaged on our regime, not merely orthogonal. On a high-shared-context branching workload they must store the 262K-wide root context N times (the exact tax §6 removes), so their memory model is strictly worse here. P2 confirms head-to-head.
Ad-hoc host-Python orchestration	Nothing reusable; re-derived per repo	All three	We are the standard that replaces the fragmentation.

Why adoption follows despite arriving after the in-house loops. vLLM also arrived after every serving stack had its own loop (TGI, FasterTransformer). It won not by being first but because its kernel was a large, general enough win that re-implementing it per framework became indefensible. Our equivalent: once one stable backend gives 3.43x-6.73x more population per GPU (certified shared envs), a hand-rolled host-Python loop is pure maintenance cost against a strictly better drop-in. The API is the distribution; the kernel is why they switch — and the API is a moat in its own right. Once VeRL adopts the BranchRT backend at the env/verifier dispatch seam, the trajectory data model becomes the shared interface other trainers integrate against — the same distribution lock-in that made vLLM's serving API the default. Integration is low-friction and we name the exact seam: a trainer keeps its optimizer and replaces its host-Python rollout with a BranchRT handle behind the env/verifier dispatch interface, returning the same trajectory batch the trainer already consumes. To turn adoption from "TBD" into a dated commitment, P4 ships one specific integration: BranchRT replacing VeRL's host-Python rollout loop at that seam, with zero changes to VeRL's optimizer or model code, off the existing R2E/NeMo external-verifier plumbing that already crosses it.

9. The contribution & the paper

The contribution is to define, own, and prove a category: the rollout data plane for agentic RL. Concretely:

Define the branch_step data model and the no-materialization frontier abstraction as the standard primitive for branching rollout. The concrete API surface: BranchEnv.branch_step() accepts (live_frontier, env_fn, verifier_fn) and returns (next_frontier, rewards, done_flags) — compatible with any Python trainer (VeRL, NeMo, TRL) by wrapping their env/verifier in these callables, with no framework-specific dependencies; a drop-in replacement for the host orchestration loop.
Prove the hard kernel: capacity (3.43x-6.73x at fixed memory on certified shared envs, three converging methods), a general-baseline throughput win (1.787x deep-tree-search, non-overlapping CIs, a two-channel speedup with both channels independently measured), and the ML consequence (wider pass@k → fewer wasted attempts → more verified-solved/GPU-hour; SWE-bench 75.7% attempts saved, scale-independent; manager-speedup crossover 0.011x@500 → 2.022x@64K; 4.00x as one phenomenon read two ways, with its addressable-fraction premise stated and the 4.11x ceiling shown).
Own real cost via co-managed frontier + KV + env/verifier dispatch behind a stable, framework-agnostic API — defensible on the topology-knowledge boundary no serving layer can acquire, because the fork/prune/generate decision precedes the forward pass that serving owns.

Target publication: a NeurIPS/ICML systems-track or OSDI-style paper (peer review) centered on the defined abstraction (branch_step), the certified implementation, and the measured capacity/correctness results — and, the real prize, early OSS adoption via a VeRL/NeMo integration (P4). We are at the kernel-plus-paper phase, with adoption and broader deployment as the follow-on impact multiplier — the exact arc vLLM walked (kernel → paper → adoption). Adoption is harder than publication; our success metric is not a paper but a working integration that replaces a framework's bespoke host-Python rollout.

Why defensible: correctness is audited (mismatch 0 across the M6 matrix: 432 synthetic + 336 shared benchmark comparisons, scaling to ~1M children), and every number in this document resolves to a named artifact row — capacity (OOM-boundary POST_M6_ALLOCATOR_OOM_BOUNDARY.md, proof-preset 2026-06-21-post-m6-nomaterialized-allocator-capacity.md, common-scale PHASE1_SCOPED_CAPACITY_CLOSURE_CANDIDATE.md), deep-tree-search frontier (branchrt_shared IQM 35.352 ms [35.29, 35.42] vs torch_general_tree_batcher IQM 63.190 ms [63.18, 63.20], 1.787x, mismatch 0, per 2026-06-22-phase2-deep-tree-search-frontier.md), SWE-bench live frontier (75.7% / budget-fraction 0.243 under low_empty_first, crossover to 2.022x@64K, per 2026-06-22-phase2-swebench-live-frontier-ablation.md), R2E manager 27.380x@65K with e2e ~1.00x (per 2026-06-22-phase2-r2egym-frontier-adapter.md) and the R2E live-worker smoke 1.986x vs host 1.978x (per 2026-06-22-phase2-r2egym-live-frontier-worker.md), and the NeMo crossover (per 2026-06-22-phase2-nemo-gym-live-blackjack-scale.md) — including the baselines we lose to and the host-Python control rows. The mechanism is novel: no-materialization frontier state, with the bet to co-manage it with KV via the topology-knowledge boundary.

10. Roadmap to claim the slot

Phase	Goal	Tied to	Measurable outcome
P1 — Prove the hard kernel	Certify `branch_step`; demonstrate capacity and a general-baseline throughput win in the shared-state regime	Handoff 1 (capacity → ML consequence), Handoff 2 (executed)	✅ mismatch 0 across M6 matrix (432+336 benchmark rows; 141 post-M6 focused tests, 341 broader pre-scope suite, invariant held across both); ✅ 3.43x-6.73x capacity (certified shared envs, backend-private hook, three converging methods); ✅ 1.787x deep-tree-search vs general batcher, non-overlapping CIs, scaling 1.075x@64 → 1.787x@128 roots (high-shared-context regime; external single-batch workloads remain negative vs vectorized CUDA — see §4)
P2 — Close the ML chain	At fixed peak GPU memory, show verified-solved-rate rises monotonically as pass@k widens on a real verifier-gated task; measure the addressable fraction p on a fresh live-model SWE-bench 64K run	Handoff 1, Handoff 3 (SWE-bench)	SWE-bench: 75.7% attempts saved (scale-independent budget fraction), manager crossover 0.011x@500 → 2.022x@64K, p measured directly to resolve the 4.00x-vs-capacity scenario, 0.688 single-run / 0.828 oracle-union; extend deep-tree curve to 256-1024 roots; head-to-head vs EnvPool/Brax/Madrona
P3 — Own real cost	Generalize the no-materialization shared-state contract to an external env class; lift the KV exclusion; co-manage frontier + generation + env/verifier	§6-7, extends Handoff 2	Certified no-materialization for an external env (R2E/NeMo-class) at ≥3x branches/GiB, mismatch 0; KV-bet validation on a generation-bearing run — measure `pre_gen_prune_rate`, skip decode for doomed branches when it exceeds ~20%, and confirm joint-scheduling e2e exceeds the 1.787x frontier-only ceiling (or document the boundary if <5%)
P4 — Adoption	Ship a stable, framework-agnostic rollout API; land the first external trainer integration	Handoff 3 (R2E + NeMo touchpoints)	VeRL runs rollout on BranchRT replacing its host-Python loop at the env/verifier dispatch seam, with zero changes to VeRL's optimizer/model code; reproducible speedup at scale

Sequenced, measurable, and already two phases deep with results in hand.

11. Risks & how we win

Risk	Honest read	Winning move
"A CUDA loop beats you."	True on early-stop-free, fixed-width Phase-1 workloads (ft06 0.9156x, routing 0.7618x, SAT 0.6251x) — by design, this is the boundary of our slot.	Don't fight there. The baseline that ships is host Python, and we crush it (75.14x SAT — within 11.3% of the CUDA latency floor while carrying lifecycle metadata it omits; 69.7x-266.7x RLVR). Compete on capacity + branching + plane ownership, where the loop can't follow — exactly where 3.43x-6.73x and 1.787x land.
"A host-Python frontier matches you on live tasks."	True today on env-latency-bound touchpoints, and at tiny scale the host edges ahead (NeMo 512-candidate touchpoint, overhead-dominated, ~0.7%) — KV not yet lifted there.	We measured the crossover and say it first: early-stop is architecture-independent, but the host edge flips as population grows — host wins ≤4K, BranchRT overtakes by 16K and holds at 32K. The win that never ties is 3.43x-6.73x capacity and 1.787x vs a general batcher — the GPU-resident, high-context regime, plus the topology-knowledge boundary.
"The win disappears on real workloads / R2E e2e is ~1.0x."	Correct, and we own it: on verifier-bound R2E, the manager is 27.380x but the Docker verifier is ~14.7h of wall-clock, so e2e ≈ 1.00x — Amdahl by design, and the deliverable there is capacity, not wall-clock.	The capacity→throughput conversion already holds against a general batcher (1.787x, non-overlapping CIs) and the frontier crosses real verifier boundaries (R2E, edging the host) and overtakes at scale (NeMo). P2 measures p directly and benchmarks EnvPool/Brax/Madrona head-to-head. The path to R2E e2e wins is moving the verifier on-GPU (P3).
"VeRL/AReaL/slime already build this."	They own real rollout layers — but on a different, composable axis (placement/disaggregation), none with a portable certified no-materialization frontier kernel or a measured capacity multiple.	Be the framework-agnostic standard: they disaggregate where inference runs; we deduplicate what the frontier stores — AReaL could run on top of our frontier. The certified ≥3x capacity result is the standing differentiator even if they ship a portable interface tomorrow. P4 names the VeRL seam and ships it.
"KV co-management is subsumed by prefix caching / vLLM can just add a hook."	RadixAttention/APC already cache the shared prefix and pay only marginal new-token cost; "don't submit the request" is available to any controller; vLLM could add a scheduling hook.	We grant all three. Our incremental win is the term APC cannot reach — skipping new-token decode for branches the frontier will prune this round (= prune_rate × decode_cost) — but the structural moat is amortization across a large frontier with no per-step host round-trip, gated on env/verifier state serving sees only after generation. The moment vLLM ingests topology, it is building a rollout plane — our category. Falsifiable P3 threshold (~20% pre-gen prune); degrades gracefully, never worse.
"The API never gets adopted."	Adoption is the unproven ingredient — and harder than publishing.	Land the VeRL integration (P4) at the named env/verifier seam, zero optimizer/model changes, off the existing R2E/NeMo plumbing — a low-friction drop-in. The certified contract is the expensive part competitors won't rebuild; one credible adoption flips the narrative, as it did for vLLM displacing TGI/FasterTransformer.

Every risk is a wedge we have either already split or have a concrete, measurable tool to split.

12. The bet

Serving got vLLM. Rollout has nothing — and agentic RL is making the live branching population, not the token, the bottleneck. The rollout loop is contested; the framework-agnostic frontier kernel-slot — a portable, certified, capacity-proven primitive — is genuinely unclaimed; the cost is migrating into rollout across three layers no one spans; and the framework world is re-inventing the same broken host-Python loop a dozen times over.

We have the hard kernel proven in mechanism — capacity 3.43x-6.73x (certified shared envs) from three converging methods; deep-tree-search 1.787x with non-overlapping CIs and a two-channel speedup with both channels independently measured; mismatch 0 across the M6 matrix. We have the ML consequence demonstrated on a real task via candidate replay — SWE-bench 75.7% attempts saved (scale-independent), a manager-speedup crossover 0.011x@500 → 2.022x@64K, and a 4.00x projection stated as one phenomenon read two ways, sitting just under its 4.11x attempts-only ceiling, with its premise named — and a fresh live-model 64K run as the P2 deliverable. And we have the live plumbing crossing real agentic environments today — edging the host control on R2E Docker (mismatch 0) and overtaking it at NeMo scale (3.21x vs 2.94x, a measured crossover). We are further along the adoption axis than vLLM was at its inflection: it had a kernel and no live integrations; we already run live across real verifier boundaries. What remains is the campaign: close the fixed-memory pass@k chain, measure p, lift the KV exclusion, co-manage the plane on the topology-knowledge boundary, ship the API, and land the VeRL integration.

That campaign is hard — and that difficulty is the moat. Hard for us is hard for everyone, and we are already inside the abstraction with a certified kernel none of them have.

BranchRT is the rollout data plane for agentic RL — the paged-attention of the frontier. The slot is open. We take it.