Motivation
An LLM agent acting in an environment spends most of its wall-clock time waiting. Each
step — a model call, a tool or MCP request, a browser action, sometimes a human reply —
must complete before the next can be issued, and the round trips dominate end-to-end
latency: a chess game between two reasoning agents runs for hours, and an
operating-system tuning task for tens of minutes. When a training or prompt-optimization
loop repeats such a run thousands of times, the waiting is the cost.
Speculative actions (Ye, Ahuja, Liargkovas, Lu, Kaffes, Peng, ICLR 2026)
transplants a classical systems idea — speculative execution in microprocessors, and
speculative decoding for LLM inference — to the agent's environment loop. A cheap, fast
speculator guesses the action a slow, authoritative actor is about to produce,
the guess is used to launch the next environment call early, and the work is committed
only when the actor's real action confirms the guess. The interface stays sequential and
lossless; the internals run in parallel.
What makes this a formalization target rather than an engineering report is the paper's
§5 cost–latency analysis. Speculating more branches buys hit probability but costs
tokens, and the paper derives closed-form expressions for both sides of that trade — a
self-contained piece of applied probability sitting underneath a systems paper. This
mission asks for those expressions, machine-checked.
Setting
Fix a horizon T and index steps t=0,1,…,T−1. At each step a policy maps the
state to an API call; the actor executes it with latency Exp(β), while
the speculator proposes candidate actions with latency Exp(α), where
β<α (the speculator is faster in expectation). A speculative branch
hits when the action it guesses implies the same next call the actor's true action
would have implied; branches hit independently across steps with probability p.
Two knobs define the two regimes analyzed. Breadth k: at each step, launch k
independent one-step speculations in parallel, each immediately followed by a real call.
At least one of the k succeeds with probability
p(k)=1−(1−p)k.
Depth: follow a single branch, extending it whenever a speculative or real call
returns and pruning subtrees the actor contradicts.
The quantity driving both results is Sn, the expected number of hits by round n. A
hit consumes the following step's speculation window — after a correct guess the next
call is already cached, so no new speculation is launched there — which yields the
two-term recursion
S0=0,S1=p,Sn=p(1+Sn−2)+(1−p)Sn−1.
Write Tseq,Mseq for the latency and token cost of strictly
sequential execution, and Tspec,Mspec for their speculative
counterparts. In the depth regime latencies are taken deterministic: a for a real call,
b<a for a speculative one.
Target
The goal theorem is the finite-horizon latency ratio for breadth-focused speculation
(Proposition 1), with p(k) abbreviated pk:
E[Tseq]E[Tspec]=1−T1α+βα[1+pk(T−1)pk+(1+pk)2pk2−(1+pk)2pk2(−pk)T−1].
The supporting targets, ordered as the analysis builds them:
- the closed form Sn=1+ppn+(1+p)2p2(1−(−p)n) solving the recursion;
- the per-hit saving E[(B−A)+]=β(α+β)α for independent A∼Exp(α), B∼Exp(β);
- the T→∞ limit 1−1+pkpk⋅α+βα, and the resulting 50% ceiling: the latency reduction is strictly below 21 for every pk≤1;
- the cost counterpart (Theorem 4), finite-horizon and in the limit, with k~ the number of distinct actions across the k branches;
- the depth-focused time and cost identities (Theorem 6), whose latency coefficient is p rather than 1+pp — raising the speedup ceiling from 21 to 1;
- the structure of confidence-aware selective speculation (Theorem 3 and Corollary 5): with sorted per-branch confidences, the marginal hit-probability gain is non-increasing, so the optimal breadth is the greedy threshold rule "add a branch while Δ⋆δq(m)≥c".
Significance
The analysis is what turns speculation from a trick into a tunable system. Proposition 1
and Theorem 4 are governed by the same quantity pk, so a practitioner who can
estimate hit probability can choose k offline against a latency/cost budget rather than
by trial. The 50% ceiling is a genuine negative result — it says breadth alone cannot do
better, and motivates the depth regime, where the ceiling becomes 1. Theorem 3 explains
why confidence-based branch selection is cheap in practice: the whole dynamic program
collapses to one scalar continuation value, so a runtime system sorts confidences and
adds branches greedily in O(k) per step.
The paper's proofs are pen-and-paper and, as far as we are aware, none of these results
has a machine-checked proof. Three parts reward formalization specifically. The
recursion's closed form is derived by a characteristic-equation argument with a
particular solution that collides with the homogeneous part — routine but error-prone.
The per-hit saving is an honest two-dimensional integral over independent exponentials.
And Theorem 6's cost expression is stated in the paper with a floor function and then
immediately replaced by an approximation, so formalizing it forces a decision about which
claim is actually being asserted (see Formalization scope).
Difficulty
The obvious first move on the recursion — guess a constant particular solution — fails,
because r=1 is a root of the characteristic polynomial r2−(1−p)r−p and a
constant trial collides with the homogeneous family; the particular solution is linear in
n, and the (1+p)2p2 coefficient comes out of matching both initial
conditions, not one.
The interesting hypothesis is the one the recursion's shape encodes and the prose states
only in passing: a hit at round t removes the speculation window at round t+1. Drop
it and the recursion becomes one-term and the answer changes.
For the per-hit saving, the difficulty is analytic rather than algebraic: the inner
antiderivative of (b−a)αe−αa must be handled, and the outer integral runs
over an unbounded interval, so integrability has to be established rather than assumed.
The asymptotic statements need the oscillating term (−pk)T−1 controlled uniformly —
it is bounded, not vanishing termwise in an obvious way — before the T1 prefactor
can be taken to zero.
Formalization scope
Everything is over R. The model lives in one definition bundle,
Def_SpecActions_model, in namespace SpecActions; the mission's Lean names match the
prose symbols (Sn is hits, p(k) is phit, k~ is kt).
The model is formalized at the level the paper's own proofs use: E[T] and
E[M] are defined by the expressions Appendix A derives for them
(specTime, specCost, and their depth analogues), and the theorems assert the
algebraic and asymptotic identities relating those quantities. Deriving those
expressions from a measure-theoretic model of the execution trace is deliberately not
in scope — with one exception: milestone 2 states the per-hit saving as a genuine
iterated integral against the exponential densities, so the one probabilistic step the
paper actually computes is formalized as an integral rather than assumed.
Conventions a solver should know before starting:
- Statements are quantified over α,β>0 and 0≤pk≤1; the standing
assumption β<α is not imposed, since none of the identities need it.
- Finite-horizon statements carry 1≤T, and T−1 is natural-number subtraction —
the T=0 case is excluded rather than silently truncated.
hits takes pk (the per-step hit probability p(k)), not the per-branch p;
phit relates the two, and Thm_SpecActions_phit_bounds supplies the
0≤p(k)≤1 range facts the other statements assume.
- Theorem 6's cost is stated as the exact identity, not the paper's approximation.
The paper gives an exact expression involving ⌊a/b⌋ and then an
≈ form with 2ba−21; these coincide only when a/b is an
integer. The milestone asserts the exact floor version, which is what the proof
establishes.
- The 50% ceiling is stated as the strict bound
1+pkpk⋅α+βα<21, which holds for all
admissible parameters; the paper's "upper bound of 50%, occurring when p=1 and
α=∞" describes an unattained supremum.
- Theorem 3's dynamic program is formalized as the two facts that carry its content —
diminishing marginal returns, and optimality of the greedy threshold breadth — rather
than as a Bellman recursion over a mode process, which would require a full MDP
development.
Reusable beyond this mission: the two-term linear recursion solved in milestone 1, and
the E[(B−A)+] computation for independent exponentials, which is a standard
fact absent from Mathlib. Contributions extending the model toward an actual measure on
execution traces — deriving specTime rather than defining it — are welcome as
follow-on work.
Selected references
- Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, Tianyi Peng. Speculative Actions: A Lossless Framework for Faster Agentic Systems. ICLR 2026. arXiv:2510.04371 — Proposition 1 (p. 4), Appendix A (pp. 13–14), Theorem 3 (p. 10), Theorem 4 (p. 19), Corollary 5 (p. 22), Theorem 6 (p. 23).
- Yaniv Leviathan, Matan Kalman, Yossi Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192 — the speculate-verify pattern at token level.
- Wenyue Hua, Mengting Wan, Shashank Vadrevu, Ryan Nadel, Yongfeng Zhang, Chi Wang. Interactive Speculative Planning. 2024. arXiv:2410.00079 — depth-oriented speculation on a single planning branch.
- Yilin Guan et al. Dynamic Speculative Agent Planning. 2025. arXiv:2509.01920 — online RL for choosing speculation depth under a cost-latency trade-off.
- Robert M. Tomasulo. An Efficient Algorithm for Exploiting Multiple Arithmetic Units. IBM Journal of Research and Development, 1967. DOI:10.1147/rd.111.0025 — speculative execution in hardware.