This is the second note in a series on long-horizon agent design, prompted by a close read of BEACON (Zhejiang University, arXiv:2605.06078).
The first note argued that loop detection cannot catch a stalling agent, because repetition is an input-side signal while progress is an output-side one. That argument only works if there is something to measure progress against. This note is about that something.
Ask the question the right way
The question is not "how does the agent know it finished a step." It is "how does the system know it really finished, rather than announcing that it did."
One layer of difference, and the credibility is not comparable.
We already had two halves and never connected them
Reading our own implementation, this was the surprising part.
The first half. Milestones already exist in the product. A tool maintains a durable task plan: how a long task decomposes, and what state each step is in — pending, in progress, completed, blocked. Its stated purpose is to give a long task a stable progress anchor. But how does a step become "completed"? The model declares it.
The second half. After every tool call, the host records a set of deterministic facts: whether a file's content hash actually changed, what a command's exit code was, whether it timed out. These are recorded host-side and never enter model context, which is exactly why they cannot be fabricated.
That is where the gap sits. One line says I finished. The other says a file's hash went from A to B. Nothing has ever put them side by side.
What is missing is not a data structure, and not observability. It is the bridge between the two.
The most useful thing in the paper is not the formula
BEACON is best known for its dual-scale advantage, but the part worth stealing is how it positions the detector Φ.
Φ needs no trained model and no human annotation. It reads only state changes observable in environment feedback: object state transitions in ALFWorld (successfully picked up, heating finished), page transitions in WebShop, and in ScienceWorld it simply consumes the subgoal signal the environment already emits.
Zero extra model, zero extra sampling cost. Compare the alternatives: a process reward model needs expensive annotation and can be gamed, and Monte-Carlo value estimation needs extra rollouts at every decision point. That cost is what Φ avoids.
Anyone building an agent product has an advantage the paper's settings do not: tool calls are structured to begin with, with explicit success and failure. The paper had to mine signals out of an environment. Ours are already there.
Three layers, if you were going to build it
Layer one: collect deterministic evidence into one stream. No semantic judgment at all, just a mechanical record of irreversible verifiable changes. A file's content hash changed. A command exited zero. An external API call returned success. A row was committed. All of this exists today. It is simply scattered, never gathered into a single record of facts established so far.
Layer two: connect the two halves. The highest-value step. When the model declares a step complete, stop taking it on faith and go look for layer-one evidence in that window. Evidence present, mark it verified complete. No evidence, mark it declared complete. This does not stop the model from declaring anything; it only separates the supported from the unsupported.
Layer three: define Φ per capability type. The authors concede that Φ requires domain knowledge and generalizes poorly, so do not expect one universal detector. Coding tasks watch test exit codes. Data-analysis tasks watch whether the output file materialized. Messaging tasks watch the API response. This layer gets built out slowly, one capability at a time.
One criterion we added: irreversibility, not importance
This one is not in the paper.
BEACON's milestones work because they mark state transitions you cannot walk back. Once you have the key, the world is different. So the criterion should be did this action produce an irreversible external side effect, rather than was this step important.
Irreversible: writing a file, committing code, sending a message, calling a paid API, committing to a database. Reversible: reading a file, searching, fetching a page, thinking.
Two things follow. It is mechanically decidable, because the action type settles it — no semantic understanding required, and no "the model thought this step mattered" subjectivity leaks in. And it coincides with recovery points: resuming from an irreversible boundary is the only kind of resume that means anything, since reversible actions can simply be redone.
Two numbers: one for nerve, one for caution
The nerve-steadying one is the degradation experiment. Randomly drop half the milestones and the score is still 82.8, against a 72.8 baseline — ten points ahead. Degradation is smooth, not a cliff. For anyone shipping this, that is the important part: you do not have to get Φ perfect before you dare turn it on. Covering half already pays.
The cautionary one is the partitioning comparison.
| Partitioning | Score | vs. baseline (72.8) |
|---|---|---|
| Random 5-way split | 74.2 | +1.4 |
| Real milestones | 91.4 | +17.2 |
The first note quoted these numbers as well, but here the implication is more direct. If your milestones are set by intuition, they are approximately a random split, and the work is wasted. Milestones are valuable precisely when they line up with the real structure of the task.
Where this has to be discounted
The paper's own appendix lists automatic milestone discovery as an open problem. All three benchmarks obtain their milestones from rules: pattern-matching environment responses, page transitions, or a signal the environment hands over directly. Genuinely open-ended settings — browser operation, codebase refactors, deep research — have no such ready-made verifiable transition.
So this is a paradigm validated inside structured environments, not a solution you can lift wholesale. What makes it land in an agent product is the structured boundary of the tool call, not an answer the paper already supplied.
The other trap is granularity. Too sparse and you have done nothing; too dense and the segment-level signal becomes noise. In product terms that is the granularity of task decomposition, and here there is a convenience the paper's settings lack. Plan steps are user-facing by design, so granularity can be anchored to whether a person can understand the step, instead of being tuned purely by algorithm.
If you only do one thing
Do layer two.
It is cheap, because layer one's data already exists and layer two only correlates it with the model's declarations. It is independently verifiable, because the ratio of verified to declared is itself a metric worth watching. And it is the precondition for everything else: without a notion of a verified milestone, the stall detection from the first note and the compaction boundary in the next one both have nothing to stand on.
It also has a comfortable property. Shipping it changes no behavior — the model declares steps exactly as before, there is simply an extra mark. Once the data accumulates, you can decide whether to intervene on the declarations that arrived with no evidence.
The next note covers context compaction: why cutting at a token threshold has little to do with what is safe to forget, and a correction to the most natural way to extend this note, which is to assume that once a milestone is verified, everything before it can be folded away.