Your AI agent got it right once. Can it do it again?

A successful attempt is only one kind of success

An agent finishes a task, says it is done, and produces a tidy explanation. The important question may be somewhere else: which records changed, which stayed unchanged, and whether the same task works reliably when repeated.

Microsoft and Hugging Face's October 22 ThinkingBox-Training write-up describes using executable checks of final backend state as reinforcement-learning rewards. In their reported Qwen3.8-27B experiment, average success increased from 51.70% to 60.78%, while tasks passing all twenty recorded attempts increased from 38 to 107 out of 507. At-least-once success across twenty attempts declined slightly. These are the authors' results, not a reproduction by Signal Scroll.

The trade-off is the story. Getting a task right occasionally and getting it right repeatedly are different measurements. Neither a best-case video nor a single average tells you the entire shape of performance.

This article offers a proposed small evaluation exercise. It does not ask you to train the released model or connect an experimental agent to a production database. The repository's reference training setup uses substantial GPU resources. You can learn the measurement lesson with a paper scenario or an isolated toy app.

Our hypothetical app lends equipment to a student club. A member requests a camera, an approval rule determines whether a reservation may be made, and the inventory must remain consistent. All people, records, and outcomes in this example are invented for explanation. We will define success before asking an agent to act, then distinguish a correct state from a persuasive story about that state.

Source: Microsoft and Hugging Face: ThinkingBox-Training results and methods ↗ · Microsoft: ThinkingBox-Training repository and requirements ↗ · SQLite: SELECT queries ↗ · SQLite: Transaction control ↗

Write the expected state before the prompt

Begin with three fictional records: an item, a member, and a reservation request. Give them stable identifiers so that a name change cannot make your check point to the wrong object. Record the initial state in a small table or a local test fixture.

For the example, the camera is available, the member has no current reservation, and the request is waiting for approval. The task is to prepare the request for review, not to approve it. A successful result leaves the camera available and creates or updates the pending request in the agreed way.

Now write the expected final state in plain language. The request exists once, the status remains pending, no checkout has occurred, and unrelated inventory records are unchanged. This is your proposed contract for the exercise. It is intentionally stricter than the agent saying it created a request.

Add one forbidden side effect. The agent must not mark the camera as borrowed. That makes the difference between progress and premature completion visible. You can ask whether the final state meets the contract without requiring the agent to follow one exact sequence of internal steps.

Then write the user prompt. Keep the expected-state check separate from the language you give the agent. Otherwise you risk changing the definition of success to fit whichever result appears.

For a paper-only version, ask a friend or an AI to describe the actions it would take, then inspect the resulting fictional records yourself. Label this a reasoning rehearsal rather than an execution test. For a real local toy app, inspect the actual stored records. The exercise becomes more informative when the verdict depends on what happened, not how well the explanation was written.

Check changes and non-changes

A task can require both doing something and leaving something alone. In the camera example, a pending request should appear, but the inventory status should not change yet. Checking only the new request would miss an accidental checkout. Checking only the inventory would miss a request that was never recorded.

For a local SQLite exercise, SELECT is the documented read-only query statement: it returns rows without modifying the database. Use appropriate queries or the toy app's own inspection interface to check the relevant records. Keep the evaluator separate from the agent's completion text.

A proposed verdict could contain four independent observations. The intended request exists. Its fields match the task. The camera remains available. No unrelated record changed. Each observation should point to evidence you can inspect. Do not compress all four into a single success badge before you understand their differences.

Also distinguish a missing result from an unreadable result. If the local test environment fails to return records, the attempt may be inconclusive rather than a normal task failure. Record the infrastructure issue separately. Otherwise you can blame the model for a test setup problem or excuse a model mistake as a setup problem.

When designing the toy app, transactions may help group related database work. SQLite's transaction documentation explains explicit commit and rollback behavior. A transaction is not itself proof that the application performed the correct operation; it is a mechanism whose use still needs to match the intended task.

The human question remains straightforward: did the final state satisfy the request and preserve its constraints? The more precisely you can answer that, the less dependent your evaluation becomes on a particular wording or a favored sequence of tool calls.

Repeat from the same starting point

To compare attempts, reset the toy environment to the same initial state each time. Otherwise attempt two may inherit the request from attempt one and become a different problem. A persistent workspace is useful in real life, but an uncontrolled starting state makes a repeatability experiment hard to interpret.

Choose a small number of repetitions that you can inspect thoroughly. For a learning exercise, even a handful of carefully logged attempts can reveal a failure pattern. Do not call that a statistically complete benchmark. Record the number of runs, the reset process, and the conditions.

Keep three separate counts: successful attempts, tasks with at least one success, and tasks that succeeded on every recorded attempt. In a single-task exercise, the latter two are simple yes-or-no statements. Across several tasks, they reveal different patterns.

Consider an invented example with two tasks and five attempts each. One task succeeds five times; the other succeeds once. Average attempt success is six out of ten. Both tasks succeed at least once, but only one succeeds every time. Those descriptions are all true, and they suggest different deployment decisions.

Do not turn five successes into a promise that the sixth attempt will work. Observed consistency is evidence within recorded conditions, not a guarantee. The point of repeated trials is to make variation visible so you can decide what further evidence matters.

Our recommendation is to save failures as carefully as successes. A failed attempt can show a specific missing condition, ambiguous instruction, or unwanted side effect. A montage of only the successful runs would erase exactly the information the exercise was meant to discover.

A grader can be wrong even when it runs perfectly

An executable check feels objective because it produces the same verdict for the same input. But the rule may still measure the wrong thing. In our example, a check that only looks for a pending request might reward an agent that also marks the camera borrowed. The code could run flawlessly while the definition of success remains incomplete.

Try to break your proposed grader using invented final states. Create a correct state and several near-misses. One has a duplicate request. One uses the wrong member identifier. One reserves the wrong item. One changes an unrelated record. Ask whether the check rejects each case for the intended reason.

This is a design exercise for your evaluator, not adversarial activity against someone else's system. Use only data and environments you control. The aim is to find where your own success definition fails before you rely on it.

Avoid making the grader unnecessarily specific. If two legitimate paths leave the correct state, the check should not reject one merely because the agent used a different order of harmless operations. On the other hand, if the task explicitly requires approval before a change, a final-state snapshot alone may not establish the order. You may need additional process evidence.

Write that limitation directly in the result. Final state verifies the stored outcome is a narrower claim than the entire process followed policy. Different questions require different observations.

The strongest practical lesson from reward-based training is also a lesson for ordinary automation: the system will be judged through the criteria you choose. Spend real thought on those criteria. A better model cannot rescue an evaluation that confidently rewards the wrong outcome.

Keep practice separate from the final exam

The ThinkingBox-Training repository documents a fixed training pool selected through probes and a separate benchmark evaluation. It says the benchmark tasks were not included in training. Its reference requirements include Linux, Python 3.12, NVIDIA GPUs, backend services, and model endpoints. This is a research training stack, not a one-click upgrade for a casual toy app.

For our proposed exercise, the analogous habit is simpler. Use some fictional tasks while designing prompts and checks, then reserve a few related tasks you have not used to tune the workflow. For example, develop the camera request scenario first and hold back a different item with a different approval condition.

Do not repeatedly adjust the prompt against a held-back case and continue calling it unseen. Once you use the result to make changes, it becomes part of development. You can still learn from it; just describe its role honestly and choose another suitable test when needed.

Also hold the evaluator stable during a comparison. If you change the model, prompt, fixtures, and grading rules together, you will not know which difference caused the observed result. Record changes deliberately and compare only what the evidence supports.

A useful learning log includes the hypothesis behind each change. Perhaps you expect an explicit identifier to reduce wrong-record edits. Perhaps a clearer approval instruction should reduce premature checkouts. Evaluate those hypotheses rather than collecting improvements without an explanation.

The point is not to mimic a large research lab's scale. It is to adopt a small version of the separation between development and assessment. That separation makes your final claim more useful because it tells another reader what the system had already practiced and what it encountered later.

Make a decision that matches the evidence

At the end, write a short result for the hypothetical club app. State the task, the starting state, the checks, the repetitions, and the observed failure types. Then say what you would permit next. Drafting a request for human review may be acceptable before automatic reservation changes are.

A useful decision might be that the agent can prepare pending requests in the sandbox, while every actual checkout still requires a person. Another could be that the prompt is too ambiguous to evaluate fairly. A third could be that your grader cannot yet detect duplicate records. These are actionable outcomes even without a dramatic score improvement.

Keep the agent's final message as one piece of the record, not the verdict. Compare it with the stored state. If the explanation says the task succeeded but the records disagree, preserve both. The discrepancy is an important finding for your application design.

When sharing results, avoid a universal reliability label. Describe the version, task set, and conditions. State whether the test involved real execution or a paper rehearsal. Include the denominator and do not hide the failed runs behind the best example.

The new training results are interesting because they show average success and repeated success moving differently. The practical habit is available without training a model: define the outcome, check the real state, reset fairly, and count consistency separately.

You should finish with a more precise answer to a modest question. Can this agent perform this task under these conditions, and what happens when it does not? That answer is more valuable for a working application than a polished demonstration that happened to go right once.

YOUR NEXT MOVE

Try this, then make it yours.

Invent a three-record equipment-request task. Write the expected final state and one forbidden side effect before drafting the prompt. Rehearse several attempts from the same initial state and keep each verdict separate from the agent's explanation.

Explore the tool ↗

Follow the signal.

Our reporting starts here. Practical suggestions are our analysis, and vendor performance statements are claims unless independently verified. We haven’t hands-on tested this release.