Two teams use the same model. One agent finishes the task. The other repeats the same mistake at step three. If the model is identical, the remaining variable is the working environment around it: prompts, tool descriptions, memory, skills, subagents, and execution scripts.
If the concept is new, What is harness engineering? starts from the basics.
Harness², released by a Google Research team on October 9, is a framework for improving that environment. The superscript stands for two levels of recursion. One loop refines the harness for the task at hand. The other accumulates lessons for future tasks. Model weights stay frozen.
The interesting part is not the headline score. It is the method: how do you compare harness edits in an open world where no ground-truth verifier is available? This article walks through the paper and the official repository, then reduces the idea to a small experiment a team can run without the full benchmark stack.
What is public, and what is not
The repository is Apache 2.0 and the paper is arXiv:2610.14224. The repo includes code, benchmark guides, and a demo. The README also says the project is not an officially supported Google product. Treat it as research code, not a hosted service.
The evaluation covers:
- Three benchmarks: WorkBuddy-Bench (code, web, office), JobBench (professional work), and LAB (legal work)
- Two executor models: GLM-5 and Gemini 3.5 Flash
- Two executor harnesses: OpenCode and Codex
- A harness editor: Claude Opus 4.8, running under OpenCode
- Three runs per configuration, with shuffled task order
One condition matters. The executor is frozen, but the model that edits the harness is stronger. The paper applies the same editor to all baselines, so the comparison is controlled. Still, this is not a weak model improving itself. It is a stronger editor repairing the environment around a weaker executor.
Two loops: fix this task, keep the lesson
Task-level recursion has three stages: Propose, Probe, Compose.
[Task-level recursion inside one task]
Current harness H0
|
+- Propose: generate invariant and domain-specific edits
|
+- Probe: run the same task with candidate harnesses
|
+- Compose: compare behavioral differences and assemble H1
No ground-truth label. The signal is what changed on the same task.
There is no verifier. Instead, the system runs the same task under several harnesses and compares the deliverables and execution traces. One edit may reduce tool calls; another may make file handling more consistent. The signal is behavioral contrast, not a score. The paper calls this a contrastive update.
The second loop carries experience forward. After a task, the pipeline stores the adopted harness and evidence about why an edit worked in a persistent improvement harness. Future tasks retrieve reusable components and lessons from that library. It does not inherit the previous execution harness wholesale; it selects the parts and conditions that are likely to transfer.
This shifts the unit of self-improvement from the model to the working environment. That is cheaper and more reversible than retraining. It also creates a failure mode: a bad lesson can contaminate later tasks. The paper therefore ablates the library separately.
How to read the numbers
The headline is "up to +11.5 points from a single improvement step." That number comes from WorkBuddy-Bench, where GLM-5 under OpenCode moved from 64.2 to 75.7. On the same benchmark, Gemini 3.5 Flash under OpenCode moved from 72.4 to 77.5, a gain of 5.1 points. The weaker executor gained more in absolute terms.
| Benchmark | Evaluation set | Metric | Best one-step gain | Example |
|---|---|---|---|---|
| WorkBuddy-Bench | 40 code, 35 web, 25 office test tasks | Composite verifier pass fraction | +11.5 points | GLM-5/OpenCode 64.2 → 75.7 |
| JobBench | 34 professional tasks | Weighted rubric score | +4.5 points | Gemini 3.5 Flash/OpenCode 31.5 → 36.0 |
| LAB | 100 held-out tasks from 1,251 | Per-criterion pass rate (Partial) | +11.8 points | GLM-5/OpenCode 59.6 → 71.4 |
Increasing the recursion budget from k=1 to k=5 adds more. The paper reports +3.9 points on WorkBuddy, +4.2 on JobBench, and +6.7 points on LAB. These are additional gains on top of the one-step result, and they come with additional execution cost. For Gemini 3.5 Flash under OpenCode, moving from k=1 to k=5 raises the gain from +4.5 to +7.5 on JobBench and from +5.1 to +8.4 on WorkBuddy. The improvement is bought with compute.
The ablations show where the gain comes from. Removing the improvement library costs 3.1 points on WorkBuddy and 1.8 on JobBench. Removing scoped editor roles costs 2.9 and 3.7. Removing composition costs 1.6 and 1.9. All three mechanisms contribute.
The cost structure of a harness
Harness² exposes eight editable components. The split is useful even if you never run the framework, because it separates what sits in context on every call from what is loaded on demand.
| Component | When it enters context | Cost and role |
|---|---|---|
| System prompt | Always | Every token competes with output budget |
| Guardrails | Always | Standing rules, permanent context cost |
| Long-term memory | Always | Domain facts, permanent context cost |
| Tool guidance | Always | How to use tools, permanent context cost |
| Skills | When invoked | Near-zero cost on tasks that do not need them |
| Scripts | Via bash | Deterministic execution instead of token-by-token reasoning |
| Subagents | When delegating | The only component that adds another inference context |
| Plugins | During the tool loop | Middleware that interposes on behavior |
The practical lesson is that adding system prompt text is the most expensive way to improve a harness. It is always on, so every extra sentence raises cost and latency on every call. Skills and scripts are conditional. Moving a repeated procedure into a skill, and a deterministic calculation into a script, is often better token economics than explaining it again in the prompt. Subagents are powerful, but each one creates another context.
The ablations reflect this structure. The library, scoping, and composition each contribute, which suggests the harness is a system of components, not a single prompt.
A one-day harness audit
Running the full framework requires Vertex AI credentials, Docker, and benchmark dependencies. A team can start smaller. The procedure below is a manual adaptation of the paper's method. It is not an execution of the official tool.
- Pick three recurring failure types from recent agent logs. For example: wrong test command, reading the same file twice, summarizing before completion.
- Define one small task where the failure is checkable. The deliverable should be a file or a test result, not a subjective impression.
- Run the current harness on the task three times. Record the score, the number of model turns, and an approximate token cost.
- Change one component at a time. Instead of lengthening the system prompt, move a repeated procedure into a skill or pin a verification command in a script.
- Run the same task three more times. Look at the trace, not only the score. Which step disappeared? Which error stopped?
- Write one lesson for the next task, including the condition under which the edit worked.
A simple experiment log is enough:
| Edit | Expected change | Observed change | Adopt? |
|---|---|---|---|
| Pin the test command in a script | Fewer wrong command choices | Turns 12 → 9, test failures 2 → 0 | Yes |
| Add a completion rule to the system prompt | Fewer early stops | No change, more prompt tokens | Hold |
| Move file-reading rules into a skill | Fewer duplicate reads | Duplicate calls 3 → 1 | Yes |
This makes the first question "what can we fix in the harness?" rather than "which model should we buy?" The paper's library is ultimately a formal version of this habit.
To try the official demo, the README gives the commands below. They are quoted from the official documentation and were not executed for this article.
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2
The demo uses deterministic editors and real Python execution and requires no model credentials, according to the README. Model-backed benchmarks need Node.js 20+, the Google Cloud CLI, and Vertex AI configuration. WorkBuddy-Bench needs Docker; LAB needs Pandoc and Poppler. The README warns that model-backed runs incur API costs and recommends starting with a small task selection.
Limits and cautions
First, this is not an independent reproduction. Every number here comes from the paper and repository. I did not find a third-party replication at the time of writing.
Second, the editor matters. Claude Opus 4.8 proposes the harness edits. Because the editor is stronger than the executor, the absolute gain is not purely "the harness got better on its own." The paper applies the same editor to baselines, but the numbers should not be transplanted to another system without re-testing.
Third, benchmark contamination is a real risk. The README states that executor network access is unrestricted and that Codex runs with its sandbox disabled. Public benchmark rubrics may be online. The documentation tells you to configure network restrictions before evaluation. If you skip that step, a high score may reflect access to the answers.
Fourth, cost and generalization. The paper acknowledges that recursive refinement needs extra executions and editor sessions. The optimal budget was characterized only over a bounded range, not for every task. The method also depends on execution-based feedback, so it is weaker when behavioral differences are hard to observe.
Fifth, this is not a product. The repository says it is not an officially supported Google product. Check the code, license, and maintenance status before putting production dependencies on it.
Conclusion: audit the harness before swapping the model
The real message of Harness² is not "agents improve themselves." It is that harness improvement can be turned into a measurable loop: run the same task under several harnesses, compare behavior, and keep the lessons that transfer. You can start that loop without the full framework. The model/harness distinction is covered in Model vs. harness, and cost per outcome is covered in AI agent KPIs.
Self-improvement still has boundaries. Humans set the objective, budget, constraints, and adoption criteria. The system proposes, execution compares, and a person remains accountable. Blur that line and you get the illusion of automatic progress.
If you are considering a model upgrade, first pick three recent failures and fix the harness once for each. That experiment is often cheaper than a new model contract.
References
- Harness²: Recursive Agent Harnessing for an Open World (arXiv:2610.14224, Oct 9, 2026)
- google-research/harness2 (GitHub, Apache 2.0)
- Harness² paper HTML
- Harness² official site
Go deeper with a course
Designing a harness as a set of components, and building loops where execution and verification repeat, makes the paper's choices concrete. The useful skill is deciding when to use a skill, a script, or a subagent instead of adding another paragraph to the prompt.