When selecting AI coding infrastructure, engineering teams instinctively look at foundation model leaderboards—MMLU, HumanEval, or SWE-bench. The conventional assumption is straightforward: the highest-ranked model on public benchmarks will naturally yield the highest success rate inside any IDE or CLI workflow.

The FrontierHarness Eval v1.1 benchmark published by independent research collective RUNTA directly disproves this assumption. Measuring 22 distinct agent harnesses across 5 frontier models over 27,280 controlled runs, the study demonstrates that no single harness architecture is universally optimal across all models.

Key Findings from 27,280 Executions

FrontierHarness Eval v1.1 pinned 5 frontier models (including Claude 3.5 Sonnet, GPT-4o, and Codex families) under identical task suites while systematically swapping the surrounding agent harness—altering prompt framing, tool schemas, and error propagation.

The empirical results show:

  1. Intra-Model Success Divergence (up to 35pp): Holding model weights and temperature constant, changing only the harness orchestration produced task completion swings between 44% and 79%.
  2. Cost Ranking Inversions: A model ranking as the most cost-effective under Harness A frequently collapsed to the most expensive under Harness B due to redundant tool-calling cascades and context bloat.
  3. Incompatible Harness Optimizations: A prompt compression strategy tailored specifically for Model A triggered a 2.3x increase in debugging cycles when transplanted onto Model B.

What Causes the Harness Discrepancy?

The agent harness acts as the Control Surface mediating between LLM reasoning and system tools. Minor architectural choices compound into massive variances across multi-turn loops.

1. Tool Schema Conciseness vs. Verbosity

Certain models struggle with extensive JSON Schema descriptions, misplacing positional constraints within nested blocks. Yet, when presented with concise TypeScript interface definitions, their argument validity surges. Conversely, other models overlook implicit constraints in types and require verbose markdown docstrings. When a harness's tool definitions conflict with a model's pretraining bias, schema repair loops consume excessive tokens.

2. Context Truncation and Log Scrubbing

As an agent traverses file trees and executes test suites, terminal output rapidly accumulates.

  • Sliding Window Buffers: Retaining merely the last 10 turns deprives the agent of early structural assumptions during multi-file refactoring.
  • Periodic Summarization: Condensing historical turns saves token volume but risks discarding compiler line numbers critical for precise patching.

3. Surgical Diagnostic Feedback

Re-injecting the full raw stderr output into the model prompt floods the context window with stack noise. In contrast, harnesses that parse compiler logs and inject only failing assertions alongside relevant git diffs improved error recovery rates by over 40%.

Comparative Benchmark Matrix

The following table summarizes core findings from the RUNTA dataset:

Dimension Harness Architecture A (Raw Terminal Feed) Harness Architecture B (Structured Diagnostic Feed) Observed Impact
Task Completion Rate 52.4% 81.2% +28.8pp success rate
Avg Tokens per Task 142,000 tokens 58,000 tokens 59.1% cost reduction
Turns to Resolution 7.4 turns 3.1 turns 58% latency reduction
Cross-Model Portability Ranked #1 on Model X, #4 on Model Y Ranked #1 on Model Y, #2 on Model X Refutes universal harness thesis

Practical Engineering Guidance

  1. Decouple Model and Harness Benchmarks: Never benchmark a new LLM release by merely swapping an API endpoint into an existing wrapper. The prompt harness, tool interfaces, and error feedback must be tested as independent variables.
  2. Build Model-Specific Adapter Layers: Teams deploying multi-model routing should implement adaptive harness layers that tailor tool schema formatting to each model family's strengths.
  3. Account for Effective Cost per Solved Task: A cheaper per-token model running in a poorly matched harness can easily rack up a higher total invoice than a premium model resolving tasks in a single turn.