Why Octomind Solves 24/25 with the Same Model: Harness Matters More Than Model

Many developers look at coding agent benchmarks and check only the model name. "Claude Code + Opus 5 scores 23/25, Codex + GPT-5.6 Sol scores 21/25." Different models are mixed in, so harness-to-harness comparison is impossible.

Octomind's benchmark takes a different approach. It runs the same glm-5.2 model through multiple harnesses. Result: Octomind solves 24/25, OpenCode solves 19/25. Same model, different harness, 5-problem gap.

Benchmark structure: 25 real PR fix cases

Octomind uses octobench, which runs 25 real fix cases harvested from merged PRs across Python, PHP, Rust, C++, and JavaScript. Each agent works in the pre-fix repo; held-out tests from the merged fix decide pass or fail.

Official benchmark results (published July 30, 2026):

The key comparison is between lines 1 and 4. Same glm-5.2 model, same endpoint. Octomind solved 24 problems, OpenCode solved 19. Cost: $63.43 vs $129.54 — nearly half.

This gap comes from the harness design, not the model.

Octomind's core design decisions

Octomind claims "one binary, any model, MCP-native." Here's what the design actually does to achieve 24/25.

1. Policy as code, no popups

# .agents/guardrails.toml
[[guard]]
match   = "shell(command=^rm\\s+-rf?)"
message = "rm -rf blocked."

[[guard]]
match   = "shell(command=git push)"
when    = ["+shell(command=git status)"]
message = "Review changes before pushing."

Guardrails declared in a TOML file operate deterministically. No approval popups, works in CI, works in daemons.

For developers familiar with Claude Code hooks: guards are PreToolUse denies in TOML, hooks are PostToolUse, validators are Stop — no scripts needed for common cases.

2. One harness, five surfaces

# Interactive
octomind run developer:general

# Piped — CI, scripts, automation
echo "Explain the auth module" | octomind run developer:general --format plain

# Daemon — background agent
echo "watch the build" | octomind run --name watcher --daemon --format jsonl
octomind send --name watcher "run the test suite"

Interactive CLI, stdin (CI), daemon, WebSocket, ACP — the same engine exposed everywhere your workflow needs it.

3. Cost limits and session management

[model]
name = "anthropic:claude-sonnet-5-5"

[supervisor.model]
name = "anthropic:claude-haiku-5-5"

[compression.model]
name = "anthropic:claude-haiku-5-5"

Different models per role, spend capped per request and per session. Over budget: interactive sessions ask y/N, non-interactive runs stop with exit code 75.

Same model, different results

The most important insight from this benchmark: harness design matters.

OpenCode used glm-5.2 but solved only 19 problems. Octomind solved 24. Both ran stock invocations with no tuning. This benchmark measures the full harness, not individual model capability.

More interesting: under "worst-case pricing," glm-5.2 ran without prompt caching while Opus billed ~97% of context re-reads at cache rates. Even then, Octomind led on solves, cost, and wall time.

What developers should check

When reading this benchmark:

• Same-model comparison: Octomind vs OpenCode with glm-5.2 shows the pure harness difference.
• Held-out tests: Held-out tests from merged PRs prevent contamination.
• Cost vs. performance tradeoff: Codex + GPT-5.6 Sol is cheapest at $14.86 but solves 21/25. Octomind + glm-5.2 is $63.43 for 24/25.
• Independent reproduction needed: The published results are from July 30, 2026, not a measurement of the current checkout.

Model benchmarks alone don't determine agent quality. The performance gap between harnesses using the same model spans 5 problems and 2x cost. When evaluating agent performance, check the harness design before the model name.