Kimi K3.1 Pro's 65.7%: Why the Benchmark Harness Matters Before the Number

Kimi K3.1 Pro's Terminal-Bench 4.0 score of 65.7% was measured under the Claude Code harness. The competing models' scores listed alongside it, however, are not re-runs under the same conditions. For developers considering a model swap, the comparison conditions matter more than the ranking.

Kimi's official footnote states that competing results were taken from each announcement or leaderboard's best record, not from re-runs under identical conditions. The 0.7 percentage point gap between 65.7% and 66.4% is too small to call one model inferior without re-running under the same harness and repeated trials.

What was published, and what was not

Based on Kimi's technical blog and K3 documentation, here is what can be confirmed:

The distinction between "3T-class" and "2.8T" matters here. The K3 documentation states 2.8T parameters with 16 active experts out of 896. K3.1 Pro is a post-trained variant of K3, so assuming the same parameter count is reasonable. However, the K3.1 documentation does not restate the total or active parameter count, so confirming those numbers from official sources is important.

65.7% under what harness, under what scoring?

Terminal-Bench 4.0 gives agents realistic terminal tasks and scores them by task-level pass/fail. The model must select tools, execute commands, read failures, and retry. The score includes not just the model but:

• How many tools are available
• When context overflows, whether compression occurs
• How many retries are allowed after failure
• Reasoning budget and timeout settings
• What the scorer counts as a failure

Kimi's footnote clarifies this point: K3.1 Pro was evaluated under the Claude Code harness, while competing model scores were taken from each vendor's blog or the official Terminal-Bench leaderboard's best records. This is not a re-run under the same harness under the same conditions. It is comparing each vendor's best recorded score under different harnesses.

BenchCAD is similar. The published table emphasizes "tool use" scores, but the same model's no-tool score is 72.8%. The gap is 19.9 percentage points. However, the execution mode also changes: tool-use harness versus single-pass generation. You cannot isolate the tool's causal effect from the execution mode difference. If your agent does not have Blender or CAD scripts as available tools, and does not have a view-render-feedback loop, 92.7% is someone else's number.

The evaluation setup also needs scrutiny. The BenchCAD figure comes from a random 1,000-subset of 17,900 Vision2Code cases, measured by Voxel IoU. If the subset composition changes, the score shifts, and the selection was made by the manufacturer. Also, Voxel IoU measures geometric overlap between two shapes. You cannot read 92.7% as "927 out of 1,000 CAD tasks succeeded." Without per-sample scores and repeated runs, confidence intervals cannot be calculated from this text alone.

Designing a same-harness comparison

Experiments to decide whether to swap models should be controlled comparisons, not benchmark reproductions. Fix at minimum:

Pay attention to reasoning budget. Kimi states the default configuration for unspecified results as reasoning effort "max", temperature 1.0, and top-p 0.95. Lowering effort to save costs produces different scores from the same model. Comparing your workload's default settings to a competitor's "max" score misunderstands actual operational cost and performance.

Starting with 30 tasks

You do not need to reproduce the full benchmark. Select 30 tasks that fail repeatedly in your repository and add hidden tests. The key metric is not just "pass rate" but "cost per success" and "latency" together.

The following is an educational aggregation example that does not call any real models. Save as compare-runs.mjs and run with node compare-runs.mjs. Requires Node 18+, no other dependencies or network calls. The output below was verified as a local run. A/B prices and times are all hypothetical data.

// compare-runs.mjs — Node 18+, no dependencies or network calls
const runs = [
  { model: "A", task: "parser", passed: true, usd: 0.12, ms: 1000 },
  { model: "A", task: "timeout", passed: false, usd: 0.18, ms: 5000 },
  { model: "B", task: "parser", passed: true, usd: 0.20, ms: 800 },
  { model: "B", task: "timeout", passed: true, usd: 0.20, ms: 1200 }
];
for (const model of ["A", "B"]) {
  const group = runs.filter(r => r.model === model);
  const passes = group.filter(r => r.passed).length;
  const total = group.reduce((sum, r) => sum + r.usd, 0);
  const times = group.map(r => r.ms).sort((a, b) => a - b);
  const p95 = times[Math.ceil(0.95 * times.length) - 1];
  const perPass = passes ? (total / passes).toFixed(2) : "NA";
  console.log(`${model}: pass=${passes}/${group.length} usd/pass=${perPass} p95_ms=${p95}`);
}

A: pass=1/2 usd/pass=0.30 p95_ms=5000
B: pass=2/2 usd/pass=0.20 p95_ms=1200

A appears cheaper per execution, but including failure cost, the cost per success is $0.30. B is $0.20. This demonstrates a difference in aggregation method, not Kimi's or any competitor's actual pricing or performance. The nearest-rank p95 from two samples is the maximum value, not a stable latency estimate. When passes equal zero, division returns NA.

Real logs should record the exact model ID, harness commit, initial repository commit, budget, task and repetition IDs, and independent scoring results. Each run should start from an isolated working directory at the same initial commit so that one run's modifications do not carry forward to the next. API errors and timeouts should be included in the failure row and cost. Do not judge success by exit code 0 alone.

Four metrics are sufficient:

1. pass@1: Pass rate of hidden tests from a single execution
2. Cost per success: (input tokens × unit price + output tokens × unit price + tool call cost) ÷ successful runs
3. p50/p95 completion time: Tail latency, not just average
4. Human intervention count: How many approvals, re-directions, and manual fixes occurred

Cost per success must include failed executions. High pass rates with many retries mean actual cost is higher. This article does not quote Kimi K3.1 Pro's unit price, as billing rates may have changed. Use the values from your invoice or API console.

Reading 1M tokens and Highspeed

One million tokens means you can load a large repository in one shot, not that the model understands 1M tokens uniformly. Long context degrades:

• Mid-document function name or constraint retrieval
• Architecture rule adherence past 300K tokens

You should also check the context compression method. If the harness summarizes old conversations, 1M tokens may not be fully utilized, and requirements may disappear from the summary.

Highspeed's "2–3× faster" claim is ambiguous: throughput or latency? For agents, p95 matters more than average. Run 20 identical tasks three times each, comparing p50, p95, and retry rates. If retries double, a 2× speed gain disappears.

Weight release also means something different from "I can run this on my laptop." The K3 documentation recommends 64+ accelerators for a cluster and inference stack. Serving 3T-class models directly requires a cluster and inference stack. What to check after November 4 is file size, license, and actual serving cost, not "can run locally."

When not to adopt

• When reproducible benchmark rankings are needed: Wait for independent leaderboards and harness-specific results.
• When switching models mid-session: K3 documentation warns that saved thought history is needed, and switching models mid-session may cause quality instability. This may apply to K3.1; verify harness compatibility first.
• When latency equals cost in interactive products: 1M context and long reasoning increase first-token latency. Do not adopt before seeing p95.
• When self-hosting is needed before November 4: Weights have not been released yet.

Conclusion: 0.7 points does not replace an experiment

The 65.7% figure is "measured under Claude Code harness, with max reasoning budget, scored by the manufacturer." Competing model scores were taken from each vendor's best records. Without running under the same conditions, saying "nearly on par with Opus" may not reproduce under your harness.

Make the decision based on a two-week experiment: 30 tasks, hidden tests, 3 repetitions, cost per success, and p95. These four metrics produce a far more useful answer than a 0.7 point gap.

The paid course promotion at the end of this article is about learning Claude Code harness rules and verification flows. It is not presented as a course for using Kimi or reproducing this benchmark.

Articles to read together

• Kimi K3's 2.8T parameters: what matters more
• How to evaluate open-weight model cost and performance together

References

• Kimi: Introducing Kimi K3.1 Pro
• Kimi: Kimi K3: Open Frontier Intelligence
• Terminal-Bench