A browser-report feature works. Then someone asks for lowercase filenames, and the previously correct implementation becomes wrong. MaxxBench asks whether a coding agent notices and implements that change.
The October 20 research introduction and repository describe modified instructions or starting code for 89 DeepSWE tasks. We checked the material on November 8; we did not rerun the full model evaluation.
What the two scores leave out
The briefing's 73.8% and 53.4% for Gemini 3.8 Flash refer to different task sets. They are not repeated measurements on the same test.
| Observation | Value | Qualification |
|---|---|---|
| Public DeepSWE figure cited in the briefing | 73.8% | Original task set |
| Gemini 3.8 Flash, high, on MaxxBench | 53.4% | Modified tasks, measured by the researchers |
| Simple subtraction of the displayed values | 20.4 percentage points | Neither a causal estimate nor a controlled effect |
Source: MaxxBench results and research. At our check, its table displayed an estimated benchmaxxing gap of 20.5 points. That differs from subtracting the briefing's displayed values. Raw values, rounding and baseline versions need checking before either number is used as a precise effect size. A reproducible comparison also needs matching model versions, prompts, tools, budgets and trial counts.
The defensible conclusion is narrower than a model ranking: performance on familiar public tasks and adaptation to a changed contract deserve separate evaluation.
A small change moves the correctness boundary
The reporting task instructions contain several requirements, including filename normalization. This original CodeBridge example isolates the lowercase condition. It is not the official reference patch or a reproduction of the whole task.
import assert from 'node:assert/strict';
const previous = name => name.replace(/[^a-zA-Z0-9]/g, '_');
const revised = name => previous(name).toLowerCase();
const input = 'Chrome (Headless)';
const oldContract = value => /^[a-zA-Z0-9_]+$/.test(value);
const newContract = value => oldContract(value) && value === value.toLowerCase();
assert(oldContract(previous(input)));
assert(!newContract(previous(input)));
assert(newContract(revised(input)));
console.log(previous(input), revised(input));
Requires Node.js 18 or later. Save as contract.mjs and run node contract.mjs. Our local output was Chrome__Headless_ chrome__headless_. The example checks strings without an API call, network access or edits to a real repository.
The old test checks a character set, so both implementations can pass it. The new test forbids uppercase characters within that set. A large existing suite does not automatically cover a newly introduced requirement.
Keep “Did existing behavior regress?” separate from “Was the new condition implemented?” for every change request.
Keep the evaluator outside the answer
The README describes grading patches in a separate container with feature and regression checks. The runner also handles passing a rebuilt image to verifiers that require it. The environment handoff is part of reproducibility, alongside the tests themselves.
Agents can legitimately edit tests. But a new test file is not independent evidence that the acceptance criteria hold. An agent could miss the lowercase requirement and update expected filenames to uppercase; its patch and its tests would agree while the request remained unmet.
For a team evaluation, preserve acceptance conditions and verifier inputs before starting. Give the agent the requirements, then run regression checks, changed-contract checks and a scope review separately. Secrecy is not the goal. Keeping the judgment criteria from drifting along with the answer is.
Three mutations to try in your own repository
Start with small real tasks before operating an entire benchmark.
| Original task | Changed condition | Failure to check separately |
|---|---|---|
| Generate report files | Lowercase names and empty-name handling | Reverting to the old naming convention |
| Implement a list API | Sort direction and tie-breaking | Matching one sample only |
| Export data | UTC date boundaries | Depending on local time |
This table is a CodeBridge evaluation proposal, not a MaxxBench measurement. Pair original and modified runs from the same starting commit and model settings. Preserve repeated-run variation and failure explanations. If product versions differ, record that model and harness effects cannot be separated.
Preserve the contract before chasing the score
MaxxBench alone cannot establish training contamination or deliberate cheating by a particular model. Public trajectories help investigate failures; they do not represent every model or codebase. Our string example does not measure model adaptation either.
There is still a useful change a team can make today: store requirement revisions alongside acceptance tests before switching models. The next evaluation should tell you under which conditions a changed contract was missed, rather than return only another pass rate.
Background: What AI benchmark saturation means.