Finding a suspicious function and repairing it without breaking valid behavior are different achievements. Gemini 4 Carbon's security results test different achievements too. Combining them into one “security accuracy” loses the distinctions a deployment decision needs.
Google's October 28 announcement reports 88.2% on SEC-bench Pro, 54.8% on ExploitGym, and 71.0% on CWE-bench, with these external cyber evaluations performed without guardrails. Initial access is restricted to selected cyber defenders and Fairwind partners; general developer access is planned. This October 29 review involved no Carbon use.
Read the task before the percentage
| Primary source | What its definition establishes | Question for a development team |
|---|---|---|
| SEC-bench Pro | Vulnerability discovery and PoC generation with containerized targets and a checker harness | Can the finding be reproduced in a fixed environment? |
| ExploitGym | Exploit development from real vulnerabilities across userspace, V8, and Linux | Are exploitability validation and discovery being confused? |
| CWE-bench | Audit and repair, checking both vulnerability blocking and preserved functionality | Does the patch also reject legitimate requests? |
Benchmark definitions and vendor results support different claims. The benchmark owner explains the task; Google's announcement reports Carbon's performance on it. The SEC-bench page inspected here did not separately corroborate Carbon's 88.2%. None of the three numbers is presented as independently reproduced by CodeBridge.
Their average has no established common sample or weighting. Multiplying them is also invalid as an estimate of end-to-end remediation success: that would require conditional measurements on the same cases at successive stages.
Attempts and denominators change the result
CWE-bench's current v1 protocol describes 120 tasks, four rollouts per model, high reasoning, and a one-hour limit per rollout. It distinguishes programmatic grading from an agentic judging panel. That is the benchmark owner's current protocol, not evidence that all three Google evaluations used identical settings.
A patch that succeeds on the first attempt and one selected after several attempts offer different operational experiences. Retrying requires a validation budget and a way to recover from bad patches. Treating pass@1 and pass@4 as interchangeable hides that cost.
A narrow exploit check can also reward deleting the vulnerable feature entirely. Existing functionality and grading integrity matter alongside blocking the specific reproduction. When inspecting a high score, ask what its checker might accept that your users would reject.
Require an evidence bundle for each patch
The following is CodeBridge's proposed acceptance matrix, not Carbon output or an official configuration. Apply it to authorized, isolated test code with synthetic data rather than a live service.
| Stage | Artifact to retain | Reason to reject or continue review |
|---|---|---|
| Discovery | Affected function, version, triggering conditions | Suspicion without a causal explanation |
| Reproduction | Minimal regression test failing before the fix | Failure cannot be reproduced in the fixed environment |
| Repair | Diff and explanation of the removed cause | Checker bypass or deleted functionality |
| Preservation | Tests for valid and boundary inputs | Previously valid behavior stops working |
| Handoff | Unchecked scope, dependencies, remaining risk | A blanket safety claim with no scope |
For a path-validation repair, for example, verify both that an invalid path is rejected and that a valid relative path remains usable. Blocking every file read could pass one reproduction while failing the product requirement. This is a review scenario; no vulnerable program or exploit was executed.
The same standard applies to long tasks. Elapsed time is not evidence of correctness. Retain attempt-level commits, failure reasons, and reproduction results so a reviewer can trace the final patch. Our security benchmark article offers related context on distinguishing a model ranking from an operational result.
Prepare the acceptance test before access arrives
There is no reason to invent an API model ID or installation command while access is restricted. Assemble reproducible historical defects instead. In evaluation copies with the original repair withheld, establish a failing test before repair, a passing test afterward, and checks for preserved functionality.
Do not equate an evaluation without guardrails with the behavior of the eventual product. Access rules and execution conditions can differ. The full sample, budget, and configuration behind every reported number were not independently established here, and no model evaluation was reproduced.
Carbon's useful engineering question is what evidence makes a patch acceptable. Defining that standard now gives you a stable comparison when access and models change.
The published paid course below covers related agent execution and verification structures. It does not promise Carbon access or a security benchmark curriculum. The link is promotional.