An AI patch is a candidate, not proof the vulnerability is closed

On August 6, 2026, 1Password’s Off-by-1 Labs published FLAWED research on AI-generated vulnerability patches. The study shows why a generated diff cannot prove that a vulnerability is closed. A patch can stop one exploit, leave another path open, change expected application behavior, or create a new vulnerability.

Application-security and engineering leaders need a stricter acceptance decision. Treat each generated security patch as a candidate until separate evidence shows complete exploit-path closure, expected behavior preservation, and no introduced vulnerability. This article explains what the experiment measured, where its measurements are uncertain, and how to keep patch generation separate from patch acceptance.

FLAWED tested difficult vulnerabilities against known upstream fixes

The FLAWED paper studied six recent, high-impact, complex open-source vulnerabilities. The sample deliberately favored hard cases, including vulnerabilities with more than one exploit path. The researchers had the official upstream fixes from the projects’ maintainers. They used those fixes as the comparison standard but hid them from the patch-generating agents.

Two model configurations attempted the work through several generation modes and prompt styles. Candidates ran in isolated containers. Each model checked its own output, and another model also checked the candidate. The pipeline then classified each result by comparing it with all code paths addressed by the known upstream fix. This last point matters because the target was full closure of the known vulnerability, not success against one test input.

The paper reports 6,080 valid attempts. It reports an average 10.8% cheat rate across the two configurations and excludes flagged iterations from the headline outcome values because looking up the real patch can inflate the clean-fix rate. The resulting dataset measures two model configurations on six selected vulnerabilities during this campaign. It does not estimate success rates for all security patches, simpler defects, or ordinary coding work.

The five outcomes separate clean fixes from three different kinds of risk

FLAWED assigns each valid attempt one of five outcomes. These definitions preserve distinctions that a single pass-or-fail score would hide:

• S1, clean fix: 26.0%. The patch fixed all known exploit paths addressed by the upstream patch and did not cause an unrelated behavior change.
• S2, fixed with changed behavior: 20.1%. The patch fixed the vulnerability, but application behavior differed from both the vulnerable version and the canonical patched version.
• S3, incomplete fix without a new vulnerability: 49.3%. The patch left at least one known exploit path available and did not introduce a new vulnerability.
• S4, original issue fixed with a new vulnerability: 2.3%. The patch closed the original vulnerability but introduced a different vulnerability.
• S5, incomplete fix with a new vulnerability: 2.2%. The patch failed to close the original vulnerability and introduced a different vulnerability.

Two aggregate values are useful for acceptance decisions. S3 plus S5 gives a 51.5% fail-to-fix rate, meaning at least one known exploit path remained. S4 plus S5 gives a 4.5% introduced-new-vulnerability rate. Calling the complement of S1 a single 74% failure rate would combine behavior-changing fixes, incomplete fixes, and newly vulnerable fixes even though they require different responses.

A passing reproducer can test one path while another remains open

A reproducer is a program or request that triggers a specific vulnerability. It is valuable because it turns a security claim into observable behavior. Its limit is scope: one reproducer can exercise one input, call site, parser branch, permission path, or system state while the same defect remains reachable elsewhere.

Consider a generic multi-path defect. A candidate patch adds a check at the location reached by the supplied reproducer, so that test stops triggering the bug. The upstream fix may address the same unsafe operation at several call sites or correct a shared function below them. If acceptance reruns only the original reproducer, the local check looks successful even though another known route still reaches the vulnerable operation.

This is also why vulnerability localization and remediation are separate tasks. As our analysis of Cisco’s vulnerability-localization benchmark explains, finding likely vulnerable files can reduce search time without establishing exploitability or proving a fix. FLAWED extends that boundary: even code that changes the right area needs path-level evidence before approval.

Model-judged outcomes make the rates useful but conditional

Most FLAWED outcomes were judged by models. The researchers used self-validation and cross-model validation, manually reviewed at least 10% of patches in every campaign, found systematic grading errors, and added a second-pass rubric. Manual review therefore improved the evaluator, but it did not adjudicate every attempt.

The reported agreement values show the remaining measurement limit. The second-pass validator matched manual review on the exact S1–S5 grade in 65.9% of reviewed cases. Agreement was 87.7% on whether the original bug was fixed and 70.5% on whether the patch introduced a new bug. The exact outcome depends on both questions plus behavior preservation, so component disagreements can move a patch between materially different categories.

These figures do not make the experiment unusable. They set the confidence boundary around its aggregate rates. The study provides strong evidence that evaluator design and exploit-path coverage affect a patch benchmark, while the displayed percentages remain conditional on this sample and grading process. Our review of third-party agent evaluations reaches the same operational point: a result carries the definitions, runtime, retries, and evaluator that produced it.

Patch generation and patch acceptance require different evidence

A patch generator searches the repository, proposes code, and can explain or test its candidate. Those actions can reduce remediation time. Acceptance has a different purpose: it decides whether the organization can merge and release the change. Letting a generator’s own success signal make that decision joins two jobs with different incentives and failure modes.

Teams should define acceptance evidence before the agent starts. The exact tools will vary by language, architecture, and vulnerability class, but the decision must answer three separate questions:

Scroll sideways to see all 3 columns.

Acceptance question	Evidence to require	Why the generated result is insufficient
Did the patch close every known exploit path?	Map paths from the advisory, upstream fix when available, code analysis, and reproducers; test each known route.	A candidate can stop the supplied reproducer at one location while another path remains reachable.
Did expected behavior remain intact?	Run relevant unit, integration, and regression tests; compare the result with approved application behavior and patched semantics.	Security closure alone does not detect unrelated behavior changes such as S2.
Did the patch introduce a vulnerability?	Review the changed trust boundaries and data flows; run targeted security tests and suitable static, dynamic, or fuzz testing.	A self-check focused on the original defect can miss a new weakness created by the diff.

A domain expert should own the acceptance decision because exploit paths and intended behavior depend on the application. Expert review directs the tests, interprets conflicting results, and decides whether remaining uncertainty is acceptable. It does not guarantee correctness, and it should not replace reproducible evidence.

The surrounding workflow still matters. Before generation, triage must establish asset relevance, exploit evidence, and ownership; our article on agent-assisted vulnerability triage covers that earlier decision. After acceptance, release and deployment controls must show that the approved fix reached affected systems. The Chrome AI security-fix analysis explains why an accepted patch and an applied update are also different states.

Use the research to tighten one decision, within its stated limits

FLAWED supports a narrow operating change: do not let a generated security patch pass because its reproducer stopped failing or its model validator approved the diff. Require independent evidence for known exploit paths, expected behavior, and introduced vulnerabilities. Record who reviewed that evidence and what conditions would block merge or release.

Teams that want to inspect or reproduce the research should also respect the FLAWED repository’s execution boundary. Its README says the harness mounts the Docker socket, which is root-equivalent on the host, while it executes untrusted third-party code. Run it only in a disposable, appropriately isolated environment, not on a normal developer workstation.

Keep the result within the study’s scope. Its rates apply to the six selected vulnerabilities and two tested configurations, and they provide no model-vendor ranking. The useful action is to separate faster candidate generation from the evidence required to accept consequential code. For these patches, the decision belongs to the acceptance system.

If your team is adding coding agents to vulnerability remediation, BaristaLabs can help design and test application-security acceptance controls for your repositories, CI checks, and release process. Discuss your application-security workflow with BaristaLabs.

Sources

• 1Password: Why AI-generated patches still require human review
• FLAWED paper: Frontier Models’ Vulnerability Patches Contain Embedded Defects
• Off-by-1 Labs: FLAWED repository