Skip to main content

AI agents at 17 million PRs: the gate that held

AI agents open 17 million pull requests monthly. A gate rejecting unproven fixes cut 46.41% of agentic fixes, showing proof beats promises.

Kodetra TechnologiesKodetra Technologies
6 min read
Nov 19, 2026
0 views

Passing tests are not proof that a bug was fixed. One team learned that the hard way: its first gate waved through AI-written pull requests whenever the tests were green, and unreliable fixes kept arriving. Once the gate demanded concrete evidence of each fix, the picture changed. In one published dataset, 46.41% of agent-written fixes were rejected. The lesson is clear: proof beats promises, and deterministic checks should come before human review.

The 17 million-PR problem that sets the constraint

AI agents now open more than 17 million pull requests a month. A pull request is a proposed code change submitted for review before it is merged. That volume sets the constraint early: a gate has to decide before a human spends review time.

This is not a generic argument about AI. It is queue math. If agents can file at that volume, any status check, an automated result attached to a pull request that can pass or fail, has to screen for evidence first.

MeasureFigure
Agent pull requests per monthmore than 17 million
Curated pull requests in the AIDev dataset33,596
Projects in the AIDev dataset2,807
AI-agentic fix pull requests3,225
Rejected fixes1,497
Rejected share46.41%
Rejected pull requests studied in the sample306

The published rejection numbers make the operational risk plain. In that dataset, 1,497 agentic fixes were rejected, or 46.41% of the fixes.

A deterministic program gives the same answer from the same inputs every time. That is the kind of gate this problem needs, because the decision point comes before review, and because the research focused on pull requests that were closed and not merged.

For your own process, watch the handoff point. If a human is the first real verifier, the expensive step has already happened.

Why trusting passing tests fails

Failing tests are an obvious filter, and a common one. They are among the most common reasons agent-authored fix pull requests never get merged.

Starting there is reasonable, because it matches how many teams already screen changes. If tests pass, the pull request is treated as probably safe from a regression, a new bug introduced by a change that breaks behaviour that worked before.

That gate is too weak. It catches broken submissions, but it still lets through fixes that only claim success and cannot prove they addressed the reported problem.

Before and after

Before: a pull request that said "fixed it" was accepted when the tests were green, even if it offered no direct proof of the fix.
After: a pull request was rejected when it could not prove the fix, even if the tests still passed.

For anyone reviewing these changes, that shifts the burden early. Human review time stops going to promises and goes only to pull requests that carry evidence the fix actually happened.

What the gate checks when the promise is not enough

GitHub, the service where developers store code and review pull requests, is where the gate has to make a plain call: trust or reject, based on proof in the change itself.

The working rule set checks the commit rather than the agent's promise about what it fixed.

  • The change has to include a test, a small automated check that confirms whether code behaves as expected, and that test has to fail on the old code and pass on the new.
  • No existing test can be weakened to make the bug disappear on paper.
  • No function, a named block of code that performs one job, can be deleted as a shortcut around the failure.
  • Nothing else can regress, meaning the change cannot break behaviour that worked before.

That list makes the gate useful on a real bug because each item asks for evidence the code can supply.

In practice, a pull request does not pass because it sounds confident or because an agent says it fixed the issue. If you build a similar gate, start with checks that can be shown from the old and new code, then let the verdict follow the evidence on the table.

Why the judge is plain Python, not another model

Seventeen million pull requests a month is the boundary condition. At that scale, a gate where one model grades another model is not acceptable.

The design follows from a short chain of decisions.

  • The volume comes first: 17 million pull requests a month.
  • The rule is explicit: no model grades the model.
  • The implementation follows: Python, a programming language used here to write the rule-based gate, decides the result.
  • When a commit lands, the gate posts its verdict on the pull request as a GitHub status check plus a comment.

That sequence matters in practice. At this scale, an LLM judging an LLM stops looking clever and starts looking like extra uncertainty.

So the deciding logic is deterministic Python, and the verdict appears on the pull request. If you copy this pattern, keep the judge rule-based and make the decision visible where review starts.

What transfers from this gate, and what does not

Verificate Gate is an AI-powered CI gate, an automated checkpoint in continuous integration that can block a merge until checks pass, and that part transfers cleanly to other teams. A rejected change turns the check red, and the merge stays blocked until it is fixed.

What transfers

  • Put the check before human review when machine-written code is arriving faster than people can read it carefully.
  • Make the gate reject unsupported claims, then let humans spend time on the smaller set that survives.
  • Keep the rules tied to the kind of change you expect. This rule set is useful for bug-fix pull requests, because it asks for proof that a stated fix is real.
  • Use published benchmarks only as calibration. In one September benchmark, a leading AI model caught 0 of 6 planted problems, while Verificate Gate caught 6 of 6 on every run.

What does not transfer is the idea that this gate can judge every code change. It reviews up to 25 changed code files per pull request by default, which fits one workflow and may not fit yours.

Do not copy the rule set blindly for refactors, new features, or design changes. Its checks are aimed at hallucinated bug fixes, plus security, reliability, performance efficiency, and maintainability on each pull request.

The verdict is simple: proof beats promises

1,497 rejected fixes say it plainly. A gate that asks code for proof beats any system that accepts a passing story, and the lesson travels: put deterministic evidence checks at the first handoff, before review time gets spent.

Apply the most specific rules for each change type. For bug-fix pull requests, demand a test that breaks on the previous code but succeeds on the updated version. Reject weakened tests and shortcut deletions. Display the verdict as a visible status check at the start of the review.

What to do this week

  • Put the check before human review when machine-written code is arriving faster than people can read it carefully.
  • Make the gate reject unsupported claims, then let humans spend time on the smaller set that survives.
  • Start with checks that can be shown from the old and new code, then let the verdict follow the evidence on the table.
  • Keep the judge rule-based and make the decision visible where review starts.
  • Keep the rules tied to the kind of change you expect.

Sources