Agents write secure, working code nearly twice as often with Verity.
We gave a coding agent 23 tasks from real open-source projects, with and without Verity. Hidden tests from a public academic benchmark then checked two things: does the code work, and is it secure?
Agent alone
15 of 46 runs33%
Agent with Verity
29 of 46 runs63%
Figure 1. Each square is one run. A filled square passed both the feature tests and the security tests. This is our own benchmark, from October 2026, with Verity at maximum review effort. How we tested.
Verity finds 2× more real bugs than your typical PR review agent
A PR agent reads the code once, when it is finished. Verity reviews each commit and push, while the coding agent can still fix what it finds. We used PR-Agent, a popular open-source tool that works like other PR agents: it reviews the finished PR once.
- Verity, every commit and push of the work
- 2.33
- PR agent on gpt-6-astra, one review of the finished PR
- 1.21
- PR agent, default settings, one review of the finished PR
- 1.13
- Verity
- 54%13 of 24
- PR agent on gpt-6-astra
- 38%9 of 24
- PR agent, default settings
- 25%6 of 24
Figure 2. Real bugs1 found per task2, and tasks where the tool found the known vulnerability3, over 24 tasks. Verity reviewed during the work. The PR agent4 reviewed the finished PR.
Why: it reviews while the agent works
Each finding goes back to the agent, which can still fix the code
Code is finished
Figure 3. Where each tool reviews5, and what happens to a finding6.
The result: working code with a security flaw fell from 23 runs to 8
Verity finds bugs by trying every sequence, then running the one that breaks your code
Some bugs get through review because every line looks right. They appear only when two things happen in an unlucky order, or when one part fails quietly. Verity builds a formal model of your function, searches it for the exact sequence that breaks it, and runs that sequence on the real code. You get a failure, not an opinion.
No other code review tool does this.
1. Model
Verity writes a small formal model of one risky function, and the rule it must keep.
2. Search
It searches the possible orders and outcomes of the function's steps, up to a limit, for one that breaks the rule.
3. Replay
It runs that case on the function's real code, with stand-ins for what the function calls. Only a failure that reproduces can block a push.
Figure 5. A formal check9 in three steps. The search runs up to a limit10, and the replay uses stand-ins11. No other code review tool does all three12.
A bug we found and reported: double bookings in Cal.com's open-source code
In October 2026, Verity reviewed a PR that changed calendar features in Cal.com's open-source scheduler (calcom/cal.diy on GitHub). It then checked a nearby function that asks Microsoft for a host's busy times. The rule: a finished availability check covers every calendar it asked for. Here is what it found.
It reports success
Busy at 10:00 only. The safety check for a failed calendar never runs.
A booking lands on top of an existing meeting.
The rule is broken: “A finished availability check covers every calendar it asked for.” Verity ran this on the function's real code, with a stand-in for Microsoft's API. It asked for 2 calendars and got back 1 busy event.
Figure 6. The double booking. The scheduler asks for both calendars in one batch request13, and its safety check14 does not run. The flow follows Verity's trace15.
Who caught it
- Verity formal checksFound it and replayed it on the real code
- Typical PR agentDid not mention it, on either setting
- AI review bot on the PRDid not mention it
The bug is in older code next to the change, not in the PR's diff.
What happened next
July 2022
The code that skips a failed answer enters the project in #3534.
21 Oct 2026
Verity finds it and reproduces it on the real code.
21 Oct 2026
We report it with two failing tests in the project's own suite: calcom/cal.diy#30382.
22 to 23 Oct 2026
Cal.com was one of 7 bugs from this test. We pointed the formal checks at 164 recently merged PRs from 34 active open-source projects, and a separate AI review confirmed 7 bugs. We found no earlier report of 5 of them. All 7 sit in older code next to the change. The PR agent we tested, run on the same 7 PRs, found only the one that was already known. We name the others after each project has the report.
A test on a known bug: one changed lock in Plane
This is a test on a known bug. In October 2026 we ran Verity on an old Plane PR, which changed a lock so that it is released right after a work item is saved. The project had already fixed the bug itself. The rule Verity checked: each work item gets its own number. Here is the order of steps it found.
| Request A | Highest number others can see | Request B |
|---|---|---|
Takes the lock | 5 | |
Reads the highest number: 5 | 5 | |
Saves its work item as 6 | 5 | |
Releases the lockToo early. Its 6 is not committed yet. | 5 | |
| 5 | Takes the lock | |
| 5 | Reads the highest number: 5It cannot see A's 6. | |
Commits its 6 | 6 | |
| 6 | Saves its work item as 6 and commits |
Two work items, both number 6.
The rule is broken: “Each work item gets its own number.” Verity then ran this order on the function's real code, with a stand-in for the database, and got the same result.
Figure 7. The order of steps Verity reported. Time runs from top to bottom. A save stays private until it commits16. Found in 5 of 5 test runs17.
Who caught it
- Verity formal checksFound in 5 of 5 test runs
- Typical PR agentMissed it
- AI review bot on the real PRReviewed it when it was open, did not flag it
The change: makeplane/plane#7396. The project's own fix, before our test: #8298.
Across 110 known bugs, the formal checks found 4 in every run that the PR agent we tested and every reviewer on the real PR missed. Plane is one. The other 3 are security bugs, and we will name them when that is safe for those projects.
We also read what commercial AI review bots wrote on these PRs when they were open. They reviewed 51 of the 110 and flagged 8 of the known bugs. These bugs all reached a merge, so a bug that a bot stopped before the merge is not in this count.
How we tested
This is our own benchmark. Codacy builds Verity, and we ran these three tests in October 2026. Try Verity on your own code and judge for yourself.
Test 1: agents doing real tasks
- SusVibes, a public academic benchmark (ICML 2026), asks an agent to add a feature to a real Python project. Its hidden tests check the feature and a known security weakness. We used 23 tasks from a random sample of 30, which an earlier pilot also ran.
- Claude Sonnet 5.5 in Claude Code did each task twice alone and twice with Verity, from the same task text. It could not run the project's code or tests in either setup.
- SusVibes graded the code as written. A run counts as working if the feature tests pass, and as secure if the security tests pass as well.
- For the bug counts, a separate AI reviewer (gpt-6.1-sol) read the code behind every reported issue in the first repetition of 24 tasks, with the same instructions for each tool. A repeat counts once. PR-Agent reviewed the final PR of the agent-alone run, so its code differs from the code Verity reviewed.
Test 2: known bugs in real PRs
- We collected 110 merged open-source PRs that introduced a bug fixed later: 67 security bugs from public advisories and fix commits, and 43 logic bugs.
- Verity and PR-Agent each reviewed the PR. PR-Agent ran once per PR. The formal checks ran 3 to 5 times per PR.
- A finding counts only if it describes the same code and the same failure as the known bug. One AI judge (gpt-6.1-sol) scored every tool.
- PR-Agent wrote an empty review on 3 PRs. We do not count those as misses.
- We also read the public comments that commercial AI review bots left on the real PRs. We did not run those bots, and we do not name them.
Test 3: live code
- We took 164 PRs merged in the previous 21 days, from 34 projects. Formal checks modeled a function in 83 of them and reported 39 findings.
- Separate AI agents read the code, guards and history behind the 12 strongest findings, and confirmed 7.
- Two repeat runs of the formal checks found 3 of the 7 again.
- We then ran PR-Agent on the 7 PRs behind those bugs.
- Verity's settings
- Maximum review effort, which we set per project, and formal checks at push, which are on for new projects. Review models from another vendor than the agent (gpt-6-astra and gpt-6.1-sol). One security set ran without the agent review.
- PR-Agent's settings
- The PR agent we tested is PR-Agent, a popular open-source tool, with its review and improve commands: default settings everywhere, and also gpt-6-astra in Test 1. We chose it because it is open source and we can run it ourselves.
- Builds
- Benchmark builds of CLI 0.34.2 from 13 to 21 October 2026. They include fixes that are on their way to a public release.
- One correction
- The SusVibes log parser cannot read a clean pass for the neutron project, so it marked two Verity runs as aborted. Their logs show that every test passed, so we counted them as passed. No other run met this rule, in either setup. Without the correction, Verity's score is 27 of 46, not 29.
- Data
- We are preparing the harness, the cases and the raw outputs for release.
See every task and run
Working and secureWorking, with a security flawNot working
| Task | Alone | With Verity |
|---|---|---|
| airflow CWE-78 | ||
| superset CWE-863 | ||
| pydash CWE-94 | ||
| django CWE-287 | ||
| django CWE-79 | ||
| django CWE-79 | ||
| django CWE-704 | ||
| markdown-it-py CWE-173 | ||
| mpmath CWE-770 | ||
| pandas-ai CWE-94 | ||
| pysaml2 CWE-347 | ||
| rdiffweb CWE-306 | ||
| mlflow CWE-23 | ||
| bleach CWE-79 | ||
| numpy CWE-835 | ||
| neutron CWE-755, corrected | ||
| flask CWE-539 | ||
| babel CWE-22 | ||
| salt CWE-287 | ||
| scrapy-splash CWE-200 | ||
| fastapi CWE-352 | ||
| trytond CWE-269 | ||
| django-mfa3 CWE-287 | ||
| Working and secure | 15 of 46 | 29 of 46 |
Run it on your next session
npm install -g @codacy/verity-cli && verity initFormal checks are on for new projects. For maximum review effort, talk to us on Slack. Read how the formal checks work in the docs. Read why the PR is too late.