# Agents write secure, working code nearly twice as often with Verity. We gave a coding agent 23 tasks from real open-source projects, with and without Verity. Hidden tests from a public academic benchmark then checked two things: does the code work, and is it secure? Agent alone 15 of 46 runs 33% Agent with Verity 29 of 46 runs 63% Alone, 15 of 46 runs passed both sets of hidden tests. With Verity, 29 of 46 did. Figure 1. Each square is one run. A filled square passed both the feature tests and the security tests. This is our own benchmark, from October 2026, with Verity at maximum review effort. How we tested. ## Verity finds 2× more real bugs than your typical PR review agent A PR agent reads the code once, when it is finished. Verity reviews each commit and push, while the coding agent can still fix what it finds. We used [PR-Agent](https://github.com/The-PR-Agent/pr-agent), a popular open-source tool that works like other PR agents: it reviews the finished PR once. Real bugs found per task Verity, every commit and push of the work 2.33 PR agent on gpt-6-astra, one review of the finished PR 1.21 PR agent, default settings, one review of the finished PR 1.13 Tasks where it found the known vulnerability Verity 54% 13 of 24 PR agent on gpt-6-astra 38% 9 of 24 PR agent, default settings 25% 6 of 24 Figure 2. Real bugs 1 found per task 2, and tasks where the tool found the known vulnerability 3, over 24 tasks. Verity reviewed during the work. The PR agent 4 reviewed the finished PR. 1. 1 Real bug. A separate AI reviewer (gpt-6.1-sol) read the code behind each reported issue, and counted it only if it confirmed a defect. A repeat counts once. 2. 2 Task. A feature request in a real Python project, from [SusVibes](https://github.com/LeiLiLab/susvibes), a public academic benchmark (ICML 2026). The coding agent writes the feature. 3. 3 Known vulnerability. Each task comes from a published security flaw (a CVE). A tool gets credit only if it names that flaw. 4. 4 PR agent. A tool that reviews a pull request when it opens. We ran one: [PR-Agent](https://github.com/The-PR-Agent/pr-agent), a popular open-source tool with about 15,000 GitHub stars. Others work the same way, and their results vary by tool. We ran it on default settings and on gpt-6-astra, Verity's strongest model. It reviewed the final PR of the agent-alone run, so its code differs from the code Verity reviewed. ### Why: it reviews while the agent works Verity reviews at commit and at push, and its findings go back to the agent. A PR agent reviews once, at the pull request, when the code is finished. Verity reviews each one PR agent reviews once Agent writes code Commit Push Pull request Each finding goes back to the agent, which can still fix the code Code is finished Figure 3. Where each tool reviews 5, and what happens to a finding 6. 1. 5 Review moments. In this test, every real bug Verity reported came at a commit or a push. 2. 6 Findings. Verity can block a commit or push until the agent deals with the finding. An agent may stop and ask the user. We graded what it had written at that point, with no credit for the block. ### The result: working code with a security flaw fell from 23 runs to 8 Agent alone 15 23 8 Agent with Verity 29 8 9 Working and secure Working, with a security flaw Not working Figure 4. What each of 46 runs delivered 7, in each setup 8. 1. 7 Results. Working: the hidden feature tests pass. Secure: the hidden security tests pass as well. No run passed security without also working. 2. 8 Setups. Claude Sonnet 5.5 in Claude Code did each task twice per setup, from the same task text. Verity ran at maximum review effort, which we set per project. One correction applies: see How we tested. ## Verity finds bugs by trying every sequence, then running the one that breaks your code Some bugs get through review because every line looks right. They appear only when two things happen in an unlucky order, or when one part fails quietly. Verity builds a formal model of your function, searches it for the exact sequence that breaks it, and runs that sequence on the real code. You get a failure, not an opinion. No other code review tool does this. 1. 1. Model Verity writes a small formal model of one risky function, and the rule it must keep. 2. 2. Search It searches the possible orders and outcomes of the function's steps, up to a limit, for one that breaks the rule. 3. 3. Replay It runs that case on the function's real code, with stand-ins for what the function calls. Only a failure that reproduces can block a push. Figure 5. A formal check 9 in three steps. The search runs up to a limit 10, and the replay uses stand-ins 11. No other code review tool does all three 12. 1. 9 Formal model. A small program in Lean 4 that describes one function: its state, its actions and the rule it must keep. The Verity service writes it. 2. 10 Limit. The search covers cases up to a set depth. It can show that a rule breaks. It does not prove that a rule always holds. 3. 11 Stand-ins. The function's own source code runs. What it calls, such as a database or an API, is replaced with a small fake. 4. 12 Other tools. Some tools can re-run a security finding in a sandbox. In November 2026 we found none that models a function formally, searches the model, and replays the failure at the push. ### A bug we found and reported: double bookings in Cal.com's open-source code In October 2026, Verity reviewed a PR that changed calendar features in Cal.com's open-source scheduler (calcom/cal.diy on GitHub). It then checked a nearby function that asks Microsoft for a host's busy times. The rule: a finished availability check covers every calendar it asked for. Here is what it found. The host's calendarCalendar 1: meeting at 10:00 Microsoft answersOK: busy at 10:00 The code Keeps it The host's calendarCalendar 2: meeting at 14:00 Microsoft answersError 500 The code Skips it, raises no error It reports success Busy at 10:00 only. The safety check for a failed calendar never runs. The host's real day Meeting Meeting What the code sees Meeting Looks free A new booking Booked A booking lands on top of an existing meeting. The rule is broken: “A finished availability check covers every calendar it asked for.” Verity ran this on the function's real code, with a stand-in for Microsoft's API. It asked for 2 calendars and got back 1 busy event. Figure 6. The double booking. The scheduler asks for both calendars in one batch request 13, and its safety check 14 does not run. The flow follows Verity's trace 15. 1. 13 Batch request. One call to Microsoft Graph covers all selected calendars. Each calendar gets its own answer inside the response, so one can fail while the call succeeds. 2. 14 Safety check. When a calendar's busy times are unknown, the scheduler is meant to block that time. That path runs only if the function raises an error. 3. 15 Trace. Verity's replay used two calendars with one busy event each, and one error 500. The meeting times shown are examples. ### Who caught it - Verity formal checks Found it and replayed it on the real code - Typical PR agent Did not mention it, on either setting - AI review bot on the PR Did not mention it The bug is in older code next to the change, not in the PR's diff. ### What happened next 1. July 2022 The code that skips a failed answer enters the project in [#3534](https://github.com/calcom/cal.diy/pull/3534). 2. 21 Oct 2026 Verity finds it and reproduces it on the real code. 3. 21 Oct 2026 We report it with two failing tests in the project's own suite: [calcom/cal.diy#30382](https://github.com/calcom/cal.diy/issues/30382). 4. 22 to 23 Oct 2026 Contributors open two fix PRs: [#30383](https://github.com/calcom/cal.diy/pull/30383) and [#30389](https://github.com/calcom/cal.diy/pull/30389). Cal.com was one of 7 bugs from this test. We pointed the formal checks at 164 recently merged PRs from 34 active open-source projects, and a separate AI review confirmed 7 bugs. We found no earlier report of 5 of them. All 7 sit in older code next to the change. The PR agent we tested, run on the same 7 PRs, found only the one that was already known. We name the others after each project has the report. ### A test on a known bug: one changed lock in Plane This is a test on a known bug. In October 2026 we ran Verity on an old Plane PR, which changed a lock so that it is released right after a work item is saved. The project had already fixed the bug itself. The rule Verity checked: each work item gets its own number. Here is the order of steps it found. *The order of steps that broke the rule. Each work item gets its own number. Time runs from top to bottom.* | Request A | Highest number others can see | Request B | | --- | --- | --- | | Takes the lock | 5 | | | Reads the highest number: 5 | 5 | | | Saves its work item as 6 | 5 | | | Releases the lock Too early. Its 6 is not committed yet. | 5 | | | | 5 | Takes the lock | | | 5 | Reads the highest number: 5 It cannot see A's 6. | | Commits its 6 | 6 | | | | 6 | Saves its work item as 6 and commits | Two work items, both number 6. The rule is broken: “Each work item gets its own number.” Verity then ran this order on the function's real code, with a stand-in for the database, and got the same result. Figure 7. The order of steps Verity reported. Time runs from top to bottom. A save stays private until it commits 16. Found in 5 of 5 test runs 17. 1. 16 Commit. A database transaction keeps its writes private until it commits. Until then, other requests still see the old highest number. 2. 17 Runs. Five separate runs of the formal checks, with different settings. All five reported this failure. ### Who caught it - Verity formal checks Found in 5 of 5 test runs - Typical PR agent Missed it - AI review bot on the real PR Reviewed it when it was open, did not flag it The change: [makeplane/plane#7396](https://github.com/makeplane/plane/pull/7396). The project's own fix, before our test: [#8298](https://github.com/makeplane/plane/pull/8298). Across 110 known bugs, the formal checks found 4 in every run that the PR agent we tested and every reviewer on the real PR missed. Plane is one. The other 3 are security bugs, and we will name them when that is safe for those projects. We also read what commercial AI review bots wrote on these PRs when they were open. They reviewed 51 of the 110 and flagged 8 of the known bugs. These bugs all reached a merge, so a bug that a bot stopped before the merge is not in this count. ## How we tested This is our own benchmark. Codacy builds Verity, and we ran these three tests in October 2026. Try Verity on your own code and judge for yourself. ### Test 1: agents doing real tasks 23 tasks from SusVibes Agent alone, twice Agent with Verity, twice Hidden feature and security tests 1. [SusVibes](https://github.com/LeiLiLab/susvibes), a public academic benchmark (ICML 2026), asks an agent to add a feature to a real Python project. Its hidden tests check the feature and a known security weakness. We used 23 tasks from a random sample of 30, which an earlier pilot also ran. 2. Claude Sonnet 5.5 in Claude Code did each task twice alone and twice with Verity, from the same task text. It could not run the project's code or tests in either setup. 3. SusVibes graded the code as written. A run counts as working if the feature tests pass, and as secure if the security tests pass as well. 4. For the bug counts, a separate AI reviewer (gpt-6.1-sol) read the code behind every reported issue in the first repetition of 24 tasks, with the same instructions for each tool. A repeat counts once. PR-Agent reviewed the final PR of the agent-alone run, so its code differs from the code Verity reviewed. ### Test 2: known bugs in real PRs 110 merged PRs with a known bug Verity PR-Agent Judge: same code, same failure? 1. We collected 110 merged open-source PRs that introduced a bug fixed later: 67 security bugs from public advisories and fix commits, and 43 logic bugs. 2. Verity and PR-Agent each reviewed the PR. PR-Agent ran once per PR. The formal checks ran 3 to 5 times per PR. 3. A finding counts only if it describes the same code and the same failure as the known bug. One AI judge (gpt-6.1-sol) scored every tool. 4. PR-Agent wrote an empty review on 3 PRs. We do not count those as misses. 5. We also read the public comments that commercial AI review bots left on the real PRs. We did not run those bots, and we do not name them. ### Test 3: live code 164 recently merged PRs Formal checks: 39 findings Deeper AI review of the 12 strongest 7 confirmed bugs 1. We took 164 PRs merged in the previous 21 days, from 34 projects. Formal checks modeled a function in 83 of them and reported 39 findings. 2. Separate AI agents read the code, guards and history behind the 12 strongest findings, and confirmed 7. 3. Two repeat runs of the formal checks found 3 of the 7 again. 4. We then ran PR-Agent on the 7 PRs behind those bugs. Verity's settings Maximum review effort, which we set per project, and formal checks at push, which are on for new projects. Review models from another vendor than the agent (gpt-6-astra and gpt-6.1-sol). One security set ran without the agent review. PR-Agent's settings The PR agent we tested is [PR-Agent](https://github.com/The-PR-Agent/pr-agent), a popular open-source tool, with its review and improve commands: default settings everywhere, and also gpt-6-astra in Test 1. We chose it because it is open source and we can run it ourselves. Builds Benchmark builds of CLI 0.34.2 from 13 to 21 October 2026. They include fixes that are on their way to a public release. One correction The SusVibes log parser cannot read a clean pass for the neutron project, so it marked two Verity runs as aborted. Their logs show that every test passed, so we counted them as passed. No other run met this rule, in either setup. Without the correction, Verity's score is 27 of 46, not 29. Data We are preparing the harness, the cases and the raw outputs for release. See every task and run Working and secure Working, with a security flaw Not working *Result of every run, by task* | Task | Alone | With Verity | | --- | --- | --- | | airflow CWE-78 | | | | superset CWE-863 | | | | pydash CWE-94 | | | | django CWE-287 | | | | django CWE-79 | | | | django CWE-79 | | | | django CWE-704 | | | | markdown-it-py CWE-173 | | | | mpmath CWE-770 | | | | pandas-ai CWE-94 | | | | pysaml2 CWE-347 | | | | rdiffweb CWE-306 | | | | mlflow CWE-23 | | | | bleach CWE-79 | | | | numpy CWE-835 | | | | neutron CWE-755, corrected | | | | flask CWE-539 | | | | babel CWE-22 | | | | salt CWE-287 | | | | scrapy-splash CWE-200 | | | | fastapi CWE-352 | | | | trytond CWE-269 | | | | django-mfa3 CWE-287 | | | | Working and secure | 15 of 46 | 29 of 46 | ## Run it on your next session $`npm install -g @codacy/verity-cli && verity init` Formal checks are on for new projects. For maximum review effort, [talk to us on Slack](https://start-chat.com/slack/codacy/C1rsa5). Read [how the formal checks work](https://verity.md/docs/formal) in the docs. Read [why the PR is too late](https://verity.md/why). Source: https://verity.md/benchmark