Is your app next?

The six smoke alarms

/claude-security scan the whole repository at high effort. I accept the time and token cost; do not ask.

/security-review

Run a Codex Security deep scan on this repository.

Find the security vulnerabilities in this repository.

Find the security vulnerabilities in this repository.

semgrep scan --config auto --metrics on --json

Round 1: smoke or toast

What the tools dropped

/security-review reported 10 real cases and held back 9 more. Endpoint05 was not mentioned.

The Claude plugin dropped the real path traversal on Endpoint05 because it judged that an HTTP header name could not carry a path separator under the HTTP rules and Tomcat. It does not count as caught.

On fast-jwt, /security-review rated the CVE finding 6/10 and filtered it out. It does not count as caught.

Round 2: fresh bugs

Final: php2shell

None caught php2shell. Codex Security spent 83.4 minutes on PHP. The Claude plugin spent 25.0 minutes and reported one finding in a session handler that needs a non-default build, a different weakness.

How the test worked

The runs took place on October 21 and 22, 2026. A script ran each tool in a sandbox with the network off, and a second script scored the reports against answer keys written before any run. Web search was off too: in a trial run, plain Claude reached a public vulnerability database even with web search turned off. The network block closed that route.

Targets were copied without tests, docs, changelogs or git history, with one exception: fast-jwt kept its pre-fix README. The Benchmark cases were renamed and comments removed to hide the answers. Each tool saw the same target scope. The five CVE targets use the commit before the fix; PHP uses version 8.4.26. A fresh repository made the full target look like one code change to /security-review.

A reported finding needs the right weakness and location. For a CVE it must name a function the fix touched, or its matching line range. For php2shell it must name a core function and the matching kind of state reuse after an out-of-memory error. Every judgment call is listed here. No model graded another model.

Minutes are elapsed time, rounded to one decimal place. Costs are list-price equivalents in US dollars, not a bill or a claim about subscription charges. Semgrep has no model cost; its cost field is unmeasured. Youden is real caught divided by real cases covered, minus safe flagged divided by safe cases covered. Scores use the cases each tool covered. Costs in the download retain source precision.

Every judgment call

Only reported findings count. Candidates a tool drops do not count. A finding must match the weakness and location in the answer key. Unrelated findings count as other findings; they have not been confirmed as bugs.

Semgrep findings with a different weakness on the same endpoint count as other findings. It missed Endpoint33 because the hash algorithm came from a properties file. On curl and PHP it scanned the files but reported partial-parse warnings: 37 on curl and 14 on PHP.

Plain Claude flagged safe Endpoint19 as a guessable token. The key calls it safe because it uses SecureRandom. Plain Codex flagged the same endpoint in all three runs, and safe Endpoint25 in run 1 for its password hashing.

Plain Claude described part of the seroval mechanism, but named another file and a denial of service. It did not name the affected promise functions, so it did not catch the CVE.

Plain Claude caught datamodel-code-generator: its line range points inside the affected function. Its visible report was a summary while another agent was still working. Both closing PHP reports said there were no findings.

Some plain Codex findings needed manual matching because the wording omitted the category keyword. The matched cases were Endpoint24 and Endpoint26 in runs 1 to 3, Endpoint28 and Endpoint36 in runs 2 and 3, Endpoint06 and Endpoint30 in runs 2 and 3, and Endpoint20 in run 3. Some runs executed target code inside the sandbox.

Codex Security called Endpoint07 and Endpoint12 constant XPath selections, so neither counts as a safe case flagged. Its fast-jwt scan failed while saving, but kept a report naming createVerifier and the matching weakness, so that CVE counts as caught.

Codex Security kept a partial seroval report after a scan error. Its 78 other findings included allocation and rejected-promise issues, but none matched the CVE in the answer key.

Codex Security PHP findings about destructor reentry and a memory limit did not describe state reused after an out-of-memory error, a bailout or across requests. Another finding named a function outside the key. None caught php2shell.

The Claude plugin reported information exposure on safe Endpoint04. That is a different weakness from its cipher case, so it counts as an other finding. It ran Benchmark as four scans of ten cases each, which together cover all 40, so its time and cost are the four scans added up. On fast-jwt it named createVerifier with an empty or null key accepting unsigned tokens, so that CVE counts as caught.

The Claude plugin's PHP scan reported one finding: a pointer to freed request memory in the mm session handler (ext/session/mod_mm.c), which needs a non-default build. It names no function in the key and no state reused after an out-of-memory error, so it is not php2shell.

Not completed, discarded or skipped

A Claude plugin Benchmark scan of Endpoint11 to Endpoint20 and a plugin PHP scan were cut off by my plan's usage limit, and a second plugin PHP scan ended when its session restarted. All three were discarded and rerun to completion.

Earlier plain Claude and /security-review attempts on PHP, and /security-review on Benchmark, ended at my plan's usage limit. They were discarded and rerun. Early fast-jwt scans from both Claude modes and a plain Claude seroval scan loaded personal instructions during setup checks. Those attempts were discarded and rerun.

Claude runs 2 and 3 were skipped. Plain Claude did not run on curl or pydantic-ai. /security-review did not run on datamodel-code-generator, pydantic-ai or curl. Among the fresh-CVE targets, the Claude plugin ran only on fast-jwt. The Fable 5.1 bonus was skipped.

A first Codex Security PHP attempt was interrupted and discarded, then rerun. Its datamodel-code-generator scan stopped after 55 minutes for budget with no report. Its pydantic-ai scan stopped after 42 minutes at a cost cap with no report. Its curl scan was skipped for budget. These incomplete scans are not scored.

OpenAI's safeguard stopped 15 of 21 plain Codex runs: all three on curl, datamodel-code-generator, fast-jwt, seroval and PHP. Those runs ended with no final message. They count as stopped, apart from completed scans that missed a bug.

Download the data

JSON includes endpoint truth, covered cases, per-run hits, CVE functions and PHP answer-key functions. CSV has one row per run with counts, time and cost. No report bodies are included.