THE HYPE KEVIN / SECURITYOCTOBER 21 AND 22, 2026

SIX TOOLS / THE FULL RECORD

Round 1: smoke or toast

Round 1: smoke or toast

Per-run results. Counts refer to the answer key.
ToolTargetRunReal caughtSafe flaggedOther findingsMinutesList-price US$Youden
Claude Code's security pluginbenchmark40run116 of 200 of 20340.445.640.80
/security-reviewbenchmark40run110 of 200 of 2009.25.030.50
Codex Securitybenchmark40run111 of 200 of 20267.650.860.55
Codex Securitybenchmark40run212 of 201 of 20269.965.150.55
Plain Claude Codebenchmark40run120 of 201 of 2031.40.660.95
Plain Codexbenchmark40run120 of 202 of 2044.00.420.90
Plain Codexbenchmark40run219 of 201 of 2053.40.370.90
Plain Codexbenchmark40run319 of 201 of 2063.30.350.90
Semgrepbenchmark40run119 of 208 of 20120.1not measured0.55
Semgrepbenchmark40run219 of 208 of 20120.1not measured0.55
Semgrepbenchmark40run319 of 208 of 20120.1not measured0.55
Each cell lists the runs that hit the case. Real hits are caught; safe hits are false alarms.
EndpointCategory / CWETruthClaude Code's security plugin/security-reviewCodex SecurityPlain Claude CodePlain CodexSemgrep
Endpoint01sqli / CWE-89safeclearclearclearclearclearrun1, run2, run3
Endpoint02xpathi / CWE-643realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint03hash / CWE-328safeclearclearclearclearclearclear
Endpoint04crypto / CWE-327safeclearclearclearclearclearclear
Endpoint05pathtraver / CWE-22realmissedmissedmissedrun1run1run1, run2, run3
Endpoint06trustbound / CWE-501realrun1missedmissedrun1run1, run2, run3run1, run2, run3
Endpoint07xpathi / CWE-643safeclearclearclearclearclearrun1, run2, run3
Endpoint08trustbound / CWE-501safeclearclearclearclearclearclear
Endpoint09hash / CWE-328realrun1missedrun1, run2run1run1, run2, run3run1, run2, run3
Endpoint10crypto / CWE-327realrun1missedrun1, run2run1run1, run2, run3run1, run2, run3
Endpoint11cmdi / CWE-78safeclearclearclearclearclearrun1, run2, run3
Endpoint12xpathi / CWE-643safeclearclearclearclearclearrun1, run2, run3
Endpoint13crypto / CWE-327safeclearclearclearclearclearclear
Endpoint14pathtraver / CWE-22realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint15securecookie / CWE-614safeclearclearclearclearclearclear
Endpoint16xss / CWE-79realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint17sqli / CWE-89realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint18ldapi / CWE-90safeclearclearclearclearclearrun1, run2, run3
Endpoint19weakrand / CWE-330safeclearclearclearrun1run1, run2, run3clear
Endpoint20securecookie / CWE-614realrun1missedmissedrun1run1, run2, run3run1, run2, run3
Endpoint21sqli / CWE-89safeclearclearclearclearclearclear
Endpoint22pathtraver / CWE-22safeclearclearclearclearclearrun1, run2, run3
Endpoint23xss / CWE-79safeclearclearclearclearclearrun1, run2, run3
Endpoint24ldapi / CWE-90realrun1run1missedrun1run1, run2, run3run1, run2, run3
Endpoint25hash / CWE-328safeclearclearrun2clearrun1clear
Endpoint26ldapi / CWE-90realrun1run1missedrun1run1, run2, run3run1, run2, run3
Endpoint27cmdi / CWE-78safeclearclearclearclearclearclear
Endpoint28cmdi / CWE-78realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint29sqli / CWE-89realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint30trustbound / CWE-501realrun1missedmissedrun1run1, run2, run3run1, run2, run3
Endpoint31xss / CWE-79safeclearclearclearclearclearclear
Endpoint32xpathi / CWE-643realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint33hash / CWE-328realrun1missedrun1, run2run1run1, run2, run3missed
Endpoint34weakrand / CWE-330realmissedmissedmissedrun1run1, run2, run3run1, run2, run3
Endpoint35securecookie / CWE-614safeclearclearclearclearclearclear
Endpoint36cmdi / CWE-78realrun1run1run1, run2run1run1, run2, run3run1, run2, run3
Endpoint37weakrand / CWE-330safeclearclearclearclearclearclear
Endpoint38pathtraver / CWE-22safeclearclearclearclearclearrun1, run2, run3
Endpoint39weakrand / CWE-330realmissedmissedmissedrun1run1, run2, run3run1, run2, run3
Endpoint40crypto / CWE-327realmissedmissedrun2run1run1, run2, run3run1, run2, run3

Claude Code's security plugin

0055101015152020run1: 16 real caught; 0 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

/security-review

0055101015152020run1: 10 real caught; 0 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

Codex Security

0055101015152020run1: 11 real caught; 0 safe flaggedrun2: 12 real caught; 1 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

Plain Claude Code

0055101015152020run1: 20 real caught; 1 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

Plain Codex

0055101015152020run1: 20 real caught; 2 safe flaggedrun2: 19 real caught; 1 safe flaggedrun3: 19 real caught; 1 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

Semgrep

0055101015152020run1: 19 real caught; 8 safe flaggedrun2: 19 real caught; 8 safe flaggedrun3: 19 real caught; 8 safe flaggedSafe flagged (cases)Real caught (cases)
Source: saved experiment results, October 21 and 22, 2026. Each ring is a run. See the per-run text table above.

What the tools dropped

Nine real cases held below confidence 8.
Real casesWhat /security-review held back
Endpoint09, Endpoint33MD5 hashing
Endpoint10, Endpoint40DES encryption
Endpoint34, Endpoint39Predictable random numbers
Endpoint06, Endpoint30Untrusted values stored in the session
Endpoint20Cookie missing its Secure flag

/security-review reported 10 real cases and held back 9 more. Endpoint05 was not mentioned.

The Claude plugin dropped the real path traversal on Endpoint05 because it judged that an HTTP header name could not carry a path separator under the HTTP rules and Tomcat. It does not count as caught.

On fast-jwt, /security-review rated the CVE finding 6/10 and filtered it out. It does not count as caught.