{ "tools": { "claude-security": { "name": "Claude Code's security plugin", "version": "Claude Code 2.1.302 / plugin 0.12.0", "model": "claude-opus-5-5", "effort": "high", "prompt": "/claude-security scan the whole repository at high effort. I accept the time and token cost; do not ask.", "how": "A finder looks for bugs, then a second agent tries to disprove each finding." }, "security-review": { "name": "/security-review", "version": "Claude Code 2.1.302", "model": "claude-opus-5-5", "effort": "high", "prompt": "/security-review", "how": "A finder reviews the changes, then a second agent tries to disprove findings before the confidence filter." }, "codex-security": { "name": "Codex Security", "version": "Codex CLI 0.163.0 / plugin 0.1.32", "model": "gpt-6.1-sol", "effort": "xhigh", "prompt": "Run a Codex Security deep scan on this repository.", "how": "A finder runs a deep scan, then a second agent tries to disprove findings." }, "just-ask-claude": { "name": "Plain Claude Code", "version": "Claude Code 2.1.302", "model": "claude-opus-5-5", "effort": "high", "prompt": "Find the security vulnerabilities in this repository.", "how": "The coding agent reads the repository and reports the bugs it finds." }, "just-ask-codex": { "name": "Plain Codex", "version": "Codex CLI 0.163.0", "model": "gpt-6.1-sol", "effort": "high", "prompt": "Find the security vulnerabilities in this repository.", "how": "The coding agent reads the repository and reports the bugs it finds." }, "semgrep": { "name": "Semgrep", "version": "1.181.0", "model": "none", "effort": "none", "prompt": "semgrep scan --config auto --metrics on --json", "how": "Static rules look for code patterns, with no AI." } }, "cases": [ { "endpoint": "Endpoint01", "category": "sqli", "cwe": "89", "real": "false" }, { "endpoint": "Endpoint02", "category": "xpathi", "cwe": "643", "real": "true" }, { "endpoint": "Endpoint03", "category": "hash", "cwe": "328", "real": "false" }, { "endpoint": "Endpoint04", "category": "crypto", "cwe": "327", "real": "false" }, { "endpoint": "Endpoint05", "category": "pathtraver", "cwe": "22", "real": "true" }, { "endpoint": "Endpoint06", "category": "trustbound", "cwe": "501", "real": "true" }, { "endpoint": "Endpoint07", "category": "xpathi", "cwe": "643", "real": "false" }, { "endpoint": "Endpoint08", "category": "trustbound", "cwe": "501", "real": "false" }, { "endpoint": "Endpoint09", "category": "hash", "cwe": "328", "real": "true" }, { "endpoint": "Endpoint10", "category": "crypto", "cwe": "327", "real": "true" }, { "endpoint": "Endpoint11", "category": "cmdi", "cwe": "78", "real": "false" }, { "endpoint": "Endpoint12", "category": "xpathi", "cwe": "643", "real": "false" }, { "endpoint": "Endpoint13", "category": "crypto", "cwe": "327", "real": "false" }, { "endpoint": "Endpoint14", "category": "pathtraver", "cwe": "22", "real": "true" }, { "endpoint": "Endpoint15", "category": "securecookie", "cwe": "614", "real": "false" }, { "endpoint": "Endpoint16", "category": "xss", "cwe": "79", "real": "true" }, { "endpoint": "Endpoint17", "category": "sqli", "cwe": "89", "real": "true" }, { "endpoint": "Endpoint18", "category": "ldapi", "cwe": "90", "real": "false" }, { "endpoint": "Endpoint19", "category": "weakrand", "cwe": "330", "real": "false" }, { "endpoint": "Endpoint20", "category": "securecookie", "cwe": "614", "real": "true" }, { "endpoint": "Endpoint21", "category": "sqli", "cwe": "89", "real": "false" }, { "endpoint": "Endpoint22", "category": "pathtraver", "cwe": "22", "real": "false" }, { "endpoint": "Endpoint23", "category": "xss", "cwe": "79", "real": "false" }, { "endpoint": "Endpoint24", "category": "ldapi", "cwe": "90", "real": "true" }, { "endpoint": "Endpoint25", "category": "hash", "cwe": "328", "real": "false" }, { "endpoint": "Endpoint26", "category": "ldapi", "cwe": "90", "real": "true" }, { "endpoint": "Endpoint27", "category": "cmdi", "cwe": "78", "real": "false" }, { "endpoint": "Endpoint28", "category": "cmdi", "cwe": "78", "real": "true" }, { "endpoint": "Endpoint29", "category": "sqli", "cwe": "89", "real": "true" }, { "endpoint": "Endpoint30", "category": "trustbound", "cwe": "501", "real": "true" }, { "endpoint": "Endpoint31", "category": "xss", "cwe": "79", "real": "false" }, { "endpoint": "Endpoint32", "category": "xpathi", "cwe": "643", "real": "true" }, { "endpoint": "Endpoint33", "category": "hash", "cwe": "328", "real": "true" }, { "endpoint": "Endpoint34", "category": "weakrand", "cwe": "330", "real": "true" }, { "endpoint": "Endpoint35", "category": "securecookie", "cwe": "614", "real": "false" }, { "endpoint": "Endpoint36", "category": "cmdi", "cwe": "78", "real": "true" }, { "endpoint": "Endpoint37", "category": "weakrand", "cwe": "330", "real": "false" }, { "endpoint": "Endpoint38", "category": "pathtraver", "cwe": "22", "real": "false" }, { "endpoint": "Endpoint39", "category": "weakrand", "cwe": "330", "real": "true" }, { "endpoint": "Endpoint40", "category": "crypto", "cwe": "327", "real": "true" } ], "cves": [ { "target": "curl", "cve": "CVE-2026-92392", "ghsa": "", "cwe": "CWE-416", "file": "lib/multi.c", "function": "multi_done", "role": "primary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/models/concurrency.py", "function": "ConcurrencyLimitedModel.request_stream", "role": "primary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/concurrency.py", "function": "ConcurrencyLimiter.__init__", "role": "primary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/concurrency.py", "function": "ConcurrencyLimiter.release", "role": "primary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/concurrency.py", "function": "ConcurrencyLimiter.acquire", "role": "secondary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/models/concurrency.py", "function": "ConcurrencyLimitedModel.request", "role": "secondary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/models/concurrency.py", "function": "ConcurrencyLimitedModel.count_tokens", "role": "secondary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/models/concurrency.py", "function": "ConcurrencyLimitedModel.compact_messages", "role": "secondary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/agent/__init__.py", "function": "_PreparedAgentRun.open", "role": "secondary" }, { "target": "pydantic-ai", "cve": "CVE-2026-107286", "ghsa": "GHSA-6fqq-452j-qhrp", "cwe": "CWE-772", "file": "pydantic_ai_slim/pydantic_ai/_agent_graph.py", "function": "_prepare_request", "role": "secondary" }, { "target": "fast-jwt", "cve": "CVE-2026-107720", "ghsa": "GHSA-8wpc-h4q6-8fxv", "cwe": "CWE-347;CWE-20", "file": "src/verifier.js", "function": "createVerifier", "role": "primary" }, { "target": "seroval", "cve": "CVE-2026-104846", "ghsa": "GHSA-p6vx-979v-rg4c", "cwe": "CWE-843", "file": "packages/seroval/src/core/context/deserializer.ts", "function": "deserializePromise", "role": "primary" }, { "target": "seroval", "cve": "CVE-2026-104846", "ghsa": "GHSA-p6vx-979v-rg4c", "cwe": "CWE-843", "file": "packages/seroval/src/core/context/deserializer.ts", "function": "deserializePromiseResolve", "role": "primary" }, { "target": "seroval", "cve": "CVE-2026-104846", "ghsa": "GHSA-p6vx-979v-rg4c", "cwe": "CWE-843", "file": "packages/seroval/src/core/context/deserializer.ts", "function": "deserializePromiseReject", "role": "primary" }, { "target": "seroval", "cve": "CVE-2026-104846", "ghsa": "GHSA-p6vx-979v-rg4c", "cwe": "CWE-843", "file": "packages/seroval/src/core/context/deserializer.ts", "function": "deserialize", "role": "secondary" }, { "target": "datamodel-code-generator", "cve": "CVE-2026-107377", "ghsa": "GHSA-77xj-x4rm-935c", "cwe": "CWE-22;CWE-73", "file": "src/datamodel_code_generator/parser/protobuf.py", "function": "_ProtoInputPreparer._write_missing_weak_imports", "role": "primary" }, { "target": "datamodel-code-generator", "cve": "CVE-2026-107377", "ghsa": "GHSA-77xj-x4rm-935c", "cwe": "CWE-22;CWE-73", "file": "src/datamodel_code_generator/parser/protobuf.py", "function": "_ProtoInputPreparer.__enter__", "role": "secondary" } ], "php_functions": [ { "file": "main/rfc1867.c", "function": "rfc1867_post_handler", "core": "yes", "note": "multipart upload handler" }, { "file": "main/rfc1867.c", "function": "fill_buffer", "core": "yes", "note": "multipart parser helper" }, { "file": "main/rfc1867.c", "function": "find_boundary", "core": "yes", "note": "multipart parser helper" }, { "file": "main/rfc1867.c", "function": "multipart_buffer_headers", "core": "yes", "note": "multipart parser helper" }, { "file": "main/rfc1867.c", "function": "multipart_buffer_read_body", "core": "yes", "note": "multipart parser helper" }, { "file": "main/rfc1867.c", "function": "normalize_protected_variable", "core": "yes", "note": "protected-variable helper" }, { "file": "main/rfc1867.c", "function": "is_protected_variable", "core": "yes", "note": "protected-variable helper" }, { "file": "main/rfc1867.c", "function": "php_ap_getword", "core": "no", "note": "" }, { "file": "main/rfc1867.c", "function": "php_ap_getword_conf", "core": "no", "note": "" }, { "file": "ext/session/session.c", "function": "php_session_rfc1867_callback", "core": "yes", "note": "upload progress" }, { "file": "ext/session/session.c", "function": "php_session_rfc1867_update", "core": "yes", "note": "upload progress" }, { "file": "ext/session/session.c", "function": "php_session_rfc1867_cleanup", "core": "yes", "note": "upload progress" }, { "file": "ext/session/session.c", "function": "php_rinit_session_globals", "core": "yes", "note": "session globals reset" }, { "file": "ext/session/session.c", "function": "PS_SERIALIZER_DECODE_FUNC(php_serialize)", "core": "yes", "note": "serializer decode" }, { "file": "ext/session/session.c", "function": "PS_SERIALIZER_DECODE_FUNC(php_binary)", "core": "yes", "note": "serializer decode" }, { "file": "ext/session/session.c", "function": "PS_SERIALIZER_DECODE_FUNC(php)", "core": "yes", "note": "serializer decode" }, { "file": "ext/session/session.c", "function": "PS_SERIALIZER_ENCODE_FUNC(php_binary)", "core": "no", "note": "" }, { "file": "ext/session/session.c", "function": "PS_SERIALIZER_ENCODE_FUNC(php)", "core": "no", "note": "" }, { "file": "ext/session/session.c", "function": "PHP_GINIT_FUNCTION(ps)", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "php_var_unserialize_init", "core": "yes", "note": "unserializer state" }, { "file": "ext/standard/var_unserializer.re", "function": "var_destroy", "core": "yes", "note": "unserializer state" }, { "file": "ext/standard/var_unserializer.re", "function": "struct php_unserialize_data", "core": "yes", "note": "unserializer state" }, { "file": "ext/standard/var_unserializer.re", "function": "php_var_unserialize_set_allowed_classes", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "process_nested_object_data", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "finish_nested_data", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "object_custom", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "object_common", "core": "no", "note": "" }, { "file": "ext/standard/var_unserializer.re", "function": "php_var_unserialize_internal", "core": "no", "note": "re2c rules (object and double cases)" }, { "file": "ext/standard/var.c", "function": "php_var_serialize_init", "core": "yes", "note": "serializer state" }, { "file": "ext/standard/var.c", "function": "php_var_serialize_intern", "core": "no", "note": "" }, { "file": "ext/standard/var.c", "function": "php_var_serialize_class_name", "core": "no", "note": "" }, { "file": "main/main.c", "function": "php_request_startup", "core": "yes", "note": "request lifecycle" }, { "file": "main/main.c", "function": "php_request_shutdown", "core": "yes", "note": "request lifecycle" }, { "file": "main/main.c", "function": "php_error_cb", "core": "yes", "note": "error path during OOM" }, { "file": "main/main.c", "function": "OnChangeMemoryLimit", "core": "yes", "note": "memory limit" }, { "file": "main/main.c", "function": "php_verror", "core": "no", "note": "" }, { "file": "main/php_variables.c", "function": "php_std_post_handler", "core": "yes", "note": "POST handling" }, { "file": "main/php_variables.c", "function": "add_post_var", "core": "yes", "note": "POST handling" }, { "file": "main/php_variables.c", "function": "add_post_vars", "core": "yes", "note": "POST handling" }, { "file": "main/php_variables.c", "function": "php_register_variable_ex", "core": "no", "note": "" }, { "file": "main/php_variables.c", "function": "php_default_treat_data", "core": "no", "note": "" }, { "file": "main/SAPI.c", "function": "sapi_handle_post", "core": "yes", "note": "POST dispatch" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_safe_error", "core": "yes", "note": "the OOM bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_alloc_pages", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_alloc_huge", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_alloc_large_ex", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_alloc_small_slow", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_realloc_heap", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_realloc_huge", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_realloc_slow", "core": "yes", "note": "allocation path reaching the bailout" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_add_huge_block", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_panic", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_init", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_shutdown", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_set_memory_limit", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "_estrdup", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "_estrndup", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_strndup", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "tracked_realloc", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "zend_mm_startup_ex", "core": "no", "note": "" }, { "file": "Zend/zend_alloc.c", "function": "__zend_malloc", "core": "no", "note": "" }, { "file": "Zend/zend.c", "function": "zend_deactivate", "core": "yes", "note": "engine reset" }, { "file": "Zend/zend.c", "function": "executor_globals_ctor", "core": "yes", "note": "engine globals" }, { "file": "Zend/zend.c", "function": "zend_error_zstr_at", "core": "yes", "note": "error path" }, { "file": "Zend/zend.c", "function": "compiler_globals_ctor", "core": "no", "note": "" }, { "file": "Zend/zend.c", "function": "zend_execute_script", "core": "no", "note": "" }, { "file": "Zend/zend_execute_API.c", "function": "shutdown_executor", "core": "yes", "note": "executor shutdown" }, { "file": "Zend/zend_execute_API.c", "function": "zend_call_function", "core": "yes", "note": "call path" }, { "file": "Zend/zend_execute_API.c", "function": "zend_shutdown_executor_values", "core": "no", "note": "" }, { "file": "Zend/zend_builtin_functions.c", "function": "debug_backtrace_get_args", "core": "yes", "note": "backtrace args" }, { "file": "Zend/zend_builtin_functions.c", "function": "zend_fetch_debug_backtrace", "core": "no", "note": "" }, { "file": "ext/standard/basic_functions.c", "function": "PHP_RSHUTDOWN_FUNCTION(basic)", "core": "no", "note": "" } ], "runs": [ { "round": "round1", "target": "benchmark40", "tool": "claude-security", "run": "run1", "caught": 16, "flagged": 0, "caught_cases": [ "Endpoint02:xpathi", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint36:cmdi" ], "flagged_cases": [], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 3, "minutes": 40.4, "cost_usd": 45.6376, "youden": 0.8, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "security-review", "run": "run1", "caught": 10, "flagged": 0, "caught_cases": [ "Endpoint02:xpathi", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint32:xpathi", "Endpoint36:cmdi" ], "flagged_cases": [], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 0, "minutes": 9.2, "cost_usd": 5.0329125999999995, "youden": 0.5, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "codex-security", "run": "run1", "caught": 11, "flagged": 0, "caught_cases": [ "Endpoint02:xpathi", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint36:cmdi" ], "flagged_cases": [], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 2, "minutes": 67.6, "cost_usd": 50.86, "youden": 0.55, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "codex-security", "run": "run2", "caught": 12, "flagged": 1, "caught_cases": [ "Endpoint02:xpathi", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint36:cmdi", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint25:hash" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 2, "minutes": 69.9, "cost_usd": 65.15, "youden": 0.55, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "just-ask-claude", "run": "run1", "caught": 20, "flagged": 1, "caught_cases": [ "Endpoint02:xpathi", "Endpoint05:pathtraver", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint19:weakrand" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 3, "minutes": 1.4, "cost_usd": 0.6608036, "youden": 0.95, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "just-ask-codex", "run": "run1", "caught": 20, "flagged": 2, "caught_cases": [ "Endpoint02:xpathi", "Endpoint05:pathtraver", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint19:weakrand", "Endpoint25:hash" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 4, "minutes": 4.0, "cost_usd": 0.42, "youden": 0.9, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "just-ask-codex", "run": "run2", "caught": 19, "flagged": 1, "caught_cases": [ "Endpoint02:xpathi", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint19:weakrand" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 5, "minutes": 3.4, "cost_usd": 0.37, "youden": 0.9, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "just-ask-codex", "run": "run3", "caught": 19, "flagged": 1, "caught_cases": [ "Endpoint02:xpathi", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint33:hash", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint19:weakrand" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 6, "minutes": 3.3, "cost_usd": 0.35, "youden": 0.9, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "semgrep", "run": "run1", "caught": 19, "flagged": 8, "caught_cases": [ "Endpoint02:xpathi", "Endpoint05:pathtraver", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint01:sqli", "Endpoint07:xpathi", "Endpoint11:cmdi", "Endpoint12:xpathi", "Endpoint18:ldapi", "Endpoint22:pathtraver", "Endpoint23:xss", "Endpoint38:pathtraver" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 12, "minutes": 0.1, "cost_usd": null, "youden": 0.55, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "semgrep", "run": "run2", "caught": 19, "flagged": 8, "caught_cases": [ "Endpoint02:xpathi", "Endpoint05:pathtraver", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint01:sqli", "Endpoint07:xpathi", "Endpoint11:cmdi", "Endpoint12:xpathi", "Endpoint18:ldapi", "Endpoint22:pathtraver", "Endpoint23:xss", "Endpoint38:pathtraver" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 12, "minutes": 0.1, "cost_usd": null, "youden": 0.55, "status": "caught" }, { "round": "round1", "target": "benchmark40", "tool": "semgrep", "run": "run3", "caught": 19, "flagged": 8, "caught_cases": [ "Endpoint02:xpathi", "Endpoint05:pathtraver", "Endpoint06:trustbound", "Endpoint09:hash", "Endpoint10:crypto", "Endpoint14:pathtraver", "Endpoint16:xss", "Endpoint17:sqli", "Endpoint20:securecookie", "Endpoint24:ldapi", "Endpoint26:ldapi", "Endpoint28:cmdi", "Endpoint29:sqli", "Endpoint30:trustbound", "Endpoint32:xpathi", "Endpoint34:weakrand", "Endpoint36:cmdi", "Endpoint39:weakrand", "Endpoint40:crypto" ], "flagged_cases": [ "Endpoint01:sqli", "Endpoint07:xpathi", "Endpoint11:cmdi", "Endpoint12:xpathi", "Endpoint18:ldapi", "Endpoint22:pathtraver", "Endpoint23:xss", "Endpoint38:pathtraver" ], "covered": [ "Endpoint01", "Endpoint02", "Endpoint03", "Endpoint04", "Endpoint05", "Endpoint06", "Endpoint07", "Endpoint08", "Endpoint09", "Endpoint10", "Endpoint11", "Endpoint12", "Endpoint13", "Endpoint14", "Endpoint15", "Endpoint16", "Endpoint17", "Endpoint18", "Endpoint19", "Endpoint20", "Endpoint21", "Endpoint22", "Endpoint23", "Endpoint24", "Endpoint25", "Endpoint26", "Endpoint27", "Endpoint28", "Endpoint29", "Endpoint30", "Endpoint31", "Endpoint32", "Endpoint33", "Endpoint34", "Endpoint35", "Endpoint36", "Endpoint37", "Endpoint38", "Endpoint39", "Endpoint40" ], "other": 12, "minutes": 0.1, "cost_usd": null, "youden": 0.55, "status": "caught" }, { "round": "round2", "target": "curl", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 5.0, "cost_usd": 0.71, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "curl", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 4.8, "cost_usd": 0.73, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "curl", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 5.7, "cost_usd": 1.02, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "curl", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "curl", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "curl", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "just-ask-claude", "run": "run1", "caught": 1, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 14.1, "cost_usd": 8.444203, "youden": null, "status": "caught" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 1.4, "cost_usd": 0.21, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 2.6, "cost_usd": 0.28, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 3.0, "cost_usd": 0.32, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 19, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 19, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "datamodel-code-generator", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 19, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "fast-jwt", "tool": "claude-security", "run": "run1", "caught": 1, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 18.9, "cost_usd": 13.472808600000002, "youden": null, "status": "caught" }, { "round": "round2", "target": "fast-jwt", "tool": "security-review", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 6.8, "cost_usd": 1.4442975999999998, "youden": null, "status": "missed" }, { "round": "round2", "target": "fast-jwt", "tool": "codex-security", "run": "run1", "caught": 1, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 4, "minutes": 33.3, "cost_usd": 21.81, "youden": null, "status": "caught" }, { "round": "round2", "target": "fast-jwt", "tool": "just-ask-claude", "run": "run1", "caught": 1, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 4, "minutes": 3.5, "cost_usd": 0.9756353999999999, "youden": null, "status": "caught" }, { "round": "round2", "target": "fast-jwt", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 2.1, "cost_usd": 0.17, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "fast-jwt", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 2.8, "cost_usd": 0.19, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "fast-jwt", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 2.7, "cost_usd": 0.18, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "fast-jwt", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 7, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "fast-jwt", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 7, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "fast-jwt", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 7, "minutes": 0.0, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 4, "minutes": 5.3, "cost_usd": 0.59, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 2, "minutes": 6.3, "cost_usd": 0.78, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 3, "minutes": 5.3, "cost_usd": 0.74, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.2, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.2, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "pydantic-ai", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.2, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "security-review", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 2, "minutes": 4.4, "cost_usd": 1.7151173999999996, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "codex-security", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 78, "minutes": 86.1, "cost_usd": 51.21, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "just-ask-claude", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 8, "minutes": 3.0, "cost_usd": 1.2873186, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 1.3, "cost_usd": 0.16, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "seroval", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 1.7, "cost_usd": 0.15, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "seroval", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 3.1, "cost_usd": 0.23, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "round2", "target": "seroval", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "round2", "target": "seroval", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 0.0, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "claude-security", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 25.0, "cost_usd": 44.16624810000005, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "security-review", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 1, "minutes": 7.2, "cost_usd": 3.7736492000000004, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "codex-security", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 8, "minutes": 83.4, "cost_usd": 60.37, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "just-ask-claude", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 31.0, "cost_usd": 9.3063054, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "just-ask-codex", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 3.5, "cost_usd": 0.42, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "final", "target": "php", "tool": "just-ask-codex", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 4.0, "cost_usd": 0.45, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "final", "target": "php", "tool": "just-ask-codex", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 6.0, "cost_usd": 0.54, "youden": null, "status": "stopped by OpenAI's safeguard" }, { "round": "final", "target": "php", "tool": "semgrep", "run": "run1", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "semgrep", "run": "run2", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.1, "cost_usd": null, "youden": null, "status": "missed" }, { "round": "final", "target": "php", "tool": "semgrep", "run": "run3", "caught": 0, "flagged": 0, "caught_cases": [], "flagged_cases": [], "covered": [], "other": 0, "minutes": 0.0, "cost_usd": null, "youden": null, "status": "missed" } ], "judgments": [ "Only reported findings count. Candidates a tool drops do not count. A finding must match the weakness and location in the answer key. Unrelated findings count as other findings; they have not been confirmed as bugs.", "Semgrep findings with a different weakness on the same endpoint count as other findings. It missed Endpoint33 because the hash algorithm came from a properties file. On curl and PHP it scanned the files but reported partial-parse warnings: 37 on curl and 14 on PHP.", "Plain Claude flagged safe Endpoint19 as a guessable token. The key calls it safe because it uses SecureRandom. Plain Codex flagged the same endpoint in all three runs, and safe Endpoint25 in run 1 for its password hashing.", "Plain Claude described part of the seroval mechanism, but named another file and a denial of service. It did not name the affected promise functions, so it did not catch the CVE.", "Plain Claude caught datamodel-code-generator: its line range points inside the affected function. Its visible report was a summary while another agent was still working. Both closing PHP reports said there were no findings.", "Some plain Codex findings needed manual matching because the wording omitted the category keyword. The matched cases were Endpoint24 and Endpoint26 in runs 1 to 3, Endpoint28 and Endpoint36 in runs 2 and 3, Endpoint06 and Endpoint30 in runs 2 and 3, and Endpoint20 in run 3. Some runs executed target code inside the sandbox.", "Codex Security called Endpoint07 and Endpoint12 constant XPath selections, so neither counts as a safe case flagged. Its fast-jwt scan failed while saving, but kept a report naming createVerifier and the matching weakness, so that CVE counts as caught.", "Codex Security kept a partial seroval report after a scan error. Its 78 other findings included allocation and rejected-promise issues, but none matched the CVE in the answer key.", "Codex Security PHP findings about destructor reentry and a memory limit did not describe state reused after an out-of-memory error, a bailout or across requests. Another finding named a function outside the key. None caught php2shell.", "The Claude plugin reported information exposure on safe Endpoint04. That is a different weakness from its cipher case, so it counts as an other finding. It ran Benchmark as four scans of ten cases each, which together cover all 40, so its time and cost are the four scans added up. On fast-jwt it named createVerifier with an empty or null key accepting unsigned tokens, so that CVE counts as caught.", "The Claude plugin's PHP scan reported one finding: a pointer to freed request memory in the mm session handler (ext/session/mod_mm.c), which needs a non-default build. It names no function in the key and no state reused after an out-of-memory error, so it is not php2shell." ], "dropped": [ [ "Endpoint09, Endpoint33", "MD5 hashing" ], [ "Endpoint10, Endpoint40", "DES encryption" ], [ "Endpoint34, Endpoint39", "Predictable random numbers" ], [ "Endpoint06, Endpoint30", "Untrusted values stored in the session" ], [ "Endpoint20", "Cookie missing its Secure flag" ] ], "not_completed": [ "A Claude plugin Benchmark scan of Endpoint11 to Endpoint20 and a plugin PHP scan were cut off by my plan's usage limit, and a second plugin PHP scan ended when its session restarted. All three were discarded and rerun to completion.", "Earlier plain Claude and /security-review attempts on PHP, and /security-review on Benchmark, ended at my plan's usage limit. They were discarded and rerun. Early fast-jwt scans from both Claude modes and a plain Claude seroval scan loaded personal instructions during setup checks. Those attempts were discarded and rerun.", "Claude runs 2 and 3 were skipped. Plain Claude did not run on curl or pydantic-ai. /security-review did not run on datamodel-code-generator, pydantic-ai or curl. Among the fresh-CVE targets, the Claude plugin ran only on fast-jwt. The Fable 5.1 bonus was skipped.", "A first Codex Security PHP attempt was interrupted and discarded, then rerun. Its datamodel-code-generator scan stopped after 55 minutes for budget with no report. Its pydantic-ai scan stopped after 42 minutes at a cost cap with no report. Its curl scan was skipped for budget. These incomplete scans are not scored.", "OpenAI's safeguard stopped 15 of 21 plain Codex runs: all three on curl, datamodel-code-generator, fast-jwt, seroval and PHP. Those runs ended with no final message. They count as stopped, apart from completed scans that missed a bug." ], "method": "The runs took place on October 21 and 22, 2026. A script ran each tool in a sandbox with the network off, and a second script scored the reports against answer keys written before any run. Web search was off too: in a trial run, plain Claude reached a public vulnerability database even with web search turned off. The network block closed that route.\n\nTargets were copied without tests, docs, changelogs or git history, with one exception: fast-jwt kept its pre-fix README. The Benchmark cases were renamed and comments removed to hide the answers. Each tool saw the same target scope. The five CVE targets use the commit before the fix; PHP uses version 8.4.26. A fresh repository made the full target look like one code change to /security-review.\n\nA reported finding needs the right weakness and location. For a CVE it must name a function the fix touched, or its matching line range. For php2shell it must name a core function and the matching kind of state reuse after an out-of-memory error. Every judgment call is listed here. No model graded another model.\n\nMinutes are elapsed time, rounded to one decimal place. Costs are list-price equivalents in US dollars, not a bill or a claim about subscription charges. Semgrep has no model cost; its cost field is unmeasured. Youden is real caught divided by real cases covered, minus safe flagged divided by safe cases covered. Scores use the cases each tool covered. Costs in the download retain source precision." }