Claude Mythos Bugs: The Human Verification Gap Revealed

Post Views: 195

Most of the vulnerabilities identified by Claude Mythos have not undergone human validation, highlighting gaps in automated security assessments.

Validation and Review Process

Anthropic’s Claude Mythos Preview analyzed 281 open-source projects, identifying 23,019 potential vulnerabilities. Of these, 1,900 were evaluated by external security firms. Project maintainers received 1,596 reports and confirmed 1,451. Among these, 97 were resolved through upstream fixes, while 88 resulted in published security advisories as of May 22, 2026. The remaining 21,119 candidates remain unreviewed by third parties.

Reviewed Samples and Unreviewed Candidates

Anthropic attributes this gap to limited human resources for validation. Of the reviewed cases, 90.8% of the 1,900 samples were confirmed as legitimate vulnerabilities. However, Echo, the software supply chain firm compiling these metrics, notes that the 1,900 reviewed samples likely represent a non-random subset, suggesting the 90.8% accuracy rate reflects the quality of the top-tier candidates rather than the entire dataset.

Anthropic has not disclosed validation results for the unreviewed 21,000 vulnerabilities.

Severity Discrepancies

Severity classifications showed significant discrepancies. Of the 88 advisories, 27 received Common Vulnerabilities and Exposures (CVE) designations. Mythos assigned eight as Critical, 15 as High, and four as Medium, with no Low-rated issues. Independent assessments by maintainers and third-party evaluators produced different outcomes: one Critical, 16 High, eight Medium, and two Low. Only one Critical rating from Mythos aligned with external evaluations.

Case Studies in Severity Mismatch

Two cases demonstrated the model’s limitations. For Temporal Server, Mythos classified an issue as Critical, citing potential workflow control across namespaces. Maintainers rated it 2.3 (Low) due to prerequisites requiring an attacker-controlled namespace with existing privileged credentials, limiting impact to known workflows. Similarly, MinIO’s vulnerability was rated Critical by Mythos, High by an external firm, and Medium by the project maintainers. The latter determined exploitation required an existing cluster root JWT and granted only read access.

Benchmark Testing and Cost Efficiency

Anthropic conducted a benchmark test using 50 previously discovered vulnerabilities in the SpiderMonkey JavaScript engine (Firefox 147). Each model attempted five exploits per vulnerability, totaling 250 trials. Claude Mythos Preview achieved a working arbitrary code execution exploit from a known crash in 181 trials (72.4%). In contrast, Claude Opus 4.6 succeeded in only two attempts (less than 1%). This 90-fold improvement between model versions coincided with three conditions: all trials started from pre-identified crashes, the testing environment disabled Firefox’s browser sandbox and other security controls, and Anthropic designed and executed the evaluation without independent verification.

Resource Allocation and Exploit Development

Cost efficiency also varied. Anthropic demonstrated a Linux kernel use-after-free vulnerability as a root exploit for under $2,000 in inference costs and within a day of runtime. This exploit combined two kernel flaws: a use-after-free in the main kernel and another in the traffic-control scheduler. However, identifying a viable vulnerability still requires several thousand dollars in resources, with no guarantee of high-severity outcomes.

Industry Survey and Future Implications

A July 2026 survey of over 80 senior U.S. security leaders revealed that 37% cited the challenge of detecting more vulnerabilities than they can remediate as the primary obstacle to improving software supply chain security. Eleven percent indicated plans to invest in enhanced detection or scanning capabilities.

Anthropic’s Manual Verification Efforts

Anthropic has manually verified additional vulnerabilities but has not shared them with maintainers due to insufficient review capacity within its team and external partners.

Conclusion

The analysis underscores the complexities of automated vulnerability detection. While tools like Claude Mythos can identify potential issues at scale, human validation remains critical to ensure accuracy and contextual relevance. Discrepancies in severity ratings and unreviewed vulnerabilities highlight the need for hybrid approaches that combine AI efficiency with human expertise.

FAQs

Why are most vulnerabilities unreviewed?

Anthropic cites limited human resources for validation, leaving 21,119 candidates unreviewed by third parties.

How accurate are automated severity ratings?

90.8% of reviewed samples were confirmed as legitimate, but discrepancies exist. For example, 13 out of 27 CVEs showed overestimated severity, and one was underestimated.

What are the risks of relying on automated tools?

Automated severity ratings may misallocate resources. For instance, a Critical rating from Mythos did not align with external evaluations, emphasizing the need for human oversight.