Why Code Review Is Still Essential for Identifying Bugs

NOVA, an automated system, identified 14,090 vulnerabilities across 3,915 open-source projects, highlighting new challenges in vulnerability management.

Introduction of NOVA

Code review once served as the primary method for identifying software flaws. A newly developed automated system, NOVA, analyzed the source code of 3,915 open-source projects over a two-month period and identified 14,090 vulnerabilities, each validated through a rigorous verification process. Researchers at Palo Alto Networks’ Unit 42 created the system and cross-referenced its findings with publicly documented issues. Only 85 of the discoveries aligned with existing records, and most of those had been published two to eight weeks after NOVA detected them.

Comparison with Existing Tools

Automated tools like Google’s OSS-Fuzz, launched in 2016, have long identified flaws through systematic testing, uncovering over 10,000 issues across 1,000 projects by August 2023. However, the types of vulnerabilities NOVA uncovered differ significantly. Just 8% of the findings fit categories traditionally addressed by fuzzing techniques, which focus on crashes. The remaining 92% involved issues that do not trigger immediate system failures.

Type of Vulnerabilities

Fuzzers excel at detecting memory corruption, integer overflows, and null dereferences but struggle with vulnerabilities that allow software to continue operating despite flaws. In Unit 42’s classification of frontier AI-related findings, memory and calculation errors accounted for 557 cases, or 4.0% of the total. Adding resource-management and denial-of-service issues brings this figure to 1,121 findings, still only 8.0%, according to Xu Zou, SVP of Cloud Delivered Security Services at Palo Alto Networks.

Supply-Chain Risks

The study also revealed significant supply-chain risks. Among 5,421 findings related to package ecosystems, 1,280 involved flaws directly in dependency packages, while 4,141 represented downstream exposures where applications relied on vulnerable dependencies. This ratio underscores a critical challenge: a single dependency flaw can lead to over 4,000 downstream exposures.

Vulnerability Lifecycle

Software composition analysis tools typically flag vulnerable versions, but this study emphasizes the importance of confirming exploitability in real-world contexts. The research highlights a fundamental shift in the vulnerability lifecycle. Zou noted that the traditional patch deployment window has drastically shortened, with an industry average of 55 days for implementing fixes. Attackers no longer require cutting-edge techniques to exploit vulnerabilities; once a patch is released, the difference between the old and new code versions narrows the attack surface, making it easier to reverse-engineer exploits.

Distribution of Findings

The findings challenge conventional approaches to vulnerability management, emphasizing the need for advanced detection methods and proactive mitigation strategies in modern software development. Despite the scale of findings, Unit 42 has not publicly disclosed specific affected projects, CVE identifiers, or reporting timelines. The report also reveals uneven distribution of vulnerabilities, with four of six ecosystem groups contributing over half of the findings despite analyzing fewer than 100 projects collectively.

Severity Assessments

Severity assessments vary depending on the scoring framework: over 25% of findings received high or critical ratings under CVSS 3.1, while nearly 40% fell into those categories under CVSS 4.0. The discrepancy underscores the evolving nature of risk evaluation.

About Author

Anuj

See author's posts