AI Security Scanner: Open Source Tool for AI Systems

Post Views: 10

Tencent’s Zhuque Lab has developed AI-Infra-Guard, an open-source tool designed to evaluate the security of AI systems.

Key Features and Functionality

The platform identifies active services such as Ollama, vLLM, and ComfyUI, cross-referencing them against a database of over 1,600 known vulnerabilities. It also analyzes MCP servers and agent capabilities across 14 risk categories while conducting jailbreak tests on target models. To assess potential threats, the system employs a language model to determine if a skill exhibits malicious characteristics. This evaluation is measured against SkillTrustBench, a publicly available dataset containing 5,520 human-labeled examples across nine risk classifications.

False Positives and Accuracy

The false positive rate varies between 1.20% and 18.67%, depending on the model used for analysis. The accuracy of results is influenced by the model’s ability to interpret and evaluate the data. In some cases, approximately 1.25% of legitimate skills may be incorrectly flagged, while up to 20% of alerts could be false positives requiring manual verification.

Users and Applications

The tool is utilized by financial institutions, telecommunications providers, and technology companies, including ICBC, China Merchants Bank, China Telecom, Lenovo, vivo, and Bilibili.

Technical Structure

Zhuque Lab has structured the platform into two distinct layers, with only one involving interpretive analysis. The CVE version-matching process is described as purely technical, with false positive rates directly tied to the precision of service identification. The scanner examines files created by attackers, as well as tool descriptions and skill files controlled by hostile servers.

Indirect Prompt Injection and Mitigations

A specific technique known as indirect prompt injection exploits this by embedding instructions within content that a model processes, aiming to manipulate the model’s behavior. Version 4.1.9 of the tool introduced enhancements to its scanning agents, mitigating this risk. The updated architecture isolates file and tool content into a dedicated text block within prompts, structurally separated from direct commands. Scanning agents are instructed to treat this block as data for analysis rather than executable instructions.

Security Limitations and Recommendations

While this reduces the likelihood of successful injection attacks, the team acknowledges it is a partial solution rather than a definitive fix. They emphasize that no LLM-driven system is entirely impervious to such threats. Organizations using the tool should not interpret a clean scan result as absolute confirmation of security.

The open-source version of AI-Infra-Guard lacks authentication mechanisms, a limitation highlighted by the project’s documentation.

The team acknowledges that no LLM-driven system is entirely impervious to such threats.

Availability and Conclusion

The repository explicitly advises against deploying the tool on public networks, as it is intended for internal use. The platform is designed as a single-user tool, with no built-in login or role-based access controls. To address this, Zhuque Lab recommends external access management, such as deploying a reverse proxy with basic authentication or IP-based restrictions alongside standard firewall configurations. Bypassing these measures could expose the scanner to unauthorized access, potentially granting adversaries access to API keys for all evaluated models. AI-Infra-Guard is freely available on GitHub, offering a resource for organizations seeking to evaluate their AI infrastructure. The tool’s development underscores ongoing efforts to address vulnerabilities in large language model ecosystems, combining automated scanning with human-in-the-loop validation to balance efficiency and accuracy.