AI Safety Checks Can Fail: Secrets Leaked Despite Compliance

Post Views: 3

An AI agent can pass every safety check and still leak secrets. A pull request includes a concise bug report in its description. A bot processes it before human review, extracts shell commands, gains approval, and shares the results. The maintainer reviews the interaction the following day.

The AI Agent’s Architecture

An agent consists of a model and a harness. The model generates intent, while the harness translates that intent into actions such as shell commands, file reads, API calls, and network requests. It also manages approval logic, tool permissions, path restrictions, and output handling. When workflows operate without human oversight, the harness functions as the security boundary.

Vulnerabilities in the Harness

Elad Meged, a founding engineer at Novee Security, tested this scenario against three vendors’ repositories using their default configurations. Anthropic’s pipeline exposed sensitive data. Organizations utilizing these agents in their default setups face identical risks. Meged develops AI agents for automated penetration testing at Novee and applied offensive techniques to evaluate the agents themselves.

Prompt Injection and Trust Decisions

In all cases, prompt injection served as the delivery method, but the core vulnerabilities lay in how the harness made trust decisions and how those decisions interacted across stages. A command is approved because it appears safe. Its output is published by default. Neither decision is incorrect in isolation, but together they create an exfiltration chain.

“Monitoring only the prompt layer misses the critical handoff where one safe decision enables the next,” Meged explained.

Patches and Their Limitations

Each patch introduced a new boundary. Meged reported findings against Claude Code Action, the default workflow for anthropics/claude-code. Millions of users install the package. Each reported issue prompted a patch, but the fixes shifted the definition of “safe,” revealing where to focus next. Anthropic awarded bounties for each round. While the patches addressed specific vulnerabilities, the attacks grew harder to detect over time.

Exfiltration Without Detection

By the final round, secrets were exfiltrated through a channel that bypassed all prior fixes: no outbound connections, no writes, no logs. Anthropic compensated for each finding, but the bounty model treated each issue as isolated. This approach sends a misleading signal, implying closure where underlying risks persist.

Structural Fixes and Industry-Wide Solutions

Google’s advisory framed the issue as an update to the Trust Model, highlighting the structural nature of the problem. Google’s Gemini CLI, hosted in a repository with over 100,000 stars, demonstrated a kill chain using the security configuration recommended for CI workflows handling untrusted input. A configured restriction failed to enforce at execution.

Shared Architectural Assumptions

OpenAI’s sandbox protects known paths, but multi-stage workflows sharing a workspace allow state from one stage to influence the next. The protected path list relies on assumptions about what requires protection. Vendors have implemented defenses: Anthropic added pipeline checks, Google introduced execution modes with environment sanitization, and OpenAI created a sandbox with protected paths. However, these measures fail at critical handoffs.

“A structural fix would stop trusting labels and re-validate trust at the point of consumption, not just at decision points,” Meged said.

Conclusion

Addressing this requires industry-wide dialogue, not isolated efforts. Meged will present code-level analysis and live demonstrations at Black Hat USA 2026. Teams running these agents should trace every path where agent output or influenced state is consumed by subsequent stages with different privileges. Identify where the harness deems something safe and ask: what happens to that output next? Is it published? Loaded as configuration? Passed to a tool with broader access than initially assumed?