Google's New Agent Security System Detects Tool Misuse, Loops, and Rogue Behavior

Post Views: 4

Google’s Agent Anomaly Detection is a reasoning-based oversight and audit layer designed for autonomous agents operating within the Gemini Enterprise Agent Platform.

Overview

Google’s Agent Anomaly Detection is a reasoning-based oversight and audit layer designed for autonomous agents operating within the Gemini Enterprise Agent Platform. Built using the Agent Development Kit (ADK) for Python 1.2 or later, the system is currently in private preview and requires ADK 2.1.0 or higher. It focuses on identifying deviations from expected agent behavior through continuous monitoring of traces generated during execution. The tool flags anomalies such as unusual intent, policy violations, and operational risks, assigning severity levels and providing clear explanations for each finding.

Key Features

OWASP Agentic Top 10 Risks Addressed

The system addresses specific risks outlined in the OWASP agentic Top 10, including tool misuse, identity and privilege abuse, agentic cascading failures, and rogue agents.

Tool Misuse: Unsafe practices like improper tool chaining, parameter manipulation, and indirect prompt injection.
Identity and Privilege Abuse: Unauthorized actions from dynamic trust delegation, persona forgery, memory escalation, and confused-deputy vulnerabilities.
Agentic Cascading Failures: Issues such as infinite loops, oscillating retries, and feedback-loop amplification.
Rogue-Agent Detection: Identification of agents abandoning roles, bypassing safeguards, or deviating from system instructions.

Additional operational risks include resource exhaustion and abnormal token usage.

Implementation Requirements

Google plans to allow users to define custom anomaly thresholds tailored to their business needs. This feature will enable users to create natural language-based anomaly detectors alongside deterministic rules, ensuring alignment with enterprise-specific guidelines. Validation of custom logic will be possible using historical traffic data.

To implement Agent Anomaly Detection, several prerequisites must be met. Logging and observability buckets must reside within the same US multi-region. OpenTelemetry tracing and logging must be enabled via the ADK, with raw telemetry capturing prompt inputs and response outputs. The enable_tracing parameter must not be explicitly set to false. Scanner service accounts require read access to logging buckets, and Log Analytics and Observability Analytics must be activated for the log bucket. Agents must maintain active telemetry data flows to verify configurations and enroll in the system. Discovered agents appear as monitored entities, but their logs are not analyzed until explicitly enabled. Agents failing to meet requirements remain unprocessed.

Anomaly Detection Layers

Anomaly detection operates in three layers to balance speed, cost, and coverage. The first layer identifies statistical outliers in traffic, flagging sessions with unusual patterns. A second, LLM-based reasoning layer examines these sessions in depth, analyzing context and intent. For example, an Inventory Agent retrieving an entire catalog through repeated tool calls with varying offsets may not trigger explicit policy violations but would be flagged as systematic scraping due to its large-batch, offset-jumping pattern. The third layer performs granular analysis of individual tool executions, examining execution states and parameter histories within conversation traces. Findings are generated asynchronously, ensuring they do not delay live operations.

Applications

Applications can access anomaly data via an API, using ADK callbacks or plugins to compare severity and probability against predefined thresholds. This enables automated actions such as blocking tool calls or halting subsequent interactions when thresholds are exceeded. The system emphasizes proactive risk management for agentic AI deployments, addressing both technical and operational vulnerabilities. Its layered approach ensures comprehensive coverage while minimizing performance overhead.

“The system addresses specific risks outlined in the OWASP agentic Top 10, including tool misuse, identity and privilege abuse, agentic cascading failures, and rogue agents.”