Ignoring Critical Logs: How Companies Miss Early Signs of Security Breaches
A study shows 50% of organizations discard 86% of logs, risking security breaches due to cost management strategies.
Survey Findings
A survey of 450 senior IT professionals at large enterprises revealed that 50% of organizations intentionally discard or fail to collect 86% of their generated logs after filtering and aggregation. This practice is driven by cost management strategies, but it significantly undermines the ability to detect and analyze security incidents.
Logs serve as a detailed record of system activity, capturing errors, user actions, and application behavior. When organizations limit log retention or reduce data collection, they erode the visibility necessary for forensic investigations. Attackers often remain undetected for extended periods, and by the time a breach is discovered, critical evidence may have been purged. Security teams rely on comprehensive logs to trace the timeline of an attack, yet many organizations prioritize financial constraints over this requirement.
Impact on Security
The decision to reduce log retention is typically made by teams focused on cost control, observability, or platform engineering, rather than security. These groups operate under financial targets, leading to practices such as shortening storage periods, sampling logs, or eliminating data categories deemed redundant.
The survey found that two-thirds of organizations view their log management costs as exceeding the value they derive, with annual expenses for logging tools at large enterprises averaging $2.5 million. This expenditure accounts for nearly half of the budget allocated to observability and monitoring.
AI Workloads and Log Volume
The rise of AI workloads has intensified these challenges. Organizations running AI systems report a sharp increase in log and telemetry volume, further straining resources. The additional data creates higher ingestion, storage, and query costs, which exacerbates the pressure to cut expenses. This trend not only worsens the financial burden but also reduces the depth of visibility into AI operations. AI systems often produce unpredictable outcomes, and diagnosing their behavior requires detailed records of inputs, interactions, and service calls. Many enterprises already report that their logs capture only partial activity within AI applications, and further data reduction limits their ability to investigate anomalies.
AI Agents in Log Processing
A separate finding from the research raises concerns about the role of AI agents in log processing. These agents both generate and analyze logs, creating a shared data layer between software and human operators. However, this dynamic introduces risks: tampered or manipulated log entries could mislead automated systems, leading to incorrect responses. While the study notes training data poisoning as a concern for a minority of respondents, it stops short of directly linking this risk to log integrity. This gap highlights a potential oversight for security teams to address.
Governance and Misalignment
The core issue lies in governance. Log management decisions are often made by teams with distinct objectives from security teams, creating a misalignment in priorities. Security leaders must verify what data is being collected, what is being discarded, and how long retained logs are stored. The survey suggests that many enterprises lack the visibility to confirm whether their telemetry meets investigative needs.
Conclusion: Balancing Efficiency and Security
The findings underscore a broader tension between operational efficiency and security resilience. As organizations continue to trim costs, the risk of undetected breaches grows, particularly in environments where AI and complex systems generate vast amounts of data. Without a coordinated approach to log management, the ability to respond to threats may become increasingly compromised.
About Author
English