How Scammers Bypass AI-Powered Email Filters with Hidden Text

Post Views: 6

Cyberattackers Bypass AI-Powered Email Filtering Systems

Cyberattackers have developed a novel tactic to evade AI-powered email filtering systems, using hidden text to manipulate machine learning algorithms and increase the likelihood of phishing scams reaching inboxes.

The “Indirect Prompt Injection” Method

This approach involves embedding benign content, such as snippets of romance novels or archived newsletter copy, alongside malicious links to dilute the signal and cause the filter to misclassify the email.

The “Adidas Newsletter Clone” campaign disguises phishing emails as legitimate communications from the sportswear company by incorporating actual Adidas newsletters.
Another campaign embeds a fictional story from a romance novel platform to create a sense of legitimacy for their health insurance-themed phishing scam.

The Limitations of Current AI Security Measures

These tactics exploit the limitations of current AI security measures, which rely on surface-level analysis of links and keywords. By injecting hidden text, attackers aim to influence the decision-making process of the machine learning models guarding mailboxes, rather than attempting to deceive human users directly.

According to researchers, the use of indirect prompt injection represents a significant threat, particularly as more organizations transition towards “agentic mailboxes” where AI assistants take actions on behalf of users.

The potential consequences of a compromised AI system following a malicious instruction are severe, highlighting the need for improved AI security tools that can understand the full context of messages being analyzed.

Enhancing AI Security

Researchers emphasize the importance of enhancing the underlying mechanism of these models to comprehend the nuances of language and prevent manipulation by hidden text.

As threat actors continue to evolve and refine their tactics, it is essential for security professionals to stay vigilant and adapt their strategies to counter emerging threats.