Adam Maruyama, Field CISO at Garrison, warns of the dangers of AI phishing and how AI may actually be a threat to itself.
Phishing emails are already a nuisance and a risk for enterprise systems. Anyone who knows the pain of having to weed through a trove of suspicious vendor emails or candidate resumes to see which are legitimate and which are not know how annoying they can be.
Any cybersecurity professional who has to deal with the 3-5% click rate of trained companies or even the 1-2% click rate at mature companies knows the risk associated with these clicks. At the end of the day, after all, it doesn’t matter whether an employee is new to the company or a first-time clicker: malware can have severe consequences for corporate systems and data.
The National Cyber Security Centre (NCSC)’s recent report on the impact of AI on the cybersecurity threat demonstrates that things are about to get worse, with uplifts and moderate uplifts to nation-state and cybercriminal attackers’ capabilities in reconnaissance and social engineering.
For AI phishing, the moderate uplift in reconnaissance capabilities means it can better identify and build profiles around targets in your organisation – whether they’re new executives, employees with privileged access, or simply disgruntled employees complaining about how annoying phishing tests are.
The uplift in social engineering and phishing capabilities means attackers will be able to more capably craft emails and documents that are free from the typos, translation errors, and generic content that mark most phishing emails today.
The limits of AI defence
The hopes of using AI-powered mail and file scanners to detect AI-generated content before presenting it to users are greatly overestimated. Such technologies may have severe implications for security (via false negatives) and productivity (via false positives).
Even though I never use generative AI in any portion of my writing process, anecdotal application of GPT-4 to my previously published work flagged my articles as having a greater than 60% chance of having been generated via Generative AI, and one particular article at 80%. As a reference point, I asked GPT-4 to generate an article in my voice, and that article, too, was rated at an 80% probability of being AI-generated. Of course, GPT-4 isn’t built to detect AI.
However, a recent study in the International Journal for Educational Integrity also noted the high prevalence of false positives in detecting AI-generated content.
Some may argue that further development in the realm of AI detection tools can close the gap between attackers and defenders in this area or that regulations requiring some sort of metadata indicator of AI-generated content could alleviate the issue. But even as generative AI detection models evolve, so too will the content generators – the study from the International Journal of Educational Integrity cited above also notes that detection for GPT-3.5 was more accurate than detection for GPT-4.
While regulation might help to detect AI phishing in an academic setting, cyber threat actors are already generating their own generative AI models to escape the restraints placed on content generation in commercial AI models; evading regulatory requirements for metadata and labelling could be similarly bypassed.
Adding to the complexity of AI content detection is the recent emergence of ‘conversation overflow’ attacks, which use a mixture of traditional malicious phishing content and AI-generated conversational content to bypass AI-driven phishing detection algorithms.
Attacks like this illustrate the difficulty that mixed content and content that may or may not be risky pose to any algorithm – AI or traditional – faced with a binary decision between ‘block’ and ‘allow’. A false positive for risky content could put the business at risk by blocking legitimate and time-sensitive content from reaching the intended recipient; a false negative could put the business at risk by allowing a malicious attack that could steal sensitive data or cripple critical IT systems.
These data points paint a grim picture of using AI as a defence against AI phishing. Assuming current trends continue, the high false positive rate alone could significantly degrade business outcomes. AI detection algorithms constantly flag legitimate business emails as AI-generated for faults like being well-structured and using data to back up their points – both of which GPT-4 flagged as reasons for thinking my previous articles may have been written using Generative AI.
Enhancing Zero Trust to protect against AI phishing
One solution to AI phishing attacks can be found in extending the principles of Zero Trust to the one application that needs it most: the browser. Most phishing exploits – whether technical or credential harvesting – occur after a target clicks on a link and it opens within the web browser.
The underlying trust issue is, of course, that Chrome doesn’t know whether the site a user is opening needs and should be trusted to have the system-level privileges that are required for Zoom or Office 365 to run on your system or whether it’s a weather or news site with absolutely no need to access system files and services.
The answer isn’t using AI to make better binary block versus allow decisions – it’s turning that dichotomy on its side by creating a third ‘sanitise by default’ option that allows users to view and interact with content in potentially malicious environments with prompts that alert them to the risk and without processing risky code on corporate systems.
By using technology like remote browser isolation, which pushes code processing off corporate systems and into a separate environment for the vast majority of websites, cybersecurity leaders and systems administrators can effectively apply the ‘principle of least privilege’ to the internet by ensuring that only websites reviewed and certified for native processing have the privileges needed to run code on the endpoint.
All other websites can then be ‘sanitised by default’ so employees can click with confidence, knowing that even if they click on a malicious link, they’ll be protected from any technical exploitation.
