Phishing has been the leading cause of breaches for decades. Better filters, authentication, and training have helped and still do. AI is making that work exponentially harder.
Phishing is not a solved problem, and it never was. Year after year, industry reports put social engineering at or near the top of the list of initial-access methods in real breaches. The numbers move a little, the details change, but the ranking doesn't.
Over the last two decades, the industry has done real work against it. Email authentication — SPF, DKIM, DMARC — cut off a large category of spoofing. Secure email gateways learned to filter obvious campaigns. Multi-factor authentication reduced the value of a stolen password. Awareness programs and simulations lowered click rates in organizations that ran them consistently. None of it solved phishing. All of it reduced the risk.
AI is now eroding those gains faster than the existing defenses can compensate.
What AI Actually Changes
AI does not invent a new vulnerability. The cognitive tendencies that make phishing work — trust in familiar names, response to urgency, deference to authority — are the same as they've always been. What AI changes is the cost of exploiting them.
Three shifts matter most.
Scale. Crafting a convincing, personalized message used to take a skilled operator hours of research per target. A language model does the same work for fractions of a cent and produces thousands of uniquely tailored messages in parallel. The personalization that used to define spear-phishing is now available at bulk-phishing cost.
Medium. Text was the whole game for a long time. Synthesized voice is now indistinguishable from the real thing with seconds of sample audio. Video synthesis is close behind. In 2024, a finance employee at Arup authorized roughly $25 million in transfers after a video call in which every participant — including the CFO — was synthesized. That attack used to require a specialist. Now it does not.
Persistence. A human operator moves on when an attempt fails. An AI agent can maintain a relationship with a target across weeks or months, remembering every prior exchange, adapting consistently, never showing the small inconsistencies that used to expose human attackers. The attack becomes a patient, ongoing conversation rather than a single event.
Any one of these on its own is a meaningful shift. All three at once is a different category of threat than the defense model was built for.
Where the Current Defenses Start to Slip
The two pillars of enterprise phishing defense are gateway filtering and awareness training. Both are built on assumptions AI is eroding.
Gateway filtering depends on shared patterns. The economics of traditional phishing pushed attackers to reuse infrastructure — one template, thousands of inboxes, shared sender reputations, repeated URL structures. A single detection updated the filter for everyone. AI removes the pressure to reuse. Every message can now be unique, every sender domain freshly warmed up, every pretext tailored. There is less shared signal to detect because there is less shared anything.
Awareness training depends on learnable tells. Employees learn to spot generic phishing by the tells: unexpected attachment, off-brand tone, clumsy grammar, unfamiliar domain, misplaced urgency. AI-generated phishing has fewer of those tells. The grammar is clean. The tone matches the sender's prior emails. The urgency is calibrated to the recipient. The follow-up call is in the right voice.
None of this means filters and training stop working. They still catch a large volume of low-effort attacks, and that matters — the attacker population is broad, and most of it is not yet operating at the frontier. But the gap between what defenses catch and what a well-resourced AI-enabled attacker can produce is widening. The slope of the curve is what changed.
There is a secondary effect worth naming, because it doesn't get discussed enough.
A 2025 EU audit of ChatGPT conversation records found that roughly 60% contained personally identifiable information. A parallel corporate study reported that 77% of employees have pasted company data into AI chatbots — source code, internal communications, financial projections, customer records. The reason is not malice. The chatbot is the fastest path to getting the work done. Using it well requires context. Sharing context creates exposure.
Where that context ends up varies by provider and by the data handling employees click through without reading. But the direction is clear. The same category of tool employees use to be more productive is the category of tool attackers use to be more convincing. Some of the reconnaissance work the attacker used to have to do, the organization is now doing for them.
What Practitioners Can Do Today
There is no single defense that makes this problem go away. The realistic goal is a layered posture that slows the curve, reduces the blast radius when something lands, and buys time for the industry to catch up on the parts that don't have good answers yet.
Four layers do most of the work.
Reduce the context you leak. The less internal detail the attacker's AI has to work with, the less convincing its messages are. That means inventorying which AI tools your employees actually use, putting DLP controls on chatbot traffic where it matters most, and offering sanctioned AI tools with real data-handling protections so there's a good option that doesn't require routing around the policy. It also means regular audits of what organizational context is publicly discoverable through LinkedIn, job postings, press coverage, and vendor case studies. Most organizations are more exposed than they realize until they look.
Verify across channels the attacker doesn't control. Single-channel verification — a reply to an email, a call to the number in the email, a video confirmation on the same platform the request came in on — does not hold against a synthesizer. Verification needs to cross into a channel the attacker has to separately compromise. A callback to a number from the corporate directory, not the one in the message. A confirmation through a second system that requires an independent login. A pre-shared phrase for high-stakes actions. A mandatory delay on wire transfers above a threshold. Not for every email. For the small number of workflows where a single compromised person can cause material loss.
Watch for slow-burn anomalies, not just single events. AI agents that build rapport over weeks produce different signals than traditional phishing. Low-volume, patient, consistent outreach. Gradual shifts in who an account talks to and about what. Slow drift in access patterns rather than sudden spikes. These show up in relationship graphs and long-window behavioral analytics, not in single-event rules. If your detection stack is tuned entirely for bursty, short-horizon activity, the patient attacks will slip past the baseline before anything looks wrong.
Close the incentive gap that feeds the problem. As a cybersecurity architect focused on AI governance, this is the layer I see organizations skip most often and regret most quickly. Employees paste sensitive data into unsanctioned AI tools because the sanctioned alternative is slower, weaker, or doesn't exist. They skip verification steps because the verification is designed as an obstacle to the work rather than a part of it. The fastest path is the insecure path. That gap is where the attacker lives. Closing it is harder than buying a product, which is exactly why it matters.
None of these layers solves phishing. They reduce the rate at which AI-augmented attacks succeed, and they limit the damage when one does. That is the realistic goal right now.
Being Honest About the Gap
AI security is not a one-to-one mapping with traditional security. It is non-deterministic. It has more unanswered questions. Saying so is not defeatism — it is the starting point for making good decisions.
The honest position on AI-augmented phishing looks like this. The industry has real tools against traditional phishing, and those tools still work at the low end of the attacker distribution. At the high end — a well-resourced adversary using current-generation models for personalization, synthesized media for verification bypass, and persistent agents for sustained rapport — the current defense model is outmatched. The gap is widening as model capability improves, and the defensive response is mostly linear.
That gap will not be closed by one product or one policy. It will be closed by a combination of better tools (email platforms starting to detect AI-generated content, identity providers adding liveness checks that resist current deepfakes, endpoint DLP extending to chatbot traffic) and by structural changes in how organizations handle verification, information flow, and the incentive to route around security. Both sides of that have to happen. Neither is finished.
In the meantime, the practitioner's job is to operate clearly inside a problem that doesn't have a clean solution yet. That means running the defenses that still work, adding the layers that address what AI specifically changed, and being honest internally about where the residual risk lives. The organizations that handle this well will not be the ones with the fanciest tools. They will be the ones who stopped pretending the old playbook was enough and started adapting — layer by layer — to the one that is coming.
Subscribe to the newsletter for bi-weekly analysis — substack.com/@adversarialminds
Steve Brodson is a cybersecurity architect focused on AI safety and security. He experiments with AI systems, consults with organizations navigating AI risk, and teaches practitioners how to think clearly about threats that don't fit traditional security frameworks. Connect on LinkedIn, X, or at brodson.com.