Adversarial AI

Adversarial AI: When Malice Enters

Welcome. Every failure we've looked at so far has been an accident—a model getting confused or a user making a mistake. Now, we're adding a new element: malice. You'll learn to distinguish between attacks targeting AI systems and attacks carried out using AI. Most importantly, we'll look at why the old ways of spotting fraud are becoming obsolete as AI raises the volume and the quality of social engineering at the same time.

The Analogy: The Guard Dog

Think of an AI system like a guard dog. There are two ways a criminal can deal with it. First, they can attack the dog itself: they might try to poison it or trick it into letting them through the gate. Second, the criminal can bring a dog of their own to help them commit the crime. These are two different problems that people constantly merge into one. While a dog is one animal in one place, an AI system can be approached at a thousand points at once, and an attacker's AI doesn't get tired.

Takeaways: The Guard Dog

Let's recap. We've moved from accidents to intent. Remember the two directions: you can target the guard dog, or bring your own. Understanding this distinction is the first step in identifying adversarial exposure.

Direction 1: Attacks ON AI Systems

This is the first direction: manipulating the system itself. We're only going to name these so you recognize the categories. Prompt Injection involves manipulating a system's instructions through content it reads. Data Poisoning is corrupting the training data to influence how the model behaves later. And Information Extraction is tricking a model into revealing confidential data it should keep hidden.

Direction 2: Attacks USING AI

This is where the risk landscape shifts for almost every business. Attackers are using AI to manufacture authenticity at scale. In the past, there was a trade-off: an attacker could send ten thousand low-quality phishing emails—high volume, low quality—or they could spend weeks researching one person for a targeted sting—low volume, high quality. AI collapses that trade-off. Now, they can produce high-quality, perfectly phrased, personalized attacks for ten thousand people at once.

The End of "The Tell"

For years, the standard advice for staff was to look for 'tells': poor spelling, odd phrasing, or generic greetings. That advice is now obsolete. AI produces perfect prose. If you're relying on a language barrier to protect your firm, you're relying on a defense that has already vanished.

Takeaways: Attack Vectors

We've seen how the strategic landscape has shifted. The volume is up, and the quality is perfect. This means we can no longer trust the 'shape' of a message to prove it's real.

Practical Scenario: The Manufactured Meeting

In May 2024, the engineering firm Arup fell victim to a $25 million deepfake scam (CNN Business, 2024). A finance employee in their Hong Kong office joined a video call that appeared to include the group’s Chief Financial Officer and several other familiar colleagues. They discussed a 'confidential transfer.' Every other participant on that call was an AI-generated fake, built from public footage of real executives. The employee saw their faces and heard their voices. Trusting what they saw and heard, they made fifteen transfers totaling roughly $25 million. The fraud was only discovered later during a routine follow-up with the actual head office.

Verify the Request

You're on a video call with two colleagues you recognize, and they ask you to make an urgent confidential payment. What actually verifies this request? Click the correct verification method. Not quite. A familiar face and voice can now be manufactured from public footage. Relying on what you see inside the call is exactly how the Arup fraud succeeded. Exactly. Nothing inside the conversation can prove the conversation is real. The only check that works is leaving the channel—like calling a known number on a different phone.

The New Authentication Heuristic

Since authenticity can now be manufactured, you must shift your mental model from recognition to out-of-band verification. First, assume the 'tells' are gone. Perfect writing is no longer evidence of safety. Second, verify outside the channel. If it's a video call, use a known phone number. Finally, treat voice and video as data, not identity. We can no longer trust a face without a second factor.

Common Pitfalls

Watch out for two common traps. The Expertise Trap is thinking 'my team is too smart to be fooled.' Expertise doesn't protect you from a perfect imitation of your boss's voice. Then there's over-reliance on visuals—assuming that 'glitches' mean it's fake, or 'smoothness' means it's real. Quality varies; it's the process that matters.

Summary & Final Word

We've covered the shift from accidental failure to malicious attack. Remember: there's a difference between attacking an AI and using AI to attack people. That is deliberately as far as we go. How these attacks are actually built, and what they mean for cyber underwriting, is a whole elective, and if that is your line it is waiting for you. What every insurance professional needs is this: the tells are gone, and the volume is up.