Deepfakes and hostile AI agents rarely break your models. They exploit the moment your workflow decides to trust something. The defence treats every submitted voice, face and document as unverified evidence, scores it against independent signals like device history and behaviour, and keeps a human on the contested cases. Cheap generation shifts your false-positive budget, so the whole scoring stack has to move with it.
The economics changed first. A convincing synthetic face or a cloned voice used to take a skilled operator and time. Now it takes a subscription and a reference clip. That does not make every attacker sophisticated, but it does mean the floor rose: the mediocre fraud attempt now looks like what a good one looked like two years ago. If your controls were calibrated against the old floor, they are already miscalibrated.
The failure modes worth naming
Generated media attacks the enrolment step, where you are trying to bind a real person or business to an account. The two variants we see most:
- Injected liveness feeds. The attacker does not hold a phone up to a screen. They inject a rendered video stream directly into the capture pipeline through a virtual camera or a patched SDK, so the “selfie” your liveness check scores was never in front of a lens. Presentation-attack detection tuned for printouts and replayed screens does not catch an injected feed, because there is no presentation.
- Voice cloning against callback and payment authorisation. A cloned voice defeats a phone-based step-up, or authorises a payment change in a treasury workflow where voice was the second factor. The clone does not need to be perfect. It needs to be good enough for a reviewer under time pressure at quarter-end.
Document forgery moved in parallel. Generated identity documents and fabricated financials no longer carry the tells that template-matching detectors relied on. A synthetic bank statement can be internally consistent, with balances that reconcile and a plausible transaction history, because the model that made it optimises for exactly that. Pixel-level forgery detection still helps, but a document that is generated whole rather than edited will not show splicing artefacts.
Then there is the newer class: your own agents, and the attacker’s. Agentic abuse takes two shapes. In one, an attacker points a scripted agent at your onboarding or dispute flow and runs it at machine speed and scale, probing for the input that gets an approval. In the other, the attacker plants instructions in content your agent will read, so your agent acts against you. That second shape overlaps heavily with prompt injection and is worth treating as the same problem wearing a different hat.
Guardrails we actually build
There is no single detector that survives contact with a motivated attacker, so we build the check as a scoring problem with independent signals, and we make the tools downstream of a decision least-privilege by default.
For media, injection resistance matters more than raw deepfake classification accuracy. That means capturing at a layer the attacker cannot cleanly substitute, checking hardware attestation where the platform allows it, and binding the capture to a device and session you have seen behave normally over time. A liveness score that arrives from a device with no history, on a network associated with prior attempts, is not a liveness score you weight the same as one from an aged device. The face check and the environment around it are separate evidence, and entity resolution across those signals is where most real detections come from.
For documents, we lean on provenance and cross-reference rather than the document alone. A statement gets checked against the issuing institution where an open-banking or verification API exists, so you are not grading a PDF, you are confirming a fact. Where that is not possible, the document is scored for internal consistency and reconciled against everything else the applicant has submitted, and mismatches feed the case rather than auto-declining it. The point-in-time correctness discipline matters here too: a document dated inside a window that could not have produced those figures is a signal, and you only catch it if your pipeline has no lookahead in how it validates dates.
For agentic abuse, the controls split cleanly:
- Against attacker agents, treat velocity and session shape as first-class fraud signals. A flow being driven by a script has a rhythm a human does not, and rate limiting, behavioural analytics and step-ups aimed at the contested middle raise the cost of running the attack at scale without punishing ordinary users.
- Against subverted own-agents, keep the agent’s tools scoped so that reading untrusted content and taking a consequential action are never the same permission. An agent that reads a dispute email should not also be able to release a refund. Structured output with schema validation on every tool call, plus an audit trail of what the agent saw and what it triggered, is what lets you reconstruct an incident and prove the boundary held.
The unifying stance is that you never let a single model’s confidence become the decision. Media, document, device, network and behaviour are separate views of the same claim, and the fraud call is the reconciliation of those views, with a false-positive budget you set deliberately rather than inherit from a vendor’s default threshold.
Keeping the system honest as attacks drift
Detection built against today’s generators decays, because the generators improve on a faster cycle than most fraud models get retrained. So the eval set is the product. We keep a labelled set of known synthetic and known genuine cases, refresh it with new attack samples as they appear, and measure drift in both directions: detections falling off, and false positives creeping up as legitimate users adopt the same tools attackers use. A liveness model that was fine in January can quietly stop catching the current injection technique by March, and you only see it if you are scoring against fresh adversarial examples rather than the training distribution.
Two operational habits keep this from rotting. Log the full lineage of every decision, so a reviewer or an auditor can see which signals fired and how they combined, not just the final score. And put the human review where the money is: on the contested cases the stack cannot separate, with everything it knows in front of them, while the clear approvals and clear declines flow straight through. The goal is not to eliminate synthetic media from the pipeline. It is to make sure that when a good fake gets through the first check, four other things it could not fake are already disagreeing with it.
FAQ
Can liveness detection alone stop deepfake onboarding fraud?
No. A determined attacker who defeats one presentation-attack check still has to match the injected feed to a document, a device, and a network history. Liveness is one signal in a scoring stack, not a gate you pass or fail.
How do we tell an AI agent from a human customer without adding friction for real users?
You mostly do not try to classify humanness directly. You look at pacing, session shape, device consistency and whether the entity can produce out-of-band artefacts a scripted agent cannot forge cheaply. Most legitimate users never see the extra step.
Where should the human review sit once forgeries get this good?
On the cases the scoring stack flags as contested, not on every application. Route the ambiguous middle to reviewers with the full lineage in front of them, and keep straight-through processing for the clear ends.