How to Benchmark Deepfake Detection Systems in 2026: 25 Tests Across Injection, Replay, Face-Swap, and Voice Attacks

A deepfake detection benchmark should test how an identity system fails, not whether it can identify 3 polished demo clips. For digital KYC, cover capture provenance, physical attacks, face manipulation, synthetic voices, and genuine customers.

This guide defines 20 attack tests and 5 genuine-user controls across 5 families. The matrix is an implementation template for banks and fintechs, not an industry standard, certification, universal pass threshold, or published Signzy benchmark.

NIST SP 800-63A-4 requires remote identity-proofing providers to test automated media analysis against available forged and manipulated artifacts and genuine media, establish false-positive and false-negative baselines, and document the tested artifact types and corresponding false-negative rates.

How Are Injection, Replay, Face-Swap, and Voice Attacks Different?

The same media can represent different attacks depending on where it enters the KYC session.

A printed-photo PAD pass does not prove virtual-camera detection. A face-swap pass does not prove genuine-sensor provenance.

What Should a Deepfake Detection Benchmark Prove?

A useful benchmark should prove 4 things: the system detects relevant attacks, accepts representative genuine users, keeps working under production media conditions, and returns evidence that can support an operational decision.

Evaluate the complete session, not 1 model. A classifier may detect manipulation while capture misses a virtual camera. PAD may stop a print while digital injection bypasses the sensor. Face match may still match a manipulated face to a stolen portrait.

The benchmark should therefore retain 6 outputs for every attempt:

1. Component result: capture integrity, PAD, deepfake, voice, and face-match decisions.

2. Continuous score: the score before policy converts it into pass, retry, review, or reject.

3. Operating threshold: the exact threshold and policy version used.

4. Reason code: suspected injection, replay, face swap, synthetic speech, quality failure, or another defined cause.

5. System state: model version, SDK version, device, operating system, codec, and network condition.

6. Operational result: latency, timeout, retry, manual review, and final KYC action.

Without these fields, 2 vendors can both report a pass while detecting different parts of the same attack.

Which 25 Tests Belong in a Deepfake Detection Benchmark?

The 25-test matrix uses 20 adversarial tests and 5 bona fide controls. Each family answers a different question and needs a different primary metric.

This separation follows the risk boundary documented by NIST. Its PAD evaluation distinguishes physical artifacts at the sensor from digital images or videos injected through virtual-camera software, which were explicitly outside the PAD evaluation scope.

How Should 5 Capture-Path Injection Tests Be Run?

Injection testing asks where the media came from. It should run before image quality, liveness, or facial similarity is allowed to influence the final decision.

Run these tests only in an authorized evaluation environment, never in a production KYC flow.

NIST identifies virtual cameras, device emulators, jailbroken devices, capture-sensor authentication, device attestation, and protected channels as relevant controls. Signzy's injection attack detection guide explains why capture provenance must be tested separately from facial liveness.

How Should 5 Replay and Presentation Attack Tests Be Run?

Replay and presentation attacks place an artifact in front of the expected camera. They belong in the PAD branch because the attack reaches the biometric sensor as a physical presentation.

NIST FATE PAD evaluated 82 passive, software-based face PAD algorithms on conventional 2D imagery of multiple presentation attack instruments. The result is not a reason to pool all artifacts. APCER should be reported per relevant presentation attack instrument species, with BPCER measured on bona fide attempts.

A PAD certificate should not be treated as an injection result. ISO/IEC 30107-3 states that its attacks occur at the biometric capture device, and NIST's PAD evaluation plan places digital injection outside its evaluation scope.

How Should 5 Face-Swap and Synthetic-Media Tests Be Run?

Face-manipulation tests should vary both the manipulation method and its delivery path. A detector that catches an offline face swap may miss a real-time swap after compression or a fully generated identity.

NIST OpenMFC separates image and video manipulation tasks from deepfake detection tasks and uses original high-provenance media, manipulated media, and ground truth. A bank benchmark can apply the same discipline: preserve the original, transformation record, expected label, and exact test file hash.

At least 1 generator family should remain hidden from the vendor until final evaluation. A model can perform well on familiar artifacts while failing on a new or previously unseen generation method.

How Should 5 Voice and Multimodal Tests Be Run?

Voice testing matters whenever speech, a call-center interaction, or an attended video session influences identity, recovery, or transaction approval. A visual-only pass should not clear synthetic audio.

The ASVspoof 5 evaluation plan evaluates stand-alone speech deepfake detection and spoofing-robust speaker verification. It includes text-to-speech and voice-conversion attacks, varied technical quality, and previously unseen methods. For a KYC pilot, report the deployed TPR/FPR operating point; use minDCF, actDCF, Cllr, and EER only when the voice subsystem and evaluation design support those established metrics.

Why Do 5 Bona Fide Controls Belong in an Attack Benchmark?

An attack detector can appear strong by flagging nearly everything. The 5 bona fide tests expose that failure and show whether production conditions create false alarms, retries, or timeouts.

NIST requires testing in conditions substantially similar to the operational environment and user base. Keep quality failures separate from attack detections. A blurry genuine session should not become evidence that an attack was found.

How Many Attempts Should Be Run for Each of the 25 Tests?

No single sample count makes a benchmark conclusive. Use 3 stages and show the empirical resolution created by each sample size.

An observed 0/100 miss rate does not prove zero risk. Increase samples for high-consequence attacks, calculate confidence intervals, and repeat across devices, generators, codecs, and model versions. Freeze the final evaluation set before vendors see results.

Which Metrics and Reporting Fields Make 2 Vendors Comparable?

Use the same corpus, policy, and output schema for every vendor. Do not compare one vendor's TPR at a permissive FPR with another vendor's TPR at a strict FPR.

Run 2 threshold views: the vendor's default and a policy-matched operating point. Keep raw scores so the bank can inspect the trade-off instead of accepting a vendor-selected pass rate.

Which Benchmark Failures Should Block a Vendor Decision?

A high average should not cancel a decisive control failure. Use family-level gates before any weighted scorecard.

The final scorecard can compare coverage, errors, latency, integration, review evidence, and cost, but each critical family should pass its own institution-approved gate first.

How Can Signzy Be Evaluated With the 25-Test Benchmark?

The same matrix should be applied to Signzy without removing difficult tests. Signzy's owned deepfake page states that its API covers manipulated image, video, and voice recordings; image and video injection; synthetic content; face swaps; active and passive liveness; selfie verification; and face match.

That scope allows the pilot to inspect 4 connected outputs instead of treating them as interchangeable:

1. Did capture integrity identify virtual cameras, emulators, substituted media, or reused sessions?

2. Did PAD detect print, display, replay, cutout, and mask presentation attacks?

3. Did deepfake analysis detect face swaps, reenactment, synthetic faces, and synthetic voices?

4. Did face match and decision policy preserve scores, reasons, and hard-stop evidence?

Use Signzy's layered liveness guidance to define the PAD/injection boundary, then test its biometric verification workflow with representative US traffic. Signzy's breadth supports 1 connected pilot across capture integrity, PAD, manipulated media, voice, and identity binding. Treat its published 1B+, 2x, and <5s figures as operating context, then compare results by test ID, threshold, model version, device, and attack family.