DeepSafe: every deepfake detector, one URL

The deepfake detection literature has a distribution problem. The models exist, the papers are good, and almost nobody outside a lab can run any of it: every research repo ships its own broken environment, its own weight files on a link that dies within a year, and its own opinion about which Python it needs. Meanwhile the fakes ship as consumer apps. DeepSafe is my answer to that asymmetry: take the best open detectors, make them run in one place, and put a single URL in front of them. The code is open source.

One process, twenty-one models

The current build runs twenty-one detection services across three modalities: seven image detectors, three audio detectors, and nine video pipelines, two of which are image detectors marched across frames. They load into a single GPU process at startup and answer as in-process function calls.

The first architecture was the obvious one, a container per detector, and it was a mistake worth documenting. A container per model means a CUDA context per model, a resident copy of PyTorch per model, and VRAM fragmentation that no scheduler fixes. Collapsing the zoo into one process traded all of that for a single context and function-call latency between gateway and model.

Getting research code from that many authors into one process is its own project, because every research repo ships a module named models, a module named utils, and a module named data, and Python’s import system permits exactly one of each. The fix is namespace isolation, with each detector’s code imported under its own prefix so the duplicate utils modules never meet.

imported = namespaced_import_multi(
    code_path=detector_dir,
    namespace="_ds_detector_07",
    import_specs=[{"module": "models.net", "attr": "Detector"}],
)

Every wrapper subclasses one BasePredictor, so a detector is a directory, a weights file, and a class with a predict method, and adding one never touches another. All the model code is vendored into the repo, and roughly 34 GB of weights live in one storage bucket, so setup is one command and the machine never touches the internet at inference time.

The other shared plumbing is preprocessing, because the expensive work is identical across detectors and should run once. Video gets decoded once into eight uniformly sampled frames plus a dense track capped at ninety; face detection runs once with SCRFD, which benchmarks several times faster than the MTCNN every research repo defaults to, and every face-hungry detector receives the same crops. Audio gets resampled once to 16 kHz mono and padded to 64,600 samples, the four-second window the ASVspoof lineage standardized.

Threads, VRAM, and shared fate

The concurrency model looks naive and is deliberate. The fan-out uses a plain Python thread pool, which works because inference spends its time inside CUDA kernels, where the interpreter lock is released; the GIL only hurts workloads that live in Python, and these do not. The payoff shows up in latency. A request costs the slowest detector’s forward pass, the max of the pool’s durations, where sequential dispatch would cost their sum, and each wrapper call carries its own timeout so one hung forward pass cannot hold the verdict hostage.

VRAM is spent as a latency budget. Every model stays resident from startup, because cold-loading a multi-gigabyte checkpoint costs seconds while a resident model answers in tens of milliseconds, and a detection product earns its keep at the second kind of latency. Half-precision residency is what makes the arithmetic close, near 20 GB for the whole zoo, with the models that misbehave in FP16 held at full precision, and one shared PyTorch allocator manages the pool, which beats twenty-one allocators fragmenting twenty-one arenas.

Shared fate is the monolith’s real price, and it is the strongest argument the container design ever had. The mitigations are boring on purpose. Every wrapper call is fenced by its own exception boundary, a model that starts failing gets marked unhealthy and its vote dropped, and the ensemble imputes the missing feature at that model’s training-set mean, so the platform returns a degraded verdict with a flag on it, never a 500. The health endpoint exists because the degraded state is normal. In a zoo this size, something is usually limping.

Every verdict is auditable after the fact. The response carries each model’s probability and latency alongside the final score, the per-model endpoints let you re-run the one detector whose vote looks strange, and the ensemble artifacts, meta-learner, scaler, and calibrator, are versioned files, so any verdict can be replayed against the exact model set that produced it. A score you cannot decompose is an opinion; a score you can decompose is a measurement.

Detectors disagree, so the ensemble is a model too

Averaging the zoo’s probabilities would assume the detectors are equally good and equally independent, and they are neither. Some watch frequency-domain artifacts that generators leave behind, some classify features from large pretrained vision backbones, some reconstruct a face and score the reconstruction error, and one specializes in mouths, because lip sync is where video fakes still stumble. Each family fails differently on different fakes.

So the aggregation layer is itself a trained model: a separate ensemble meta-learner per modality, gradient boosting for images and video and a random forest for audio, each taking the per-model probabilities as features and finished with Platt scaling so the final number behaves like a probability. The meta-learners train on a held-out pool of about 15,500 labeled files, 1,465 effective samples for image, 3,346 for audio across dozens of languages, 987 for video, and the calibration is checked, because a verdict that says 0.9 should be wrong about one time in ten.

The image ensemble reaches 0.9967 AUC on the held-out evaluation, audio reaches 0.9565, and video manages 0.7366. I publish all three numbers because the third one is the honest headline: video deepfakes remain the hard problem, and any product that claims otherwise is selling something.

Building the ensemble was mostly a firing process. One audio detector graded at 0.199 AUC, worse than a coin flip and anti-correlated with the truth, and it survived a week longer than it should have because its README was excellent. Another scored 0.367 on speech because it was built for AI-generated music, a nuance its abstract mentions once. An ensemble is a hiring decision, and every model has to earn the seat on the evaluation set, never on the paper.

The second opinion is provenance

Classifier scores are one kind of evidence. The other kind is written into the file: C2PA content credentials, generator metadata, and the invisible watermarks some generation tools embed in their output. DeepSafe runs these provenance checks as a second stage, and the merge is deliberately asymmetric, on the logic that a watermark is a confession while its absence proves nothing: provenance can push a score toward fake, never toward real.

boost = (max_provenance - 0.5) * weight  # 0.20 strong signal, 0.05 weak
final = score + boost * (1.0 - score)    # moves toward 1, never past it

A moderate signal nudges, a definitive signal overrides, and the shape of the formula guarantees the classifier’s score is the floor.

The API is the product

Detection is useless behind a conda environment, so the models sit behind a FastAPI inference server with a gateway in front for auth and rate limiting. POST /v1/detect takes a multipart file and returns one calibrated verdict. Every model also exposes its own endpoint, so a single detector can be interrogated when its vote looks strange, and GET /health reports per-model status. The web app for humans keeps its manners: drag a file in, or paste a URL, and read a number.

The training-data lineage runs through the public benchmarks the field grew up on, FaceForensics++ and the Deepfake Detection Challenge among them, and the generators keep moving. GANs made faces; this summer’s diffusion models make everything. A detector zoo is a subscription to an arms race, and the subscription does not lapse quietly.

What the zoo taught

1. Evaluate every detector yourself before it touches the ensemble. Two published models scored below a coin flip on my test set, and one of them was anti-correlated enough to be useful upside down. The paper’s number is the paper’s number.
2. The ensemble’s weakest modality sets the product’s honesty. Image AUC of 0.9967 is a nice line until a video arrives, so the video number, 0.7366, is the one that belongs in every conversation about what the tool can promise.
3. Calibrate or stay silent. An uncalibrated 0.93 means nothing to the person deciding whether to trust a video of their CEO; Platt scaling is twenty lines and it is the difference between a score and a statement.
4. Vendor the code, mirror the weights. Half the detectors I evaluated had a dead weight link within a year of publication, and reproducibility that depends on someone else’s cloud folder is a countdown.
5. Namespace isolation beats containers for a model zoo. One process holding all the models pays for a single CUDA context where containers would pay for twenty-one, and the import prefix trick costs forty lines.
6. Provenance beats inference when both are present. A watermark check is cheap, definitive, and boring, and boring evidence should always run before exciting evidence.
7. Design the degraded mode before the demo. In a zoo this size one animal is always limping, so the missing-vote path, imputation plus a flag in the JSON, is a first-class feature, and it costs a fraction of the incident it prevents.