Payment card fraud across the European Economic Area reached EUR 4.2 billion in 2024 (ECB & EBA, 2025). The central question: does deep learning offer advantages over traditional ML baselines that justify its added complexity?
To answer this, six model configurations were evaluated on the ULB dataset — four core models spanning supervised and unsupervised paradigms, plus two hybrid extensions — alongside drift-mitigation experiments.
Dataset and the imbalance problem
The ULB dataset contains 284,315 legitimate and only 492 fraudulent transactions (fraud rate: 0.173%, imbalance ratio ~1:578). Under such extreme imbalance, accuracy is meaningless — a majority-class classifier hits 99.83% trivially. ROC-AUC is similarly misleading. PR-AUC was adopted as the primary metric throughout.
Features V1–V28 are PCA-transformed (anonymised). Only Amount and Time are in their original form.
A 70/15/15 chronological split was used rather than random — fraud is temporally clustered, and random
splitting would leak future patterns into training.
Feature drift
PSI was computed for all 29 features across the two halves of the 48-hour window. 15 features exceed PSI > 0.10, and 8 exceed PSI > 0.25. V1 alone has PSI = 1.430.
This matters for two reasons:
- It validates the chronological split — random splitting would spread drift across train/test sets and overstate performance.
- It raises whether distributional shift causes measurable degradation — examined in the drift analysis below.
The six models
| Model | Paradigm | PR-AUC | 95% CI | F1 | Recall@P=0.9 |
|---|---|---|---|---|---|
| LR | Supervised ML | 0.692 | [0.550, 0.852] | 0.753 | 0.673 |
| FFN | Supervised DL | 0.766 | [0.640, 0.876] | 0.813 | 0.731 |
| IF | Unsupervised ML | 0.054 | [0.035, 0.080] | 0.133 | 0.000 |
| AE | Unsupervised DL | 0.069 | [0.041, 0.124] | 0.129 | 0.000 |
| W1B | Semi-supervised | 0.770 | [0.651, 0.871] | 0.821 | 0.750 |
| ENS | Ensemble | 0.772 | [0.656, 0.873] | 0.821 | 0.750 |
Test set: n = 42,722, 52 fraud cases. Thresholds target Precision ≥ 0.90. McNemar test LR vs FFN: p = 0.581 (ns).
Supervised: LR vs FFN
FFN outperforms LR on PR-AUC and recall, but the McNemar test is non-significant (p = 0.581) and confidence intervals overlap substantially. The gap may partly reflect sampling variation on 52 test cases rather than a robust generalisation advantage.
The FFN’s real advantage is operational: mini-batch retraining supports efficient adaptation to concept drift, and its architecture supports federated learning — model parameters rather than raw transaction data are shared across institutions. Both claims are tested directly, not left theoretical — see Incremental and federated learning below.
Unsupervised: IF vs AE
Both models performed near the no-skill baseline. This isn’t an implementation failure — it’s a dataset constraint. The ULB’s PCA-anonymised features preserve population-level variance rather than exposing fraud-specific distributional structure, violating the off-manifold assumption that reconstruction-based anomaly detection requires.
The AE’s 11.7× fraud-to-legitimate reconstruction error gap confirms a residual anomaly signal exists, but it’s too weak to produce a useful precision-recall curve.
Precision–Recall curves
Hybrid extensions
W1B appends the AE’s reconstruction error as a 31st FFN input feature. The near-zero gain is explained by SHAP: the reconstruction error feature ranks last of 31 with 0.0% of top-feature importance. The features the AE struggles to reconstruct for fraud are not those that discriminate fraud from legitimate transactions — confirmed by Spearman ρ = −0.433 (p = 0.017).
ENS averages normalised LR and FFN scores. Despite near-identical threshold-level decisions, ENS achieves the highest PR-AUC overall (0.772). LR and FFN show only partial feature-ranking agreement (ρ = +0.305, ns), so averaging smooths individual model biases — score-level complementarity even when classification decisions coincide.
SHAP interpretability
The FFN’s feature importance ranking closely mirrors the KS-based separability analysis (Spearman ρ = +0.727, p < 0.001). V14, V12, V10, and V4 dominate both rankings. One exception: V27 is highly ranked by KS but assigned near-zero SHAP importance — a concrete target for future refinement.
Hard cases: 12 of 52 fraud cases (23%) were missed by all four core models simultaneously — a floor that architectural improvements alone cannot resolve. FFN uniquely caught 5 cases no other model detected; LR, IF, and AE caught none uniquely.
Drift analysis
Both supervised models degrade over the chronological test split, but the raw PR-AUC drop is partly confounded by a 67% fall in fraud prevalence between halves. Normalised AP removes this confound and reveals FFN’s relative discrimination improved in the late period, while LR’s did not.
Three mitigation strategies were tested:
- LR-SW (sliding-window resampling) — worse than baseline
- FFN-TW (time-weighted loss) — worse than baseline
- ENS — the only successful strategy (+0.003 full, +0.004 late Δ)
The targeted mitigations failed because the sliding window reduces training fraud from 384 to ~139 cases — too few for stable optimisation under imbalance. Model diversity, not temporal reweighting, is the more reliable mitigation when recent labelled data is scarce.
Calibration
FFN achieves strong calibration (ECE = 0.00024, Brier = 0.00045) — suitable for direct threshold-setting in production. LR’s ECE of 0.051 requires post-hoc correction (Platt scaling or isotonic regression) before deployment.
Incremental and federated learning
Two deployment-oriented extensions to the FFN were tested against full retraining as a baseline.
Incremental retraining split the training set into five chronological blocks (20% each) and compared a model retrained only on the newest block against one retrained cumulatively on all blocks seen so far. Incremental retraining tracked full retraining within a maximum gap of 0.018 PR-AUC across all five blocks (block 5: 0.755 incremental vs 0.772 full) — evidence that block-wise updates capture most of the benefit of full retraining without needing to revisit the entire training history each time.
Federated learning simulated three participating banks, each training the FFN locally for 12 epochs before FedAvg-style weight averaging over three rounds. The federated model reached PR-AUC 0.750, a 2.1% relative gap versus the 0.766 centralized FFN — trading a small performance cost for not requiring raw transaction data to leave any single institution.
MLOps simulation
The final section simulates a production ML lifecycle around the six trained models: a versioned registry with an automated promotion gate, ongoing drift monitoring, a retraining trigger rule, and a canary rollout with automatic rollback.
Model registry & promotion gate. Each model is registered with its PR-AUC, and a candidate only replaces the current production champion if it beats it by at least 0.01 PR-AUC — a deliberate buffer against promoting a model on noise rather than a real improvement. FFN (0.766) cleared this bar over the LR baseline and became champion. Notably, neither W1B (0.770, Δ = +0.003) nor ENS (0.772, Δ = +0.006) — both of which technically outrank FFN on the headline leaderboard — cleared the +0.01 minimum-delta bar, so the gate correctly declined to promote them. This mirrors the McNemar result above: the small numerical gains between FFN, W1B, and ENS are not statistically distinguishable, and the gate is designed not to churn production models on such margins.
Drift monitoring dashboard. PSI was recomputed against the training reference across five chronological monitoring windows on the test set. Mean PSI drifted gradually from 0.167 to 0.183 (window 1 → 5), staying below the 0.25 alert threshold on average — but individual features (chiefly V1, V3, V28, V11, V25) crossed the alert threshold in every window, consistent with the feature-level drift already flagged in the EDA. PR-AUC per window is volatile rather than monotonically declining: all three tracked models (LR, FFN, ENS) dip sharply in window 4 before recovering in window 5, a reminder that small per-window fraud counts make these estimates noisy.
Retraining trigger. The orchestration rule retrains only when PSI alerts and a relative PR-AUC drop exceeding 15% co-occur. Neither condition was met in this run (mean PSI never breached 0.25; PR-AUC recovered by the final window), so no retrain was triggered — correctly avoiding an unnecessary retrain on transient noise.
Canary rollout and rollback. The champion (FFN) was tested against a challenger (LR-SW, the sliding-window retrain from the drift-mitigation experiments) on the held-out late-window test split. FFN scored PR-AUC 0.597 versus the challenger’s 0.420; McNemar’s test on their disagreements was not significant (p = 1.0), so the gate keeps the champion rather than promoting a challenger on an unproven difference. A separate rollback demo sets an error-budget floor at 90% of the champion’s offline PR-AUC (0.537) and shows the challenger — which falls to 0.420 — breaching that floor and triggering an automatic revert to the champion.
Latency. Batched inference cost is 0.0001 ms/transaction for LR and 0.0008 ms/transaction for FFN, both far inside a ~200ms card-present authorisation budget — though this is measured on a batch, not under realistic single-request serving load, a limitation worth noting before treating it as a true SLA guarantee.
What I’d try next
True streaming updates — the block-wise incremental retraining tested above already tracks full retraining within ~0.02 PR-AUC, validating that partial retrains capture most of the benefit. The natural next step is per-transaction online learning with frameworks like River, closing the gap between periodic batch retraining and continuous adaptation. This still requires low-latency labels; where disputes-based labelling delays exceed 24 hours, semi-supervised online methods are preferable. Visa and Mastercard both run real-time model updating at transaction scale, so this is operationally proven beyond academia.
Graph neural networks — fraud rings invisible at the individual transaction level become detectable as anomalous subgraphs. A card–merchant bipartite graph with V-features as node attributes could expose ring-level patterns unavailable in the flat feature space. Practical caveat: the ULB’s PCA anonymisation removes the raw merchant/cardholder identifiers needed to construct graph edges.
LLM-based tabular detection — TabLLM demonstrates few-shot classification without retraining, addressing concept drift through in-context adaptation. Key constraints for real-time fraud: inference latency typically exceeds 100ms (vs. <20ms required for live authorisation), and GDPR Article 22 explainability obligations are harder to satisfy than with SHAP-annotated networks. Better suited to asynchronous review workflows than real-time scoring.
Key takeaway
Deep learning offers conditional, not categorical advantages here. FFN’s non-linear representation is empirically real — traceable via SHAP, evidenced by unique fraud catches — but predictive superiority over LR is statistically uncertain. The strongest case for FFN is operational: mini-batch adaptability, validated federated-learning compatibility (a 2.1% PR-AUC cost for keeping data on-premise), and superior calibration.
ENS is the practical recommendation: highest PR-AUC, no additional training cost, robust to drift through model diversity — though the simulated CI/CD gate correctly declines to promote it over FFN, since its +0.006 PR-AUC edge falls below the 0.01 minimum-delta threshold meant to prevent promoting models on statistical noise.