AD-LLM Benchmark: GPT-4o Hits 0.93+ AUROC Zero-Shot for Text Anomaly Detection

The last two entries in this series covered AnoLLM and CausalTAD — fine-tuned and prompt-engineered approaches to tabular anomaly detection. Before deploying either at production scale, you need to know where LLMs actually stand across a broader range of anomaly detection paradigms. That is the explicit goal of AD-LLM, which benchmarks LLMs across three distinct roles: zero-shot detector, data augmentation engine, and model selection advisor. The focus is NLP text data rather than tabular ledger entries, but the methodological lessons transfer.

The paper

Tiankai Yang, Yi Nian, and colleagues at USC and Texas A&M introduce AD-LLM (arXiv:2412.11142, ACL Findings 2025), the first benchmark to evaluate LLMs systematically across three anomaly detection paradigms on NLP datasets. The setting is one-class classification: training data contains only normal samples, and the model must flag anomalies at test time. The five datasets — AG News, BBC News, IMDB Reviews, N24 News, and SMS Spam — all derive from text classification tasks with one category designated as anomalous. The paper pits two LLMs, GPT-4o and Llama 3.1 8B Instruct, against 18 traditional unsupervised baselines that span end-to-end methods (CVDD, DATE) and two-step embedding-plus-detector combinations (OpenAI embeddings + LUNAR, LOF, Isolation Forest, etc.).

Key ideas

• Zero-shot detection works well for text. GPT-4o scores AUROC of 0.9293–0.9919 across the five datasets in the Normal+Anomaly setting; Llama 3.1 reaches 0.8612–0.9487. The best traditional baseline, OpenAI + LUNAR, scores around 0.92 on AG News — GPT-4o matches or beats it without any training.
• Synthetic augmentation helps, consistently but modestly. LLM-generated synthetic samples improve the OpenAI + LUNAR pipeline on all five datasets. Category description augmentation also improves most baselines, though gains are uneven — Llama 3.1 improves AUROC by +0.07 on IMDB Reviews, but results elsewhere are smaller.
• Model selection is the weak link. GPT-o1-preview recommends models that surpass the average baseline performance on most datasets, and occasionally approaches the best method (e.g., on IMDB Reviews and SMS Spam). But it never reliably identifies the top performer, and the authors acknowledge the recommendations are based on simplistic inputs that lack dataset-specific statistics.
• Open-source versus proprietary gap is real. GPT-4o's AUROC advantage over Llama 3.1 8B is 4–13 points depending on dataset, a gap consistent with the pattern seen in zero-shot tabular anomaly detection papers.
• NLP anomaly detection still lacks a definitive benchmark. Five datasets, all derived from classification corpora, is thin. The companion NLP-ADBench paper (EMNLP Findings 2025) broadens to eight datasets and 19 algorithms but still uses the same semantic-category-as-anomaly construction that makes these tasks somewhat artificial.

What holds up — and what doesn't

The zero-shot findings are credible. Using LLMs as scorers without fine-tuning on labelled anomaly data is genuinely useful when the anomaly class is semantically coherent — a spam message differs from a ham message in ways a well-trained language model understands. The AUROC numbers are high, and the comparison against strong OpenAI-embedding-based baselines is fair.

The scope, though, is narrow in ways the paper undersells. All five datasets encode anomalies as a different topic category — spam versus legitimate SMS, news from a held-out publisher versus in-distribution outlets. This means the LLM is essentially doing topic classification, a task it is explicitly pre-trained on. The benchmark does not include semantic anomalies within a single category (e.g., unusual transactions within the same account type), which is precisely the kind of anomaly that matters for financial auditing.

The data augmentation and model selection tasks are evaluated on the same five datasets, so the paper ends up benchmarking whether LLMs can make slightly different slices of the same narrow problem marginally better. The authors freely list six limitations — including that they test only a subset of LLMs, exclude few-shot and fine-tuning regimes, and rely on simplistic inputs for model selection — which is intellectually honest but also flags how preliminary this benchmark is.

One result worth flagging for skeptics: the AUPRC scores are substantially lower than AUROC for both models. Llama 3.1 on BBC News reaches AUROC 0.8612 but only AUPRC 0.3960, reflecting the class imbalance in the one-class setup. In high-precision auditing contexts, AUPRC is the more meaningful metric, and here the picture is less flattering.

Why this matters for finance AI

The Bean Labs agenda involves two anomaly detection use cases: catching unusual ledger entries in real time (tabular, structured) and flagging suspicious narrative text in invoices, memos, or support tickets (unstructured NLP). AD-LLM speaks directly to the second case and gives us a realistic ceiling: GPT-4o can zero-shot detect topic-level anomalies in text with AUROC above 0.93 on clean, balanced datasets. That is a useful prior, but ledger narrative anomalies are subtler — an invoice memo that describes a routine service but belongs to a vendor flagged for suspicious patterns is not a topic-classification problem. The benchmark provides a starting point, not an answer.

The model-selection finding is separately interesting for system design. The dream of asking an LLM "which anomaly detector should I use on this dataset?" and getting a reliable answer does not yet pan out. That means choosing between AnoLLM-style fine-tuning, CausalTAD-style causal prompting, or a classical embedding method still requires human judgment or systematic empirical evaluation — it cannot be delegated to an LLM advisor.

What to read next

• NLP-ADBench (arXiv:2412.04784, EMNLP Findings 2025) — the companion benchmark from the same group, covering eight datasets and 19 algorithms; provides the broader classical baseline context that AD-LLM's five-dataset scope cannot.
• Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey (arXiv:2409.01980, NAACL Findings 2025) — surveys the full landscape of LLM-based AD approaches across text, image, and tabular modalities; fills in the context around where AD-LLM sits relative to prior work.
• AnoLLM: Large Language Models for Tabular Anomaly Detection (OpenReview:7VkHffT5X2, ICLR 2025) — the tabular counterpart; comparing its likelihood-based approach to AD-LLM's prompt-based zero-shot strategy clarifies which paradigm is more appropriate for Beancount ledger entries.