#1
2025 model set, threat-intelligence reasoning
Range across the models in Meta and CrowdStrike's paper (Sep 2025); per-model scores only in its figures
43% to 53%
CyberSecEval 4 Leaderboard and Benchmark Guide
CyberSecEval 4 is Meta's security benchmark suite, including CyberSOCEval for malware analysis and threat intel. What it tests and what the scores show as of October 2026.
SecurityUpdated 2026-10-30
What It Measures
CyberSecEval 4 is Meta's open benchmark suite for the cybersecurity risks and capabilities of language models, published in its PurpleLlama repository. It builds on CyberSecEval 3 and covers two sides. The risk side tests how often a model suggests insecure code, how it handles prompt injection, whether it helps with cyberattack requests mapped to MITRE ATT&CK, and how it performs on offensive tasks. The defensive side was new in version 4.
The defensive benchmarks are the reason most security teams look at the suite. CyberSOCEval, built with CrowdStrike, tests two jobs a SOC actually does: malware analysis, where the model reads a sandbox detonation report and answers questions about the sample's behaviour, and threat-intelligence reasoning, where it reads a threat report (including its images) and draws security conclusions. AutoPatchBench, the third new benchmark, tests whether an agent can patch vulnerabilities in native code.
How It Works
CyberSOCEval is scored as multiple choice with several correct answers per question, and a response counts only when it selects exactly the right set. That keeps the random-guess baseline low: about 0.63% on malware analysis and about 1.7% on threat-intelligence reasoning, according to the paper (Meta and CrowdStrike, arXiv 2509.20166).
The questions were generated with Llama models and then filtered, and the authors flag this as a possible bias in Llama's favour, since third-party API terms ruled out using other models for generation. The malware reports are long, often near or past a 128,000-token context window, which makes the benchmark a test of long-document reading as well as security knowledge.
Current Leaders
#2
2025 model set, malware analysis
Same paper and model set; random guessing scores about 0.63%
23% to 34%
Which model fits which security job: Best LLM for Cybersecurity (October 2026)
How to Interpret Scores
The paper tested 2025 models from OpenAI, Google, Anthropic, Meta and DeepSeek, open and closed. All of them landed between about 23% and 34% on malware analysis and between about 43% and 53% on threat-intelligence reasoning, so open and closed models were bunched together, and per-model numbers appear only in the paper's figures. The authors found that larger, newer models did better, that reasoning models gained less here than they do on coding and maths, and that current models are far from saturating the suite.
As of 30 October 2026 we found no published 2026 results for either frontier or open models on CyberSOCEval, so the leaders below are the paper's ranges. That makes it one of the clearest open questions in defensive AI: a benchmark with room at the top, built from real SOC work, and no current scores. The rows below will change as 2026 results appear.
Related Resources
Put an open model on your SOC's own reports
We deploy GLM, Kimi and Qwen models for security teams on infrastructure we manage, in your own cloud tenancy, or on-premise, so malware reports and threat intel stay private.
Book a call