#1
Claude Opus 4.6 (high effort)
Microsoft's run, same environment for all 16 configurations (Mar 2026)
0.637
CTI-REALM: Microsoft's Detection Rule Benchmark
CTI-REALM is Microsoft's benchmark for AI agents that turn threat intelligence into detection rules. How the CTI-REALM benchmark works and who leads it as of October 2026.
SecurityUpdated 2026-10-30
What It Measures
CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is Microsoft's open-source benchmark for detection engineering, the work of turning a threat report into rules that catch the attack in real telemetry. It was published in March 2026 (arXiv 2603.13517) and mirrors what a SOC detection engineer does: read the cyber threat intelligence (CTI) report, map it to MITRE ATT&CK techniques, find the right data sources, write and test queries, and deliver a working detection.
Of the public security benchmarks, it sits closest to a defender's daily job. A high score means the agent can carry a detection from report to deployable rule.
How It Works
The agent works in a sandbox with authentic attack telemetry from emulated attacks across Linux, Kubernetes (AKS) and cloud logs, plus reference databases of MITRE techniques and Sigma rules. It must produce both a Sigma rule and an executable KQL query (Kusto Query Language, used by Microsoft Sentinel and Azure Data Explorer).
Scoring is a reward between 0 and 1 built from checkpoints along the way: deterministic checks for the techniques and data sources identified, F1 for whether the detection fires on the right events, and an LLM judge (GPT-5-Mini in the paper) for the quality of the report alignment and the Sigma rule, with a sample of judge outputs reviewed by security researchers. The benchmark is also available in the UK AI Security Institute's inspect_evals collection, so anyone can run it against a new model.
Current Leaders
#2
Claude Opus 4.5
Microsoft's run (Mar 2026)
0.624
#3
Claude Sonnet 4.5
Microsoft's run (Mar 2026)
0.587
#4
GPT-5 (medium effort)
Microsoft's run; GPT-5.2 at medium effort also scored 0.572
0.572
#5
GPT-4.1
Microsoft's run; the most token-efficient configuration, about 120,000 tokens per task
0.419
Which model fits which security job: Best LLM for Cybersecurity (October 2026)
How to Interpret Scores
Microsoft evaluated 16 model configurations, all from Anthropic and OpenAI. Claude Opus 4.6 at high reasoning effort led with a reward of 0.637, followed by Opus 4.5 at 0.624 and Sonnet 4.5 at 0.587. The best OpenAI results were GPT-5 and GPT-5.2 at medium effort, both 0.572, and GPT-4.1 scored 0.419.
Cost per task differs as much as the score. GPT-4.1 used about 120,000 tokens per task, GPT-5 at low effort about 179,000 for a reward of 0.541, and the Anthropic models about 442,000 to 524,000. The paper also found that seeding a smaller model with workflow memory closed 33% of the gap between GPT-5-Mini and GPT-5, with the biggest gain on MITRE technique mapping, a sign that task knowledge can make up part of the gap that model size opens.
As of 30 October 2026, no open-weight model has a published CTI-REALM result. Because the benchmark runs in inspect_evals, that gap can be closed by anyone with an open model and the telemetry environment, and it is the clearest place to test whether a right-sized open model trained for detection work can approach the frontier rows.
Related Resources
Run detection models on your own telemetry
We deploy GLM, Kimi and Qwen models for security teams on infrastructure we manage, in your own cloud tenancy, or on-premise, so telemetry stays private and usage costs become more predictable.
Book a call