A detection standard is only as trustworthy as the numbers behind it — and self-published numbers are the easiest kind to game. So we did the opposite: we ran Agent Threat Rules (ATR) through two independent, open agent-security benchmarks we did not build, using their own harnesses, and we're publishing the results as they came out — including where ATR is weak.
Two benchmarks, because they measure two different things.
OpenGuardrails: runtime detection
The OpenGuardrails benchmark scores detectors on a shared corpus of agent runtime events — prompt injection, malicious commands, data exfiltration, secret leakage — and ranks them by macro-F1 with a p95-latency budget. This is ATR's home turf: it is a rule standard for detecting attacks in an agent's live input, output, tool calls, and MCP exchanges.
Running the current ATR rule set through the OpenGuardrails harness, ATR placed first among the seven reference detectors in the seed suite — ahead of the composed config-plus-LLM detector — with zero false positives on the benign set. It was strongest on malicious-command and data-exfiltration detection, and its weakest lane, paraphrased natural-language injection, is exactly where a pure pattern layer has known limits and where a semantic layer helps.
OASB: static skill and package scanning
The Open Agent Security Benchmark (OASB) measures something different: detection over a labeled corpus of skill and package artifacts — the static source an agent would install — with over 3,800 benign samples and a few hundred malicious ones.
Here the result is two-sided, and worth stating plainly. Across every one of the 3,881 benign skills, ATR raised zero false positives — the lowest false-positive rate of any scanner on that board, better than the purpose-built static scanners. But its recall was low: it caught roughly one in five of the malicious skill artifacts.
That gap is not a bug, and we are not going to dress it up. ATR detects agent behavior at runtime. It is not a source-code or package scanner, and most of the malicious artifacts in OASB are static supply-chain patterns — persistence baked into a manifest, exfiltration logic in code that only runs later — that a runtime detector is not designed to find before execution. On this corpus the AST-based static scanners rightly beat ATR on recall, because that is what they are built for.
What we take from it
ATR is a precise runtime detection layer, not a static package scanner, and the two benchmarks say so in opposite directions. Where ATR is designed to work — live agent traffic — it leads. Where it isn't — static supply-chain source — it is honest, quiet, and low-coverage. The through-line across both is precision: on thousands of benign inputs across both benchmarks, it did not cry wolf once.
The practical read for anyone building agent security: use a runtime detection standard for what happens at runtime, pair it with a source and package scanner for what ships in the supply chain, and don't trust any single layer that claims to do both. Defense in depth is not a slogan here; the benchmarks make it measurable.
We're publishing the numbers we got, not the numbers we wish we had. That is the only kind worth standing behind.