Skip to content
Agent Threat Rules (ATR)中文
Research

The numbers, and what ATR misses.

The evasion tests are public. So is the list of attacks the rules can't catch.

April 2026 · 25 pages · 67 references
The Collapse of Trust: Security Architecture for the Age of Autonomous AI Agents

Why ATR exists, argued from the start. Once agents act on their own, trust can't be assumed, and detection has to become a standard layer that anyone can audit and the community versions and maintains. It covers the RFC-001 quality specification, the in-the-wild ecosystem scan, and how the standard keeps updating instead of stopping at one snapshot. The scan figures in the paper are out of date. The ones to cite are 101,280 items scanned and 1,434 flagged (engine v2.0.0, 2026-04-13).

NewApril 2026
A Coordinated Malware Campaign in the OpenClaw Registry: Three Mass-Publishing Threat Accounts

The wild scan flagged 1,434 items, clustered in three accounts that published in bulk: hightower6eu (354 skills), sakaen736jih (212) and 52yuanchangxing (137). Payloads include encrypted zips, base64 shell commands and C2 callbacks. They were reported to NousResearch and added to a public blacklist. The verdicts are made per publisher and applied to the whole account, not file by file. The report's totals are out of date; use the figures on this page.

April 2026 · 7 pages · 32 references
A Large-Scale Security Audit of the AI Agent Ecosystem

A large scan of agent skills and MCP definitions across five public registries. It found three coordinated threat accounts. One rule, ATR-2026-00121 (credential access through tool descriptions), accounts for 53% of flagged skills. The paper's totals are out of date. The ones to cite are 101,280 items scanned and 1,434 flagged (engine v2.0.0, 2026-04-13).

NewApril 2026 · 18 pages · 30 references
30 CVEs in 60 Days: The Model Context Protocol Attack Surface

The MCP attack surface, measured. 30 CVEs in 60 days, and 38% with no authentication at all. The paper sorts attacks into 7 classes and scans 53K ecosystem items. That pace is 15x faster than Docker's first two years.

Benchmarks

Measured on ATR's own corpus and on public attack datasets: deepset, Lakera Gandalf, HackAPrompt and garak.

PINT format, self-built corpus
100.0%
Precision · hunt
65.4%
Recall · hunt
enforce 0.4%
79.1
F1 · hunt
850 samples · ATR 4.1.5, measured 2026-10-15

This is not Lakera's official PINT benchmark. That corpus is private and about 5x larger. This one is self-built in PINT's format: 850 samples from deepset/prompt-injections (660) and Lakera/gandalf_ignore_instructions (190). Only a small subset of rules fire on it, and one rule accounts for most detections. Read it as a score for the prompt-injection family, not for ATR's overall coverage. Partly closed-book: some hunt-lane rules were mined from this corpus's own misses, so the hunt figure is not fully out-of-sample.

Regression check on ATR's own test cases
100.0%
Precision · hunt
96.6%
Recall · hunt
enforce 3.4%
341
Samples
ATR 4.1.5, measured 2026-10-15

This corpus is built partly from the rules' own true positives, so a high score is expected. It checks that the rules still catch what they were written to catch. It is not a benchmark and does not compare with the corpus on the left.

In the hunt lane, precision is 100.0% and recall is 65.4%. That gap is expected. Regex catches the patterns it knows and misses paraphrases and attacks in other languages. The enforce lane loads only stable rules, and it is the one to run with blocking on. Its recall is 0.4%.

SKILL.md detection benchmark

An internal, labeled SKILL.md corpus of 498 files (32 malicious + 466 benign; ATR 4.1.5, measured 2026-10-15). Some of the malicious files were written for this benchmark. Layer A is obvious payloads: curl | bash, base64 exec, reverse shell. Layer C is semantic attacks: natural-language instructions, social engineering. The corpus has no Layer B (obfuscated) samples.

Overall
100.0%
Recall · hunt
enforce 0.0%
97.0%
Precision · hunt
98.5
F1 · hunt
0.20%
False positive rate · hunt
498
Labelled samples
By attack layer · hunt lane
Layer A100.0%
Obvious payloads: 24/24 detected
Layer C100.0%
Semantic attacks: 8/8 detected
Benign1 FP
466 benign SKILL.md files, 1 false positive (0.20%)

The enforce lane loads stable rules only, and it's the one to run with blocking on. It detects 0 of the 32 malicious samples (0.0%). The measurement has no per-layer enforce split.

Ecosystem scans

The rules, run against real MCP skill registries.

Wild ecosystem scan (5 sources)
101,280
skills scanned
1,210
CRITICAL findings
282
HIGH findings
1,434
Items flagged

CRITICAL and HIGH count findings: the 1,434 flagged items produced 1,507 findings, and one item can have several.

ClawHub registry scan
36,394
skills crawled
182
Skills rated CRITICAL
1,124
Skills rated HIGH
9,676
With source code

The wild scan above ran on 2026-04-13. Rules also come from CVE and advisory sweeps, from benchmark misses and from contributors. The ruleset has grown from 462 to the 836 effective rules on main today, plus 8 inert (latest npm release [email protected], as of 2026-11-20).

How the numbers are made

A number only counts if someone else can reproduce it. Everything here can be rerun. The datasets, scan scripts and evaluation scripts are open source under the MIT license. Run them on the same ATR version and you get these figures, or you show they're wrong.

Scan coverage

Five sources, 101,280 items in total (engine v2.0.0, 2026-04-13). The largest are OpenClaw 56,480, ClawHub 36,394 and Skills.sh 3,115. The other two are Hermes Agent and an MCP registry index. Each per-source count comes from that crawler's own report, taken at a different time from the total, so they won't add up to it. Every source is crawled through a public HTTP API or git repository.

Detection engine

The detection core is deterministic. Rules run regex and AST matching with no LLM inference, so the same input gives the same result anywhere. That core is what ships on by default. Since v3.1.0 there is also an optional, experimental semantic judge stage for paraphrased attacks the patterns miss. You have to turn it on, and it never changes the deterministic core. Before a new rule merges, it has to pass a safety gate: zero false positives on a corpus of benign skills.

Benchmarking

Precision and recall run on a self-built corpus in PINT's format. Recall is reported for both lanes, precision for the hunt lane only. The samples come from public datasets (deepset/prompt-injections, Lakera/gandalf_ignore_instructions). Some hunt-lane rules were later mined from this corpus's own misses, though, so the hunt figure is partly closed-book. The corpus is ATR's own, not Lakera's official PINT benchmark. A separate internal SKILL.md benchmark uses labeled files, and some of its malicious files were written for that benchmark.

False positive measurement

False positive rate is measured on real benign samples, meaning skills that passed manual or community review: the benign samples flagged, divided by all benign samples scanned. It is reported per lane, not as one flattering number. The enforce lane runs only the most mature rules. The default hunt lane runs every rule as an advisory signal. The per-lane figures once published here are withdrawn until they are re-measured: the benign corpus behind them turned out to contain real jailbreak samples, so the baseline didn't hold. No figure appears here until the re-measurement is done. Every documented FP context still goes into the rule YAML's false_positives field and shows on the rule page.

Reproducibility

Scan checkpoints, test sets and evaluation scripts are public under data/ and tests/. Rerun the scan with the same ATR version and you get identical results.

External citations

ATR is meant to be cited, embedded and built on. External citations are recorded here, and so is the fact when there are none.

No external citations recorded yet. If your paper, technical report or product docs cite ATR, open a GitHub issue to tell the maintainer.

Cite the standard

Cite as: Agent Threat Rules (ATR) Project (2026). Agent Threat Rules: An open detection standard for AI agent threats. DOI: 10.5281/zenodo.19178002

Originating paper

Cite as: Lin, Kuan-Hsin (2026). The Collapse of Trust. DOI: 10.5281/zenodo.19178002

What's next

Next, detection looks at runtime behavior and meaning, not only static patterns, and the hard cases get written back into deterministic rules. Each step shows up in GitHub releases and paper updates, so anyone outside can follow the progress.

Tier 3 Behavioral Evaluator
In progress

Takes detection past static regex to runtime behavior. The first rules look for env variable reads followed by network calls, odd tool-call rates, and shell access nobody expected.

Tier 4 Semantic Evaluator + Crystallization Pipeline
Design

Cases Tier 3 can't decide go to an LLM-as-judge for semantic analysis. What the LLM finds is written back into ATR as Tier 2 regex rules.

Sigma / YARA output
Partly available

Sigma: the repo has a converter script (scripts/generate-sigma.py) that turns ATR rules into Sigma YAML. There is no ratified Sigma logsource for agent traffic yet, so it uses a custom ai_agent logsource, and you map the fields yourself when wiring it into a SIEM. ATR is also listed in the SigmaHQ tools directory. YARA: ATR has no official compiler; the third-party atr-to-yaral converts ATR rules into YARA-L 2.0 for Google SecOps.

The gap between training and runtime
Open problem

ATR matches text: static artifacts such as skills and tool descriptions (the ecosystem scan above is one), and runtime inputs, outputs and tool calls. A backdoor planted in a model's weights during training is not in any of that text, so it is invisible to ATR by design. Closing that gap needs new techniques that tie supply-chain provenance (model cards, training data audits) to runtime behavioral fingerprints.

Sources: ATR-FRAMEWORK-SPEC.md Phase 2-4 roadmap and the main paper's future work section.

What ATR can't detect

Before you rely on these rules, you should know where they miss.

Paraphrases

Any regex rule can be dodged by saying the same thing another way. “Ignore previous instructions” is caught. “Please set aside the guidance you were given earlier” is not.

Other languages

Nearly every pattern is in English. A few rules also match phrasing in Chinese, Japanese, Korean, Russian or Arabic, and only for the attacks they were written for. An injection in any other language, or a new one in those, mostly gets through.

Attacks that depend on context

“Delete all records” can be a normal request or an attack. Regex sees the words. It can't tell whether the agent was allowed to do it.

Protocol-level attacks

ATR reads content, not the transport. Message replay, schema manipulation and a man-in-the-middle on the MCP transport never reach it.

Attacks spread over many turns

Trust built up slowly over 20 turns, where no single message looks wrong, goes unnoticed. ATR judges each event on its own and doesn't connect them.

New attacks

Regex can't match a pattern nobody has seen yet. A new technique needs a new rule.

Interests: the maintainer founded and controls PANGUARD AI, INC., which sells AI agent security products built on ATR.