Data, benchmarks, and honest limitations.
ATR publishes evasion tests openly. We tell you what we can't catch.
The full argument for why ATR exists: once agents act on their own, trust can no longer be assumed, and detection has to become a standard layer anyone can audit, version, and maintain. Covers the RFC-001 quality specification, a 96,096-skill ecosystem scan (1,302 flagged, 552 confirmed malware after manual review), and the mechanism that turns the standard from a one-time snapshot into a living one.
1,302 flagged across 96,096 skills scanned in six registries; 552 confirmed malware after manual review. Three coordinated threat actors (hightower6eu 354, sakaen736jih 212, 52yuanchangxing 137). Reported to NousResearch and blacklisted.
The largest AI agent security scan to date. 96,096 skills across 6 registries, 1,302 flagged, 552 confirmed malware after manual review. Three coordinated threat actors. Credential access via tool descriptions accounts for 53% of detections.
Empirical analysis of the MCP attack surface. 30 CVEs in 60 days, 38% zero authentication, 7-class attack taxonomy, 53K ecosystem scan. 15x faster than Docker's first two years.
Benchmarks
Tested against our own corpus and public adversarial datasets (deepset, Lakera Gandalf, HackAPrompt, garak).
The gap between 99.7% precision and 63.6% recall is expected. Regex catches known patterns but misses paraphrases and multilingual attacks.
SKILL.md Detection Benchmark
Tested against 498 real-world OpenClaw SKILL.md files (32 malicious + 466 benign). Layer A = explicit malicious instructions. Layer C = obfuscated / hidden attacks.
Ecosystem Scan Data
Real scans of real MCP skill registries.
This scan is not a report — it is a loop that runs every day. New attack appears → crystallizes into a detection rule → flows back into the standard. A red-team mega-scan flywheel and a CVE-ingestion flywheel each completed a full sweep, then moved to daily updates, taking the ruleset from 462 to the current 708 (246 new rules, npm [email protected]). This is what it means for a standard to be alive: not a one-time snapshot, but a layer that evolves at the speed of the threats it covers.
Research Methodology
A standard's numbers only count if someone else can reproduce them. All research here is reproducible — datasets, scan scripts, and evaluation scripts are open-source under MIT license, so anyone can rerun them on the same ATR version and arrive at these figures, or prove them wrong.
Six registries totaling 96,096 skills. Largest subsets: OpenClaw 56,480, ClawHub 36,394, Skills.sh 3,115, plus three additional MCP / skill indexes. Each registry is crawled via public HTTP API or git repository.
The detection core is deterministic: rules execute regex / AST matching with no LLM inference, so the same input produces the same result in every environment — reproducibility is a prerequisite, and it is what ships on by default. Since v3.1.0 an optional, experimental semantic judge stage recovers paraphrased attacks the patterns miss; it is opt-in, off by default, and never alters the deterministic core. Every rule is wild-validated against 36,394 ClawHub skills before publication.
Precision / recall uses the external PINT dataset (850 samples) rather than self-generated tests — this avoids overfitting to our own test cases. A separate SKILL.md benchmark uses 498 real-world OpenClaw files, with malicious samples hand-labeled as ground truth.
False positive rate is measured against real benign samples (skills vetted by manual or community review), divided by total detections, and reported lane by lane rather than as one flattering number: the enforce lane runs only the most mature rules (~0.24% on a ~65,000-sample benign corpus), while the default hunt lane runs everything as an advisory signal (~9%). A standard earns trust by publishing its worst figure, not hiding it — so both are stated here. Every documented FP context is written into the rule YAML's false_positives field and surfaced on the rule page.
Scan checkpoints, test sets, and evaluation scripts are public under data/ and tests/. Any researcher can rerun the scan with the same ATR version and obtain identical results.
External Citations
ATR is built to be cited, embedded, and depended on by other people's work. This section records external citations honestly — including where there are none yet.
No external citations recorded yet. If your paper, technical report, or product documentation cites ATR, please let us know via GitHub issue.
Cite as: Agent Threat Rules (ATR) Project (2026). Agent Threat Rules: An open detection standard for AI agent threats. DOI: 10.5281/zenodo.19178002
Cite as: Lin, Kuan-Hsin (2026). The Collapse of Trust. DOI: 10.5281/zenodo.19178002
Upcoming Research
The next turns of the flywheel: pushing detection from static patterns toward runtime behavior and semantics, then crystallizing the hard cases back into deterministic rules. Each step ships in GitHub releases and paper updates, so the progress is auditable from the outside.
Extends detection from static regex to runtime behavioral signals. Initial rules: env variable access combined with network calls, tool-call rate anomalies, unexpected shell access.
Cases Tier 3 cannot decide escalate to LLM-as-judge semantic analysis. LLM findings crystallize into Tier 2 regex rules and flow back into ATR — the adaptive-to-innate immunity transition.
Compile ATR rules into Sigma (SIEM-side) and YARA (file-side) formats so agent threat detection plugs into existing security pipelines without rebuilding infrastructure.
ATR detects runtime attacks. Model backdoors planted during training are architecturally invisible at inference time. Bridging this gap requires new techniques combining supply-chain provenance (model cards, training data audits) with runtime behavioral fingerprinting.
Sources: ATR-FRAMEWORK-SPEC.md Phase 2-4 roadmap and the main paper's future work section.
What ATR Cannot Detect
We publish this section because honest limitations build more trust than false confidence.
Any regex rule can be bypassed by semantically equivalent rephrasing. "Ignore previous instructions" is detected; "please set aside the guidance you were given earlier" is not.
All patterns are English-only. Injection payloads in Spanish, Chinese, Arabic, or any other language bypass all rules completely.
"Delete all records" might be legitimate or malicious. Regex matches patterns without understanding authorization context.
ATR inspects content, not transport. Message replay, schema manipulation, MCP transport-level MITM are invisible.
Gradual trust escalation across 20 turns, where no single message is detectable, is not correlated. ATR evaluates events independently.
By definition, regex cannot detect attack patterns that don't exist yet. New techniques require new rules.