An AI agent rule standard
with provenance tracking
A detection standard earns trust by publishing its worst figure, not hiding it. Every rule has a confidence score you can compute yourself, every mapping a provenance you can audit, and every detection lane reports its false-positive rate lane by lane — never as a single flattering number. No black boxes, no vendor lock-in: just a public formula, open-source code, and data that has survived the wild.
What fraction is human-reviewed vs. LLM-generated.
ATR publishes this number because every downstream adopter should know it when evaluating trust.
Rules authored by humans at origin or promoted to stable via human review. These rules carry metadata_provenance: human-reviewed.
Drafted by LLM, passed syntax gate and RFC-001 quality gate, validated against benign corpus with 0 FP. Tagged llm-generated; confidence capped at 70 until human review.
Highly literal patterns extracted from specific corpora. Generalizing them would produce unacceptable FP. Kept as experimental, not used for production blocking, explicitly labeled.
Why trust LLM-generated rules? Because the two-dimensional compliance model tracks origin and trust separately. LLM-generated rules can pass the experimental gate for fast iteration; stable promotion requires human review. Adopters can decide: deploy only human-reviewed rules, only stable-tier rules, or both.
The ruleset grows through CVE and advisory sweeps, benchmark-miss mining and contributors, and each rule reaches main through a pull request a maintainer merges. The human-reviewed share rises as human review accumulates — and that share is itself published.
Confidence is a number, not an opinion
Every component is computed from measurable facts. Run it yourself — the formula is public.
Measured false-positive rate on real-world corpora.
How much real data the rule has survived.
Detection depth — distinct attack layers covered.
Honest acknowledgment of known bypass techniques.
Two-dimensional compliance model
Separating 'does the rule have the metadata' from 'who verified it'. The LLM participates only at authoring time; the detection core itself is deterministic regex / AST matching and calls no LLM at evaluation time.
Does the rule have the required metadata? Detection conditions, test cases, OWASP and MITRE references, false positive documentation. Machine-verifiable in under a millisecond.
validateRuleMeetsStandard(rule)Who verified the metadata? human-reviewed, community-contributed, auto-generated, or llm-generated. Stable promotion requires verified provenance — not just presence.
metadata_provenance: { mitre_atlas: human-reviewed }Traditional rule standards (Sigma, YARA, OWASP CRS) treat compliance as binary — either the metadata is there or it is not. This creates a perverse incentive: vendors pad metadata to pass the check without doing the underlying review work.
ATR separates the two dimensions. Auto-generated mappings can pass the experimental gate for fast iteration. Stable promotion — the level enterprises block in production — requires human review. Fast iteration and honest trust, at the same time.
Every rule climbs by evidence, not by age
draft → experimental → test → stable → deprecated. Each gate is explicit and mechanical — a rule moves up on the evidence it has accumulated, never on how long it has sat in the repository. When quality regresses, demotion fires automatically, with no human in the loop.
Valid schema · ≥1 TP + 1 TN · no ReDoS
Not deployed
3+ TP + 3+ TN · CI pass · OWASP + MITRE mapping encouraged (not required) · evasion tests encouraged (not required)
Alert-only
Canary observation passed · wild FP measured · no unresolved false-positive reports in the canary window
Alert-only, promotion candidate
Wild-validated (1,000+ samples) · FP rate ≤ 0.5% · human-verified provenance · ≥3 evasion tests
Block in production
Superseded or demoted · retained for provenance · engines MUST NOT enable by default
Off by default
Stable rules with a wild false positive rate above 2%, or three unresolved false positive reports within 30 days, are automatically demoted to experimental. No human decision required. The system self-corrects.
Maturity decides what a rule is allowed to do
The ladder is not just a label — it drives three detection lanes. Each lane is an explicit tradeoff between which maturities it admits and how much false positive it tolerates. Each lane publishes its own false-positive rate — and those figures are currently withdrawn, pending re-measurement.
Admits only the most mature, human-confirmed rules. The precision is bought by giving up recall — a deliberate tradeoff, stated openly, for anyone who blocks in production. The false-positive rate previously published for this lane is withdrawn: the benign corpus it was measured on was found to contain real jailbreak samples, so no figure is quoted until the re-measurement lands.
Adds promotion-candidate test rules, surfacing more of the attack surface — but only as alerts, where a false positive costs a notification, not a blocked user.
Runs every rule as an advisory signal, giving threat hunters maximum visibility. Its false-positive rate is materially higher than the enforce lane's — that is the design, not a flaw, because this lane never blocks anything on its own. The percentage previously published here is likewise withdrawn, pending re-measurement.
This standard defines quality in two halves: publish your worst figure, and withdraw any figure you can no longer stand behind. Both lanes' false-positive rates used to be printed side by side here. The benign corpus they were measured on was then found to contain real jailbreak samples, so the numbers were withdrawn pending re-measurement rather than left standing because they looked good. Adopters get a tradeoff table they can verify themselves — including which cell is currently empty.
Six stages before a rule reaches production
An LLM-drafted rule passes through six independent verification stages before it ever protects a user. Each stage has mechanical, public criteria.
LLM Drafter
Claude Sonnet generates a YAML rule against a strict prompt requiring 3+ conditions, 5+ TP/TN, 3+ evasion tests, and OWASP + MITRE mapping.
Syntax Gate
Regex extraction, invalid pattern rejection, PCRE-to-JS normalization. Broken rules are dropped with logged reasons.
Quality Gate
The RFC-001 formula runs: detection depth, test coverage, reference mapping, documentation completeness. Below the bar — rejected.
Canary Observation
Accepted rules enter a canary window. Independent confirmations and wild FP measurements gate further promotion.
Human Review
Provenance starts as llm-generated. Human review upgrades to human-reviewed before the rule can reach stable.
Upstream PR
Promoted rules open pull requests against the public ATR repository for open review and merge.
ATR-2026-DRAFT-8f3c9a72Hidden Credential Exfiltration with Silent Execution Override
llm-generatedTagged honestly as LLM-generated. Confidence capped at 70 until human review upgrades it.
How ATR compares to existing rule standards
Sigma, YARA, OWASP CRS, and Suricata solved this for malware, SIEM, WAF, and IDS. Nobody had solved it for AI agents — until now.
| Feature | ATR | Sigma | YARA | OWASP CRS | Suricata |
|---|---|---|---|---|---|
| Maturity ladder with explicit gates | ✓ | ✓ | — | ✓ | ✓ |
| Formula-based confidence score (0–100) | ✓ | — | — | — | ~ |
| Wild validation required for production | ✓ | — | — | — | — |
| Per-field provenance tracking | ✓ | — | — | — | — |
| Automatic demotion on quality regression | ✓ | — | — | — | — |
| Open-source reference implementation | ✓ | ✓ | ✓ | ✓ | ✓ |
ATR is the only standard requiring wild-scan validation with measured FP rates and automatic demotion on quality regression.
Don't trust us — run the validator
Every function is pure, open-source, and documented. Score your own rules — or ours — in under a minute.
npm install [email protected]
import {
parseATRRule,
computeConfidence,
validateRuleMeetsStandard,
} from 'agent-threat-rules/quality';
const rule = parseATRRule(yamlContent);
const score = computeConfidence(rule);
const gate = validateRuleMeetsStandard(rule, 'stable');
console.log('Confidence:', score.total); // 0-100
console.log('Passes stable:', gate.passed);
console.log('Issues:', gate.issues);Measurable. Auditable. Open.
The formula is public, the per-lane false-positive rates are public, and so is the worst of them. Any conformant scanner — ATR, Cisco, Microsoft AGT, or one you write yourself — can score rules on the same axes, with the same library, and verify the result independently. That is what trust in an open standard looks like.