NVIDIA garak × ATR

garak is a red-teaming tool; ATR is a detection standard. One finds the attack, the other turns it into a versioned, MIT-licensed detection rule, so ATR does not have to build its own attack corpus — every probe's provenance stays on the author line.

The idea

garak finds attacks at the LLM layer. ATR defends at the agent layer (tool calls, skills, MCP). One attack payload, two detection surfaces — one standard measures whether it breaks the model, the other measures whether it can be caught on a real agent artifact. The bridge is just a Python script: garak reports in, ATR proposals out, the crystallisation loop drafts rules and a maintainer reviews them. A rule written here reaches every engine that consumes ATR downstream.

garak (red team)                    ATR (standard)           downstream consumers
─────────────────                    ──────────────           ────────────────────
Probes claude-3.7                    828 rules today          npm: agent-threat-rules
Probes gpt-5                 ──▶     +auto-crystallised       PyPI: pyatr
Probes gemini-2-pro                  from garak evidence      your CI pipeline
Writes report.jsonl                  canary 24h
                                     maintainer review

garak PR #1676 (ATR detectors inside garak): withdrawn by its ATR author on
2026-08-04, not merged. garak does not ship ATR.

One-time setup

Email [email protected] with your org name to receive a partner API key (manual issuance during early-partner phase, no cost, MIT terms).
The bridge script, garak-to-tc.py, was moved out of the public repository on 2026-06-20 and is not published at the moment.
Set the key in your environment: export ATR_PARTNER_KEY=…

Run it

After a garak eval session:

# Native garak JSONL report
python3 garak-to-tc.py \
    --input ~/.local/share/garak/runs/2026-04-18-claude-3-7/report.jsonl \
    --partner-name  your-org \
    --target-model  claude-3-7-sonnet \
    --garak-version 0.14.1

# Or ATR-style eval (from scripts/eval-garak.sh in this repo)
python3 garak-to-tc.py \
    --input data/garak-benchmark/garak-eval-report.json \
    --partner-name  your-org \
    --dry-run   # inspect before submitting

Two modes

The script has two submission modes. The default (--mode drafter) is what you want.

drafter (default). POSTs each failed probe to /api/atr-proposals/from-payload. Server-side runs a tool-use LLM drafter (grep existing rules for dedup, fetch research for grounding, write YAML with 3+ conditions / 5+ TP / 5+ TN / 3+ evasion tests), passes RFC-001 quality gate, self-tests its own regex, then files as a proposal. Each call takes 30-60s. This is the mode that produces real ATR rules.
proposal (legacy). POSTs a client-built literal draft. Fast but the LLM reviewer typically rejects these as too narrow (literal fingerprint ≠ detection rule). Only use if you have your own YAML-generation step upstream.

What happens downstream

Each unique failed probe → one proposal in /api/atr-proposals with full garak provenance embedded.
TC crystallisation LLM reads the attack payload + garak metadata and drafts a generalised detection regex.
Rule enters a 24h canary on Threat Cloud.
Survives canary → safety gate (0 FP on the benign skill corpus, test_cases present, ≤ 10 rules per PR; the gate reads every .md file under data/skill-benchmark/benign/, subdirectories included) → opened as a PR against agent-threat-rules main, which a maintainer reviews and merges by hand (the repository has auto-merge turned off) → ships in the next release, which CI stages on npm and a maintainer approves with 2FA → every downstream consumer upgrades on next install.
Every resulting rule carries author: your-org (via garak vX.Y.Z) and metadata_provenance.garak_probe: <probe-id>. Attribution is permanent and machine-readable.

What counts as a failed probe

The script treats a garak attempt as a rule candidate when either:

At least one garak detector returned a score above 0.5 (the target model fell for the probe).
The input is an ATR-style eval report and the prompt appears in missed_samples — garak flagged it malicious and ATR had no pattern for it yet (highest-signal candidates).

Rate limits & safety

Default 100ms delay between POSTs; a run takes about a minute. Both corpora are published, in both lanes. On the public garak in-the-wild jailbreak set (650 samples) recall is 92.3% (hunt) / 0.9% (enforce); on the full garak corpus (3,475 samples, every probe family) it is 57.2% (hunt) / 0.5% (enforce) (ATR 4.1.5, measured 2026-10-15). The hunt lane is the engine default and loads every rule maturity; the enforce lane, the one to run with blocking on, loads only stable rules, so its figure is the one to read before turning blocking on. Both are partly closed-book: many of the rules that fire on them were written from them, and the full suite contains the in-the-wild set twice, so neither hunt figure is an out-of-sample result. Part of the gap between the two corpora is deliberate — the regex detection layer does not chase some probe families (pure semantic jailbreaks, for instance); forcing it to would buy brittle signatures and false positives. Part of it is that the in-the-wild figure rests partly on rules written from that corpus. We report the lower figures too, because a standard earns trust by publishing its worst numbers, not hiding them.
patternHash is a sha256 of the prompt, so re-running on the same report is idempotent — TC recognises duplicate submissions and just bumps a confirmation counter, never creates duplicate rules.
All traffic is authenticated with your partner key. Key compromise? Email and we revoke and re-issue.
Every submitted proposal is reviewable and rejectable via the TC admin dashboard before it enters canary.

/integrate — general adopters (npm + PyPI), no partner key needed
/partner-sync — live rule pull for platforms that consume ATR downstream
scripts/garak-miss-to-proposals.ts — the pipeline in the public repository that turns the prompts ATR misses on the garak benchmark into rule proposals