Once is luck. Ten times is evidence.
ATR Eval tests whether what your agent reads can steer it into leaking data, running code, changing records or spending money. Each test runs at least ten times, each time in a fresh environment. A fixed check decides the outcome, and the result is signed.
The checks hold. The run starts.
The checks get checked first
Before every run, two scripted agents go first. One obeys every attack and must succeed on every test. The other does nothing and must succeed on none. If either result is off, the run doesn't start.
Every test has a twin
The same task runs the same number of times with the attack taken out. That's how you tell a setting that stops attacks from one that just stops the agent from doing its job.
No model grades the result
Every test sets up a small mock world: mail, files, a payment account, a memory store, none of it real and none of it on the network. A fixed check reads what happened in that world and decides whether the attacker got what they wanted.
A receipt anyone can recompute
The result is an in-toto statement signed with Ed25519, and the signature covers every verdict and the evidence behind it. Whoever holds it can check the signature and recompute every verdict from the evidence. The report comes in English and Traditional Chinese.
Where it falls short
Reperformance, running everything again from the receipt, has only been shown on the same machine. The library has 10 tests of its own plus tests imported from public benchmarks. The specification is a 0.9 draft maintained by one project. It is not a standard.
Interests: the maintainer founded and controls PANGUARD AI, INC., which sells AI agent security products built on ATR. Results the maintainer produces are not independent.
ATR Eval is not public yet, and there is no release date.