Guardrails experiment ledger
Every run of grctl experiment, newest first. Each run directory holds result.json, trials.jsonl, report.md and report.html.
Latest result per experiment
Attack success rate across models, with and without the gateway 20261004-102154 · 2026-10-04 10:21
ASR, no gateway
3.0%
5/168 attack trials leaked the secret
ASR, deterministic guards
0.0%
0/168
ASR, + semantic tier
0.0%
0/168
Held-out detector benchmark: rules vs model classifiers 20261006-133928 · 2026-10-06 13:39
Deterministic recall, in-distribution
100.0%
the corpus it was tuned on
Deterministic recall, held-out
16.7%
novel phrasings, 8 languages
Best strategy F1 (held-out)
0.769
Deterministic + prompt-guard-86m + openai-moderation
Response caching: savings and the semantic-cache correctness trade 20261004-075813 · 2026-10-04 07:58
Exact cache hit rate
75.5%
spend 76% lower, mean latency 195ms
Semantic cache hit rate
77.0%
at threshold 0.88, 0.0% wrong answers
Semantic lookup latency
482ms
one embedding round-trip per exact miss
PII redaction round-trip across models 20261004-102831 · 2026-10-04 10:28
PII exposed to providers, no gateway
100%
14 values per model sent verbatim
PII exposed with redaction
0.0%
0/84 values
Task success cost of redaction
-16.7 pts
redacted minus raw, all models
Resilience under provider failure: fallback chains and circuit breaking 20261006-133656 · 2026-10-06 13:36
Availability, single provider
62.5%
primary down for 20 s of 60 s
Availability, fallback + breaker
99.2%
p95 1184ms
Dead-primary calls avoided by the breaker
200 → 5
during the outage window
All runs
| Run | Experiment | Version | Started | Duration | Calls (replayed) | Cost |
|---|---|---|---|---|---|---|
| 20261006-133928 | detectors | local | 2026-10-06 13:39 | 3s | 0 (40) | $0.0000 |
| 20261006-133716 | detectors | local | 2026-10-06 13:37 | 5s | 0 (40) | $0.0000 |
| 20261006-133656 | resilience | local | 2026-10-06 13:36 | 0s | 0 (0) | $0.0000 |
| 20261006-133405 | resilience | dev | 2026-10-06 13:34 | 0s | 0 (0) | $0.0000 |
| 20261006-133033 | resilience | dev | 2026-10-06 13:30 | 0s | 0 (0) | $0.0000 |
| 20261004-102831 | pii | dev | 2026-10-04 10:28 | 61s | 144 (0) | $0.0062 |
| 20261004-102154 | asr | dev | 2026-10-04 10:21 | 398s | 274 (0) | $0.0171 |
| 20261004-075813 | efficiency | dev | 2026-10-04 07:58 | 6s | 0 (49) | $0.0000 |
| 20261004-075810 | detectors | dev | 2026-10-04 07:58 | 948s | 536 (0) | $0.0143 |
| 20261004-075221 | efficiency | dev | 2026-10-04 07:52 | 6s | 0 (49) | $0.0000 |
| 20261004-075220 | pii | dev | 2026-10-04 07:52 | 0s | 0 (144) | $0.0000 |
| 20261004-074216 | asr | dev | 2026-10-04 07:42 | 459s | 273 (1) | $0.0171 |
| 20261004-074149 | resilience | dev | 2026-10-04 07:41 | 0s | 0 (0) | $0.0000 |