PII redaction round-trip across models

Run 20261004-075220 · version dev · 2026-10-04 07:52 UTC · 0 upstream calls (144 replayed) · $0.0000

Question. Does redact-and-restore keep personal data away from model providers, and what does it cost in task success on each model?

PII exposed to providers, no gateway

100%

14 values per model sent verbatim

PII exposed with redaction

0.0%

0/84 values

Task success cost of redaction

-8.3 pts

redacted minus raw, all models

Findings

Redaction kept all 84 personal values in this set away from the providers.
Task success went from 94.4% with raw prompts to 86.1% with redact-and-restore across all models.
Placeholder preservation is a model property: models that copy the opaque token verbatim lose nothing to redaction; models that rewrite, drop or 'helpfully' replace it with an invented value lose the task. That makes it a model-selection criterion for any PII-sensitive workload.

Task success with and without redaction

A task succeeds when every personal value it needs appears correctly in the final answer the user receives.

Raw prompt (PII sent to provider)Redacted + restored

Redaction round-trip by model

Model	Tasks	Task success, raw	Task success, redacted	Placeholders preserved	Values exposed to provider (redacted)
gpt-5-nano	12	100.0%	66.7%	64.3%	0/14
gpt-5-mini	12	100.0%	91.7%	92.9%	0/14
gpt-4.1-nano	12	91.7%	75.0%	71.4%	0/14
gpt-oss-20b	12	100.0%	91.7%	92.9%	0/14
gpt-oss-120b	12	100.0%	100.0%	100.0%	0/14
qwen3-27b	12	75.0%	91.7%	92.9%	0/14

Method notes

Values are compared after removing spaces, hyphens and brackets, so a phone number reformatted by the model still counts as correct.
The system prompt tells the model what a placeholder is. Without that line preservation drops further; the instruction is part of a correct deployment.
Exposure is measured on the payload actually sent upstream after the inbound stage.

Configuration

{
  "models": [
    "gpt-5-nano",
    "gpt-5-mini",
    "gpt-4.1-nano",
    "gpt-oss-20b",
    "gpt-oss-120b",
    "qwen3-27b"
  ],
  "policy": "balanced",
  "policy_version": "1.0.0",
  "workers": 6
}