Confirmation as Disclosure

Large language models are increasingly deployed as customer service agents with direct access to personally identifiable information. We identify and systematically evaluate confirmation-based PII disclosure, a vulnerability class in which an attacker states customer data and asks the model to confirm it, causing the model to verify the data without authenticating the requester.

We evaluate eight models across two providers. The original campaign used more than 1,060 stateless API calls across 715 test conditions; a subsequent harness added 104 controlled security and utility trials. We identify four named attack classes, including two novel techniques that require zero prior knowledge of correct data. Paired correct/incorrect testing yields an oracle score of 0.90, confirming that model responses are record-dependent rather than sycophantic.

We propose a taxonomy rooted in CWE-204 (Observable Response Discrepancy) and observe that the vulnerability persists despite system prompt verification instructions, which the model reinterprets as satisfied by data-matching alone.

In our tests, whether PII was disclosed was determined by application-layer authorization rather than by model-mediated verification. Where the model was the only barrier between the requester and the records, the records were disclosed. Where a lookup tool required an authenticated session before returning a record, disclosure fell to zero.

Only Claude Opus 5.5 requested identity verification across all reported conditions at default settings. The mechanism is not established, and the divergence between it and Sonnet 5.5 cautions against inferring disclosure behavior from release recency or model family alone.