Data: what flows in and what leaks out

Every stop in the pipeline is a copy

Identity: whose keys your AI holds settled who your product acts as. This chapter follows what it keeps, because a pipeline that looks like one flow writes the same message down at four stops.

• Typed. The message travels to your model vendor, which typically holds API traffic for a window of days for abuse review; your contract names the number.
• Stored. The conversation lands in your transcript or ticket database, where retention defaults to forever because nobody chose otherwise.
• Embedded. Chunks of the text become vectors for retrieval, with the original text riding along as metadata, so the index holds a searchable copy of everything users ever pasted.
• Logged. The request and response land in application logs and monitoring tools, often as full payloads because that made debugging easier in week one.

Each stop keeps its own retention clock and its own list of readers: every stop in the pipeline is a copy, and every copy can leak on its own schedule.

The copies fail independently. In 2025, Lasso researchers showed Microsoft Copilot still answering with code from GitHub repositories that had since gone private, because a search cache had copied thousands of them while they were public: an indexed copy that outlived its source, with readers nobody had counted.

Data leaves through the answer, not only through break-ins

In August 2024, researchers at PromptArmor showed how Slack AI could be turned into the exit. The attacker posts one message in a public channel they create themselves, one the victim never opens. When the victim later asks Slack AI about their own private channel, an API key stored there in the demonstration, retrieval pulls the planted instructions into the same context as the private content, and the answer renders a clickable link with the secret packed into the URL. One click delivers it to the attacker's server. No access control failed; the answer itself was the exit. Slack AI had also just begun ingesting uploaded documents, and every source you add is another place where a stranger's text can wait.

The planted message is prompt injection, covered in depth in Injection: the input is the attack surface. What this chapter takes from the incident is the exits, because data leaves through doors that look like features:

• Rendered output. A link or image in an answer can carry data inside its URL, so the moment your product renders markdown it has an exit.
• Tool calls. A web fetch or webhook whose argument the model composes carries data out the same way.
• Sharing features. Shared ChatGPT conversations marked discoverable were indexed by Google, thousands of them, some deeply personal, until the option was removed in 2025.

In a shared index, filter by who is asking

Most B2B products put every customer's documents into one index, the multi-tenant pattern: many customers, one system. Similarity search ranks chunks by how closely they match the query and applies no permission check of its own, so unless the query filters by tenant and by the asking user's entitlements, the best-matching chunk wins even when it belongs to another customer. One shared index plus one missing filter equals another customer's data in your answer.

What the model can retrieve, the user in front of it can read, so enforce entitlements when the index is queried, not in the prompt after the data has reached the context.

A prompt line like "never reveal other customers' data" guards nothing, because by the time the model processes it, retrieval has already done the revealing. The wall is a retrieval-time filter: every query names who is asking, and the index returns only rows that person could open at the source.

Asana learned this in 2025, when a bug in its new AI integration let users see other organizations' projects and files; roughly a thousand customers were affected and the feature stayed offline for two weeks. The wall has to hold everywhere the copies live, not only in the index. If you build in a regulated category, Security and guardrails carries the audited version of these walls.

Write a retention number at every stop

Retention sounds like a compliance chore, but it is a security control you set with a product decision. For every stop, we write a number of days and one line defending it, and when two numbers are defensible, the shorter wins, because a copy that no longer exists cannot leak.

• Your stores. Transcripts, the index, the logs: each gets its own number, and "forever" needs the strongest defense of all.
• Vendor clocks. Your model vendor and monitoring vendor keep copies on schedules you did not pick, and those windows can stretch: in 2025, a court order in The New York Times copyright case required OpenAI to preserve consumer chats, deleted ones included, and the hold stood for months.
• The delete button. A user-facing delete is a promise every stop has to keep, so wire it through all four.

Logs deserve the most suspicion, because a log pipeline holds the same secrets as your database with none of its access controls and usually the longest default retention. In 2025, Wiz researchers found a DeepSeek database open to the internet with no authentication: over a million log lines of user chat history and API keys in plain text. People tell an assistant things they would never put on a form, so the transcript store needs database-grade protection even if nobody calls it a database.

Redact before embedding, scrub before logging

The cheapest defense shrinks what each copy contains at the moment it is written. Secrets have recognizable patterns, API keys, card numbers, government IDs, and the same pattern matching that secret scanners run in CI can run at your write paths: mask the value in the transcript, store a placeholder in the embedded chunk so retrieval stays useful, and log metadata, request IDs, latency, token counts, instead of payloads. None of this needs a platform purchase; a redaction pass is one function call at each stop. Run it at the first stop and the cold open never happens: the key gets masked on arrival, never reaches the other copies, and the security team gets a one-line answer instead of a week of archaeology.

Try it now

The drill takes about fifteen minutes and runs on your own product.

Pick one real user message. Pull yesterday's traffic and find a message with something sensitive in it. An email address qualifies, though something worse usually shows up.

Follow it to all four stops. Point Claude Code at your repository and ask where chat payloads are sent, stored, embedded, and logged. Confirm what it finds against the code: the model call, the database write, the index upsert, the log line.

Write the number and the readers at each stop. Record the retention in days and the list of people and vendors who can read that copy. "Forever" and "anyone with log access" are honest answers, so write them down.

Fix the worst stop this week. It is usually the log: cap its retention, scrub payloads at write time, or remove the debug flag that outlived its incident.

Chapter Summary

• An AI feature copies every message it handles to four stops: the model vendor (typed), your transcript database (stored), your retrieval index (embedded), and your logs (logged).
• Every stop is a copy with its own retention and its own readers, and every copy can leak on its own schedule.
• Data leaves through doors that look like features, rendered links, tool calls, and sharing options, not only through break-ins; the Slack AI disclosure showed one public-channel message steering a private value into a rendered link.
• Every new source your feature ingests, files, drives, tickets, is another place where an outsider's text can arrive.
• Similarity search applies no permission check, so in a shared index the filter for tenant and user entitlements has to run at retrieval time.
• What the model can retrieve, the user in front of it can read; a prompt instruction cannot take back what retrieval already delivered.
• Write a retention number at every stop and let the shortest defensible number win, remembering that vendors keep copies on clocks you do not control.
• Logs are a second database with the same secrets and none of the access controls, so scrub payloads before logging and redact secrets before embedding.
• The copies you keep are only half the exposure; The supply chain you didn't build follows the models, packages, and tool servers other people built into your product.

Sources

• PromptArmor, disclosure of a prompt-injection data-exfiltration path in Slack AI (August 2024).
• Lasso Security, with TechCrunch coverage, on Microsoft Copilot returning content from GitHub repositories that had been made private (February 2025).
• Asana disclosure and UpGuard analysis of the MCP server bug that exposed data across organizations (June 2025).
• Wiz Research, on DeepSeek's publicly accessible database holding over a million log lines including chat history and API keys (January 2025).
• Court filings and OpenAI statements on the order to preserve ChatGPT conversations, including deleted chats, in The New York Times copyright case (May to June 2025).
• OpenAI statement and press reporting on shared ChatGPT conversations indexed by Google and the removal of the discoverable option (July to August 2025).