Defense in layers: what the prompt cannot stop

That team was not careless; it reached for a defense that has never once held, and this chapter is about what to build instead.

Why asking the model nicely fails

A system prompt shapes behavior statistically. Instruction-following dominated the model's training, so it usually complies, but the right input can make some other completion more likely, and your rule and the attacker's text sit in the same context window with the same standing.

In July 2023, researchers at Carnegie Mellon and collaborating labs generated adversarial suffixes automatically, strings that read as line noise, and appending one to a blocked request reliably defeated the safety training of the major chatbots. Computed against open models, they transferred to closed ones the team never touched: months of safety tuning fell to strings a program produced in bulk. That is the cleanest evidence anyone has for the rule this chapter stands on: instruction-following is a behavior, not a contract.

The prompt is not a secret either.

• Snapchat's My AI had its system prompt extracted by users soon after launch in 2023; it now sits in a public GitHub archive beside prompts from dozens of other products.
• A study of more than two hundred custom GPTs pried the system prompt out of nearly every one and the uploaded knowledge files out of all of them.
• Major vendors now publish their own. Anthropic has published Claude's consumer system prompts since August 2024, and xAI moved Grok's prompts onto public GitHub in May 2025 after an unauthorized edit pushed one inflammatory topic into unrelated replies for a day.

Write the prompt as if it ships in your public docs, because in practice it does.

One layer never holds, so you stack them

Security engineering met this problem long before AI inherited it: when no single control is reliable, you stack controls so an attacker has to beat several in a row. The pattern is called defense in depth, and OWASP's Top 10 for LLM applications reaches the same verdict, listing mitigations that reduce prompt attacks and none that remove them. For an AI product, we build the stack four layers deep, from softest to hardest.

No layer in this stack has to be perfect, because each layer exists to catch what the layer above it let through.

The prompt earns its keep on cooperative traffic

Most of your traffic is not attacking you, and for that majority the prompt is the highest-leverage text you own: it sets tone, scope, format, and refusal style, and one edit reaches every conversation. The Grok incident shows the reach: one unauthorized line sent a deployed product sideways for a day, and xAI's announced fix was process, published prompts and reviewed changes, not better wording. Maintain the prompt like code, versioned, reviewed, and tested against your eval, and assign it no security jobs.

Classifiers cut the attack volume

Classifiers are small, cheap checks on every request: one reads the input for known attack patterns before the model does, another reads the output for secrets, policy violations, or your own prompt on its way out the door. They are fallible by construction, attackers iterate offline until something slips past, but fallible is not worthless. When Anthropic put constitutional classifiers in front of a model in February 2025, jailbreak success in automated testing fell from 86 percent to 4.4 percent, and 183 red-teamers spent more than three thousand hours chasing a 15,000 dollar bounty without finding a universal bypass. Read both halves: the flood became a trickle, and 4.4 percent still got through. A classifier buys volume reduction and an alarm bell, never a guarantee.

The deterministic floor is enforced by code, not the model

Below the classifiers sits the layer this chapter exists to sell you: rules enforced by ordinary code, outside the conversation entirely. The send tool accepts only addresses already on the ticket, the database credential is read-only, the API key stops at its daily budget, and outbound calls reach three approved hosts and nothing else. None of it is decided by the model, so no phrasing changes it: even when a hostile instruction works perfectly, the action it asked for stays impossible. OpenAI built ChatGPT's code execution this way from its 2023 launch: users talked the model into running arbitrary code, and the sandbox had no internet access, so nothing the code produced had anywhere to go.

The prompt is a request; the deterministic floor is a fact. Wording shifts what the model usually does, while scopes, allowlists, caps, and egress rules decide what can happen at all.

Human gates hold the actions you cannot take back

Some allowed actions stay catastrophic when wrong: moving money, deleting records, emailing outsiders, deploying to production. The control for those is a person who approves, placed where The autonomy ladder: place every action deliberately told you to put one, on the irreversible rungs. OpenAI's Operator shipped in January 2025 built this way: it fills a cart on its own, stops for confirmation before any purchase, makes the user watch every step on banking pages, and declines bank transfers outright. If your product lives in a regulated domain, Security and guardrails covers the version examiners will ask to see.

Every bypass becomes a test case

Layers decay quietly: a prompt rewrite weakens a refusal, a model upgrade changes what a classifier sees, a new tool widens the floor. Whenever anything gets past a layer, in a pen test, a red-team session, or a production transcript, that exact input becomes an eval case, with the layer's correct response as the bar; the screenshot from the opening becomes a permanent case the day you find it. The regression gate: no change ships blind then carries the load, because every future rewrite and model swap has to beat every bypass you have ever caught.

Caps turn a catastrophe into a bad day

One member of the floor deserves a headline. Why attackers love AI products opened this part with the meter that runs on every request, and attackers have noticed the meter is yours. When an attacker got into Sourcegraph in 2023 with a leaked admin token, the early moves included raising API rate limits and opening a free proxy to the underlying model; the usage spike is what gave the intrusion away.

Caps do not stop any of this from starting; they decide what it costs once it has started. Capped, the worst case is a per-user, per-day number you chose in advance; uncapped, it is whatever the meter reads when a human finally looks.

Try it now

The drill takes about fifteen minutes and runs on your own product's threat model.

Pull up your top three threat lines. Take the three attacks you ranked highest in Threat-model your AI feature. If you skipped that chapter, write the three attacks you would try first against your own feature.

Name three defenses per threat. For each one, write a line per layer: the deterministic stop (the scope, allowlist, cap, or egress rule that makes the damage impossible), the classifier (the check that would flag the attempt), and the human gate (the approval before anything irreversible). Claude Code makes a quick sparring partner: paste each threat and its defenses and ask for the attack it would try next.

Circle every prompt-only row. Any threat whose entire defense is wording in the system prompt is, as of today, undefended. Expect to circle at least one.

Date the fix. For each circled row, pick the cheapest deterministic stop or human gate that closes it and put it in the backlog with a date inside this week. The wording can stay; the layer underneath it is the fix.

Chapter Summary

• A "never do X" line in the system prompt is one more piece of text in the context, not a rule the system enforces.
• Researchers proved the point in 2023 with automatically generated suffixes that defeated the safety training of major models and transferred to closed ones. Instruction-following is a behavior, not a contract.
• Treat the system prompt as public. Users extract prompts routinely, and major vendors now publish their own.
• Defend in layers: the prompt steers cooperative traffic, classifiers cut attack volume, the deterministic floor blocks what must never happen, and human gates hold the irreversible.
• The prompt is a request; the deterministic floor is a fact. Scopes, allowlists, caps, and egress rules hold no matter what the model produces.
• No layer has to be perfect, because each one exists to catch what the layer above it let through.
• A refusal you watched once is not a control. Count a threat covered only when a layer beneath the prompt would stop it.
• Every bypass you catch becomes a case in the regression gate, so patched holes stay shut through every rewrite and model swap.
• Spend caps decide in advance what abuse can cost, which turns a catastrophic incident into a bounded one.
• The layers are now worth attacking, and Red-team your product before strangers do shows you how to do that yourself before strangers volunteer.

Sources

• Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, Carnegie Mellon University and collaborators (July 2023).
• Community-maintained GitHub archive of leaked system prompts, entry for Snapchat's My AI (April 2023).
• Yu et al., Assessing Prompt Injection Risks in 200+ Custom GPTs, arXiv (November 2023).
• Anthropic release notes publishing Claude's system prompts, with press coverage by SiliconANGLE (August 2024).
• xAI statements and CNBC reporting on the unauthorized Grok prompt modification and the publication of Grok's prompts on GitHub (May 2025).
• Anthropic, Constitutional Classifiers: red-team and automated-evaluation results (February 2025).
• OpenAI, Operator launch announcement and system card (January 2025).
• OpenAI documentation and independent technical write-ups on the ChatGPT code-execution sandbox running without internet access (2023).
• Sourcegraph security update and BleepingComputer reporting on the leaked admin token incident (August 2023).
• OWASP Top 10 for Large Language Model Applications (2023, updated 2025).