The demo is going beautifully. The team has spent six weeks on a support chatbot for the bank's retail customers, and on the staging screen it is handling a question from real transcripts: a customer asking whether she should move her savings into the bank's index funds before interest rates drop. The answer is warm, specific, and accurate. It explains how the funds work, compares current yields, and suggests she make the move before the next rate decision. Someone says it reads better than the scripted flow ever did.
Then the compliance officer, quiet until now, asks one question. Did it just give investment advice?
The room goes still because nobody is certain. Explaining how the funds work and comparing their yields is education, which the product may do. Telling one customer what to do with her savings is, in a licensed business, a sentence that can belong to a different legal regime entirely. Every team in a regulated industry makes this discovery early. The gap between a helpful answer and an illegal one can be a single sentence, and the model that produced it cannot reliably flag which side of the line that sentence landed on.
What makes an industry high-stakes
This part of The Frontier level teaches you to ship AI in rooms like that one. Finance, healthcare, insurance, legal, and government are the canonical cases, and what makes them different is structural rather than cultural.
- A regulator with rule-making and enforcement power. A public agency writes binding rules and backs them with fines, license revocations, and bans, so your release process answers to someone who has never attended your standup.
- Mandatory record-keeping. Customer communications must be retained and produced on request, which makes every chatbot transcript a record an examiner can read back to you years later.
- Licensed roles. Some communications may lawfully come only from a person who holds a license, such as an adviser, a physician, or an attorney, and the duties travel with the person rather than the software.
- Penalties that attach to the communication itself. In most software a wrong answer is a quality bug. In these industries a single sentence can violate a statute even if nobody acts on it.
The same posture appears anywhere a mistake is expensive and public. If your product can produce a sentence that costs someone money, health, or legal standing, this part applies to you whether or not a public agency has your name yet.
Where education ends and advice begins
The most important boundary in this part is the advice line, and a version of it runs through every regulated room.
- Finance. A product may educate and inform, explaining what an index fund is, showing published yields, defining the jargon. Advice, a recommendation to a specific person about a specific action, carries suitability and best-interest duties (obligations to fit the recommendation to that person's circumstances and to put her interests first), and those duties attach to licensed people and licensed processes.
- Healthcare. The line runs between informing and diagnosing. A product may explain what a symptom can indicate, and it may not tell a patient what condition she has.
- Legal. The line runs between giving information and practicing law. Describing what a statute says is allowed, but applying it to one person's specific facts is practicing law, which only a licensed attorney may do.
The model cannot hold a license. It produces educational sentences and advisory ones with equal fluency, and no duty stands behind either.
Because no license sits behind the model, the advice line has to be enforced in the product rather than left to chance in whatever the model produces.
In a regulated industry, the law writes your must-never list
Every product has a must-never list, the set of behaviors that are unacceptable from any user, under any prompt, on any day. A consumer app writes that list from brand judgment. In a regulated industry the list is written for you, in statute and rule, and each entry arrives with a penalty attached.
That changes the engineering target. Ordinary quality is an error rate you manage, while the must-never list is a set of behaviors you prevent outright with controls, because the first public violation is the one an enforcement action quotes.
What you claim about the AI is regulated speech
The rules reach your marketing before they reach your model. In March 2024 the SEC brought its first enforcement actions against AI-washing, the practice of overstating how much AI a product actually uses, fining the investment advisers Delphia and Global Predictions a combined 400,000 dollars for overstating their use of AI. The cases turned on what the firms claimed rather than on anything a model produced, so treat the landing page and the sales deck as regulated speech held to the same standard as the product's answers.
Why a disclaimer does not protect you
A public failure makes the stakes concrete. In March 2024, reporting revealed that New York City's MyCity business chatbot was telling employers they could take a cut of their workers' tips and could fire workers for reporting harassment, both of which are illegal. The city kept the bot online with a disclaimer.
Where regulation is heading
The direction of travel is global. The EU AI Act entered into force in August 2024, with obligations phasing in through 2027. It classifies uses of AI into risk tiers, categories ranked by how much harm a failure can cause, and attaches duties such as human oversight, logging, and documentation to high-risk uses. The practices this part teaches are the practices those regimes expect.
The path through this part
From here the part walks the work end to end: you will set the boundaries your product may never cross, choose and constrain the model, govern the context it draws on, build the gates that keep a bad answer from reaching a customer, test for fairness, assemble the evidence regulators and buyers ask to see, and run the governance that keeps all of it true after launch. One scope note before we start: this part teaches the product work, and your counsel and compliance teams rule on the law.
Try it now
Write a one-page stakes memo for your own product. It is paper only, takes about 15 minutes, and becomes the reference document for the rest of this part.
Name your regulator. Write down who actually governs your product. If no public agency claims jurisdiction, name the de facto regulator with the power to stop your release: app store review, enterprise procurement, or your biggest customer's security team. Someone already holds a veto over your launch, and the memo begins with their name.
Write your must-never list. List at least five behaviors your product must never exhibit, in concrete language, so an entry reads like recommending a specific fund to a specific customer rather than something vague like giving bad advice.
Attach a penalty to each entry. Next to every line, write what happens if the behavior occurs once in public: a fine, a platform ban, an audit, a lost deal. Any entry where you cannot name the penalty is your first question for your compliance partner.
Read the page back as a spec. Every line is a requirement your controls will have to enforce, not a hope that the model produces the right output on the day it matters.
Chapter Summary
- An industry is high-stakes when it has a regulator with real enforcement power, mandatory record-keeping, licensed roles, and penalties that attach to the words your product says.
- In these industries a single sentence can break the law even if no customer acts on it, so a wrong answer is not just a quality bug.
- Every product needs an advice line, the point where helpful information turns into a recommendation that only a licensed person may give.
- The model produces educational sentences and advisory ones with equal ease, so you enforce that line in the product rather than rely on its output landing on the right side of it.
- In a regulated industry your must-never list is written for you in law, with a penalty on each entry, so you prevent those behaviors with controls instead of managing them as an error rate.
- What you claim about your AI in marketing is regulated too: overstating how much AI you use has already drawn fines.
- A disclaimer does not make an illegal answer legal, because regulators treat the output itself as the communication. Build controls that change what the product says.
- Your stakes memo names the regulator, the must-never list, and the penalty behind each entry. Next, Setting boundaries: what your AI may and may never do turns that page into enforced behavior.
Sources
- The Markup and Associated Press (2024). Reporting on New York City's MyCity business chatbot.
- SEC (2024). Enforcement actions against the investment advisers Delphia and Global Predictions for overstating their use of AI.
- European Union (2024). The EU AI Act, in force August 2024 with obligations phasing in through 2027.