AI Product Guardrails: Incidents, Architecture and a 90-Day Plan

Published: 3 November 2026 | Updated: 4 November 2026

Quick answer: AI product guardrails are the controls that decide what an AI assistant may commit to, which systems it may read or write, and what it does when it isn’t sure. Rules that must hold under pressure belong in code, and the prompt handles tone and scope. This guide maps eight real incidents, including Replit, Air Canada, Cursor, DPD and Chevrolet, to a six-layer architecture, eval suites and a 90-day plan.

Key takeaway: If a rule has to hold when a user pushes against it, such as a refund limit or a production freeze, enforce it in code. Use the prompt for tone and scope.

Who this is for: product managers, engineering leads, and security and compliance owners shipping chatbots or agents that handle money, accounts or regulated advice, especially in banking and other BFSI settings.

When to use it: before external users get access to an assistant that touches money, accounts or regulated advice, or when auditing an assistant or agent that is already live.

Use cases covered: customer-support chatbots, bank and share-registrar assistants, and agents with write access to production systems.

Background

Air Canada’s chatbot told a grieving customer he could claim a bereavement fare after travelling, which the airline’s own policy page ruled out. Replit’s coding agent deleted a production database during a code freeze. Both trace back to product decisions about scope, permissions, which sources to trust, and whether anything checks an answer or action before it goes out.

We worked through the same three questions on two recent engagements, for a Japanese bank and for a large Indian share registrar: what the assistant may commit to, which systems it may read or write, and what it does when it isn’t sure. A paragraph in the system prompt and a moderation API call can’t settle any of them.

If a rule has to hold when a user pushes against it, such as a refund limit or a production freeze, enforce it in code. Use the prompt for tone and scope.

The full starter code for everything below (policy files, prompt templates, the six layers, eval suites and a CI workflow) is on GitHub at bluetickconsultants/ai-gurdrails-starter. This article shows one policy file and one prompt template; the repo has the rest.

Eight incidents

Each row gives what happened, the root cause, and the control that would have caught it.

Chevrolet and DPD are the only two where users deliberately pushed a bot off script. Replit, McHire and Air Canada cost money or ended in a ruling, and access control, environment separation and a checked policy source would have prevented each of them.

Five failure modes

Grouped by cause, the incidents fall into five failure modes. Each needs its own control and owner, so name them in the PRD.

1. Fabrication. Air Canada, Cursor and NYC all had bots stating a policy or legal fact the organisation never published. The fix sits with product and with whoever maintains the knowledge base.
2. Manipulation. At Chevrolet the user typed the override directly into the chat. In enterprise products, indirect injection is harder to spot: instructions are planted in a document, email or web page your system retrieves. Security should own the defence.
3. Overreach. Replit’s agent ran destructive commands it was never meant to run. Any agent with write access can do the same, so permissions belong to platform engineering.
4. Bad sourcing. Google’s AI Overviews summarised a Reddit joke and an Onion article accurately. Source trust is a data and search problem.
5. Drift. DPD’s bot changed behaviour after a system update and customers found it first. Release owners need an eval gate.

For each user-facing intent, list which of the five can occur and point to the code that prevents it. Intents protected only by prompt text go on the backlog.

The six-layer architecture

Map your product onto the six layers below. For each layer, write down which decisions code makes and which a model makes; routing and permissions should always be on the code side.

guardrail architecture · 6 layers, 3 deterministic off-ramps

Each side exit (refusal, approval, escalation) is triggered by code reading a model’s structured verdict, and shows text or a queue that was set up in advance.

• Redact PII in L1, before any model sees the text. With a hosted model, redaction after the call does nothing for DPDP or APPI exposure, because the data has already left your network.
• The L2 router is a small model that returns intent, risk and an injection flag as one JSON object. Low-risk turns skip the L5 grounding check, and that skip accounts for most of the cost difference between the two paths.
• In L3, content from outside your trust boundary (web pages, uploaded PDFs, customer emails) is wrapped in a tag and quoted as data. Nothing retrieved is concatenated into the instructions.
• No prompt takes part in the L4 decision. Permissions, environments and limits are YAML checked in code, and the model can only submit a request.
• Run L5 as a separate call with its own prompt. A call that both drafts and grades an answer is biased toward passing it.
• Keep L6 traces for every layer. They are how you tell a false block from an attack, and how an incident becomes a test case.

Repo structure

Keep guardrails in their own package, versioned and tested like any other service. Prompt strings scattered through app code can’t be audited, A/B tested or rolled back. This is the layout of the starter repo:

• Policies are data, and prompts reference them by ID. The system prompt says “follow policy refund.v2” and the rule sits in YAML, so legal and compliance review a short file instead of the whole prompt.
• Refusal and escalation wording lives in responses/. The model flags that a refusal is needed and code picks the approved text, which matters most on money, health and legal topics.
• Treat evals/incidents/ as append-only. A production fix is complete when its test case is in that folder and CI passes.

Prompt template: the grounding check

The repo has four prompt templates (system prompt, input router, grounding check, output judge) plus the tool gate, which is code with no prompt. The grounding check is the one shown here because it targets the failure behind Air Canada and NYC. It lives at prompts/verifiers/grounding_check.md and runs as a separate call from generation.

This check targets the kind of failure seen at Air Canada and NYC. On a share registrar’s assistant, high-risk questions include dividend payment dates, the document list for a transmission request and KYC deadlines. Run the check on every answer in that category.

You are checking an answer against source documents. You did not write the answer.

<sources>
{{sources}}
</sources>

<answer>
{{answer}}
</answer>

Split the answer into atomic claims. For each claim return:
{"claim": "...", "supported": true or false, "source_id": "..." or null, "contradicted": true or false}

Rules:
- A claim is supported only if a source states it or directly implies it.
- A claim that any source contradicts is "contradicted": true, even if another source agrees.
- Numbers, amounts, dates, time windows and eligibility conditions must match the source exactly.
- A citation such as [policy fee.v3] is supported only if that policy_id appears in the sources.

Return JSON only:
{"claims": [...], "verdict": "pass" or "fail"}

"verdict" is "fail" if any claim about money, fees, eligibility, deadlines or legal duty is
unsupported or contradicted.

On fail, regenerate once with the failing claims listed, then fall back to the escalation response. Don’t rely on a disclaimer instead: NYC’s MyCity bot displayed one while giving answers that contradicted city law.

If the verifier returns anything that doesn’t parse, the pipeline treats it as a fail. The system prompt, input router, output judge and tool gate are in the repo under prompts/ and layers/l4_tools.py, each with tests.

Policy file: product_policy.yaml

This is a trimmed version of the product policy for a sample bank support assistant. Each block names the layer that enforces it, so a reviewer can trace a rule to the code that applies it. Legal and compliance review this file, not the prompts.

# policies/product_policy.yaml
version: 2026.10.2

scope:                             # enforced in layers/l2_router.py
  intents:
    card_status:
      description: Delivery or activation status of the customer's own card
      examples: [track, new card, card delivery, where is my card]
    fee_schedule:
      description: Published fees and charges for accounts and cards
      examples: [annual fee, fee, charges]
    dispute_how_to:
      description: How to raise a dispute on a card transaction
      examples: [dispute, chargeback, unauthorised transaction]
    branch_hours:
      description: Branch opening days and times
      examples: [branch, timings, opening hours]

high_risk_intents:                 # strict path: grounding check on every answer
  - fee_schedule
  - dispute_how_to

never_generate:                    # always escalated to a human, never answered by the model
  refunds_or_waivers:   {examples: [refund, waive, waiver, reverse the charge]}
  interest_rate_quotes: {examples: [interest rate, fd rate]}
  credit_eligibility:   {examples: [loan eligibility, credit limit, pre-approved]}
  legal_or_regulatory_advice: {examples: [legal, lawyer, ombudsman]}

commitments:
  allowed: false                   # output judge blocks any promise of money or terms

disclosure:
  ai_label: required               # appended to every reply, including refusals
  label_text: "This answer was written by Northwind Bank's AI assistant."

The repo has three more policy files: tool_permissions.yaml (environments, argument schemas, destructive verbs, approvals), source_trust.yaml (trust tiers and quarantine patterns) and pii_rules.yaml (redaction for India, Japan and generic identifiers). One detail from tool_permissions.yaml applies to every product with customer data: the customer ID is injected by the server. If the model could choose which record to fetch, any user could ask for someone else’s data by wording the request carefully. McHire’s exposure came from the same kind of bug, an insecure direct object reference, in a web admin panel.

Evals and red-team suites

DPD said the AI part of its chat had worked for years before a system update caused the incident. Gating every release on an eval run catches that kind of regression before it ships.

Starting suites are below. The sizes and pass bars are defaults; adjust them once you have your own failure rates.

• Run golden, grounding and incident suites on every PR that touches prompts/, policies/ or the model version. Red-team suites take longer and can run nightly and before each release.
• Grade with code where possible: exact match on policy IDs, JSON schema checks, tool-call logs. An LLM judge is fine for open-ended quality if you recalibrate it against human labels every quarter.
• Paste the eval diff into the PR description, for example “Golden 96.1% to 95.8%, injection unchanged, 2 new grounding fails on fee_schedule”, so the reviewer sees which suite moved.
• Pin model versions in config and put provider snapshot updates through the same gate as prompt changes.

The 90-day roadmap

External users shouldn’t reach money, accounts or regulated advice until all six layers are live. The plan below spreads the build over 90 days, with a gate at the end of each phase.

90-day guardrail roadmap · 3 phases, 3 gates

Policies are written in phase 1 because prompts and evals need a written rule to test against. Shadow mode at gate 1 runs the router and checks on live traffic and logs their decisions without blocking, which gives a false-block baseline. Red-teaming is scheduled for phase 3, once enforcement exists to test.

For a product that is already live, run phases 1 and 2 together and build the tool gate first, because destructive agent actions are the hardest to reverse.

Metrics, latency and cost

Guardrails add latency and token cost. Agree a budget for both with stakeholders before launch.

Most of the cost saving comes from routing. If only high-risk intents go through the full grounding check and output judge, the heavy path runs on a minority of turns. Run the router and judge on a small, fast model and keep the larger model for generation and strict-path grounding. Cache grounding results for repeated FAQ answers, keyed on question cluster and source version.

Checks that don’t depend on each other can run in parallel. PII redaction and the router can run together, and on the output side the brand judge and format validation can run alongside the grounding check. Added latency is then set by the slowest check.

Metrics to review weekly:

Measure false blocks by sampling the refusal log each week and having a reviewer label each sampled turn as a correct or wrong refusal.

Pre-launch checklist

Tick every item before external users get access. The incident behind each item is in brackets.

• The assistant identifies itself as AI in every channel (Cursor)
• Answers on price, refunds, eligibility and legal duty are either deterministic or pass a grounding check against an authoritative source (Air Canada, NYC)
• Questions about policies that don’t exist produce “not covered” or escalation, verified by eval (Cursor)
• Retrieved content and tool output are wrapped as untrusted data, and indirect injection evals pass (Chevrolet, enterprise RAG)
• Retrieval sources have trust tiers, and high-risk intents only use authoritative ones (Google AI Overviews)
• The agent cannot reach production with write access, and destructive verbs need human approval outside the chat (Replit)
• Freezes, limits and budgets are system state checked in code, not instructions in a prompt (Replit)
• Customer and tenant IDs are injected server-side; the model never picks whose data to fetch (McHire)
• Admin panels, test accounts and transcript stores went through a normal security review (McHire)
• Prompt, policy and model changes are gated by an eval run in CI (DPD)
• Every refusal and escalation message is pre-written and brand-approved
• There is a human queue behind escalations, with an owner and an SLA
• A kill switch can disable the generative path and fall back to deterministic answers in minutes

Keep this checklist next to evals/incidents/. Each new production failure should add a line to both.

Frequently asked questions

What are AI product guardrails?

AI product guardrails are the controls that settle what an AI assistant may commit to, which systems it may read or write, and what it does when it isn’t sure. Routing, permissions and limits are enforced in code. The prompt handles tone and scope.

Why isn’t a system prompt enough to keep an AI assistant safe?

A prompt can be overridden by a determined user or ignored by an agent. Replit’s code freeze was only a chat message, and Chevrolet’s bot followed a user who told it to agree with everything. Any rule that has to hold under pressure, such as a refund limit or a production freeze, should be enforced in code.

What are the six layers of an AI guardrail architecture?

L1 redacts PII before any model sees the text. L2 is a small router model that returns intent, risk and an injection flag. L3 wraps retrieved content as untrusted data. L4 checks permissions, environments and limits in code. L5 runs a separate grounding check and output judge. L6 keeps traces so an incident becomes a test case.

Which real incidents show why guardrails matter?

The article covers eight: Replit’s deleted production database, the McHire admin exposure, Cursor’s invented login policy, Google AI Overviews’ glue-on-pizza answer, NYC’s MyCity bot, Air Canada’s bereavement-fare ruling, DPD’s swearing bot and Chevrolet’s $1 Tahoe. Each is matched to a root cause and the control that would have caught it.

How do you test guardrails before a release?

Run eval suites in CI: golden, grounding, injection, overreach, fabrication bait, tone and incident cases. Golden, grounding and incident suites run on every pull request that touches prompts, policies or the model version, and red-team suites run nightly and before each release. Pin model versions so provider updates go through the same gate.

How long does it take to add guardrails to an AI product?

The article lays out a 90-day plan in three phases, with a gate at the end of each. External users shouldn’t reach money, accounts or regulated advice until all six layers are live. For a product that is already live, run phases 1 and 2 together and build the tool gate first.

Sources

• Air Canada tribunal ruling coverage, CBS News
• Air Canada ruling detail, MobileSyrup
• Replit database deletion, Fortune
• Replit incident, TechTarget
• Cursor support bot, eWeek
• McHire exposure, BleepingComputer
• Chevrolet of Watsonville, AI Incident Database
• NYC MyCity investigation, The Markup
• MyCity shutdown plan, The Markup
• DPD chatbot, ITV News
• Google AI Overviews changes, Engadget