The 10-Point AI Safety Checklist Before Launching a Customer-Facing Chatbot

For three weeks your support bot has been quick, polite and almost flawless. Then a stressed customer mentions a prescription, and the reply comes back with specific dosage advice delivered in a confident, helpful tone. That single message can undo months of careful work. Chatbots fail loudly, and screenshots travel fast.

The checks below are the ones worth doing before real customers arrive. None of them require a large team. They require decisions, written down, and evidence that someone tested them.

1. List the answers your bot must never give

Before you tune tone or personality, write a short refusal list. It should name the categories where a wrong answer causes real harm, plus the ordinary ones that cause embarrassment. Typical entries:

• Medical dosing, diagnosis or symptom interpretation
• Legal advice and anything resembling a contract term
• Financial recommendations, credit or investment guidance
• Instructions involving weapons, chemicals or self-harm
• Anything about a minor's account, location or identity
• Claims about competitors, prices or guarantees that are not already published

Then test the refusals with prompts a real person would type: indirect phrasing, hypotheticals, "asking for a friend", roleplay framing, and requests in another language. A refusal that only works when the question is asked politely is not a refusal. When the bot declines, it should acknowledge the question, say plainly that it cannot help with that, and offer the next step, usually a human.

2. Test for bias using your customers' own words

Fairness problems rarely show up in the demo. They appear when the same question gets a different quality of answer depending on who is asking. Build a small test set from real transcripts, support tickets and chat logs, then vary the surface details: names, dialects, spelling errors, non-native grammar, regional slang, mixed languages.

Ask a blunt question: for the same underlying request, does the bot give equal length, equal warmth and equal accuracy? Watch for the bot treating some phrasings as suspicious, dismissive or low priority. Watch too for stereotyping when a name or location appears in the conversation.

Turn every failure into a permanent test

Keep a regression sheet. Each time you find a bad answer, add the prompt and the expected behaviour to the list and run the whole sheet before every release. It grows into the most useful document your team owns. Bias work is never finished, but it becomes routine.

3. Decide what conversations you are willing to keep

Privacy is a design decision, not a cleanup task. Long before launch, answer these in writing:

• Which parts of a conversation are stored, and for how long?
• Who inside the company can read full transcripts, and is that access logged?
• Do past conversations feed future training, tuning or evaluation?
• How are names, addresses, card details and account numbers detected and removed?
• What do customers see before they start typing, and can they ask for deletion?

Test redaction with deliberately messy input, including numbers split across messages and details buried in uploaded documents. Remember that anything placed in a system instruction can eventually be surfaced in a reply, so treat those instructions as public. If you operate across jurisdictions, data protection duties differ by country and change over time; get advice from a qualified professional rather than guessing.

4. Make the exit to a human obvious and fast

Customers forgive a bot that admits limits. They do not forgive one that traps them. Define the triggers that hand a conversation to a person: repeated misunderstanding, an explicit request, sensitive topics, account changes, refunds above a set value, or two failed attempts at the same question.

Then check the shape of the handoff. Does the agent receive the transcript and a summary of what was already tried? Is the wait time honest? Does the customer get told what happens next? Never dress the bot up as a person, and answer directly if someone asks whether they are talking to a human. Test the phrase "let me speak to someone" in several forms, including a typo, before launch day.

5. Assume people will try to break it

Some users will paste instructions into the chat, upload documents, ask the bot to ignore its rules, or send very long messages hoping something slips through. You cannot stop curiosity, but you can limit the damage. Keep user text strictly separate from your own instructions, give the bot only the permissions it needs for the current task, and require confirmation before anything that changes an account or sends money.

Run a short session where you play the attacker: hidden text inside an uploaded file, a link, a fake system message, a request in another language. Note what broke, fix what you can, and decide what you will accept as residual risk.

6. Watch the first weeks closely

Launch is the start of testing, not the end. Sample conversations daily at first, then weekly. Track a small dashboard: refusal rate, escalation rate, messages containing complaint words, and how often customers ask the same thing twice. Give one person the responsibility and the authority to switch the bot off, and know in advance how a rollback works. Rate limits and a simple kill switch are worth more than an extra feature.

The 10-point launch checklist

Run this in order and keep the evidence. If you cannot answer a line with something you tested, you are not ready.

1. A written refusal list covering health, legal, financial, safety and minors.
2. Evidence that refusals hold up under indirect, translated and roleplay phrasing.
3. A bias test set built from real customer language, with results reviewed by someone outside the build team.
4. A regression sheet that runs before every release.
5. Documented retention, access and redaction rules for transcripts, reviewed by a qualified professional where the law applies.
6. Verified redaction on messy, multi-message input and uploaded files.
7. Defined escalation triggers, with the bot's limits stated honestly to customers.
8. A tested handoff that passes context to a human and gives the customer a clear next step.
9. Notes from an abuse session covering injected instructions, long inputs and file uploads.
10. A named owner, a monitoring view, and a working way to switch the bot off.

Sign off with evidence rather than confidence. A checklist you can show someone is the difference between launching a helpful assistant and launching an incident.

Photo: Matheus Bertelli / Pexels