Table of contents
An AI fail is any moment an artificial intelligence system produces output that is confidently wrong, plainly unusable, or legally binding in a way nobody intended. The best-documented ones are not jokes on Reddit — they are court decisions, product rollbacks and public apologies.
Key Takeaways
- A Canadian tribunal ordered Air Canada to pay $812 because its support chatbot invented a refund policy — the airline's own website counted as its word.
- A New York federal judge imposed a $5,000 penalty after lawyers filed a brief citing 6 judicial opinions that did not exist.
- Google paused Gemini's image generation of people on 23 February 2024, three weeks after launch.
- Microsoft took its Tay chatbot offline within 24 hours of release in 2016 after a coordinated attack on its learning loop.
- Public patience is thin: Gallup found 7 in 10 Americans oppose an AI data centre in their own area, 48% strongly.
- Nearly every high-profile failure traces to 1 of 5 root causes, and 4 of those 5 are fixable before launch.
The six kinds of AI fail (they are not all the same problem)
Lumping every bad output into "the AI hallucinated" hides the fix. A tool that invents a citation needs a different control than one that quietly drifts as its input data changes. The taxonomy below is the one we use with clients before anything reaches live traffic.
| Failure type | What the system does | Typical trigger | Control that actually works |
|---|---|---|---|
| Fabrication | States facts, citations or policies that do not exist | Open-ended prompt with no grounded source | Retrieval from an approved source set plus a human double-check on anything citable |
| Bias amplification | Reproduces or exaggerates skew in the source corpus | Past decisions treated as neutral ground truth | Subgroup testing before launch, not aggregate accuracy alone |
| Over-correction | Refuses reasonable requests or distorts specifics to force variety | Blunt safety tuning applied to every prompt | Case-split evaluation sets covering prompts that must be literal |
| Brittleness | Works in the demo, collapses on real-world input | Accents, noise, edge formats, adversarial users | Staged rollout with a measured fallback to a human |
| Unbounded authority | Makes commitments the business must honour | A chatbot allowed to discuss policy, price or refunds | Hard scope limits and canned answers for anything contractual |
| Silent drift | Degrades slowly after launch with no alarm | No monitoring on live model output | Weekly sampled review of production output against a rubric |
Ten documented AI fails and what each one cost
Every case below is drawn from a primary record — a tribunal decision, a court docket, a company statement or a national survey. We deliberately skipped the viral screenshots nobody can source.
| Case | Year | What went wrong | Documented consequence |
|---|---|---|---|
| Air Canada support chatbot | 2024 | Chatbot told a passenger bereavement fares could be claimed retroactively | Tribunal ordered $812 in damages on tickets of $1,630.36 |
| Fabricated case law filing | 2023 | A brief cited six invented opinions produced by a chatbot | $5,000 penalty plus mandated letters to every misattributed judge |
| Google Gemini image generation | 2024 | Tuning for variety produced historically inaccurate images and refusals | Feature paused after 3 weeks; public apology from a company SVP |
| Microsoft Tay | 2016 | Public learning loop exploited by coordinated abuse | Offline in under 24 hours; formal apology |
| Résumé screening models | 2018 onward | Historical hiring data encoded as a target to imitate | Abandoned tools and a decade of regulator scrutiny on automated hiring |
| Voice ordering in drive-thrus | 2024 onward | Accents, background noise and prank orders broke speech pipelines | Multiple chains rolled tests back to human-assisted ordering |
| AI-written publisher articles | 2023 onward | Unlabelled AI-generated content with factual errors and fake bylines | Mass corrections, retracted archives, lasting trust damage |
| Automated notification summaries | 2025 | Condensed headlines into statements the publisher never made | Formal complaints from news organisations; features restricted |
| Algorithmic pricing and valuation | 2021 onward | Forecast models kept buying while the market turned | Programme shutdowns and nine-figure inventory write-downs |
| Public trust in AI infrastructure | 2026 | Deployment outpaced consent from the communities affected | 70% of Americans oppose a local AI data centre, 48% strongly |
The chatbot that wrote its own refund policy
In February 2024 the British Columbia Civil Resolution Tribunal ruled that Air Canada owed Jake Moffatt $812, the gap between the bereavement fare and the $1,630.36 of full-price tickets Moffatt bought after their grandmother died. The chatbot had said a reduced rate could be claimed within 90 days of travel; a static page on the same site said the opposite. The airline argued the bot was "a separate legal entity that is responsible for its own actions". Tribunal member Christopher Rivers called that "a remarkable submission" and held the company responsible for everything on its website, chatbot included. The full decision is public, and it is now the cheapest governance training a support team will ever read.
The brief that cited six opinions nobody wrote
In Mata v. Avianca, a filing quoted judicial opinions named Varghese, Shaboon, Petersen, Martinez, Durden and Miller. None existed. On 22 June 2023 Judge P. Kevin Castel imposed a $5,000 penalty jointly on the responsible lawyers and ordered them to write to each judge falsely credited with a fake opinion, attaching the fabricated text. The docket record is the clearest artefact of the pattern: the model produced source-shaped output, and nobody opened the sources.
When safety tuning becomes its own failure
Google's own post-mortem is unusually candid. Gemini's image feature, built on the Imagen 2 model, was tuned so that generic prompts returned a range of people. Two things then went wrong: the tuning failed to account for prompts that should not show a range, and the model grew more cautious than intended, refusing anodyne requests. Google paused image generation of people on 23 February 2024, three weeks after launch, and coverage of the historical inaccuracies ran for weeks. The lesson is that a guardrail applied uniformly to every prompt is a product decision, not a neutral safety step.
The 2016 fail everyone still repeats
Microsoft's Tay learned from public replies. Within the first 24 hours, as Microsoft wrote in its apology, "a coordinated attack by a subset of people exploited a vulnerability", and the bot began posting abusive material. The company had shipped a similar assistant in China at scale without the same outcome — proof that the failure was contextual, not technical. Ten years later, brands still connect models directly to unmoderated public input and are surprised by the result.
Why AI systems fail: five root causes
Cause 1 — the data is a record, not a rule. Training data describes what happened, including every human decision that was unfair or simply lazy. Anything asked to predict that record faithfully will reproduce it. This is the mechanism behind algorithmic bias in consumer systems, and it is why aggregate accuracy is a misleading test.
Cause 2 — fluency is not knowledge. Large language models generate plausible sequences, so a fabricated citation looks exactly like a real one. Hallucination is a property of the method, not a bug awaiting a patch, which means the control has to sit outside the model.
Cause 3 — the objective is a proxy. Systems optimise the metric they are given: engagement instead of satisfaction, clicks instead of revenue, throughput instead of order accuracy. Each proxy eventually diverges from the real goal.
Cause 4 — no human in the loop where it matters. Human review is often placed where it is cheap rather than where the risk is. Reviewing tone while nobody verifies a legal claim is theatre.
Cause 5 — no monitoring after launch. Most damage is done by tools that worked on day one and were never sampled again. NIST's AI Risk Management Framework, released 26 January 2023 for voluntary use, exists largely to force this discipline: govern, map, measure and manage, continuously.
AI fails by business function
The same five causes surface differently depending on where the system sits. This table is a fast way to run a risk review across a company without a formal audit.
| Function | Most common AI fail | Who notices first | Cheapest guardrail |
|---|---|---|---|
| Customer support | Invented policy, price or refund promise | The customer, in writing | Block policy questions; hand off with the transcript attached |
| Content and SEO | Unverified statistics and dead citations | A reader, a journalist or a competitor | No number ships without a live source URL |
| Paid media | Automated creative or copy that breaches platform policy | The ad platform, via disapproval | Pre-flight policy checklist per channel |
| Sales | Personalisation that merges the wrong records | The prospect | Field-level validation before send |
| Hiring | Screening models trained on biased history | A regulator or a candidate | Subgroup testing plus documented human decision rights |
| Finance | Forecast models blind to a regime change | The next quarter's results | Scenario bounds and a manual override threshold |
| Product images | Generated image contradicts the physical product | Returns and review volume | Ban generated imagery on product-truth surfaces |
| Operations | Speech and vision systems failing on real-world noise | Frontline staff | Fallback to human within two failed attempts |
| Legal and compliance | Fabricated authorities in a filing or contract | A judge or counterparty | Mandatory citation verification, no exceptions |
| Healthcare and safety | Recommendations outside the validated population | Clinicians | Restrict use to the documented indication |
| Translation and localisation | Fluent output that inverts the meaning | Local market teams | Native-speaker review on public copy |
| Analytics | Confident narrative built on a broken data pipeline | Nobody, for months | Automated freshness and volume alerts on every source |
What an AI fail actually costs
The direct invoices are small and the indirect ones are not. Air Canada's damages were $812; the reputational cost of the ruling, quoted in every AI governance deck since, is far larger. The $5,000 sanction in Mata was trivial next to the professional consequences of the order to write to six judges. Reputational exposure compounds because the artefact is permanent — a screenshot, a docket entry, a cached page.
Regulatory cost is rising in parallel. The EU's regulatory framework for AI ties obligations to risk tier, so the same chatbot carries different duties depending on what it decides. Sentiment is the third cost line: Gallup reported in May 2026 that seven in ten Americans oppose an AI data centre in their local area, with 48% strongly opposed, and Pew Research Center found in July 2026 that 43% of Americans consider US leadership in AI extremely or very important while overall views tilt negative. Consumer tolerance for a visible AI mistake in 2026 is lower than the internal enthusiasm for shipping one.
A seven-step pre-launch check that prevents most of them
Step 1 — write down the decision the system is allowed to make. One sentence. If it includes price, policy, eligibility or safety, add a human approver now.
Step 2 — build an adversarial test set of at least 50 prompts. Include the prompts a bored teenager would try, the ones a lawyer would try, and the ones only your worst-tempered customer would try.
Step 3 — test subgroups, not just averages. Report accuracy by segment, accent, region and product line. An 88% aggregate can hide a 40% failure rate in one cohort.
Step 4 — ground every factual claim. Retrieval over an approved source set, with links surfaced to the reviewer. Anything without a source is treated as unpublished.
Step 5 — cap the blast radius. Start at 5% of traffic, define the rollback trigger before launch, and make sure one named person can switch it off.
Step 6 — sample production output weekly. Twenty real outputs, scored against a written rubric by a human who did not build the system. This is how drift gets caught.
Step 7 — log everything a user was told. The Air Canada case turned on a screenshot. Your own transcripts should be easier to retrieve than the customer's.
Teams that run this checklist typically find 2 or 3 issues serious enough to delay launch — which is the entire point. Our data intelligence and growth marketing teams run exactly this sequence before any client-facing automation goes live, and the delay is measured in days against a remediation cycle measured in quarters.
What AI fails mean for marketers specifically
Marketing is where machine output meets the public fastest, with the least review. Three habits carry most of the risk: publishing machine-written copy without verifying its numbers, letting a bot speak about policy, and using generated imagery on surfaces where customers expect the literal product.
The mitigation is unglamorous. Every statistic in a published piece carries a live source. Every claim about a product is checked against the spec sheet, not the model's memory. Every scripted reply that touches money routes to a person. Then double-check the boring things: dates, names, prices, and whether the cited page still exists. Automations degrade quietly for months, and a full archive of machine-written pages with broken citations is much harder to fix than a slow publishing calendar. Browse our marketing blog for the sourcing standards we apply, or talk to us if you need an existing automation workflow reviewed before it becomes the case study.
FAQ
What is an AI fail?
An AI fail is output from an artificial intelligence system that is factually wrong, biased, unusable or unintentionally binding on the business that deployed it. The category spans fabricated citations, image generation that misrepresents history, chatbots inventing refund policies, and forecasting models that keep making decisions after conditions change. What makes it a fail rather than an error is that the system presents the output with full confidence and no signal to the user that it is unreliable.
What is the most famous AI fail?
The two most cited are the Air Canada chatbot ruling of February 2024, where a tribunal ordered the airline to pay $812 because its chatbot invented a bereavement-fare policy, and Mata v. Avianca in 2023, where a US federal judge imposed a $5,000 penalty over a brief citing six non-existent judicial opinions. Both are widely quoted because the primary records are public, which makes them far more useful than a viral screenshot.
Why do AI systems fail if the technology keeps improving?
Capability and reliability improve on different curves. Better models produce more fluent output, which makes wrong output harder to spot rather than less frequent. The failures that reach the news are almost never about raw model quality — they come from missing grounding, an unbounded scope, a proxy metric, absent human review at the risky step, or no monitoring after launch. Those are deployment decisions, so they do not get solved by the next model release.
How do I stop my team shipping an AI fail?
Write down the single decision the system may make, test it against at least 50 adversarial prompts, measure accuracy by subgroup rather than in aggregate, ground every factual claim in an approved source, launch to 5% of traffic with a named rollback owner, and review 20 sampled production outputs every week. Most teams running this sequence find two or three launch-blocking issues, which is cheaper than discovering them publicly.
Are AI fails a legal risk for my company?
Yes. The Air Canada decision established that a company is responsible for what its chatbot tells customers on its own website, and courts have sanctioned professionals for filing AI-fabricated sources. Risk-tiered regimes such as the EU framework attach obligations to what the system decides, not to how it was built, so the practical exposure depends on whether your automation touches price, eligibility, employment or safety.
Sources
British Columbia Civil Resolution Tribunal, Moffatt v. Air Canada, February 2024 · CBC News reporting on the same decision · CourtListener docket, Mata v. Avianca, Inc., opinion and order of 22 June 2023 · Google, "Gemini image generation got it wrong. We'll do better", 23 February 2024 · The Verge coverage of Gemini image generation · Microsoft, "Learning from Tay's introduction", 25 March 2016 · NIST AI Risk Management Framework, released 26 January 2023 · IBM, AI hallucinations explainer · Brookings Institution on algorithmic bias detection and mitigation · European Commission, regulatory framework for AI · Gallup, artificial intelligence topic page, May 2026 · Pew Research Center, artificial intelligence topic page, July 2026. All sources accessed August 2026.
