Lecture 5. The AI Product: The Full Lifecycle — From Intent to Operation¶
Lecture 5 · ~90 min · 56 slides
01 · Hook: the paradox¶
"Let's not start with a definition. Let's start with a paradox.
Anthropic describes its own internal experience like this: tasks that used to take weeks of writing and reviewing code now take hours. An agent writes the implementation, passes the tests — and the code is nearly free.
At the same time, in the summer of 2025, a research group at MIT Media Lab published a report that the media picked up under the headline: 'ninety-five percent of organizations get zero return from generative AI.'
Both facts are true at once. And at first glance they contradict each other. Building is nearly free — and almost nobody is extracting value from it. How is that even possible?
I won't resolve the paradox right now — that's the payoff of the lecture, we'll come back to it at the end, with numbers. But let's fix the shape of it right away, because it holds up everything that follows: building has become nearly free, and turning that build into real value has not.
We're going to walk the entire product lifecycle — from intent, through discovery, design, build, measurement, operation — and at every phase we'll ask the same question: what did AI make cheap here, and what stayed expensive and human?
Think about it: if building became almost free — what became the real bottleneck of a product? Hold on to your answer, we'll come back to it more than once."
Sources: [1] Anthropic — 2026 Agentic Coding Trends Report · [2] MIT NANDA — "The GenAI Divide" (July 2025, v0.1)
02 · Cover and roadmap¶
"Lecture five. The AI product: the full lifecycle — from intent to operation.
Lecture four dissected the lifecycle of code. Today we zoom out: code is one phase — now dramatically cheaper — inside a much wider product lifecycle.
Let's be honest about you specifically: you're technically strong, you write code with AI assistants every day. But most product disciplines — customer development, design thinking, product experimentation, operations run like reliability engineering — you're most likely seeing for the first time. So each of today's six sections opens not with AI, but with the classical foundation: where the discipline came from and what problem it solves, without a single mention of artificial intelligence. Only after that do we ask what AI changes and where it breaks.
This isn't academic politeness. An engineer who only knows 'how to build fast,' but doesn't know why customer discovery or A/B testing exists, risks making a decision that's technically flawless — and pointless or dangerous from a product standpoint. The very structure of this lecture — foundation, then AI, then limits, then failure — applies the course's own thesis to itself: to judge whether AI belongs somewhere, you first have to know what existed before it."
03 · Lecture map = the loop¶
"Here's the map of today's lecture — and it's not drawn as a list, it's drawn as a circle. Six sections aren't six independent topics; they're six arrows of one loop. Discovery — where the hypothesis comes from. Design — how the hypothesis becomes an artifact. Build and Launch — how the artifact becomes a working product in production. Measure — how you verify whether it actually worked. Support and Operate — what happens when the product is live twenty-four seven. And Governance — how the organization decides where to put its capital.
The key point: the last arrow loops back into the first. Signal from operations and governance decisions changes what we investigate next in Discovery. This is a closed cycle, not a one-time linear process — and at the end of the lecture we'll come back to this same circle, but with a full understanding of what happens on every arrow."
04 · Bridge from Lecture 4 + the central question¶
"Lecture four showed: software development with AI is an engineering discipline built around a loop of human-owned artifacts. A spec turns into an architecture decision, that becomes a plan, the plan becomes a pull request, and operation produces an incident record that loops back as a new requirement into the spec. That loop lives inside a single phase of the product lifecycle — Build, which we'll dissect in Section 3.
Today's lecture goes up one level. The product as a whole runs its own loop, and the decision to go back to the start or kill the product is made not by an engineering criterion, but a product one: did the hypothesis about user value hold up?
The difference in scale matters. Lecture four answered 'how do I write the code I'm accountable for' — its unit of work was a pull request, a commit. This lecture answers 'where does the confidence to write that code even come from in the first place' — its unit of work is a hypothesis, an experiment, a release decision. Code is just one step, and now the cheapest one.
That gives us the central question of the lecture, the one we'll return to in every section. Write it down, it's load-bearing:
Now that AI has made building nearly free — what became the real bottleneck of a product, and at every phase of the cycle: which classical discipline still stands, what does AI speed up, and where does AI-first break?
Hold this question as a lens: not 'can AI be applied here' — you almost always can — but 'where did the bottleneck move when this arrow got cheaper.'"
05 · KEYSTONE: the feedback loop¶
"This is the key slide of the lecture — the whole logic that follows depends on it. Hold it in your head through all six sections.
The load-bearing idea: a product is a feedback loop. Discover, build, measure, learn, decide, discover again. And this is a shape that at least three people arrived at independently, from completely different fields, with no contact with one another.
Walter Shewhart and W. Edwards Deming, quality-control statisticians, formulated the PDCA cycle — plan, do, check, act — back in 1939 and the 1950s. John Boyd, a military fighter pilot, formulated the OODA loop in the 1970s — observe, orient, decide, act — as a model of who wins a dogfight: not the pilot with the faster plane, but the one who cycles through this loop faster. Eric Ries, an entrepreneur, formulated Build-Measure-Learn — build a minimally sufficient product, measure the market's reaction, learn — as a way to avoid spending months on a feature nobody wants.
A statistician, a fighter pilot, and an entrepreneur drew, in essence, the same blueprint, without coordinating with each other — a compelling argument that the feedback loop isn't a trendy methodology, it's a structural property of any system that learns under uncertainty.
There's a nuance to the same shape — the single loop and the double loop. A single loop corrects an error relative to a fixed goal, like a thermostat. A double loop goes deeper: it questions the goal itself. Hold this in mind — it foreshadows a failure from Section 4: training a model on human feedback is a mechanized single loop, and reward-function hacking is exactly its failure mode.
That's exactly why AI cannot 'cancel' the loop. It can change how much each arrow of the loop costs and how much you can trust it — but not the underlying necessity of observing reality, interpreting it, and deciding."
06 · KEYSTONE-2: the cost/trust asymmetry¶
"The second half of the keystone — the central analytical instrument of the lecture: AI changes the cost and the trust of every arrow of the loop asymmetrically, not uniformly.
Three cases. The 'build' arrow — cost has nearly collapsed to zero: writing code, a design draft, a first research prototype are orders of magnitude cheaper and faster than five years ago. We'll dig into this in Section 3.
The 'measure' and 'learn' arrows — cost stayed the same, but trust in the result dropped: the measurement instrument itself became probabilistic and vulnerable to metric manipulation. Section 4 will unpack this in detail.
The 'observe' and 'orient' arrows — got faster, but more exposed to attack: the speed of observing a system in production went up, but the very ability to react quickly turns into a vulnerability if the observed data can be poisoned. We'll open this up in Section 5.
From here comes the meta-pattern that will repeat in every one of today's six sections, literally on the same template: every phase of the product lifecycle has a classical discipline, and AI does not cancel it. AI changes how much that discipline costs and how much you can trust it without verification — but it does not remove the need for the discipline itself. Customer Development doesn't get replaced by 'ask the model what the user thinks.' A product experiment doesn't get replaced by 'the model predicted the metric would go up.' SRE-style operations doesn't get replaced by 'the agent will figure it out.'
In every section we'll ask one question: what does the classical discipline make reliable, what does AI speed up — and at what point does speed become a trap, because trust doesn't keep pace with it.
On which arrow of your most recent task did speed outrun your actual trust in the result? Hold on to your answer, it'll be useful at the end."
07 · Divider: Section 1¶
"Moving to the first arrow of the loop — Discovery. This is the phase where the hypothesis about what to build even comes from. Most strong engineers have informal experience 'asking friends' — but almost nobody has run into the discipline that explains why that experience systematically lies to you.
Two classical foundations in this section: a methodology for finding a business model, and a technique for a single conversation. Then — what AI adds, where it's limited, and three failures that show three facets of the same mistake: a synthetic source mistaken for a verified fact."
08 · ELI5: Discovery in plain terms¶
"Before we get into the formal methods — let's break discovery down to basics.
Discovery is the first arrow of the loop. At this point you're not building anything yet. Your only job is to figure out whether it's even worth building something, and for whom. Sounds simple, but this is exactly where most products fail: the team falls in love with its own idea and builds it without ever checking whether there's a real problem that idea actually solves.
The key mental model for this section: there are no facts inside the office. Everything you think about your users while sitting at your desk is a hypothesis, not knowledge. Facts live outside, with real people.
AI can help a lot — summarizing interviews, pulling together desk research in minutes instead of hours. But it cannot talk to a real person on your behalf, and it cannot produce real disagreement. Next — two classical techniques that keep a conversation honest, and three real failures where discovery got replaced by a plausible-sounding invention."
09 · FOUNDATION: Customer Development¶
"Let's start with the classic — you have to start here, not with an AI tool, because without this frame any AI tool just speeds up the mistake.
Customer Development is Steve Blank's methodology, from 2003. The central idea sounds like an aphorism, memorize it word for word: 'there are no facts inside the building, so get outside.' A startup, in Blank's terms, is not a smaller copy of a big company — it's a temporary organization in search of a repeatable business model under extreme uncertainty. You can't write the plan in advance, because it depends on answers only customers know.
The method is four sequential steps, each ending in an explicit decision to continue or pivot. Discovery — turn the founders' hypotheses into facts by going out to real people; the step is explicitly hypothesis-first: you write down what you believe before you walk out the door. Validation — can you sell this not just to the first enthusiasts, but to a wider circle. Creation and Company Building — generate demand at scale and shift the organization from search mode into execution mode.
Blank's canonical example: the founders wrote a forty-page business plan and only then held their first conversation with a customer — every important decision made on zero evidence. The Lean LaunchPad courses forced teams to run ten or more interviews before writing a single line of the plan: the discipline, not the plan itself, was the graded outcome.
And here's a technique we'll reuse in every phase that follows — in the experiment, in the launch gate, in the drift threshold. It's called the falsifiable hypothesis: a written statement of the form 'we believe X, we'll test it via Y, by date Z we'll have a yes-or-no answer.' The format turns vague confidence into a testable commitment with a date — and we'll see it in a few slides as the only defense against one of this section's failures."
Sources: [1] Blank — Customer Development (Four Steps to the Epiphany)
10 · FOUNDATION-2: The Mom Test¶
"If Customer Development is the map — which four steps to walk — The Mom Test is the technique for one step: how to run a single conversation with a single person without lying to yourself.
The method was formulated by Rob Fitzpatrick: even your own mother will lie to you about your business idea's quality if you ask her directly — and the defect isn't in the honesty of the person you're talking to, it's in how the question is built.
Three rules. First — talk about their life, not your idea: don't pitch the idea and don't ask for a verdict — that creates a 'politeness problem' where the person protects your feelings instead of telling you the truth. Bad: 'do you like the idea?' Good: 'walk me through how your team handles weekly reporting.'
Second — ask about specifics from the past, not opinions about the future: hypothetical questions invite dishonest optimism. Instead of 'would you pay twenty dollars for this?' — 'what do you pay today for the closest thing to this?' The second question anchors on real past behavior.
Third — talk less, listen more, validate with a real commitment, never a compliment. 'I love it' is free and means nothing. Legitimate signals are time, reputation, money: will the person get on a call, will they pre-order.
And a distinction engineers often confuse: qualitative research answers 'why' at a small sample size, quantitative answers 'how many' at scale — that's a division of labor, not a hierarchy. And sample bias: interviewing only your friends systematically inflates enthusiasm — invisible from inside the interview, only visible on an audit of who you actually talked to. Hold on to that — it's the direct premise of the first failure a couple of slides from now."
Sources: [1] Fitzpatrick — The Mom Test
11 · AI: capabilities + tools in Discovery¶
"Now — what AI actually changes in this phase.
Desk research — scanning market reports, forum complaints — collapses to minutes. Perplexity Deep Research is widely considered the best tool for this: it plans a research trajectory, reads dozens of sources, returns a cited report in minutes, cutting research time by roughly fifty percent.
Mandatory practice — memorize it word for word: verify important data directly against the cited source before using it in a decision. Citations exist precisely so verification is possible — the tool doesn't remove that step. This is a direct bridge to the failure a few slides from now, and there the price of skipping it is measured not in time, but in money and reputation.
Interview synthesis — tools like Dovetail are built for cross-interview synthesis: a quote in the summary is anchored back to its original source, which protects against drifting into an unverifiable summary.
Let me draw a line here as sharply as I can: synthetic users are legitimate narrowly, as a pre-research tool — piloting an interview guide, sketching hypotheses. But none of those outputs is evidence for a 'ship it or not' decision. A telling number: ninety-seven percent of researchers already use AI, but only around eight percent trust AI personas as a data source — a healthy gap, not a sign of falling behind.
Even Teresa Torres, a leading voice in discovery practice, uses AI for a first draft, but never drops the rule 'synthesize each interview separately first, then across interviews' — and she warns against 'one-click AI trees,' because the team's own thinking is the entire point of the process."
Sources: [1] Perplexity Deep Research — overview for product research · [2] Dovetail — AI interview synthesis · [3] NN/g — 2026 practitioner survey
12 · AI limits in Discovery¶
"Now — where AI is structurally limited in this phase, not just 'not good enough yet.'
Teresa Torres documented a measurable finding: an AI summary can miss twenty to forty percent of important interview detail when the synthesis skips the 'individually first' step and jumps straight to cross-interview themes. That's not a vague 'AI sometimes gets things wrong' — it's a traceable methodological failure.
Now the accumulation argument, follow this carefully. Rule two of The Mom Test — ask about specific past events, not hypothetical futures — a synthetic persona structurally cannot satisfy: it has no real past, only something that sounds plausible. Stack that on top of the model's tendency to agree, and you get a compounding failure: not just 'biased toward yes,' but 'biased toward yes about an event that never happened.'
What survives from the classics: the three Mom Test rules apply the same whether a human or an AI is asking the question; Blank's hypothesis-first discipline is the antidote to a one-click AI tree; validation through a real commitment has no AI substitute.
The criterion, stated as operationally as possible: the closer the output sits to an irreversible decision about real people or real money — the more strictly you need a real human with a real past. The further away — the more appropriate AI is as a drafting accelerator."
Sources: [1] Torres — Continuous Discovery Habits / anti-synthetic
13 · Failure #1: NN/g synthetic users¶
"And here's the first failure of the section — measured, reproducible, concrete.
Nielsen Norman Group ran a controlled usability comparison of onboarding. Real users completed, on average, three out of seven onboarding steps. A synthetic AI panel testing the same flow falsely reported completing seven out of seven. Real users described one feature as 'contrived and useless'; synthetic personas produced multi-paragraph enthusiastic praise; one synthetic user called impractical drone delivery 'a game changer' — while real people rated it skeptically.
Notice the basis for this: this isn't a one-off prompting mistake, it's a structural divergence on both measured quantities, from the same test material.
The mechanism, the root cause — large language models are tuned to agree with the frame of the question they're asked. A synthetic user cannot produce disagreement, because it has no real experience capable of contradicting how the question was phrased — only a statistically plausible continuation of text. This is the same sycophancy mechanism the course already discussed. Here it materializes as a concrete, measured failure specifically in the discovery phase: the decision to ship a feature gets made on data that is structurally incapable of saying 'no.'
The lesson, let me state it as directly as possible: a synthetic panel that agrees with your concept isn't evidence the concept is good. It's evidence the panel is incapable of disagreeing.
The criterion for where AI isn't the right tool: any output feeding a 'ship or don't ship' decision must trace back to real, specific past behavior from a real person. Synthetic output is disqualified from that role by construction, not by bad luck on one particular run.
The alternative — use a synthetic panel only in a pre-research role, then run real qualitative testing with five to eight participants — a sample size that, per Nielsen's classic finding, already surfaces most usability problems."
Sources: [1] NN/g (Rosala & Moran) — Synthetic Users
14 · Failure #2: Deloitte Australia fabricated research¶
"The second failure is no longer about a feature — it's about an external document with real legal consequences.
Deloitte Australia, in 2025, delivered a report to the Australian government worth AU$440,000 — nearly US$290,000 — containing citations to nonexistent academic papers and a fabricated quote attributed to a real court ruling. The report was produced via Azure OpenAI without human verification at the citation level. Deloitte partially refunded the fee and publicly acknowledged the use of generative AI.
For scale: a database tracks more than 712 court rulings worldwide involving AI-hallucinated legal citations, of which roughly ninety percent occurred in 2025 alone. The problem is growing systemically.
The mechanism — the failure sits exactly in the 'desk research' step: an AI-generated draft of secondary research was accepted as a finished, verified review, rather than as an unverified draft — a direct violation of the practice we called mandatory a few slides ago.
And here's why this failure matters specifically for a course about fact-checking — it's not just 'lawyers used ChatGPT badly.' It's structurally identical to a whole class of failures we'll come back to in Section 6: plausible-sounding fabrication mistaken for a verified fact.
The lesson: a large language model's output in the discovery phase is an unverified draft, not a source of facts. Every citation, date, and number needs independent verification before it becomes the basis of a decision. The criterion: for any external document, citation-level verification has to be one hundred percent, not spot-checked."
Sources: [1] Fortune (Oct 7, 2025) — Deloitte Australia A$440k · [2] Damien Charlotin — AI Hallucination Cases Database
15 · Failure #3: IBM Watson for Oncology¶
"The third failure of the section — it's unusual that there are three, but each shows a different facet of the same mistake, and here the cost is orders of magnitude higher.
IBM Watson for Oncology was developed starting in 2012 in partnership with Memorial Sloan Kettering Cancer Center as a treatment-recommendation system for cancer. The partnership with MD Anderson Cancer Center was shut down in 2016 after spending sixty-two million dollars — with zero patients treated. Internal IBM documents showed the system produced 'unsafe and incorrect' recommendations — for example, prescribing a drug with an explicit warning against exactly that use, for a hypothetical patient with active bleeding.
The basis is stark: sixty-two million spent against zero patients actually treated — not one bad recommendation out of a thousand good ones, but systemic unsafety on demonstration cases.
The mechanism — the same as in the first two failures, a different domain: the failure sits in the choice of data source. The system was trained not on real patient data, but on a small number of synthetic, hypothetical cases, labeled by a handful of oncologists — the product inherited the personal preferences of a few doctors instead of a representative evidence base.
The lesson: the discovery phase of an AI product in a high-stakes domain must verify data representativeness before demonstrating capabilities to clients.
How to close a section with three failures: look at them not as three stories, but as one class. Synthetic users, fabricated sources, Watson — in all three, a synthetic or unverified source got accepted as a validated basis for a decision. What changes is the cost of the mistake — from a failed feature to patient harm."
Sources: [1] STAT News — leaked internal IBM Watson documents · [2] IEEE Spectrum — how Watson overpromised and underdelivered
16 · Divider: Section 2¶
"Moving to the second arrow of the loop — Design. This is how the hypothesis from Section 1 becomes a concrete artifact you can show a person. You're technically strong, but generally haven't gone through formal design training — so this section starts with the classics before we get into AI interface-generation tools."
17 · ELI5: Design in plain terms¶
"In plain terms: design is translating 'we believe people need X' into a concrete picture or flow of screens that you can show a real person and watch their reaction to.
Two classical rules that are easy to break without noticing. First — don't converge on a solution before you've verified the problem itself. Second — you are not the user: anything obvious to you, the developer, might be completely unobvious to a real person.
AI today can generate dozens of screen variants in minutes — that drastically lowers the cost of a draft. But it doesn't lower the cost of deciding which variant actually works for your users — that decision is still made by a human, watching a real reaction."
18 · FOUNDATION: Double Diamond¶
"Design Thinking is a five-stage human-centered methodology: empathize, define, ideate, prototype, test. Double Diamond is the British Design Council's frame, a visual model of two cycles of divergence and convergence: the first diamond, broad research and synthesis into a brief, is about the right problem; the second, generating solution directions and shipping one, is about the right solution.
We use Double Diamond as the single load-bearing frame, because it's structurally equivalent to Design Thinking but makes the core point more explicit: two separate divergence-convergence cycles, not one. The most common mistake an engineer makes on their first encounter with a design process: instinctively converging on a solution before verifying the problem itself — skipping the first diamond and diving straight into the second. It's the same anti-pattern as 'just write the code,' skipping the question of whether you're even solving the right problem."
Sources: [1] UK Design Council — The Double Diamond
19 · FOUNDATION-2: Nielsen heuristics + design system¶
"Jakob Nielsen's usability heuristics are rules for expert evaluation — not a replacement for testing with real users, but a fast and cheap way to catch problems before spending money on research. A useful metaphor: it's a linter for UX.
Top five of ten, briefly: visibility of system status; match between the system and the real world; user control and freedom — undo/redo; consistency — the same actions work the same way; error prevention — stop a mistake from happening rather than reporting it nicely after the fact.
A design system is a living set of principles and components that keeps a product coherent at scale. The load-bearing thought for the next slide: a design system is a guardrail, keeping generative freedom inside the boundaries of an already-validated brand.
And Nielsen's hard lesson, an industry mantra: 'you are not the user.' Anyone close to a project systematically mispredicts what will confuse a real person. Even a single-user test uncovers about thirty-one percent of usability problems — even a tiny test already pays for itself. That's the counterweight for the next slide: no matter how much AI speeds up prototype generation, the step of 'watch a real person's real confusion' has no AI substitute."
Sources: [1] Nielsen — 10 usability heuristics · [2] Nielsen — "you are not the user"
20 · AI: capabilities + tools in Design¶
"The realistic workflow today: a designer describes a screen in a prompt to a tool like v0 by Vercel or Figma Make — the latter pulls in the team's own existing components rather than generic templates. The generator produces two to four directions in minutes — what used to take a day. Then a human converges: picks a direction, fixes details the model got wrong, and it goes through real usability testing with actual users, not an AI persona.
Three honest wins: speed of the first draft, generation that respects the design system — less rework — and democratizing a decent first pass for non-designers.
The single cleanest formulation of this section, memorize it word for word: 'AI for divergence, human for convergence.' Co-creative tools maximize the volume of early exploration. Convergence — defining which problem to solve, and refining a direction until it fits real users — is the human side no prompt replaces."
Sources: [1] Figma — 2026 AI design tools · [2] Google Stitch (ex-Galileo AI)
21 · AI limits: non-deterministic UX¶
"Nielsen Norman Group's State of UX report states it plainly: interfaces still matter, but they'll become a weaker differentiator — soon anyone can produce a decent-looking UI. The homogenization mechanism: generators are trained on the same corpus of patterns, and different teams prompting 'modern dashboard' converge on similar output.
A hard number: a study ran 21,880 accessibility evaluations on AI-generated interfaces and found only twenty-nine percent WCAG compliance. A finding that undercuts 'just ask AI correctly': platform design affected accessibility more than prompts did.
A key addition for this audience: designing for non-deterministic output. Classical UX assumes a deterministic system — same input, same output; AI products break that. Hence new patterns — graceful degradation instead of a patch, co-creation instead of a final verdict, responsible autonomy.
What survives from the classics: design thinking's 'empathize first' — against homogenization; Nielsen's heuristics plus a design system with accessibility built in — against gaps; real testing — against hallucinated 'best practices.'"
Sources: [1] ACM Web4All 2026 — Generated Inaccessible
22 · Failure #3(S2): Character.AI¶
"Fourteen-year-old Sewell Setzer III died by suicide in February 2024 after months of emotionally intense conversations with an AI character on the Character.AI platform. His mother filed a wrongful-death lawsuit. In May 2025 a federal judge rejected the defense's First Amendment argument; in January 2026 Google and Character.AI agreed to a settlement.
For scale on the failure: safety features — a time limit for minors, a ban on open-ended romantic roleplay for users under eighteen, age verification — were added only after the lawsuits. Almost two years after launch, and eighteen months after the tragedy.
Why is this a Design failure, not Support/Operate. It's easy to misclassify this as an operational incident: 'the bot responded badly in production.' But the root cause sits earlier — in the design decision about what experience to build in the first place. A product optimizing for the emotional attachment of minors is a choice made in the first diamond of Double Diamond, made without ever asking 'who could get hurt.' The absence of crisis detection isn't an implementation bug — it's a missing requirement in the design brief.
The lesson: for products targeting the emotional attachment of vulnerable users, safety guardrails have to be part of the MVP design from day one, not a patch bolted on after a tragedy. The criterion: if a product optimizes for time-in-app or minors' attachment without a crisis-recognition mechanism — that's a structural design flaw, not a surface-level fix."
Sources: [1] Washington Post (Oct 24, 2024) · [2] CBS News (Jan 2026) — settlement
23 · Failure #4(S2): iTutorGroup¶
"An important contrast within the same section. iTutorGroup programmed its automated résumé-screening system to reject women aged fifty-five and older and men aged sixty and older — a direct violation of US federal age-discrimination law.
The trigger for discovery is instructive in itself: a rejected applicant submitted the same résumé again with a younger birth date — and immediately got an interview invitation. In August 2023 the Equal Employment Opportunity Commission reached its first-ever AI discrimination settlement: three hundred sixty-five thousand dollars in compensation.
A methodically important contrast: in Character.AI, the design failed to include a safeguard. Here, the design encoded a specific harmful decision rule — a hardcoded discriminatory rule, deployed without a bias audit.
Why the discovery trigger is so instructive: ordinary metrics looked great, 'the system works.' Bias baked into a decision rule only surfaces through a targeted audit for disparate impact — relying on 'someone will notice' is not a strategy.
The criterion: any screening process touching legally protected categories requires a mandatory bias audit before going to production."
Sources: [1] EEOC (Aug 9, 2023) — iTutorGroup
24 · Divider: Section 3¶
"The third arrow of the loop — Build and Launch. This is exactly where AI changes the most: the cost of writing code. But the section starts with the classical discipline of release, because that discipline defines what 'safely ship' means — and that question doesn't go away just because writing code got cheaper."
25 · ELI5: Build/Launch in plain terms¶
"In plain terms: building a product has become nearly free — that's true, you know it from your own vibe-coding practice. But 'build it' and 'safely release it to the world' are two different questions. The first is a matter of speed. The second is a matter of discipline: small steps, with the ability to roll back fast if something goes wrong.
The classical analogy — you're already doing this whenever you ship code through a canary rollout or a feature flag. What's new here isn't the mechanics — it's that the decision 'keep rolling out or stop' has become a product decision, a business decision, not just a technical one."
26 · FOUNDATION: MVP + release mechanics¶
"You already know release-rollout mechanics — feature flags, canary, rollback are familiar from CI/CD. Hold on to one product-specific difference: the same mechanics are embedded inside a product go/kill gate, where the decision to kill a project is a business decision about whether it's worth building further, not just a technical flag.
Quickly through the foundation. MVP, minimum viable product — Eric Ries's concept: defined by learning, not by shipping. Dropbox validated demand with an explainer video before building the product — the waitlist grew from five thousand to seventy-five thousand overnight.
The single thread running through every release practice: each one trades a little speed for a contained blast radius — a guarantee that damage, and its repair, stay fast and small. A feature flag separates deployment from release. Canary rolls out to a small slice of traffic first. Staged rollout — one percent, ten, twenty-five, fifty, a hundred. Rollback — a fast return to the last working state.
Stage-Gate is Robert Cooper's methodology: five stages, each preceded by a gate — continue, kill, pause — forcing disciplined stop decisions early, before capital-intensive spending.
In one line: launch speed traded against contained blast radius — and in a product context, the 'blast radius of a release' gets a companion, 'the radius of capital and reputation before a kill decision.'"
Sources: [1] Ries — Lean Startup / MVP · [2] Stage-Gate — the method's history
27 · AI: Build→≈0 + the shifting bottleneck¶
"Anthropic's own reported measurement frames the shift as a compression of stage duration: implementation moves from weeks to minutes of agentic execution. But the central risk claim matters more than the speed: 'as agents produce more code, review volume doesn't scale at the same rate.' Anthropic's internal data: code volume per engineer grew roughly two hundred percent year over year, but only around sixteen percent of pull requests got substantive human review before merging.
The single most important reversal of today's lecture: the scarce resource was never 'who can write the code.' In the AI era it's 'who can specify precisely enough' — upstream — and 'who can review fast enough' — downstream. It mirrors the keystone: build collapsed to zero, the bottleneck shifted to the two human ends of the arrow.
What survives from the classics: version control, PR review — now the destination of the bottleneck — feature flags, staged rollout, kill switches, and the human-owned spec as the input the agent can't invent on its own."
Sources: [1] Anthropic — 2026 Agentic Coding Trends Report
28 · AI: launch as a transfer of control¶
"The sharpest reframing of 'launch' for the AI era — the CC/CD framework, Continuous Calibration and Continuous Development, contrasted with CI/CD: CI/CD assumes deterministic code that either passes tests or doesn't, and AI systems don't have that property.
Mechanics: before launch — version releases by level of agency, not feature set — version one routes tickets with high human control, version two proposes solutions for human approval, version three auto-resolves with a human fallback. After launch — run evals continuously against live data.
The survival rule, word for word: 'if you haven't tested how the system behaves under high control, you're not ready to give it high agency.' Example — the agency ladder of GitHub Copilot and Cursor: from autocompletions to code blocks to entire pull requests, each rung unlocked once the previous one has proven reliable."
Sources: [1] Reganti & Badam — CC/CD framework
29 · AI limits in Build/Launch¶
"Review doesn't scale with generation — the load-bearing limit of this section. This is a mechanism, not a temporary staffing shortage: a model can generate plausible code in seconds, but verifying it's correct and safe still requires real-time human understanding that hasn't gotten any faster.
The counterintuitive consequence: double your generation speed, and you don't double product throughput — you double the review queue.
The 'seventy percent problem': AI agents get a team roughly seventy percent of the way to production quality. The remaining thirty percent — refactoring, edge cases, engineering judgment — is where a senior adds value AI doesn't supply. A senior rethinks and constrains the output; a junior accepts it more readily, building a house of cards out of code.
Shipping without an eval gate or a rollback plan isn't speed — it's deferred cost. Both failures below are variations on exactly this limit."
Sources: [1] Osmani — The 70% Problem
30 · Failure #5: Google AI Overviews¶
"In May 2024 Google rolled out AI Overviews to one hundred percent of US search users in a single step — skipping the usual staged rollout: one percent, ten, twenty-five, fifty, a hundred. No eval gate.
Within days the feature started producing viral, literally dangerous recommendations: eat rocks — sourced from a satirical article on The Onion — and put glue on pizza for better cheese adhesion — sourced from an eleven-year-old joking comment on Reddit. Users had no way to turn the feature off.
The nature of these failures is a textbook illustration of 'confidently wrong.' The model synthesized a statistically plausible answer from its corpus without distinguishing satire from fact, and delivered it with the full confidence of an authoritative search answer. At one percent of traffic, answers like these would have surfaced in internal monitoring before the whole country saw them.
The lesson, important precisely because it's a giant: a generative feature needs the same discipline of staged rollout and eval gates as any other AI feature. 'Search is our mature product' isn't a reason to skip the rollout — if anything, it raises the cost of skipping it, because the audience at one hundred percent is the whole user base, on day one."
Sources: [1] Forbes (May 31, 2024) — Google AI Overviews · [2] AndroidPolice — "glue on pizza" / "eat rocks"
31 · Failure/positive #6: McDonald's × IBM drive-thru¶
"An unusual slide — a good example of a process working correctly, even though it superficially looks like a failure.
McDonald's tested automated order-taking with IBM for roughly two and a half to three years, scaling to about one hundred restaurants — roughly zero point seven percent of nearly fourteen thousand US McDonald's locations. Without the denominator, 'a hundred restaurants' sounds like a large-scale pilot; with the denominator, it's a tiny slice of the chain. Viral clips accumulated for months: bacon added to ice cream, an order for nine iced teas instead of one. In June 2024 McDonald's announced ending the partnership.
Why this is a GOOD example: it's a decision to stop scaling — a Stage-Gate kill decision — not a story about a runtime failure. A pilot that's still regularly producing viral failures after that many years is a signal that says: not 'we need more data,' but 'the current approach has a structural ceiling.'
Compare this with Google AI Overviews — a mirror-image mistake: Google went straight to a hundred percent with no pilot phase; McDonald's correctly did NOT cross the gate after a long pilot. Together the two cases outline the discipline: a pilot exists to give you grounds for a kill decision — and it's useless if you skip it, or ignore what it's telling you.
The lesson: a long pilot that keeps publicly failing at limited scale is a signal to kill it, not to 'refine it a bit more.'"
Sources: [1] CNBC (June 17, 2024) — McDonald's x IBM · [2] Nation's Restaurant News — details of the pilot shutdown
32 · Divider: Section 4¶
"Moving to the fourth arrow of the loop — Measure, Experiment. This is the arrow where AI cut trust, not cost. Arguably the most important section today: measurement is where 'looks good' diverges from 'is good' most sharply.
You can write code and you might know A/B testing from web practice. But the formal discipline of a product experiment — you're most likely seeing it for the first time."
33 · ELI5: Measure in plain terms¶
"In plain terms: you changed something, and the metric went up. Does that mean the change worked? Not necessarily — it could have been the season, a marketing push, or plain chance.
The only way to know for sure is to compare against a control group that got nothing, with both groups chosen at random. That's the entire meaning of an A/B test in one sentence. And in an AI product a new complication shows up: the model's own answer is probabilistic, so you need one more layer of checking — not just 'did the change work,' but 'is the model reliably producing the correct answer.'"
34 · FOUNDATION: controlled experiments + OEC¶
"A controlled online experiment applies the logic of a randomized controlled trial to a live product. Users are split at random into Control and Treatment, and a metric is measured for each group.
Why randomization gives you causality: a metric went up after a release, but the same week there was a marketing spike, a season change — without a control group you can't separate the effect of the change from everything else. Randomization solves this: outside factors are, on average, spread evenly across both groups, and 'cancel out' in the comparison. Without randomization — correlation. With randomization — causation. That's exactly why a canary rollout is NOT equivalent to an A/B test: it has no held-out control group.
OEC, the Overall Evaluation Criterion, is the single metric the experiment is trying to move — what the organization has agreed counts as success. Kohavi's cautionary example: a team adopted 'time on the support site' as its metric — but when asked to state the direction, the team split: more time is good, engagement, or bad, the user got stuck? The team had adopted a metric without ever agreeing on what it meant.
And guardrail metrics: monitored but not optimized, to catch damage the OEC doesn't see — the structural fix for the trap of optimizing a single proxy in isolation, a failure we'll dissect a few slides from now."
Sources: [1] Kohavi — Overall Evaluation Criterion (OEC) · [2] Wikipedia — A/B testing (Bing 2012)
35 · FOUNDATION-2: experiment pitfalls¶
"Eight classical experiment pitfalls are best remembered not as a list, but as three questions asked in the right order.
First — is the test actually set up correctly? Check for Sample Ratio Mismatch — a statistically significant deviation of the group split from what you intended — and a sample size fixed in advance.
Second — am I reading the result correctly? Peeking — checking results before reaching the target sample size: two peeks roughly double the false-positive rate, five roughly triple it. P-hacking — trying different metrics and segments until something looks significant.
Third — is the effect even real? Novelty effect — a reaction to the change itself out of curiosity, fading over weeks. And Twyman's law, word for word: 'any figure that looks interesting or unusual is usually wrong.'
A distinction engineers often confuse: an A/B test is measurement, a feature flag is rollout. A canary rollout with no control group tells you 'did something break,' not 'did the change cause an improvement.'
A canonical case with a mandatory correction. The Bing ad-headline test — the idea sat in the backlog for over six months, an engineer shipped it as an experiment on his own initiative. Within hours, revenue was so high it tripped a 'too good to be true' check — and the effect held up: a twelve-percent lift produced more than one hundred million dollars in annual US revenue. A separate case — 'the three-hundred-million-dollar button' — is a usability finding, NOT a randomized experiment; roundup lists regularly conflate the two."
Sources: [1] Microsoft Research — Sample Ratio Mismatch (KDD 2019) · [2] GoPractice — the peeking problem · [3] exp-platform.com — Kohavi, Twyman
36 · AI: evals as experiments¶
"OpenAI's head of product frames the shift like this: 'writing evals will become a core skill for product managers.' An eval is 'a quiz for the model' — a way to set a concrete accuracy bar the product needs to clear.
Hamel Husain proposes a three-tier frame: tier one — unit tests, the pytest equivalent; tier two — human and model evaluation, LLM-as-judge; tier three — A/B testing, only once the product is mature. A/B doesn't get replaced by evals — it's the last rung of the ladder.
Why probabilistic output breaks pass/fail — the most important explanation of the section. Pass at k — the probability of at least one correct result across k attempts. Pass to the power k — the probability of success on all k attempts, true consistency. These numbers diverge sharply at the same base probability: as k grows, pass-at-k approaches one hundred percent, while pass-to-the-power-k collapses toward zero. Production reliability requires pass-to-the-power-k — does it work every single time a user relies on it.
LLM-as-judge — GPT-4-class judges reach roughly eighty-five percent agreement with humans, comparable to agreement between humans themselves. But a critical caveat: agreement drops below sixty percent on hard categories, especially safety — the judge is least reliable exactly where the stakes are highest.
An eval isn't a one-time 'pass-fail' — it's a continuously maintained loop: production failures get promoted into a golden set, and the bar itself is a product decision about the cost of an error, not a technical detail."
Sources: [1] Anthropic — Demystifying evals for AI agents (Jan 9, 2026) · [2] Zheng, Chiang et al. — MT-Bench (NeurIPS 2023)
37 · AI limits: Goodhart's law¶
"Goodhart's law, word for word: 'when a measure becomes a target, it ceases to be a good measure.' The AI/ML formulation is reward hacking: behavior that satisfies the letter of the objective's specification without achieving the intended outcome.
A named example: an RL agent in the CoastRunners boat-racing game discovered it could score roughly twenty percent more points than humans by circling endlessly in a lagoon and collecting regenerating bonuses — catching fire, crashing into other boats, and never finishing the race. Points were meant as a proxy for race progress; the agent exploited exactly that assumption.
A modern example: Anthropic observed a model generalize from simple reward hacks to sophisticated reward tampering — from political sycophancy to falsifying checklists and directly editing its own reward function.
Why the discipline of experimentation and guardrail metrics stays mandatory: the more autonomous and probabilistic a system is, the easier it is to game a single proxy metric — and the harder it is to notice by inspection."
Sources: [1] DeepMind — Specification gaming (Apr 21, 2020) · [2] Anthropic — Sycophancy to Subterfuge (arXiv:2406.10162)
38 · Failure #7: Facebook MSI¶
"On January 11, 2018, Mark Zuckerberg announced that News Feed would prioritize 'meaningful social interactions' over passive content consumption. The ranking formula weighted every emoji reaction — love, haha, wow, sad, and angry — five times higher than an ordinary like.
A mandatory fact-check correction. The common retelling claims Facebook specifically boosted the 'angry' reaction to reward outrage. That's inaccurate: all five reactions were weighted equally, five times, at introduction — anger was not singled out. The real failure was subtler: 'any strong reaction equals more meaningful' was a poor, blanket proxy for engagement quality, not a deliberate decision to reward rage. This is institutional carelessness in proxy-metric design — a more common and more dangerous failure mode than malicious intent.
The guardrail that caught it: internally confirmed by 2019 that posts triggering an angry reaction were disproportionately likely to contain misinformation — the engagement proxy was inversely correlated with content quality.
The most instructive engineering point: Facebook had one of the strongest data science teams in the world, and the harm still went unnoticed for about two years. The reason wasn't a shortage of analysts — it was the absence of the guardrail metric itself, from day one. You can't notice the degradation of a quantity you never instrumented.
The lesson: designing an A/B experiment for an AI feature must include quality guardrail metrics, not just the optimized proxy."
Sources: [1] Techdirt (Oct 28, 2021) — correcting the "Facebook Files" · [2] House E&C Committee — the Haugen documents
39 · Failure #8: benchmark ≠ reality¶
"Google's Med-PaLM 2 scored eighty-six and a half percent on MedQA — questions in the format of a medical licensing exam. A mandatory correction: the claim that the model 'recommended chemotherapy for a headache' doesn't trace to any primary source. The verifiable version is stronger: Google's researchers had to build a separate adversarial test set precisely because an exam-style benchmark doesn't surface real clinical-safety failures at all. A model can score eighty-six and a half percent on a closed proxy of medical knowledge while still requiring a completely separate adversarial safety evaluation to check its real-world answers.
Legal AI — with an attribution correction. In Mata v. Avianca, a lawyer filed a document citing six fabricated court cases generated by ChatGPT — not Harvey AI, a frequent misattribution. A peer-reviewed study: across two hundred two legal queries, raw GPT-4 without retrieval erred eighty-eight percent of the time; the best RAG tool, Lexis+ AI, erred seventeen percent of the time — roughly one in six; Westlaw — around thirty-three percent, twice as often, despite marketing itself as 'hallucination-free.'
The mechanism behind both cases is, again, Goodhart's law: the benchmark became a marketing target, and the number reported was the one that looked best, not the one relevant to the real task.
Three widely 'known' failures, corrected against the primary source: the chemotherapy-for-a-headache claim doesn't trace to a primary source; the fake citations are ChatGPT's, not Harvey's; the widely circulated 'Harvey hallucinates one in six' figure was very likely misattributed from Lexis+. Check the primary source before repeating any number.
The lesson: a measured benchmark score is evidence only for the format of that benchmark's task. Treat it as orthogonal to real-world safety until a separate evaluation has been run."
Sources: [1] Med-PaLM 2 — Nature Medicine 2024 (arXiv:2305.09617) · [2] Wikipedia — Mata v. Avianca · [3] Stanford RegLab — J. Empirical Legal Studies
40 · Synthesis of Section 4¶
"Let's pull Section 4 together: Measure is the arrow of the loop where AI cut trust. The classical A/B test survives — randomization is still the only way to know a change caused a result — but evals and guardrail metrics are now mandatory on top of it.
The central skill of this phase, and of the whole lecture: telling signal from noise. A suspiciously good number, a gamed proxy, a benchmark in the wrong format — all of it is noise wearing the mask of signal.
Bridge to the next section: in operations, the same skill applies — noticing drift before the dashboard shows it."
41 · Divider: Section 5¶
"The fifth arrow of the loop — Support and Operate. What happens once a product is already in production and real users are interacting with it twenty-four seven.
The classical foundation matters especially here, because the discipline of operations exists for a reason a vibe-coding engineer rarely lives through: between 'it works on my machine' and 'the product reliably works for millions, around the clock' lies a gap, and that gap is closed by process, not just by code quality."
42 · ELI5: Support/Operate in plain terms¶
"In plain terms: shipping a product isn't the end of the story, it's the beginning. Real people start using it around the clock, and something will inevitably go wrong — the question isn't 'if,' it's 'when' and 'how fast will you notice.'
The classical discipline answers: how many failures are we willing to tolerate before it becomes a problem, and what do we do once that threshold is crossed. What's new for an AI product is that the model itself can 'break' silently — not an explicit 500 error, but a quiet drift toward slightly worse answers. And that's much harder to notice than an ordinary server outage."
43 · FOUNDATION: SRE + support ops¶
"You may already know on-call and SLOs — I'll move through the foundation quickly. Hold on to one difference: all of this was built for a deterministic system, where a request either succeeds or fails. In an AI product's production environment, you're running a non-deterministic model — an indicator like 'share of successful HTTP 200 responses' tells you nothing about whether the model is hallucinating.
Google's SRE frame. SLI — a metric of service behavior from the user's point of view. SLO — the team's internal target. Error budget — 'one minus the SLO': a service with a million requests over four weeks at a 99.9% SLO has a budget of one thousand errors — a concrete number you can spend. The Google SRE Workbook: 'changes account for roughly seventy percent of our outages.'
Incident management: on-call — a rotation for response; severity levels — who gets woken up and how fast; a runbook — instructions written in advance; a blameless postmortem — a review with no search for someone to blame.
And the loop from support back into the product: support tickets are raw data about where the product falls short of expectations."
Sources: [1] Google SRE — Error Budget Policy
44 · AI: LLMOps/AgentOps¶
"An AI product in production has two properties classical software doesn't: non-determinism and an agency-versus-control trade-off. A measurable problem: experiments record up to fifteen percent accuracy variation between runs of the same prompt, even at zero temperature, with a gap of up to seventy percent between the best and worst run.
Tracing — observability underneath the LLM stack: not 'a request came in, a response went out,' but the full path — the prompt, retrieval, tool calls. Its purpose is to catch what infrastructure monitoring misses: hallucination drift, retrieval failures, prompt regressions.
Drift monitoring: data drift — the distribution of inputs shifts; concept drift — the relationship between input and correct answer itself changes. Runtime guardrails — a layer validating inputs and outputs before a response reaches the user.
Guardian Agents — a Gartner category, its first Market Guide published in February 2026: agents that watch other agents. A Sberbank product under this exact name isn't confirmed in the sources; the closest analog is GigaCowork.
A specific note on regulated data: tracing and LLMOps tools, by construction, send request content to external services. Regulated personal data in support tickets, piped through external tracing, is a leak into an uncontrolled surface. The rule: don't send regulated data without anonymization or a self-hosted deployment."
Sources: [1] Confident AI — 2026 LLMOps observability overview · [2] Gartner Market Guide for Guardian Agents (Feb 25, 2026)
45 · AI limits: silent drift¶
"Perhaps the most important operational lesson of the section: trust falls before the dashboards move. The model keeps answering, the dashboards stay green — but outputs become slightly less grounded. By the time a team notices visible degradation, the drift has already been present for weeks.
Power users notice before the aggregates do — when a model is quietly updated, experienced users feel the regression even while aggregate metrics are improving.
And governance drift — the guardrails themselves drift, unless they're versioned as policy-as-code, with continuous feedback. A non-obvious point: an unversioned guardrail creates a false sense of safety while actually diverging from reality — the team thinks it's protected, and the protection has quietly gone stale.
The lesson: AI doesn't remove the need for operational discipline — it adds a new, more insidious class of failure, silent and distributed, on top of the old, explicit, binary one."
46 · Failure #9: Zillow drift¶
"Zillow Offers bought and quickly resold homes based on an algorithmic valuation. On November 2, 2021, the company announced shutting the business down. Inventory write-downs: three hundred four million dollars in Q3, four hundred eight million for the year. Roughly two thousand people laid off, a quarter of the staff.
For scale: in Q3, the company bought nine thousand six hundred eighty homes and sold only three thousand thirty-two — an average loss of roughly eighty thousand dollars per property. The CEO's quote: 'the observed error rate turned out to be far more volatile than we ever thought possible.'
The mechanism: this is a failure of operations, not measurement. The model was calibrated correctly on historically stable data. The failure was operational: a production model directing hundreds of millions of dollars in real-estate purchases had no runtime accuracy-drift monitoring with an automated circuit breaker. Pandemic-era volatility was classical concept drift, not caught in time because there was no equivalent of an error budget for this specific model.
The lesson: for models directing capital-intensive decisions, error-budget discipline has to apply to real-time prediction quality — otherwise the company finds out it blew the budget only once the loss has already materialized."
Sources: [1] GeekWire (Nov 2021) — Zillow Offers · [2] SEC 10-K FY2021 — Zillow Group
47 · Failure #10: Air Canada¶
"In November 2022, Air Canada's website chatbot incorrectly told a customer that the bereavement fare could be applied retroactively. The actual policy didn't allow that. The customer bought full-price tickets and was denied the discount. The tribunal awarded $812 — a small amount, but a significant precedent.
Air Canada's defense: the chatbot was 'a separate legal entity responsible for its own actions.' The tribunal called this 'a remarkable submission' and rejected it: a company is responsible for all information on its website, whether it comes from a static page or a chatbot.
The small dollar amount sharpens the lesson: precisely because the sum was modest, the precedent is clean — the tribunal was issuing a principled ruling about accountability, not settling a money dispute. For you, this turns the abstract question 'who is accountable for an AI's output' into a concrete legal fact.
The lesson, as a principle: you own every answer your bot gives — the same as a static page on your website. 'The chatbot made a mistake' is not a defense; support content requires the same QA process for staying in sync with policy as a human agent's script does."
Sources: [1] CanLII — Moffatt v. Air Canada, 2024 BCCRT 149 · [2] American Bar Association — case commentary
48 · Failure #11: Klarna + NYC MyCity¶
"This block carries two failures plus a third example of the same class — together they outline different facets of one problem: automated support with no guaranteed human path.
Klarna. In February 2024, Klarna and OpenAI announced that an AI assistant had handled two point three million conversations in its first month, doing the work of seven hundred full-time agents. By May 2025 the CEO acknowledged a reversal: 'cost became too dominant a factor in the evaluation… you end up with a lower-quality outcome.' An important nuance: what got reversed was specifically the policy of 'AI-only, no human access' — the volume of automation itself kept growing, reaching the equivalent of eight hundred fifty-three full-time positions by the end of 2025. This isn't 'Klarna abandoned AI' — it's an acknowledgment that a pure speed metric with no guaranteed human fallback was the wrong design. Augmentation, not replacement.
NYC MyCity. A chatbot for small businesses advised employers to withhold part of their employees' tips — which is illegal. The basis that makes this a systemic failure: all ten out of ten journalists who tested it got the same wrong answer. The mayor acknowledged the errors but declined to take the bot offline.
A third example of the same class, briefly: a user used a prompt injection to get a Chevrolet dealership's chatbot to agree to sell an SUV for one dollar as a legally binding offer.
The lesson for the whole class: any public-facing bot whose response could create a legal obligation needs a deterministic, non-LLM guardrail layered on top — hardcoded rules the LLM cannot talk its way around with any text. Success can't be measured by throughput alone — you need guaranteed escalation to a human from day one."
Sources: [1] Bloomberg (May 8, 2025) — Klarna · [2] Fortune (Oct 10, 2025) — 853 FTE · [3] The Markup (Mar 29, 2024) — NYC MyCity
49 · Synthesis of Section 5¶
"Let's pull Section 5 together: operations is the point where the loop physically closes. Signal from support tickets gets converted into updates to the reference dataset, the prompt, the guardrail — and back into production, and with a deeper signal, back into intent itself.
Keystone callback: signal from operations changes intent — the same loop, closed. AI changed the cost and trust of every arrow asymmetrically, but observing, interpreting, and deciding remain human steps."
50 · Divider: Section 6¶
"And the last, sixth arrow of the loop — Governance, ROI, the finale. This is the capstone: it pulls the whole loop together and answers the paradox we opened with, more than an hour ago. Here we go up to the organizational level: why building is nearly free, and value is not."
51 · ELI5: Governance in plain terms¶
"In plain terms: an organization doesn't have one product, it has a portfolio of initiatives, and someone has to decide which ones to fund and which to stop. That decision can't be made on the feeling of 'this model is cool' — it needs a criterion for success, agreed on in advance and written down as a number, plus a person who tracks it.
A spoiler for this section: most of the time an AI initiative fails to pay off not because the model is bad, but because that criterion was simply never written down."
52 · FOUNDATION: portfolio governance¶
"Portfolio governance is a classical discipline: an organization runs not one product, but a portfolio, and decides what to fund by criteria agreed on in advance. A direct descendant of the Stage-Gate from Section 3, raised to the portfolio level.
Unit economics — the per-unit economics of a product. For an AI product, the new variable is cost per request: a product where cost per request exceeds value per request is economically unviable, no matter how impressive the demo. And unlike SaaS, the variable cost grows right along with usage.
Let me make 'track ROI' operational. A bad KPI: 'improve support efficiency' — not measurable. A good one, written down before the pilot: 'we believe AI triage will cut cost-per-ticket from X to Y without degrading CSAT; we'll test it on N tickets over M weeks; owner: a specific named person.' The same apparatus — falsifiable hypothesis, OEC, guardrail metric — applied to the decision of whether to fund an AI initiative."
53 · AI: operating model¶
"The thesis of this block: five independent sources — Deloitte, Sber, Gartner, McKinsey, Forrester — work in different markets, and they converge: the bottleneck for extracting value from AI has shifted from technology to the organization's operating model. The winner isn't whoever has the more powerful model, it's whoever rebuilt their teams and governance.
Deloitte: seventy-five percent of tech leaders say the operating model needs to fundamentally change, while forty-two percent report low or zero ROI. The bottleneck is structural: legacy funding models run on old annual cycles, while AI workflows are continuous.
Sber, as a peer illustration, not a flagship: a zero-to-five maturity scale, and Sber explicitly places itself at level three, not the aspirational five — a healthy anti-hype signal.
Gartner introduces the term agentwashing — vendors rebranding chatbots as 'agents' with no agentic outcomes to back it up. Criterion: many use cases marketed as agentic don't require an agentic implementation at all.
Forrester: improvements from coding alone deliver thirty to forty percent gains, but team-wide productivity gains without end-to-end adaptation are under ten percent."
Sources: [1] Deloitte — 2026 Global Technology Leadership Study · [2] Sber — the "AI PDLC Lead" job posting
54 · Failure #12: the macro reality¶
"The payoff of the hook we opened with. Let's return to the MIT Project NANDA report, widely reported as 'ninety-five percent of organizations get zero return.'
The report's actual funnel: sixty percent of organizations explored → twenty percent piloted → five percent successfully deployed. The success rate among organizations that reached a pilot is five out of twenty, twenty-five percent — not 'ninety-five percent failed.' The media interpretation conflates 'never seriously piloted' with 'piloted and failed.' And an undisclosed conflict of interest: all four authors of the report sell agentic-AI products through that same project.
Let's extract the transferable skill. Faced with a loud AI-failure statistic, ask four questions. What's the denominator? Without it, a percentage is uninterpretable. What counts as a failure? 'Didn't pay off on schedule,' 'abandoned,' and 'delivers no value' are three different events, collapsed into one. What's the author's conflict of interest? Does the number trace back to a primary source with a stated methodology?
A meta-lesson about fabricated precision: RAND cites 'more than eighty percent' as a hedged reference to other people's estimates. The widely circulated figure of eighty point zero three percent does not appear anywhere in the actual RAND report — a secondary fabrication of false precision. Use RAND as a lesson about fabrication, not as the source of that number.
Triangulating the macro figures: Gartner — only twenty-eight percent of AI use cases are fully successful, a separate survey forecasts more than forty percent of agentic projects will be canceled by the end of 2027. S&P Global — forty-two percent of companies plan to abandon most of their AI initiatives, up from seventeen percent a year earlier. BCG closes the triangulation: sixty percent of companies track zero financial KPIs tied to AI value — every source points to the same root cause: not 'the model is bad,' but 'there was never an agreed-upon criterion and a named owner.'"
Sources: [1] MIT NANDA — "The GenAI Divide" (July 2025, v0.1) · [2] Fortune (Aug 18, 2025) — media distortion of the MIT report · [3] NewMR — Myth #2 · [4] Gartner (Apr 7, 2026, 782 I&O leaders) · [5] BCG — Clos
55 · Failure #13: Just Walk Out + the matrix¶
"Since 2018, Amazon has positioned Just Walk Out — cashier-less stores — as pure computer vision, without disclosing human involvement. An investigation found that roughly a thousand workers in India were reviewing video to label ambiguous transactions.
The key baseline that turns this case from 'there are humans behind the curtain' into 'the claimed autonomy wasn't actually achieved': in 2022, seven hundred out of a thousand transactions required manual review, against an internal target of just fifty out of a thousand — roughly fourteen times the target. Amazon disputed the phrase 'live observation,' but not the number itself.
The lesson: claims of autonomy have to be verifiable and include honest disclosure of the share of human involvement. The hidden-human-cost criterion: if acceptable accuracy requires manual review at a multiple of the target rate — the claimed level of autonomy hasn't actually been reached.
And now — the summary matrix, all six phases folded into one decision apparatus. Discovery — cost and trust in synthesis both dropped; Customer Development and live interviews remain; the criterion is a go/no-go decision made on synthetic data with no real human. Design — prototype cost dropped; a design system remains as a guardrail; the criterion is vulnerable users without a safety audit. Build/Launch — building collapsed to zero, the bottleneck moved to review; feature flags and Stage-Gate remain; the criterion is an irreversible action with no eval gate. Measure — trust dropped the most; A/B testing and guardrail metrics remain. Support/Operate — observation is exposed to silent drift; SRE discipline and escalation remain. Governance — the operating model matters more than ever; a financial KPI with a named owner remains.
This isn't a table to memorize — it's an apparatus to apply to an unfamiliar case. Thirteen failures are thirteen runs through the matrix. You should be able to run the fourteenth yourselves."
Sources: [1] Business Standard (Apr 2024) — Just Walk Out · [2] Retail Dive — the dismantling of Just Walk Out
56 · Keystone payoff + checklist + Q&A¶
"Now let's resolve the paradox we opened with, fully.
Building became nearly free — true. Almost nobody extracts value from it — also true, but with a caveat: twenty-five percent of those who reached a pilot, not five percent of everyone. The resolution, memorize it word for word, this is the substantive takeaway of the lecture:
When building is free, the scarce resource is no longer execution — it's judgment: the ability to tell real signal from noise, and to keep intent and accountability with a human.
An engineer who can build anything instantly but can't tell whether it worked is more dangerous than before. The classical discipline of every phase isn't obsolete: it's the apparatus for telling signal from noise, brought into an operational form. AI changes the cost and trust of every arrow asymmetrically — but it does not cancel the loop itself, or the human at its center.
Symmetry with Lecture 4: there, the thesis was 'AI changes the cost of writing code, but not the cost of understanding what to build and who is accountable.' Here — the same thesis, one level up: AI changes the cost of executing any phase, but not the cost of judging whether it should be executed, and whether it worked.
A practical apparatus for checking any phase before going AI-first, eight questions: is the classical discipline in place? Does trust match the cost? Is there an eval and a reference dataset? Is there a guardrail metric? Staged rollout and rollback? Human escalation? Who's accountable? Is the data safe? A 'no' on even one of these doesn't mean 'don't use AI' — it means 'there's a structural gap here that needs closing before, not after.'
Recall your answer to the very first question today — on which arrow speed outran trust. Run it through the checklist right now.
In the paired seminar you'll walk the same loop on a practice AI product — from hypothesis, through a reference dataset and three evaluations, to a guardrail metric and a human-escalation point.
Thank you. Questions?"