Why the demo works and the product doesn't
A demo follows one path, with inputs chosen by the person who built it, in front of an audience willing to forgive a pause. A product gets every input, from people who don't know what it expects, and has to keep working while you change it every week.
Conventional software has a helpful property: code that works keeps working until someone edits it. Software built on a language model doesn't have that property in the same way. The same prompt can produce different outputs on different runs, a small wording change can shift behaviour somewhere you weren't looking, and a new model version can alter results without a single line of your code changing.
So the question that decides whether an AI prototype can become a product isn't "does it work?" It's "can we change it and know whether it still works?" The seven problems below are all versions of failing that test.
Seven structural reasons, and the habit that prevents each
1. Prompts scattered through the code
In a prototype, prompts get written where they're used: a string in a route handler, another in a helper, a third built up from fragments in three files. Nobody can then see what the product is telling the model, and a change to one prompt can't be reviewed or reversed cleanly.
The habit: keep every prompt in one place, as named templates under version control. A change to a prompt should be as visible in review as a change to code.
2. No evaluation set
Without a fixed set of examples to test against, every prompt change is judged by trying two or three inputs and deciding it "seems better". That works until a fix for one case quietly breaks another, which you learn from a user.
The habit: collect a couple of dozen real inputs with a note on what a good output looks like, and run them before any prompt or model change. It doesn't need tooling at first. A script and a spreadsheet are enough. This is the single habit I'd keep if I could only keep one.
3. Model calls tied to one provider
When a provider's library is called directly from many places, the provider's request format, response shape and quirks spread through the codebase. Changing model, even to a newer version from the same provider, becomes a project.
The habit: route all model calls through one module of your own. The rest of the code asks for a result; only that module knows which provider supplies it. This also gives you one place for logging, timeouts and cost tracking. It's the "own the seam" principle from buy vs build vs glue.
4. No handling of slow, failed or garbled responses
Model APIs are sometimes slow, sometimes unavailable, and sometimes return output that isn't in the format you asked for. A prototype assumes none of this happens. In production it all happens, and the user sees a spinner that never ends or a screen of broken text.
The habit: set a timeout on every call, retry a limited number of times, validate the structure of the response before using it, and decide what the user sees when all of that fails. A plain "that didn't work, try again" is better than silence.
5. Costs never measured
Hosted models are billed by usage, so cost depends on how long your prompts are, how much context you send and how many calls each user action triggers. In a prototype these numbers are invisible. Pricing then gets set without knowing whether each use of the product makes or loses money.
The habit: record token usage on every call and work out the cost of one core action. You don't need it to be low yet. You need to know what it is.
6. Demo data instead of real data
Prototypes are tested on clean, short, well-formed examples. Real data is long, messy, duplicated, in the wrong language, or missing the field you relied on. An approach that looks excellent on ten tidy documents can fall apart on the first real customer's archive.
The habit: get real data in front of the system as early as you're permitted to, and make the awkward cases part of the evaluation set.
7. No logging of inputs and outputs
When a user says "it gave me a wrong answer", you need to see what went in and what came out. Without that record you can't reproduce the problem, and with a system that isn't fully repeatable you may never see it again.
The habit: log the input, the output, the prompt version, the model version and the time taken for every call. Treat those logs as the user data they are: restrict access, set a retention period, and leave out what you have no right to store.
| Problem | How it shows up | Cheap habit |
|---|---|---|
| Scattered prompts | Nobody knows what the model is being told | One place, named, version-controlled |
| No evaluation set | Fixes break other cases unnoticed | A couple of dozen real examples, run before each change |
| Provider coupling | Changing model is a project | One module for all model calls |
| No failure handling | Endless spinners, broken output | Timeouts, limited retries, validation, a fallback message |
| Unmeasured cost | Pricing set blind | Record usage; know the cost per core action |
| Demo data | Works on samples, fails on customers | Real data early; awkward cases in the evaluation set |
| No logging | Can't reproduce a complaint | Log input, output, versions and timing, with care |
None of these takes long. Together they're a matter of days, which is why they belong in the first version, not a later "hardening" phase. They're the AI equivalent of the items I'd refuse to cut in the MVP scope cuts that actually save time.
When a rewrite is the right call
A rewrite has a bad reputation, mostly deserved. Joel Spolsky's Things You Should Never Do is the classic argument against it: old code contains fixes for problems you've forgotten you had. But that argument is about mature products with years of accumulated knowledge in them. A prototype of a few weeks' work is a different case.
A rewrite is likely the right call when:
- The data model is wrong. If the core records are the wrong shape for what the product turned out to be, patching around it costs more every month.
- The core approach changed. You built an open-ended chat interface and learned users want a fixed workflow with a model making a few decisions, or the reverse.
- It was built in a tool that can't carry it to production. Some prototyping tools are excellent for learning and unsuitable for running a business on.
- Nobody understands the code. If it was generated or assembled quickly and no one can explain how it works, every change is a gamble.
- It's small. Rewriting a few weeks of work, with everything you now know, is often quicker than repairing it.
A rewrite is likely the wrong call when the complaints are the seven problems above. Those can be retrofitted one at a time into a running product: gather the model calls into one module, add logging there, build the evaluation set from what you log. It's also the wrong call when the real motive is that the code is unpleasant to look at.
Whichever way you go, the prototype's most valuable output isn't its code. It's what you learned: which inputs are hard, which prompts worked, what users actually asked for. Keep that, whatever happens to the code.
What to read next
If you're still at the planning stage, start with how fast you can actually ship an MVP. The full series is under Shipping AI products fast.