Essay · AI technical debt
The AI Prototype Trap
AI technical debt compounds faster than any software debt I've managed in 30 years. Here is where it builds up, the controls I use to stop it, and the questions leaders should ask before anyone says "ship it."
The demo went well. An AI assistant answered policy questions, pulled up the right documents and impressed every executive in the room. The direction that followed was predictable: get this into production.
Within a few months, some version of this happens. The same question gets two different answers depending on the day. A user finds a phrasing that coaxes out the system's hidden instructions. The model the team built against is scheduled for retirement, and its replacement handles the carefully tuned prompts differently. An agent that was given broad tool access to make the demo smooth does something nobody approved. The monthly model bill is triple the estimate. And the engineer who wired it all together has moved to another project.
None of this is bad luck. It is AI technical debt: the future cost an organization takes on when it optimizes an AI system for time-to-demo instead of time-in-production. Every team carries some. The ones that get hurt are the ones who don't know how much they're carrying.
I learned the model part of this the hard way. On July 9, my production platform, HeadHunterAgent.AI, started getting "model no longer available" errors from the Gemini 2.5 family, more than three months before Google's announced shutdown date, as access was withdrawn in waves. My fallback chain didn't save me. Both of its models came from the same family, so they failed together and ingestion stopped. By the end of that day, every agent task was on the successor models, and the chain had a fourth fallback tier. Two lessons stuck. A fallback that can be retired on the same day as your primary model isn't a fallback. And an automatic fix has to know why a model failed: a retired model should trigger a migration, while a billing or permission error should alert a person and change nothing. I built that second rule in September.
Why AI debt compounds faster
Traditional software debt is painful but contained. The code is messy and the tests are thin, but traditional software is mostly deterministic: the same input usually produces the same output, and a passing test still means something tomorrow.
Google engineers described the core problem in 2015, in "Hidden Technical Debt in Machine Learning Systems" (Sculley et al., NIPS 2015): in a real ML system, the model code is a small fraction of the whole, and most of the debt lives in everything around it. Generative AI and agents make that surrounding area much larger, and they break the assumption of repeatable outputs in four ways:
- The outputs are probabilistic. Wording, conversation history, retrieved context and model version all shift the answer. Even with temperature set to zero (the setting meant to make answers repeatable), two runs aren't guaranteed to match.
- Your most important dependency belongs to someone else. Providers update the models behind aliases (general names like "latest" that they can repoint), publish retirement dates, change prices and throttle under load. Each of those is a change you didn't make but still have to absorb.
- Agents act. A chatbot that's wrong gives a bad answer. An agent with tool access can send the email, update the record or spend the money.
- Cost scales with every call. A loop, a retry storm or a popular feature turns directly into spend, often before anyone is watching.
Change any one variable (the model, the prompt, the data, the tools) and the effect cascades through every downstream interaction. Leave those variables unmanaged and the debt grows every month the system runs.
Six places it accumulates
AI debt doesn't live in one place, and fixing one surface while ignoring the rest gives a false sense of safety. This is the map I use, and where I've put each control into practice, including in products I own end to end, where I can run every control at full rigor.
Model dependency
An alias quietly moves to a new version. A model is retired. A provider throttles at peak.
Pin versions, and route each task to the right model class. Fall back automatically only to models that have passed the same evaluation set, and alert when it happens. If a floating alias is used at all, make it the last resort. Treat a retirement as a planned migration, with failover as the safety net.
HeadHunterAgent.AI runs 15 agent tasks behind one gateway with a four-tier fallback: the primary model, a fallback in the same class, a pinned safe floor, and only then a floating alias as the last resort.
Evaluation
No baseline. Prompt and model changes ship because they "looked better" in a few tries.
A fixed evaluation set from day one, scored by deterministic rules plus a panel of model judges and spot-checked by people, with a gate before any change reaches users.
HeadHunterAgent.AI gates its five most critical agent tasks with pass/fail checks against a fixed test set before any model change ships. When the cheapest candidate model failed that gate by dropping three qualified leads, it stayed out of production. A different lower-cost model passed, and together with the rest of the cost work, measured model costs fell 56–64%.
Prompts, inputs & tools
Prompt injection. Leaked system instructions. Agents with more permissions than the task needs.
Prompts in version control. Input and output guardrails. Least-privilege tools (only the permissions the task needs), with a person approving anything consequential.
Power Platform X-Ray treats uploaded solution files as untrusted model input, and I test it against six injection scenarios in which a file tries to redirect the analysis. My rule for the coding agents I direct: nothing reaches production or switches on an automation until I approve it.
Data & retrieval
Stale indexes (the searchable store of documents the AI draws on). Sensitive data in the model's context. Content nobody vetted.
Quality gates on what gets indexed, freshness checks, data minimization and redaction, and provenance on every answer.
The Opportunity Radar has read-only access to its sources, links to source records instead of copying them, and keeps an audit trail for every case it raises.
Cost
Unbounded consumption: loops, retries and success all show up on the bill.
Per-call cost metering, budgets with alerts, circuit breakers that stop spend automatically, and a kill switch a person can pull.
HeadHunterAgent.AI meters the cost of every call, alerts against a monthly budget, and enforces an hourly hard limit on its active job search. Circuit breakers on its paid data sources step from warning to critical to hard stop before a quota runs out, and one switch pauses the whole pipeline. CI runs against mocked model responses, so tests never pay for a model call.
Ownership & operations
No named owner. Silent failures. Nobody can explain how an output was produced.
A named owner, observability, a dry-run mode, a kill switch, and alerts when the system goes quiet.
The timesheet service has dry-run routing, a kill switch and a test suite proving nothing reaches anyone unless it's live. The Radar posts a heartbeat after every run and a warning when a run is missed.
Common vocabulary for these controls: the NIST AI Risk Management Framework and its Generative AI Profile (NIST AI 600-1), ISO/IEC 42001, and the OWASP Top 10 for LLM Applications (2025).
The cheapest AI debt is the model you didn't need
The most effective control is also the least glamorous: use a model only where it earns its place. Everything else should be plain, deterministic code you can test, audit and explain.
Two systems I built this year make the point. A timesheet-compliance service and a CRM opportunity radar both reach people where they already work, in Microsoft Teams, instead of waiting for someone to open a dashboard. I designed both and built them with AI coding agents under my gated review process. At runtime, both run on fixed rules, because "who is late" and "which won deal has nobody named" are questions with exact answers. The result is a system that behaves the same way every day and can be explained to anyone who asks.
Internal measures, reported as observed in the first weeks after practice-wide rollout.
The numbers matter less than the property behind them: the same behavior every day, explainable to anyone. Put AI where judgment, language or ambiguity is the actual problem. Keep it out of places where a rule will do.
Prototype fast, with a contract
I'm not arguing against speed. Stakeholders need to see capability early, and I build proofs of concept on aggressive timelines. The difference is that I give every prototype a short contract, so a demo can never quietly become the production system nobody designed.
The prototype contract
- A written scope boundary. What it does, what it doesn't, and which data it may touch.
- An expiration date. After that date the prototype is retired or rebuilt, never quietly extended.
- A production ticket, opened in the same sprint as the demo, with the architecture, security and operating work it would take to ship.
- An evaluation set from the first day, so "better" means something measurable.
- A spend ceiling, enforced by the system rather than remembered by a person.
This is also how I build with AI coding agents. In my engineering harness, the agents write the code and the gates decide what ships: a spec, a build, an adversarial review, CI and a human approval. Speed and rigor stop competing for the same hours.
Questions to ask before you say "ship it"
- Which model versions does this depend on, and what happens when one is retired?
- How do we know a change made it better? Show me the evaluation set and the score.
- What can this system do without a person approving it?
- When someone talks it into ignoring its instructions, and someone will, what's the worst it can do?
- What will it cost at ten times today's usage, and what stops the spend automatically?
- Who owns it, and how would we know if it stopped working tonight?
- Which parts actually need a model, and which could be plain rules?
If a team can answer all seven with evidence, the system is ready for a production decision. If it can't, the demo was the easy part.
The real test of an AI system comes a year after launch: can you still change it safely, and can you still afford to run it?
If someone you know is about to ship AI, send this to them first.