The launch is not the finish line
The demonstration went well. A mid-sized software company had built an assistant that read customer contracts and answered questions about renewal terms, notice periods and liability caps. In the boardroom it answered every question correctly, in seconds, with the relevant clause highlighted. The board approved a release to existing customers within the quarter.
Six weeks after launch, the picture was less tidy. Some customers uploaded scanned PDFs, and the extraction step quietly produced garbled input. An integration with the document management system timed out at month-end. The support desk began receiving tickets it had no script for: “The assistant says my notice period is 90 days but my lawyer says 60.” Nobody was sure whether that was a bug, a data problem, a model limitation or a user misunderstanding — or who should find out.
That scenario is illustrative rather than drawn from a specific company, but anyone who has shipped an AI feature will recognise its shape. The prototype proved that the capability existed. It did not prove that the product could be run.
The gap between those two things is where many AI initiatives stall. In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs and unclear business value (Gartner). Most of those causes are not modelling problems. They are problems of engineering discipline, operations and support — the work that begins in earnest once real users arrive.
This article argues that AI applications need a connected lifecycle: development that plans for production from the outset, and technical operations and support that feed what they learn back into the product. Treating these as separate phases, handed from one team to another, is one of the more reliable ways to turn a promising prototype into an expensive disappointment.
Why moving an AI prototype into production is difficult
A prototype is built to answer one question: can this be done? A production system has to answer many more. Can it be done reliably, for thousands of users, on data nobody curated, alongside systems that were never designed with it in mind, at a cost the business can sustain?
Some of the resulting difficulties are familiar to any software team; others are specific to systems whose behaviour depends on models and data. The distinction matters, because the familiar problems have mature solutions and the AI-specific ones often do not.
Problems every production system shares
Integration. A prototype usually runs against a sample export or a sandbox API. In production it must authenticate against identity providers, respect rate limits, handle partial failures in upstream services and fit into existing business processes. An AI feature that works perfectly but writes its output to a field the CRM truncates is, from the user’s point of view, broken.
Performance and scalability. Latency that is acceptable in a demonstration can be intolerable inside a workflow that someone repeats 200 times a day. Language-model calls are typically slower and more variable than conventional database queries, and costs scale with usage in ways that can surprise finance teams.
Problems that are sharper, or new, with AI
Changing inputs. Conventional software fails loudly when it receives malformed input. A model will usually produce something, and that something may look plausible while being wrong. As user behaviour, document formats and source data drift away from what the system was tested on, quality can degrade without any error being thrown.
Non-deterministic outputs. The same prompt can yield different answers. That complicates testing, reproduction of reported faults and the very definition of “correct”. A regression test suite built on exact-match assertions will not catch most of the failures that matter.
Entanglement. In their widely cited 2015 NeurIPS paper on technical debt in machine learning, Google researchers observed that “only a small fraction of real-world ML systems is composed of the ML code” and described a principle they called CACE: “Changing Anything Changes Everything. Adjusting a prompt, swapping a model version or changing a retrieval setting can alter behaviour across the whole application, not just the feature being tuned.
New attack surfaces. Applications built on large language models face threats that conventional applications do not. The lists prompt injection first, followed by risks including sensitive information disclosure, improper output handling, excessive agency and unbounded consumption — the last of which is as much a cost-control problem as a security one.
Monitoring what “working” means. Uptime and error rates tell you whether a service responded. They do not tell you whether its answers were any good. Monitoring an AI application means watching output quality as well as infrastructure health, and that requires deliberate design.
What good AI application development should include before launch
Most of the problems that surface after launch were decided, or left undecided, much earlier. The engineering choices that make an AI application supportable are rarely glamorous, and they are easy to postpone when a prototype is already impressing stakeholders.
Success criteria that someone can measure
“The assistant should answer contract questions accurately” is an aspiration, not a requirement. A usable criterion names the task, the data and the threshold: for example, correct identification of the notice period in at least an agreed share of a labelled test set of 500 real contracts, including scanned documents, with any answer below a confidence threshold routed to a human. The precise numbers are a business decision. What matters is that they exist before launch, because without them nobody can say whether a post-launch complaint represents a defect or expected behaviour.
Architecture that assumes change
Models will be updated, deprecated or replaced. Prompts will be revised. Retrieval sources will grow. An architecture that isolates these components — versioning prompts and model configurations as code, separating retrieval from generation, and logging which version produced which output — makes it possible to trace a reported problem back to its cause. Without that traceability, support teams are left describing symptoms that engineers cannot reproduce.
Testing that reflects real use
Because outputs vary, AI testing tends to combine several techniques: curated evaluation sets scored against agreed criteria, adversarial tests for prompt injection and data leakage, load tests that include realistic peaks, and integration tests against production-like systems rather than mocks. The 2015 Google paper made a point that still holds: unit and end-to-end tests “are not sufficient to provide evidence that a system is working as intended”, which is why the authors argued for live monitoring alongside them.
Human oversight where the stakes justify it
Not every output needs review. But where an AI system takes actions, issuing refunds, changing account settings, sending communications, or advises on matters with legal or financial consequences, teams should decide in advance where a person stays in the loop, and what limits trigger escalation.
Deployment you can reverse
Staged roll-outs, feature flags and a tested rollback path are standard practice in mature software teams. For AI features they matter more, since a model or prompt change can degrade quality in ways that only become visible days later. A rollback plan should cover model versions and prompt configurations, not just application code.
Observability and handover
Finally, development should produce what operations and support teams will need: dashboards for output quality as well as uptime, runbooks for known failure modes, documented limitations and clear incident ownership. The UK government’s Code of Practice for the Cyber Security of AI (January 2025) treats maintenance as a lifecycle phase in its own right, with principles on monitoring system behaviour and communicating with end users. The framing applies well beyond security.
This is where the choice of who builds the application has long-term consequences. Teams commissioning AI application development services, or building internally, should ask how the work will be tested, reviewed and handed over to whoever runs it afterwards, not only how quickly a working version can be delivered. A build that arrives without evaluation data, version history or runbooks transfers its risk to the support desk.
What changes when an AI application meets real users?
Pre-launch testing, however thorough, is a model of reality written by the people who built the system. Users do not read that model. Three illustrative scenarios — hypothetical, but typical of what teams report — show how production exposes what testing missed.
The unexpected input. A SaaS company adds an AI feature that summarises support conversations for account managers. Testing used English-language chat transcripts. In production, a significant group of customers write in a mix of English and another language, and some paste screenshots rather than text. Summaries become patchy. No errors are logged, because the model returns a summary every time. The first signal is an account manager telling a colleague the feature is “hit and miss”.
The fragile integration. An AI scheduling assistant books field engineers by reading a calendar system and a parts inventory. It works until the inventory vendor changes a date format in an API response. The assistant now books visits for parts that have not arrived. The model is behaving correctly on the data it receives; the defect sits in an integration nobody was monitoring for semantic changes.
The plausible wrong answer. An internal knowledge assistant answers staff questions about HR policy. A policy is updated, but the old document remains in the retrieval index. The assistant confidently cites the superseded version. Staff who rely on it make decisions on outdated rules. The fault is not in the model or the code but in content governance — and it reaches engineering only if someone reports it in a way that can be traced.
The common thread is that the first evidence of a problem frequently arrives as a user’s complaint rather than an alert. Support tickets, chat transcripts and feedback buttons are therefore not just a service obligation. They are one of the richest sources of production telemetry an AI product has, provided someone captures them in a structured way and routes them to people who can act.
That is also why the line between “bug” and “support request” blurs. A user who asks why the assistant gave two different answers to the same question may be reporting non-deterministic behaviour that is within tolerance, a retrieval fault, or a genuine model failure. Deciding which requires both support skills and engineering context.
Why technical support is part of the AI product lifecycle
In many software organisations, technical support is budgeted and managed as a cost centre that sits downstream of the product. For AI applications that framing undersells it. Support is where the product’s behaviour meets users’ expectations, and where most of the evidence about that gap is first collected.
Triage that understands the system
Effective triage sorts incoming issues by impact and urgency, but for AI products it also needs to sort them by likely cause. A useful first-line classification might distinguish between user error or misunderstanding, data or content problems, integration failures, configuration issues, and model behaviour outside agreed tolerances. Each routes to a different owner. A support team that lumps them together as “AI not working” will either escalate everything or nothing.
Troubleshooting and escalation paths
Support agents need runbooks that reflect how the system actually works: which logs show the prompt and model version behind an answer, how to check whether a source document is current, how to tell a timeout from a refusal. Escalation criteria should be explicit. A reproducible wrong answer on a legal or financial question, for instance, might go straight to engineering regardless of ticket volume.
Knowledge bases that keep pace with the product
AI features change more often than conventional ones, because prompts, models and data sources are adjusted continually. Support documentation written at launch goes stale quickly. Teams that treat knowledge-base updates as part of each release, rather than a separate task, avoid a common failure: agents giving users advice that was accurate two versions ago.
Where AI helps support — and where it does not
AI tools are increasingly used inside support operations themselves. They can classify and prioritise tickets, retrieve relevant articles, suggest troubleshooting steps, draft responses and spot clusters of similar complaints that a human might take days to connect. Used well, ai saas technical support services can absorb repetitive, well-documented requests and give human specialists more time for the problems that need judgement.
The limits deserve equal attention. An AI support agent inherits the weaknesses of the product it supports: it can misclassify an unfamiliar issue, retrieve an outdated article or give a confident wrong answer. Novel failures — the ones engineering most needs to hear about — are exactly what automated triage is least equipped to recognise. Sensible designs set confidence thresholds for automated handling, keep a person easy to reach, and have humans review a sample of automated resolutions. Vendor figures for tickets resolved without human involvement vary widely with the product and with how “resolved” is defined; they are best tested against a team’s own data.
Closing the feedback loop between developers and support teams
The most valuable connection in an AI product’s lifecycle is also the one most often left informal. Support teams see the product’s failures first. Engineering teams understand why they happen. When the two communicate only through escalated tickets, both lose information: support never learns the root cause, and engineering sees individual incidents rather than patterns.
A worked example
Consider a hypothetical B2B expense-management platform that uses an AI model to read receipts and categorise spending.
- Users report a recurring issue. Over a fortnight, several customers complain that hotel bills are being split incorrectly, with room charges and restaurant charges merged into a single category.
- Support classifies and documents the problem. Rather than closing each ticket with a manual correction, the support lead tags them under a shared problem record, attaches example receipts (with customer permission and appropriate redaction) and notes a pattern: most affected receipts come from hotels using one particular billing system that prints itemised charges in a two-column layout.
- Engineers investigate the cause. Using the logged model and prompt versions, engineers reproduce the fault. The text-extraction step reads two-column layouts row by row across both columns, so the model receives jumbled line items. The model is not the root cause; the pre-processing is.
- A fix is tested and deployed. The team adds layout detection to the extraction step, extends the evaluation set with the redacted examples from support, confirms no regression on other receipt types and releases the change behind a feature flag to a subset of customers first.
- Support materials are updated. The knowledge-base article on miscategorised receipts is revised, the triage guide gains a note on recognising extraction faults, and the affected customers are told the issue has been addressed.
- The team monitors for recurrence. The problem record stays open for a defined period while the team watches the miscategorisation rate for hotel receipts. If it falls and stays down, the record is closed; if not, the loop starts again with better data.
Two features of the example matter more than the specific fix. First, support’s decision to group tickets into a problem record — the distinction between incident management and problem management that IT service management frameworks have long drawn — turned scattered complaints into a reproducible defect. Second, the support team’s examples became permanent test cases, so the same failure would be caught before any future release.
Metrics that show whether the loop is working
Operational metrics are useful precisely because they are limited. Each answers a narrow question, and misreading one as a general verdict on product quality is a common error.
| Metric | What it can tell you | What it cannot tell you | |
| Incident frequency | Whether reported problems are rising or falling over time | Whether fewer reports mean fewer problems or fewer users bothering to report | |
| Time to resolution | How quickly the organisation restores normal service | Whether the underlying cause was fixed or merely worked around | |
| Repeat-ticket rate | Whether fixes and answers are sticking | Why they are not; that needs ticket-level analysis | |
| Application availability | Whether the service was reachable against its target | Whether the answers it gave while available were correct | |
| Error rates | How often requests fail outright | Plausible but wrong outputs, which raise no error | |
| User satisfaction (e.g. CSAT) | How users felt about an interaction | Whether their problem was actually solved, or the product improved |
Google’s site reliability engineering guidance offers a useful discipline: define service level indicators as “a carefully defined quantitative measure of some aspect of the level of service”, set objectives against them, and agree in advance what happens when they are missed, For AI applications, at least one indicator should measure output quality — for example, the share of sampled answers judged correct by reviewers — because availability and latency alone will report a healthy service that is quietly giving wrong answers.
On the engineering side, DORA’s software delivery metrics, which include change fail rate and failed deployment recovery time, show whether fixes are reaching production safely.
Build, support or combine both? Choosing an operating model
There is no single correct way to organise development and support for an AI product. The right model depends on the product’s complexity, the volume and hours of support required, regulatory obligations, and what the organisation can realistically staff. Three broad approaches are common.
The developers also run support
In small teams and early-stage products, the engineers who built the application often handle post-launch issues themselves. Feedback is immediate: the person reading the complaint can open the logs and fix the cause. Context is never lost in a handover.
The weaknesses appear as usage grows. Support interrupts development, out-of-hours cover depends on a few individuals’ goodwill, and knowledge stays in engineers’ heads rather than in documentation. This model suits pilots, internal tools and products with a small, technical user base.
Separate development and support teams
Many established software companies run dedicated support functions, sometimes outsourced, with defined service levels and escalation tiers. This brings specialist skills, predictable cover across time zones and clearer cost control. Engineers can concentrate on building.
The risk is that information stops at the boundary. Support resolves symptoms without learning causes; engineering hears about problems only once they are severe enough to escalate. For AI products, whose failures are often subtle and pattern-based, that delay is costly. The model works best for stable, well-documented products, backed by deliberate mechanisms — shared problem records, regular joint reviews, shared access to logs.
An integrated model
Here development, operations and support remain distinct roles but share processes, tooling and data. Support can see the observability tools engineering uses, problem records link directly to backlog items, and each release updates support documentation. The approach borrows from DevOps and site reliability engineering practice.
It is also the most demanding model to set up. It requires shared tooling, agreed definitions of severity and ownership, and leadership willing to treat support insight as an input to product decisions. For a small product with low support volume, the overhead may not be justified.
Weighing the options
| Decision factor | Points towards developer-run support | Points towards separate teams | Points towards an integrated model | |
| Product complexity | Low; few integrations | Moderate; stable behaviour | High; frequent model or data changes | |
| Support volume | Low | High and predictable | Moderate to high, with recurring patterns | |
| Operating hours | Business hours suffice | Extended or 24/7 cover needed | Extended cover plus rapid engineering response | |
| Compliance needs | Minimal | Documented processes and service levels | Audit trails linking incidents to changes | |
| Internal capability | Strong engineers, no support function | Established support function | Both, or a partner able to provide both | |
| Budget | Tight | Predictable operating spend preferred | Willing to invest in shared tooling and process |
Organisations can also mix approaches, building some capabilities internally and engaging external specialists for others — for example, keeping core engineering in-house while using an external provider for extended-hours support. What matters is that the boundary between the parties is designed for information to cross it.
A practical checklist for preparing an AI application for production
The questions below are intended for a launch review. A team that cannot answer one of them has found its next task.
- Acceptance criteria. Are quality thresholds written down, agreed with the business owner and tied to a labelled evaluation set drawn from real data?
- Testing and validation. Has the system been tested on messy, real-world inputs, at realistic peak load, against production-like integrations, and with adversarial prompts?
- Security and access controls. Are permissions, data retention, data residency and third-party model terms documented, and has the application been reviewed against known LLM risks such as prompt injection and sensitive data disclosure?
- Monitoring and alerting. Do dashboards track output quality and cost per request as well as uptime, latency and errors? Who receives each alert?
- Incident ownership and escalation. Is there a named owner for each class of incident — model, data, integration, infrastructure — and written criteria for escalating to engineering?
- User-facing documentation. Do users know what the feature can and cannot do, how to report a problem and how to reach a person?
- Engineering handover. Has support received runbooks, known limitations, access to the relevant logs and a walkthrough from the engineers who built it?
- Feedback loops. Is there a mechanism for grouping related tickets into problem records and turning confirmed failures into new test cases?
- Maintenance responsibilities. Who will handle model deprecations, prompt revisions, retrieval-index updates and dependency patches, and how often?
- Operational review. Is a date set, typically within the first weeks after launch, to review incidents, metrics and support themes and decide what changes?
Treat AI as a continuing product lifecycle
A working model is evidence that a product is possible. A successful launch is evidence that it can be delivered. Neither is evidence that it will keep working as users, data, integrations and the models themselves change around it.
The organisations that move AI applications from prototype to production most reliably tend to share a habit of mind rather than a specific technology: they treat development, monitoring, maintenance and support as connected responsibilities. Engineers design for the people who will run the system. Support teams are equipped to diagnose as well as respond, and what they learn shapes the next release.
Whether those capabilities are built internally, sourced from external specialists such as Flairstech, or combined in some mix depends on each organisation’s scale, skills and risk appetite. The more important decision is to plan for the whole lifecycle before the first demonstration convinces everyone that the hard part is over.