Architecture diagrams for AI products have converged on the same picture. Four stacked boxes, orchestration over model over retrieval over observability, with an arrow running through the middle. The diagram is fine. It is also not where anything goes wrong.
What goes wrong is six choices made in the first month, usually in a hurry, usually before anyone draws the diagram at all. Two of them you can undo in an afternoon. Two of them you cannot undo without a rewrite. Almost nobody sorts them into those piles before committing, which is the actual reason AI products get expensive in year two.
The model is a component, not the product
Fix one boundary early: what the model does, and what your system does.
A model interprets language and generates tokens. That is the whole job. It does not enforce permissions. It does not own durable state. It does not carry accountability for an action taken in your product's name. Your system does all three, and if it does not, you have shipped a policy engine nobody can audit.
The consequence is operational rather than philosophical. Draw the boundary clearly and a model upgrade is a config change plus an eval run. Draw it badly and every provider release becomes a production incident, because the behavior you depended on was never written down anywhere except inside a system prompt.
There is a one-question test for this. Can you describe, without opening the model call, what your product is allowed to do? If the answer lives in prose inside a prompt, that rule is not enforced. It is suggested.
This much is well covered elsewhere. Almost every serious write-up on AI product architecture, from Andreessen Horowitz's survey of emerging LLM application architectures (2023) to the GitHub Blog's walkthrough of the same territory (2023), lands on some version of it. Take it as settled. The six decisions below are the ones still being got wrong, and they are ordered by how much they cost to reverse.
Decision one, which model and how many
Three real options exist: commit to a single provider, build a routing layer across several, or fine-tune something of your own.
Our default is the first, and the reason is narrower than it sounds. Start with one strong general model behind an abstraction thin enough that swapping providers is a day of work rather than a quarter. Add routing when you have measured a cost or latency reason to route. Not before.
Premature routing is the more common mistake, and it costs you in a way that never shows up on the bill. You get a gateway nobody can debug, sitting in front of two models that behave differently on the same prompt, and every quality regression now has two suspects instead of one. Teams reach for it because routing sounds like maturity. Routing is an optimization. An optimization you cannot measure is complexity wearing a better name.
The opposite mistake costs more. With no abstraction at all, provider quirks leak into business logic: the tool-call format, the refusal behavior, the way one model handles a long system prompt while another truncates it. Six months in, "try the new model" is a two-week estimate.
What to abstract is narrower than most teams assume. Abstract the call itself, the message format, the tool schema, the retry and timeout policy. Do not abstract prompt content. Prompts are not portable across models, and pretending otherwise produces a lowest-common-denominator prompt that underperforms on every model you support.
Fine-tuning is rarely the first answer in 2026. It earns its place on narrow, stable, high-volume tasks where prompt length is the dominant cost driver, or where you need a small model to behave like a large one on one specific job. If your task definition is still moving, fine-tuning locks in a snapshot of a spec you have not finished writing.
Decision two, whether you need retrieval at all
Now the question the rest of the internet skips, because the answer is not always yes.
Retrieval earns its place when the corpus is larger than what you can fit in context, changes often enough that reindexing beats redeploying, needs per-document access control, or needs attribution the user can click. When several of those are true, build it.
When none of them are true, you are about to take on an indexing pipeline, an embedding model version, a chunking strategy, and a second failure mode, in exchange for something a well-cached prompt would have done. Small stable corpora belong in the prompt. If the documents fit and rarely change, put them there and cache them.
There is a reason the default answer online is always yes. A large share of the pages ranking for architecture questions are published by vector database vendors, and their answer to every architecture question routes through a vector database. That is not dishonesty. It is the shape of the incentive. Read those pages for the mechanics, not the recommendation.
If you do need retrieval, the sub-decisions that determine quality are not the ones teams argue about.
- What you embed. Embedding the raw chunk is the default and is often wrong. Embedding a generated summary, or the chunk plus its section heading and document title, changes retrieval quality more than any database choice will.
- Chunking. Fixed-size chunks are easy and lose structure. Structure-aware chunks are more work and retrieve better. Pick deliberately, because changing it later means reindexing everything.
- Hybrid versus pure vector. Keyword search still wins on exact identifiers, product codes, and error strings. Pure semantic search fails on exactly the queries where users are most confident they typed the right thing.
- Access control. If different users may see different documents, filtering has to happen inside retrieval, not after. Retrofitting that is painful.
One habit is worth building before any of the above: evaluate retrieval separately from generation . Ask whether the right document came back, as its own measurement, before asking whether the answer was good. Teams that skip this spend months tuning prompts to compensate for a retriever that was returning the wrong chunk the entire time, and every hour of that prompt work gets thrown away the moment retrieval is fixed. It is the cheapest instrumentation on this list. It is also the one most often skipped.
Decision three, workflow or agent
Autonomy is a spectrum with a price attached, not a binary.
The anchor points, in order of increasing autonomy: a fixed chain of steps, a chain with a routing decision in it, an agent with a bounded tool set and a loop cap, and a multi-agent system where agents call each other . LangChain's Harrison Chase made the useful point back in 2024 that these are levels of a cognitive architecture rather than different technologies, and the level you pick is a design decision you should be able to defend.
Our default is the least autonomy that solves the problem. A large fraction of things built as agents are workflows with a conditional in them.
Two arguments drive that, and the first shows up in the invoice.
Cost. A workflow's token spend is a number you can compute before you ship. Steps times average tokens per step, and you are done. An agent's spend is a function of loop count, and loop count is not a number you control directly. It emerges from how hard the task turned out to be, how good the tool descriptions are, and whether the model got confused. The distribution has a long tail, and the tail is where the surprise invoice lives .
Debuggability. A workflow failure gives you something like a stack trace. You know which step ran, what it received, and what it returned. An agent failure gives you a transcript, and working out from a transcript why the model chose the wrong tool on turn seven is a slower and much less pleasant activity than reading a log.
Agents do earn their keep when the space of valid action sequences is too large to enumerate in advance, and when a wrong sequence is recoverable rather than destructive. Research tasks. Triage across many possible tools. Anything where the next step really depends on what the last step found. The rule of thumb is blunt: if you can draw the flowchart, build the flowchart.
| Autonomy level | Fits when | Cost predictability | Typical failure |
|---|---|---|---|
| Fixed chain | Steps are known in advance | Computable before launch | A step gets bad input |
| Chain with routing | A few known paths | Computable per path | Router picks the wrong path |
| Bounded agent | Sequence depends on findings | Estimable, long tail | Loops, wrong tool choice |
| Multi-agent | Genuinely separable sub-problems | Hard to bound | Cascading confusion between agents |
Decision four, who owns memory
This is the least-discussed decision on the list and the second most expensive to reverse.
Start by separating four things teams routinely lump together as "memory":
- Conversation history, the raw turns of an exchange.
- Retrieved context, pulled fresh per request and not durable at all.
- Durable user state, the facts and preferences your product is expected to remember next week.
- Agent scratchpad, intermediate reasoning that exists for the length of one task and should be thrown away.
Only the third one is really memory. Treating the other three as memory is how context windows fill up with material nobody needed.
The architectural question is ownership. Does durable state live in your database with a schema you designed, or inside a framework's session object?
Our default is your own store, with the framework holding nothing you would miss after a restart. The failure mode of framework-owned state is specific and unpleasant: you cannot query it alongside the rest of your data, you cannot migrate it independently, and a framework version bump becomes a data migration with no migration tooling. Conversation history in a table you own is boring. Boring is the entire point.
Summarization and truncation belong here too, and they are cost decisions as much as context-window decisions. Every turn you carry forward is paid for again on the next request. A product that never summarizes has a per-conversation cost that grows quadratically, which is fine in a demo and is not fine at ten thousand users.
State schemas calcify fast, because everything downstream starts reading them within weeks. Decide this one properly on day one, or accept that you have decided it by accident.
Decision five, orchestration and the control plane
The orchestration layer owns the unglamorous parts: routing, retries, timeouts, fallbacks, rate limits, and tracing. It is also where lock-in happens, which makes it the most expensive item in this article.
The framework question has an honest answer that satisfies nobody. Frameworks buy real speed early and cost real flexibility later, and the bill arrives at exactly the moment you need behavior the framework did not anticipate. That moment is not hypothetical. It lands roughly when you first have production traffic and a specific latency or cost problem to solve.
Our default is to split it. Use a framework for the parts that are commodity, meaning provider adapters, retry and backoff, tracing instrumentation, streaming plumbing. Own the parts that encode your product's logic, meaning the control flow itself, the decision of what happens next. If the framework owns your control flow, removing it later is not a refactor. It is a rewrite.
Observability belongs in week one, and it is not the same thing as logging. The requirement is being able to answer "why did this specific response happen" for a request a customer is complaining about right now. That means span-level tracing : the prompt version, the retrieved documents, the tool calls, the model and its parameters, the token counts, and the latency of each. You can retrofit it, but you lose every incident that happened before you did, and those are the incidents you most wanted to understand.
The related discipline is prompt versioning. Prompts are code. They belong in version control, they should be reviewed, and a response should be traceable to the exact prompt version that produced it. Teams that keep prompts in a database with no history end up unable to explain a quality regression that happened last Tuesday.
Decision six, serving and the shape of the cost curve
Serving choices run from a managed API, to managed inference inside your own cloud account, to fully self-hosted open-weights models. The trade is ops burden against control over cost, latency, and data residency. Most products should start at the managed end and move only when a specific constraint forces it, because self-hosting converts a variable cost into a fixed cost plus a team.
More useful than the serving choice is knowing where the money actually goes, because it is not where teams look first. In rough order:
- Output tokens. Usually the largest single line, and usually several times the price of input tokens. A verbose response format is a permanent tax.
- Loop count. See decision three. Agentic retries multiply everything else.
- Retrieval over-fetch. Pulling twenty chunks when five would do is paid on every single request, forever.
- Retries and fallbacks. A retry policy that is too eager can double your spend during a provider's bad afternoon.
The base model price, which is what most comparisons obsess over, matters less than any of these.
The levers that work are unglamorous. Cache the stable prefix of your prompts, because system instructions and few-shot examples are identical on every call and you should not pay full price for them repeatedly. Route the easy majority of requests to a smaller model, since most production traffic is not hard. Cap loop counts explicitly rather than hoping. Stream output, because a large part of what users experience as slowness is time-to-first-token rather than total generation time.
One more pattern is worth building before you need it: per-tenant quota isolation. If one customer's batch job can eat the rate limit that everyone else's interactive requests depend on, you have a fairness problem that presents as an availability problem. It will happen on a Monday morning.
Which of these you can undo
Here is the sorting promised at the top. The six decisions do not cost the same to change, and that difference is the only ranking that matters when you are deciding what to think about this week.
- Model choice. Cheap to reverse if you built the abstraction, expensive if you did not. The abstraction is the actual decision.
- Whether to use retrieval. Moderate. Adding retrieval later is normal. Changing what you embed means reindexing the whole corpus.
- Workflow versus agent. Moderate and asymmetric. Adding autonomy is additive. Removing it means rewriting control flow you have since built on top of.
- Memory ownership. Expensive. Schemas calcify because everything downstream reads them.
- Orchestration framework. The most expensive item on this list, if the framework owns your control flow.
- Serving and cost optimization. Cheap to moderate, and the most amenable to later work. Which is exactly why it should not be the thing you over-engineer first.
That ordering gives you a sequence. Decide the system boundary, the model abstraction, and state ownership in week one, deliberately, while they are still cheap. Defer routing, agent autonomy, and self-hosting until you have production data telling you which one you need.
The principle underneath it is short. Optimize early for reversibility, and let elegance wait. You will be wrong about something. The only question is whether being wrong costs you an afternoon or a quarter.
What we reach for on day one
Across the AI products we have delivered, the default shape has stayed stable , and it is deliberately boring.
One capable general model behind a thin internal client that owns the message format, tool schemas, timeouts, and retries. Prompts in version control, versioned, with the version id attached to every trace. Durable state in Postgres with a schema we designed, and nothing important living inside a framework's session object. Deterministic control flow by default, with autonomy introduced only at the step that needs it. Span-level tracing from the first week, before there is anything interesting to trace. Retrieval added when the corpus justifies it, evaluated separately from generation on its own set of questions.
Two calls have paid for themselves repeatedly.
The first is refusing to build a routing layer until there is a measured reason. It feels like leaving performance on the table (it usually is, a little). What it buys is that every quality regression has exactly one suspect, which shortens every investigation for as long as the product lives.
The second is putting the trace in before the traffic. It is the least satisfying thing to build in week one, because there is nothing to look at yet. It is also the difference between an incident you can explain in an hour and one you argue about for three days.
None of this is exotic, and that is the point. The expensive mistakes in AI product architecture are rarely the wrong model. They are the boundaries nobody drew, the state nobody owned, and the control flow that moved inside a dependency without anyone deciding to put it there.
If you are making these calls right now and want a second opinion grounded in delivered work rather than a reference diagram, our Discovery Sprint is a one-week engagement that produces the specific architecture decisions and roadmap for your product.
