A model call is a distributed systems problem

Most conversations about putting AI into a product are about the model: which one, how good, how to prompt it. Those questions matter. But once a feature is live, the model stops being the interesting part. From your application’s point of view, it is a remote dependency. And it is a difficult one.

It has three awkward properties:

It’s slow. Seconds, not milliseconds. Longer answers take longer, because output is generated a token at a time.
It’s metered. You pay for every token in and every token out.
It’s rate-limited. Providers cap requests and tokens per minute. Go over the cap and you get errors, usually at the moment traffic peaks.

None of this is new. It describes every expensive third-party API a team has ever integrated. What’s new is how often it gets forgotten, because calling a model feels like calling a function.

A local function is fast, free and always there. A model call is none of those. So the useful question isn’t “which model?” It’s “how do we design around a dependency like this?” Six design decisions answer most of it.

1. A timeout is not a failure

Consider this sequence. Your application sends a request. The provider processes it and generates a response. The connection times out before the response arrives. Your application retries.

A timeout doesn’t mean the model didn’t run. It means the caller doesn’t know whether it ran. That is the oldest problem in distributed systems, with two extra costs here: you may pay for both executions, and the second answer may not match the first.

So the design has to separate two cases. A transport failure means the request never got there, and a retry is safe. An ambiguous outcome means it may have completed, and a retry is a decision. In practice:

Give every logical request its own ID, and record each attempt against it.
Where the provider or your gateway supports idempotency keys, use them.
If anything downstream acts on the answer, decide which answer wins. Don’t let a late retry quietly replace a result that has already been used.
Cap retries, and back off with jitter. Immediate retries turn a short rate limit into a long outage.

2. Three limits that do three different jobs

Teams often stop at the provider’s rate limit. That limit protects the provider. It does nothing for you.

Picture five hundred users triggering a model call at the same moment. Even if the provider accepts all of them, your application is now holding five hundred slow connections open, or building a queue that will take minutes to drain. By the time the last answer arrives, the user who asked has gone.

Three controls do three different jobs:

Rate limiting protects the provider. It’s their cap, and you have to respect it.
Concurrency limiting protects your application. Set a maximum number of in-flight model calls, so slow responses can’t exhaust your own resources.
Backpressure protects the whole system from work it can’t finish. Use a bounded queue. When it’s full, refuse new work early and clearly, and expire queued requests whose user has stopped waiting.

Then add priority. Interactive requests and background jobs shouldn’t compete as equals. Give them separate keys or quotas, so a batch job can never starve a customer-facing feature.

3. Plan capacity in tokens, not requests

Capacity planning for a normal API starts with requests per second. For a model, that number hides most of the picture.

	Workload A	Workload B
Requests per minute	100	100
Average input tokens	500	8,000
Average output tokens	200	2,000
Total tokens per minute	70,000	1,000,000

Same request volume. Fourteen times the token throughput, and a very different bill.

So a model integration needs two budgets: requests per minute and tokens per minute. Four practical points follow:

Cap output length. Output tokens usually cost more than input tokens, and they set the latency.
Know what fills the prompt. Retrieved context and long instructions sent on every request are often the biggest cost. Provider-side prompt caching helps with the part that never changes.
Route by difficulty. Send simple steps, such as classification, to a small, cheap model, and keep the large model for the work that needs it.
Set your own budget below the provider’s limit. The provider’s limit stops you at the edge. Your budget, per feature and per day, tells you before you get there.

4. The model is a versioned dependency

A traditional API has a contract. If it changes, you usually get a new version and a notice. A model’s behaviour can change while your code stays exactly the same: a new model version, a prompt edit, a different temperature, a change in how structured output is produced.

So write the contract down yourself. For each use of a model, record:

Part of the contract	Example
Provider and model	An explicit model identifier, never “latest”
Prompt version	v12
Output schema version	v3
Sampling settings	Temperature 0.2
Output limit	1,000 tokens
Evaluation set version	v5

A prompt isn’t just a string in the codebase. It’s part of the runtime contract, and it should be versioned, tested and visible in your logs. When answer quality changes, this record is how you find out what changed. Without it, you’re guessing.

5. Decide how each feature fails

When the provider is unavailable or too slow, the worst design is the default one: the whole user journey fails. Not every feature deserves the same answer.

Use	When the model is unavailable
A summary shown to a user	Degrade: show the source content without the summary
Background classification	Queue it and retry later
AI-enhanced search	Fall back to conventional search
Support for a high-impact decision	Fail safely and send it to a person

One caution. A fallback model is not a free swap. It is a different dependency with different behaviour, and it needs its own evaluation before you rely on it.

6. Watch three things, not one

A status code of 200 tells you the call completed. It tells you nothing about whether the answer was any good. A model can return a fast, well-formed, confident and wrong answer, and every dashboard stays green.

Observability for a model integration has three layers:

Infrastructure: latency at the median and at the tail, timeouts, error and retry rates, calls in flight.
Economics: input and output tokens, cost per request, cost per user journey, budget used.
Quality: schema validation failures, evaluation scores on sampled outputs, and what human reviewers found.

Tie them together with one correlation ID that follows the user’s request through every attempt, retry and downstream action. When something goes wrong, that ID is the difference between an hour of investigation and a week.

Where these decisions live

Put together, those six decisions describe one component: a gateway that every model call passes through.

These are logical responsibilities, not separate services. In a small system they can be one module in front of the provider’s SDK. What matters is that every model call goes through them.

The point of the gateway is ownership. If each team calls the provider directly, each team makes these six decisions separately, or doesn’t make them at all. One path means one place to set a budget, change a model version, or find out why last Tuesday’s answers were worse.

Questions I’d ask before it ships

If a call times out, do we know whether it ran, and what happens on retry?
What is the maximum number of calls in flight, and what happens to request number one more?
What is the token budget per request and per day, and who is told when it’s exceeded?
Which other workloads share this rate limit?
Which model version, prompt version and settings produced this answer?
If the model is down for an hour, what does each feature do?
How would we know the answers had got worse?

If a team can answer those seven, the model choice is almost a detail.

The skills that carry over

There’s a view that AI changes everything about how software is built. Parts of it are true. But the further an AI feature gets from a demo, the more its problems look like the ones distributed systems engineers have always had: ambiguous failures, shared limits, capacity, versioned contracts and partial failure.

That’s good news for anyone who has spent years on those problems. The model is new. The discipline around it isn’t, and it’s the part that decides whether the feature survives production.