Latency Budgets in AI Systems

The user experience of an AI system can be compromised long before the model generates its first token. Authorization checks, query rewriting, embedding computation, vector retrieval, reranking, external tool calls, task queues, and storage reads often consume a major fraction of the total available latency envelope.

Looking solely at tokens per second or raw LLM invocation duration paints an incomplete picture. Real-world end-to-end latency is best understood as orchestration + retrieval + tool execution + model time-to-first-token (TTFT) + generation—all competing against the single unified budget perceived by the user.

Decomposing Optimization Priorities

Decomposing latency this way shifts architectural priorities. Shaving 150 ms off token generation barely affects the p95 when retrieval alone injects 800 ms. In contrast, parallelizing independent searches, pre-computing embeddings, or eliminating a sequential round-trip can drastically reduce total elapsed time.

Distributed Tracing and Stage Budgets

Rather than chasing isolated metrics, production architectures benefit from defining explicit latency budgets per stage and instrumenting every boundary with distributed traces. When APIs, primary databases, vector stores, rerankers, tool runtimes, and inference providers are individually observable, the system ceases to be an opaque black box and clearly reveals where time is being spent.

Graceful Degradation and Architectural Levers

Armed with this operational visibility, architectures gain concrete fallback levers: caching stable prompt prefixes, pre-computing predictable inputs, dispatching independent operations concurrently, and gracefully degrading costly steps when a budget is exceeded—preserving a useful response rather than stalling on the ideal execution path.

AI Performance is Distributed Systems Performance

AI performance is fundamentally distributed systems performance: the foundation model is merely one hop in a broader pipeline. The metric that truly matters is not how long the LLM took in isolation, but how long the entire system took to transform a user's intent into valuable, actionable output.