Learning objectives
- Design a safe cache for a support assistant.
- How would you handle provider timeouts without a retry storm?
- Apply Semantic cache risk to a customer decision and defend the tradeoff.
The mental model
AI reliability includes more than keeping an endpoint reachable. A slow response can make an interactive feature unusable, and a cheap response can be useless if quality collapses. Caching can avoid repeated inference for suitable requests, while rate limits protect a budget or dependency from excessive demand. Fallbacks can provide an alternate route when a provider fails, but another model may interpret inputs differently. Treat each mechanism as a policy with tradeoffs. The goal is useful service under expected and degraded conditions, with clear behavior when the application cannot safely answer. A visible unavailable state can be better than a misleading substitute.
How it works
Build an end-to-end latency budget covering authentication, retrieval, context assembly, gateway processing, provider connection, inference, and response delivery. Streaming improves perceived responsiveness when partial output is useful, but it does not necessarily reduce total generation time. Bound output length and avoid needless context. Track input and output consumption using the provider's actual accounting model rather than treating every request as equally expensive.
A cache key must include every factor that changes a safe response: tenant, authorization context where relevant, prompt version, model, parameters, and source version. Never reuse private output across users because the text of their question matches. AI Gateway supports features including observability and request controls; verify the exact supported provider and feature combination before promising behavior. A fallback policy should define triggering failures, total deadline, quality acceptance, and privacy compatibility. Do not retry authentication errors blindly or cascade across providers until every deadline is exhausted. Test rate limits and degraded responses with deterministic provider fixtures before comparing real model quality.
Separate a per-user limit from a global provider limit. A customer may stay within its own allowance while the shared backend is overloaded by other tenants. Use admission control that reflects both boundaries and communicate retry behavior clearly. Avoid a queue whose waiting time already exceeds the user's deadline. When returning a retry response, ensure the client does not immediately resubmit in a tight loop that defeats the server's protective mechanism.
Applied example · hypothetical
Hypothetical scenario: a help assistant receives a burst of identical public-documentation questions while a provider slows down. Public answers may be cached under a versioned key, but account-specific answers remain isolated. The application limits concurrent inference and returns a clear retry state when its queue budget is exhausted. A fallback model is used only for tasks that passed the same evaluation set. The dashboard separates cache hits, provider failures, and fallback quality. Cost reduction is measured per useful resolved request, not only per API call.
In a customer engagement
Ask which requests may be reused, which providers are contractually acceptable, and how long users will wait. Explain that failover can change answer quality and data handling, not just availability. Agree on a degraded experience and a maximum spending envelope. Demonstrate a provider timeout without allowing retries to hide the true customer latency.
Interview relevance
An interviewer may propose adding a second model for reliability. Explain the compatibility tests, privacy constraints, retry budget, and quality gate needed before that is safe. Connect to general API resilience experience and avoid claiming that every AI Gateway feature works identically with every provider or plan without checking current documentation.
Common mistakes
- Keying only on prompt text
- Ignoring source updates
- Retrying permanent errors
Key terminology
Semantic cache risk — The possibility that reusing an apparently similar answer crosses context, freshness, or authorization boundaries.
Fallback policy — Rules governing when an alternate provider or model is used and which correctness and privacy constraints still apply.
Consumption budget — A bound on model usage or cost for a request, user, tenant, or time window.
Explain it without the page
- Design a safe cache for a support assistant.
- How would you handle provider timeouts without a retry storm?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.
A tenant-aware cache key is a starting point
const key = JSON.stringify([
tenantId,
permissionVersion,
sourceVersion,
normalizedQuestion
]);Authorization must be enforced before reuse, and revocation/expiry must be handled. A better key alone does not make a cache secure. Run the offline cache contract in exercises/cache-contract.mjs.