Gateways, rate limits, and budgets

First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.

Why centralize an entry point?

A gateway handles shared concerns such as authentication checks, routing, and admission control. Think of an API gateway as the entry to application APIs and an AI gateway as the entry to model providers. These are roles, not guaranteed product feature lists. This hypothetical chat design does not describe Kumyu's deployed infrastructure.

One logical entry point can have many physical instances. A local allowance of 100 requests per second on each of ten servers can admit 1,000 requests per second in aggregate. Envoy Gateway documents global rate limiting across instances. Before choosing an implementation, define the scope: user, tenant, model provider, region, or entire service.

Flow: three independent constraints

Read at full size (scroll horizontally)

Figure 1. Arrows trace a request. Each gate protects a different resource using authenticated tenant identity.

Ten requests per second with a 30-second duration can sustain roughly 300 active requests. Conversely, a concurrency limit alone can admit many short calls and exhaust a usage budget. Tenant and user limits separate authenticated consumers; an IP limit may be a useful secondary control but groups unrelated users behind shared networks.

Atomic buckets and reserved generation budgets

Read at full size (scroll horizontally)

Figure 2. Reserve an upper bound before generation. Refund the difference from observed usage once, after completion.

A token bucket has capacity B, refill rate r, and remaining quantity t. Its tokens represent admission permits, not LLM text tokens. After elapsed time Δ, refill with t = min(B, t + r × Δ) and subtract only if enough permits remain. Redis's guide demonstrates shared state and atomic Lua operations. Our design requires refill, comparison, and subtraction to occur together. Invariants are nonnegative balance, bounded refill, and no double debit for the same admission.

{
  "requestId": "req-example-9",
  "tenantId": "tenant-from-session",
  "reservation": {"tokens": 3000, "state": "reserved"},
  "maxOutputTokens": 1000,
  "deadlineMs": 8000,
  "attemptLimit": 2
}

This is an unexecuted design example. Reserve 3,000 for 2,000 input tokens and at most 1,000 output tokens, and enforce that output maximum at the provider. If observed consumption is 2,400, refund 600 exactly once. Missing usage after timeout is not zero usage. Persist reserved → reconciled or reserved → pending_reconciliation, then settle through provider records or an explicit accounting policy. Bound every attempt rather than granting unlimited fallback calls under one reservation.

Concurrency slots need expiration leases and renewal. Expiration alone is insufficient if old work continues after its lease: cancellation and reconciliation must prevent replacement work from silently exceeding the intended limit. Keep state transitions durable and use request identifiers to make repeated completion notifications harmless.

Failure: retries amplify demand

Read at full size (scroll horizontally)

Figure 3. Retrying and switching models both consume resources. Carry the original deadline and total attempt count forward.

RFC 6585 defines HTTP 429 for excessive requests and permits a Retry-After header. Honor a supplied delay, combine backoff with jitter, and do not retry when the wait would consume the remaining deadline. A permission error is not a quota error and will not resolve through waiting.

Two attempts at the gateway, two at the application, and two in the provider client can multiply into eight calls. Give one layer responsibility for retries and propagate the total attempt allowance. A timeout during a non-idempotent send does not prove the operation failed. Query its operation identifier before replay. Falling back to another model cannot undo tool side effects that already occurred.

If shared admission state becomes unavailable, unlimited fail-open traffic may overwhelm the dependency. This design stops strict budget admission. A separately bounded local allowance for cheap reads is possible, but its aggregate allowance and weakened global guarantee must be explicit. An unbounded waiting queue merely converts rejection into latency and memory exhaustion.

Worked hypothetical capacity example

Assume a bucket starts full with capacity 20 and refills at five permits per second. Thirty simultaneous one-permit requests admit 20 and reject ten under atomic evaluation. Two seconds later at most ten permits have returned. These are teaching assumptions, not a vendor configuration or benchmark.

With a concurrency cap of eight, the ninth execution must wait or be rejected even when rate permits remain. Bound the queue separately, for example at 16 waiting requests with an eight-second deadline. A remaining generation budget of 12,000 tokens permits only four simultaneous reservations of 3,000 tokens each. The tightest applicable constraint determines admission; passing one gate is not permission to bypass another.

For cost accounting, token quantity is insufficient when models or input/output rates differ. Use an explicitly chosen accounting unit and a conservative bound per model, then reconcile actual charges. No currency rates or price claims are implied by the token arithmetic above.

Exercise: the browser disconnects

A request reserves 3,000 tokens and the browser disconnects before any answer arrives. Can the server immediately refund the full amount?

Suggested answer. No. Generation may still run. Request cancellation, inspect the provider's completion and usage, and reconcile afterward. If the result is unknown, retain a pending reservation with a bounded reconciliation procedure. Continue with RAG to decide how retrieval, reranking, and generation share the request budget.

These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.

← Browse all chapters