Design foundations: draw boundaries from requirements

First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.

Start with the successful user action

System design is the work of making user actions succeed under constraints. The AI chat in this chapter is a conceptual teaching design, not a description of Kumyu production. Start with requirements such as preserving submitted questions, isolating private conversations, and explaining delays during overload. These establish responsibilities for storage, permission, and execution before choosing services.

Hypothetical objectives might be accepting 99.9% of valid submissions over a month and displaying the first response within two seconds at p95. These numbers are examples, not measurements or product guarantees. Define eligible requests, the observation window, timing boundaries, and treatment of errors alongside the target. Google SRE distinguishes a service indicator, the measurement, from its objective, the target.

A low average can hide a slow minority. A p95 latency is a value at or below which 95% of observations fall. It is not an average. Measure response start separately from completion, and separate short and long operations when their different workloads would obscure interpretation.

Read at full size (scroll horizontally)

Follow the request and persistence arrows. The key question is which component can decide the authoritative state, rather than how many boxes exist.

Separate identity, permission, and state

Authentication establishes who is acting. Authorization decides whether that person may read or change a particular resource. A signed-in user must not be able to fetch an arbitrary conversation ID. Check ownership or membership at submission, result retrieval, and reconnection.

Choose one authority for each state. The UI owns temporary presentation, a conversation store owns committed messages, and a run record owns execution progress. Independent services writing different versions of “complete” make partial success difficult to explain. Distinguish changes covered by a database transaction from effects across a network boundary. If an external tool succeeds and its response disappears, rolling back a local transaction cannot undo that external effect.

Follow one submission through time

Read at full size (scroll horizontally)

Define acceptance explicitly. In this design it means durably storing the message and execution intent. Persist before acknowledging, so a lost acknowledgment can be resolved through the original request ID. Assign a separate run ID and tool call ID. Retrying a request and repeating an external action are different decisions.

Estimate capacity with consistent units

For stable systems, Little's Law relates long-run averages: L = λ × W. L is the average number of items inside the chosen system boundary, λ its arrival/throughput rate, and W the average time spent there. At a hypothetical 20 requests per second and three seconds of average residence time, 60 requests are present on average, waiting or executing. This is neither queue length alone nor a count of CPUs. Substituting p95 for average W does not produce an exact concurrency estimate.

Use that estimate to investigate waiting. Adding web CPU may do little when model latency dominates. Growing database waits may require admission limits rather than accepting work indefinitely. Do not apply steady-state reasoning unchanged when arrival exceeds service capacity continuously.

Recover at the failed boundary

Read at full size (scroll horizontally)

The diagram traces overload into missed deadlines. Bound accepted work and waiting time, then shed load beyond capacity. Within that boundary, distinguish storage failures from uncertain external effects when choosing recovery.

A confirmed transaction failure means the request was not accepted. A commit timeout leaves the outcome unknown: query the original request ID before deciding what to do next. A disconnect after persistence means the existing run should be queried. An unknown external outcome calls for reconciliation rather than declaring success or failure. Restarting everything can create duplicate answers or external transactions.

Observe request IDs, run IDs, tool call IDs, durations, state transitions, and error classes. Avoid logging conversation bodies, credentials, or entire retrieved documents. Correlated IDs should explain whether the bottleneck is acceptance, scheduling, or external execution. User-visible measurements reveal experience; internal measurements help locate causes.

Start small and split for a measured reason

A single application can still enforce clear module responsibilities. Measure read volume, waiting, and change frequency before splitting components. A cache does not replace the authoritative store. For example, include a profile revision in cache keys and read the new revision after updates. For permission-sensitive content, specify where current authorization is checked rather than relying only on expiration.

Worked design: the first chat service

Keep admission, execution, and delivery modules in one application, with messages and run state in a database. Introduce asynchronous execution for lengthy operations and realtime delivery for presentation updates. Add a distributed boundary when a concrete constraint justifies its recovery and operational cost.

Exercise and answer

At ten requests per second, reducing average residence from four seconds to one changes average items inside a stable system from 40 to 10. This does not prove that CPU demand or peak concurrency falls by the same factor. Now draw recovery when storage succeeded but the browser disconnected: fetch the existing run by ID instead of unconditionally creating another execution.

These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.

← Browse all chapters