AI chat: runs that survive disconnection

First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.

Purpose and invariants

Treating chat as a function that sends text to a model and prints a response makes connection loss a data problem. This reference architecture separates conversation records, generation work, and delivery to the screen. It does not describe a deployed Kumyu system.

Start with three invariants: retrying one send must not add another user message; a disconnected client must be able to retrieve a committed answer; and every read must enforce conversation ownership. Exactly one external generation is a stronger promise than exactly one accepted request or one committed answer. State which promise the design actually provides.

Flow: persist before starting work

Read at full size (scroll horizontally)

Figure 1 is an original conceptual diagram; arrows distinguish requests, writes, and delivery. The API authenticates the caller, checks ownership, and accepts POST /runs with a request_id. One transaction stores the user message, run, and an outbox for dispatch. A successful commit precedes the acceptance response.

Use UNIQUE(tenant_id, user_id, request_id). A duplicate key returns the existing run, provided a stored payload hash matches. Scope the key to tenant and user. Hash a canonical payload containing conversation ID, normalized message, attachment identities and versions, and generation-affecting parameters. Reusing a key for the same text in a different conversation is therefore a conflict. A different payload never silently reuses an answer. See PostgreSQL unique constraints for the database mechanism. The deduplication retention period is part of the retry contract.

A worker fixes the history version, calls the model, and records events ordered by (run_id, seq). The browser subscribes to that run independently. Closing a tab therefore need not end generation.

State machine and delivery cursor

Read at full size (scroll horizontally)

Read Figure 2 downward in time; columns identify the participants. Define run states separately: The ordinary path is queued → running → completed; failed and cancelled are also terminal. A retry creates another run with a causal link rather than moving a terminal run back to running. Displayed deltas remain provisional until an assistant message is committed.

{
  "run_id": "run-example",
  "request_id": "send-example",
  "state": "running",
  "history_version": 12,
  "last_seq": 41
}

This unexecuted example separates the history used as input from the cursor applied by the screen. Wall clocks do not provide a reliable total order: clocks differ and timestamps can collide. Use per-run sequence numbers and ignore already applied events.

For SSE, the HTML event ID and reconnection specification supplies a transport mechanism. It does not store your events or implement server replay. This design authorizes a reconnect, replays retained events after the cursor, then joins live delivery without a gap. An expired cursor causes an explicit resynchronization to a committed snapshot.

Failure: completion races cancellation

Read at full size (scroll horizontally)

A disconnected socket and a cancellation request are separate events. A stop button issues a durable cancellation request. Completion and cancellation each conditionally update the run, for example with WHERE state = 'running', and inspect the affected row count. Only the winner may establish the terminal result.

This is compare-and-set reasoning. Store the final assistant message and completed state in one database transaction. A provider stop request and external tool effects are outside that atomic boundary. A cancelled run cannot imply that a previously sent email was recalled.

If a worker dies, do not promote its partial text into a completed answer. A lease and generation token reject completion from an obsolete worker. If generation cannot resume, mark failure and offer an explicit retry. Repeating a provider call can consume resources and produce different text, even when only one final answer is accepted locally.

Tradeoffs

Worked design: editable consultation history

Suppose two devices submit to one conversation concurrently. This illustrative design accepts both inputs but serializes generation per conversation. The first result commits before constructing the next input history. History edits create a new conversation version instead of silently changing the evidence used by an existing run.

Measure acceptance-to-first-event time, acceptance-to-final-commit time, replay recovery success, and the age of nonterminal runs. Choose targets from intended usage and observed behavior. Model generation speed alone does not explain the user's wait.

Exercise and suggested answer

A send succeeds on the server, but its response is lost. After refreshing, the user submits the same question. Should the service create a new run?

Persist the same request_id across refresh and retrieve its existing run. Show its committed result if complete. Reject a different payload under that key. Allocate a new key only for an explicit request to generate another answer. Distinguishing acceptance idempotency, durable results, and recoverable delivery is the central design skill here.

These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.

← Browse all chapters