Realtime delivery: separate connections from runs

First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.

Fast delivery does not establish durability

Incremental answers show progress, but an open connection does not prove that an answer has been stored. In this conceptual design, a run belongs to the server and a connection is a temporary presentation path. Separating asynchronous execution from delivery lets users recover work after closing a tab.

Read at full size (scroll horizontally)

A replacement connection attaches to the same run ID. Calling the model again on every reconnect would turn a network interruption into duplicate execution.

Choose the transport for the interaction

SSE is a server-to-client event mechanism; EventSource is the browser API commonly used to receive it. Fetch can also consume response bodies incrementally, but parsing an SSE stream and implementing reconnection become application responsibilities. WebSocket provides a bidirectional message path. See the MDN SSE guide and WebSocket overview.

Standard EventSource does not expose arbitrary request-header configuration. Choose an appropriate authentication arrangement, such as a secure same-origin session. Avoid long-lived tokens in URLs that may appear in history or logs. Choose WebSocket because frequent client messages require it, rather than because the product feels “realtime.” Occasional submission and cancellation can use HTTP alongside a one-way stream.

Define a replay contract

Give each event a run ID, a monotonically increasing sequence within that run, and a type: for example run-7 / 42 / text.delta. A cursor identifies the last applied event. Advance it after applying the event to local presentation state, not merely after receiving bytes. Deduplicate repeated sequence numbers and detect missing ones.

Read at full size (scroll horizontally)

A hypothetical policy could retain events for 24 hours while keeping final answers and snapshots longer. This is a design example, not a product limit. When a cursor expires, return an explicit replay-unavailable result and load a versioned snapshot. If the snapshot includes sequence 80, apply events from 81 onward. Updates arriving between snapshot retrieval and subscription must remain recoverable from the same history.

Ordering on a single connection does not prevent gaps across connections. Ephemeral pub/sub alone cannot replay old events. If the interface needs only the final answer, it can instead recover committed state without reconstructing every intermediate delta.

Bound slow consumers and recover disconnections

Read at full size (scroll horizontally)

The standard browser WebSocket API has no native receive backpressure. Do not assume a slow receiver automatically makes the producer wait. Set message-size, frequency, and per-connection buffer limits; coalesce replaceable progress updates. At the limit, disconnect and resynchronize from saved state. Different event types need different drop policies: a replaceable percentage update is different from an operation result.

A heartbeat helps detect connection liveness, not application progress. Reconnect with delay and jitter to reduce synchronized load. Authenticate again and recheck run access on reconnect. Validate browser WebSocket Origin, while remembering that Origin is not identity. Decide how to terminate delivery when authorization is revoked during a subscription.

Worked design: streaming an AI answer

An HTTP submission persists the message and returns a run ID. A worker generates the answer and publishes durably recorded events. The client sends cancellation through HTTP and receives progress through SSE. Keep “cancellation accepted” separate from “execution stopped.” Whether disconnecting should cancel work is a product policy, not a property of SSE.

This design pays for event retention and replay logic. Periodic state retrieval may be simpler for short, infrequent notifications. For voice conversations, low latency and continuous bidirectional media can justify another transport.

Exercise and answer

A client applied event 42, then receives 42, 43, and 45 after reconnecting. Discard duplicate 42, apply 43, and detect missing 44. Recover history or load a consistent snapshot before treating 45 as contiguous. If retention expired, obtain current committed state rather than starting another model run.

These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.

← Browse all chapters