How can deferred work survive crashes without applying its result twice?
First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.
Purpose and requirements
Asynchronous processing separates acceptance from execution. Document ingestion, conversation summaries, and audio conversion need not keep a client connection open. A queue alone, however, prevents neither lost acceptance nor duplicate effects. This is an original reference design, not a description of Kumyu production.
Require a traceable record for every accepted job, at most one committed business result for each logical job, and a way to discover unfinished work. Treat the queue as delivery infrastructure; keep authoritative job status elsewhere.
Flow: replace dual writes with a durable intent
Read at full size (scroll horizontally)
In Figure 1, the database arrow represents one commit. Publishing is a later operation. Writing a job and then sending a message leaves a crash window; sending first can announce a job whose transaction rolls back. The AWS transactional outbox explanation addresses this dual-write problem.
Our design stores job and outbox together. A relay publishes an unpublished entry and marks it delivered afterward. A crash between publication and marking can repeat the message. We have transformed a missing-work problem into a detectable duplicate problem.
Amazon SQS standard queue documentation explicitly discusses duplicate delivery. Do not transfer its guarantees to every broker: verify ordering, redelivery, and retention for the selected product.
State and ownership
Read at full size (scroll horizontally)
The ordinary state path is queued → leased → succeeded, with retry_wait for recoverable failures. A lease grants temporary ownership. Heartbeats and expiry recover work when a process disappears. Expiry does not prove that the old worker stopped, so also issue a monotonically increasing generation token.
{
"job_id": "job-example",
"state": "leased",
"attempt": 2,
"generation": 7,
"lease_until": "2026-10-23T03:05:00Z",
"input_version": 4
}
This is an unexecuted example. A completion transaction checks generation 7, stores the result, and marks success before acknowledging delivery. That order establishes the required happens-before relation: acknowledgment depends on durable completion. Acknowledging first can lose work during a crash.
On redelivery, inspect the committed job result. If complete, acknowledge without recomputation. If a valid worker owns it, respect that lease. Recording a processed ID separately from the result merely creates another crash window; commit those local facts together.
Failure and bounded recovery
Read at full size (scroll horizontally)
A crash before result commit requires recovery; a crash after commit but before acknowledgment reuses the result. An external payment or notification is outside this database transaction and requires its own tool idempotency or reconciliation.
Use bounded exponential backoff with jitter for transient failures, and reject invalid input without retrying. AWS discusses retry amplification and ambiguous side effects after timeout. Here, one layer owns retries with both attempt and elapsed-time budgets.
Exhausted work needs a quarantine record with failure reason, input version, last attempt, and an owner for recovery. After fixing the cause, decide whether replay uses the same business key or a new input version. An empty queue does not prove all accepted work succeeded.
Tradeoffs
| Choice | Benefit | Cost |
|---|---|---|
| Synchronous work | Result in the original response | Client lifetime depends on processing |
| Queue plus outbox | Recoverable dispatch after acceptance | Relay and deduplication logic |
| Global serialization | Simple ordering | One slow job blocks unrelated work |
| Serialization per document or conversation | Preserves relevant ordering | Partition skew and key management |
Worked design: document indexing
Assume a hypothetical arrival rate of 60 documents per minute and a mean processing time of 10 seconds. The steady-state average in service is approximately 1 document/second × 10 seconds = 10 documents. This is an assumption-based calculation, not a measured worker requirement. Validate variability, retries, provider quotas, and waiting-time objectives before choosing capacity.
Pin document_id and input_version in the job. Publish a new index with a conditional version change so an older completion cannot overwrite a newer document. Observe the oldest queued job, the difference between accepted and completed counts, expired leases, and quarantined work.
Exercise and suggested answer
A worker stores its result, then dies before acknowledgment. Should the redelivered job call the model again?
Read committed state. If the matching input version already succeeded, reuse the result and acknowledge. Otherwise acquire a valid generation before executing. Duplicate delivery and duplicate business effects are different problems; the architecture must explain both.
What to take away
Separate delivery attempts from business effects and make the recovery boundary explicit.
Sources & further reading
- AWS: Transactional outbox patternContent reviewed: 2026-10-23
- Amazon SQS: At-least-once deliveryContent reviewed: 2026-10-23
- AWS: Timeouts, retries, and backoff with jitterContent reviewed: 2026-10-23
These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.