Learning objectives
- How would you diagnose duplicate actions in an agent workflow?
- What would you do when queue depth grows continuously?
- Apply Idempotency key to a customer decision and defend the tradeoff.
The mental model
Distributed work can fail in an ambiguous state: the remote system may complete an operation while the caller never receives its reply. Retrying is therefore a correctness decision, not merely an availability setting. An idempotent operation can be repeated without producing an additional intended effect. Backpressure slows admission when downstream capacity cannot keep up. A queue can absorb a temporary burst but cannot solve a permanent mismatch between arrival and processing rates. In orchestration systems, track what was requested, what was accepted, and what actually completed. These distinctions matter more than the number of workers a dashboard says are running.
How it works
Assign an operation identifier before the first attempt and preserve it across retries. Store the durable result where duplicate requests can discover it. If an external provider performs the side effect, use its supported idempotency mechanism and understand retention and scope; a local duplicate check followed by an unprotected remote call can still race. Retry transient failures with bounded attempts, exponential backoff, jitter, and a total deadline. Avoid automatic retries of invalid input or unauthorized requests.
Backpressure can use admission limits, bounded queues, consumer concurrency controls, or a reduced feature mode. Estimate backlog growth from arrival rate minus sustainable processing rate. If work arrives at 120 jobs per second while capacity is 100, the queue grows by roughly 20 jobs per second until behavior changes. Track age of the oldest item and completion latency, not just depth. For a worker crash, decide whether its lease expires, how ownership transfers, and how partially completed effects are reconciled. A recovery runbook should include safe replay criteria and a stop condition for unexpected duplicate behavior.
Calculate recovery rather than guessing. If a backlog contains 12,000 jobs, new arrivals continue at 80 per second, and verified processing capacity is 100 per second, only 20 jobs per second drain the backlog. Recovery takes about 600 seconds under those simplified steady assumptions. Increasing concurrency is useful only if the downstream provider and database can sustain it. Recompute the estimate when arrival rates, failures, or processing cost change during the incident.
Applied example · hypothetical
Hypothetical scenario: an orchestration worker submits a document to a third-party signing API, then crashes before recording success. The scheduler retries the task. Without a stable provider idempotency key, the customer could receive two signing requests. You reproduce this exact crash boundary using a test provider. The improved design reuses the operation identifier and records provider status during reconciliation. You also cap concurrent submissions after a provider slowdown. The intended outcome is duplicate-safe recovery, which must be verified rather than assumed.
In a customer engagement
Ask the customer to identify irreversible or expensive effects: payments, emails, account changes, and partner API calls. Review retry behavior at every layer because client, gateway, service, and worker retries can multiply. Demonstrate the ambiguous-success case in a controlled test and agree on acceptable recovery time, queue age, and operator intervention before increasing throughput.
Interview relevance
A debugging interviewer may describe duplicated jobs and ask for a fix. Explain the failure window before proposing a lock or queue. Show why an operation identifier must survive retries and why the side-effect boundary matters. Use a real incident from a job, open-source project, or lab, and explain the setting, evidence, and your role in the resolution.
Common mistakes
- Assuming timeout means no effect
- Using a new key on retry
- Adding retries to an overloaded dependency
Key terminology
Idempotency key — A stable identifier used to recognize repeated attempts at the same intended operation.
Backpressure — A mechanism that limits incoming work when downstream processing cannot keep up.
Retry amplification — The multiplication of attempts when several layers independently retry the same failing operation.
Explain it without the page
- How would you diagnose duplicate actions in an agent workflow?
- What would you do when queue depth grows continuously?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.