Learning objectives
- Lead the first ten minutes of an API outage.
- How would you write a useful root-cause review?
- Apply Fault isolation to a customer decision and defend the tradeoff.
The mental model
An incident has two concurrent goals: reduce customer harm and discover enough about the failure to recover safely. The fastest mitigation may arrive before a complete explanation. Assign someone to coordinate decisions, someone to investigate, and someone to communicate, even when one person initially fills several roles. Build a timeline of observed events and distinguish facts from hypotheses. Change one meaningful variable at a time when circumstances permit. A rollback that restores service is evidence about a change boundary, but it does not automatically explain the precise defect. Preserve the information needed to finish the investigation after the immediate pressure ends.
How it works
Scope the incident by start time, affected workflows, regions, tenants, and error signatures. Compare the failing path with a known-good path and identify their first divergence. Form competing hypotheses with predicted observations: an origin overload should produce different evidence from a WAF block or a client-side DNS issue. Rank diagnostic actions by information value, reversibility, and customer risk. Avoid a sequence of unrelated configuration changes that destroys causal evidence.
Mitigation options include rollback, traffic shifting, disabling a nonessential feature, reducing admission, or restoring a dependency. Validate recovery from the user's perspective and watch for delayed effects such as queue backlog or corrupted retries. Maintain a decision log with timestamps, owner, action, expected result, and observed result. A post-incident review should describe contributing conditions, detection gaps, and recovery friction rather than assigning simplistic blame. Corrective actions need an owner and a verification method. A root-cause statement that says only human error or the server crashed provides little guidance for preventing recurrence.
Use recovery criteria that an operator can evaluate without interpretation: a representative transaction succeeds, errors return below the agreed threshold, backlog age falls, and no new duplicate side effects appear. Continue monitoring long enough to cover delayed jobs and normal traffic variation. When a mitigation disables a feature, name the owner who will restore it. Otherwise a temporary protective change can become an undocumented permanent limitation after everyone believes the incident is over.
Applied example · hypothetical
Hypothetical scenario: an API's error rate rises immediately after a deployment. Static content remains healthy, and requests reaching the previous version succeed. A new version logs database connection failures after a configuration change. The incident lead rolls back the application while an investigator preserves the deployment configuration and error samples. Service recovers, but a queue contains delayed jobs. The team drains it under a rate limit and checks for duplicates. Later review verifies the configuration validation gap instead of stopping at rollback succeeded.
In a customer engagement
Customers need concise updates containing impact, current mitigation, known uncertainty, and the next update time. Avoid publishing a confident cause before evidence supports it. Coordinate with support, the account team, and specialists through a shared timeline. After recovery, agree on which actions improve detection, prevention, and response, and how the customer will verify completion.
Interview relevance
In a troubleshooting interview, narrate observations and predicted evidence without guessing wildly. Start by scoping impact, then state two plausible hypotheses and the safest discriminating test. Explain how you would keep the customer informed while another engineer investigates. A strong answer distinguishes restoring service from completing a defensible root-cause analysis.
Common mistakes
- Changing many settings at once
- Waiting for complete root cause before mitigating
- Blaming an individual as the explanation
Key terminology
Fault isolation — Narrowing a failure to a component or boundary using discriminating evidence.
Mitigation — An action that reduces current impact even before the underlying defect is fully understood.
Incident timeline — A timestamped record of observations, decisions, actions, and resulting system behavior.
Explain it without the page
- Lead the first ten minutes of an API outage.
- How would you write a useful root-cause review?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.