Systems Architecture and Debugging
Reason about boundaries, ambiguous failures, telemetry, incident leadership, and recoverability.
Draw boundaries before splitting services
When would you recommend a modular monolith to an enterprise customer?
Design retries, idempotency, and backpressure
How would you diagnose duplicate actions in an agent workflow?
Use logs, metrics, traces, and profiles as complementary evidence
How would you investigate a compiler performance regression?
Run an incident with hypothesis-driven fault isolation
Lead the first ten minutes of an API outage.
Trade reliability, recovery, and cost explicitly
Design recovery for approved contracts with strict data-loss requirements.
Investigate a failed quote-generation release
Evidence packet: 09:00 deployment v42; 09:03 API 5xx rises from 0.2% to 12%; static pages remain healthy; v41 requests succeed; v42 logs DB_CONNECT_TIMEOUT; database CPU is 35%; active connections are at the configured maximum; a new client is created for every job; queue age rises to 11 minutes. The packet is fictional.
Work through the problem
- Write the customer-impact statement and two competing hypotheses.
- Choose the first three diagnostic actions and predict their results.
- Propose a reversible mitigation with validation and communication.
- Explain how connection lifecycle behavior could produce the symptoms.
- Plan backlog recovery, duplicate protection, and a regression check.
Deliverables
- Incident timeline and hypothesis table
- Customer update of at most 100 words
- Mitigation and prevention plan
Self-review rubric
- Uses packet evidence rather than generic guesses
- Distinguishes database CPU from connection exhaustion
- Protects backlog replay from duplicate effects
- Separates immediate recovery and final cause verification
Compare with the worked approach
The evidence points toward connection exhaustion introduced by v42, not necessarily database compute saturation. Roll back or constrain the new consumers, verify error recovery and connection counts, then drain the backlog within tested limits. Reproduce per-job client creation under controlled load and test corrected connection lifecycle behavior. Preserve logs and configuration; identify why pre-release checks missed the behavior.
Open the evidence packets in the troubleshooting lab to investigate a simulated incident.