Learning objectives
- Design recovery for approved contracts with strict data-loss requirements.
- How would you justify reliability spending to a CFO?
- Apply SLO to a customer decision and defend the tradeoff.
The mental model
Reliability is a property of a user workflow over time, not a promise that a server never fails. Define which requests count, what counts as success, and the time window that matters. Recovery time describes how quickly service must return; recovery point describes how much recent data loss is tolerable. These goals influence redundancy, backups, testing, and cost. A backup that has never been restored is an unverified recovery assumption. Customers may rationally choose different protection for a public brochure and an order-processing system. The engineer's role is to make the consequences visible so stakeholders can choose deliberately.
How it works
Define service-level indicators around customer-visible behavior and service-level objectives as targets for those indicators. An error budget expresses the permitted amount of unreliability within the objective window. It can guide release and remediation decisions when teams agree on how it will be used. Availability percentages alone are insufficient if measurement excludes the failures users actually experience.
Recovery design distinguishes component failover from data restoration. Replication can improve availability while rapidly copying an accidental deletion; an independently recoverable backup addresses a different failure mode. Test restoration into an isolated environment and verify business invariants, not merely that files exist. Estimate the cost of redundant capacity, retained data, operational labor, and rehearsals. Analyze correlated failures: two instances sharing one credential, network path, or database may not provide the isolation expected. For scaling, measure the bottleneck and workload shape before multiplying compute. Capacity planning should include degraded operation, dependency quotas, and the time required to provision or activate replacement capacity under realistic failure conditions.
For a simple availability exercise, define a thirty-day window with 43,200 minutes. A 99.9% time-based objective permits 43.2 minutes outside the target, under that simplified measurement. A request-based indicator produces a different budget and can weight busy periods differently. State which method is used before comparing numbers. An objective should reflect user impact and measurement quality; a precise percentage built on incomplete telemetry can create false confidence in the service's real reliability.
Applied example · hypothetical
Hypothetical scenario: a quote application can tolerate fifteen minutes of downtime but cannot lose an approved contract. A cheaper backup-only design may fail the data-loss requirement even if it restores quickly. You separate transactional durability from document regeneration and define a restore exercise that verifies approval history. The customer compares the cost of replication and retained backups with estimated disruption exposure. The financial model is a scenario, not a guaranteed loss prediction. Acceptance requires a timed rehearsal and confirmation that the recovered records are correct.
In a customer engagement
Ask stakeholders which business activities stop during an outage and which records cannot be recreated. Translate those answers into measurable recovery and service goals before discussing redundant components. Bring finance and operations into the comparison, because a lower subscription price can shift substantial labor or outage risk onto the customer's team.
Interview relevance
An interviewer may challenge an expensive high-availability proposal. Explain the user objective, the failure modes covered, the correlated risks left over, and a cheaper alternative with explicit consequences. Demonstrate that reliability engineering includes cost judgment and restore testing, rather than equating more replicas with a better architecture in every case.
Common mistakes
- Treating a backup job as a restore test
- Ignoring correlated failures
- Guaranteeing zero downtime
Key terminology
SLO — A target for a defined service-level indicator over an explicit measurement window.
RTO — Recovery Time Objective: the targeted maximum time to restore a service after disruption.
RPO — Recovery Point Objective: the targeted maximum amount of recent data that may be lost, expressed as time.
Explain it without the page
- Design recovery for approved contracts with strict data-loss requirements.
- How would you justify reliability spending to a CFO?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.