Learning objectives
- How would you investigate a compiler performance regression?
- What can a trace tell you that a metric cannot?
- Apply High cardinality to a customer decision and defend the tradeoff.
The mental model
Metrics show patterns across many events. Logs preserve details about selected events. Traces connect steps belonging to a request. Profiles reveal where a program spends resources. None is a universal replacement for the others. A latency graph can show that users are affected without explaining which dependency is slow; a single error log can explain one failure without revealing its prevalence. Begin with a question and select the evidence that can answer it. For a compiler or runtime failure, a minimal reproducer and exact build identity may be more valuable than another chart. Collect enough context to compare affected and unaffected work.
How it works
Separate symptoms from mechanisms. Request rate, error proportion, duration distributions, and saturation help locate a service-level change. Use percentiles and workload segmentation: an average can hide a small but important group of very slow requests. Correlation identifiers should cross process and queue boundaries where practical. Trace spans need meaningful names and attributes, but unbounded identifiers in metric labels can create excessive cardinality and cost.
Logs should include timestamps, operation identifiers, deployment version, and safe error context. Redact credentials and sensitive payloads. Record sampling and retention decisions so absence of a log is not mistaken for proof that an event never occurred. Use CPU or allocation profiles when resource consumption is the hypothesis, and compare representative workloads rather than isolated microbenchmarks. A runtime crash investigation should preserve compiler flags, input hash, environment, and a small reproducible case. A trace indicating time in a database call does not establish whether the cause is query execution, connection waiting, network delay, or locking; test those possibilities separately.
Write a hypothesis table before collecting more data. For database connection waiting, expect long wait time with potentially modest query execution time. For CPU-heavy parsing, expect sustained CPU and relevant stack samples. For slow artifact upload, expect network or storage spans after compilation. This table helps prevent an attractive but unrelated graph from controlling the investigation. Preserve the observations that contradict your favorite hypothesis; they are often the fastest route to the actual mechanism.
Applied example · hypothetical
Hypothetical scenario: a compiler service's median build time is unchanged, but a customer reports five-minute waits. A segmented latency chart reveals that large generated inputs are affected. Traces place the delay inside compilation, not artifact upload. A profile identifies repeated allocation in a transformation pass. You preserve a reduced input that triggers the behavior and compare the same build configuration before and after a candidate fix. The customer update reports the affected workload and mitigation; it does not claim every slow build has the same cause.
In a customer engagement
Agree on the question before requesting broad access to telemetry. Ask for time range, timezone, request identifiers, deployment changes, and an unaffected comparison. Show how each evidence item changes a hypothesis. When data is sampled or incomplete, state that limitation explicitly and propose a narrow additional measurement rather than pretending a dashboard gives complete visibility.
Interview relevance
Senior debugging answers should connect an observation to the next discriminating test. Explain when you would use a trace versus a profile and why averages can mislead. A verified application, network, or runtime example can demonstrate this skill, but distinguish the evidence you personally gathered from work performed by other contributors.
Common mistakes
- Comparing different workloads
- Treating correlation as mechanism
- Expecting every request to be traced
Key terminology
High cardinality — A large number of distinct values in a telemetry dimension, potentially increasing storage and query costs.
Trace span — A timed operation within a distributed trace, with context linking it to related work.
Minimal reproducer — The smallest practical input and environment that reliably demonstrates a failure.
Explain it without the page
- How would you investigate a compiler performance regression?
- What can a trace tell you that a metric cannot?
Further reading
Sources explain the underlying concepts; examples and practice answers are original course material. Recheck current limits and plan eligibility before a real proposal.
Correlate observations across a request
{
"request_id": "synthetic-request-42",
"tenant": "synthetic-tenant-a",
"component": "payment-client",
"attempt": 2,
"elapsed_ms": 1900,
"outcome": "timeout"
}Synthetic telemetry. Do not put credentials, payment details or unnecessary personal information in logs. Record end-to-end deadline and idempotency context separately.