Cheatsheet Observability Part 046 — Root Cause Analysis Signals

Fokus part ini: mengubah RCA dari cerita “sepertinya karena X” menjadi analisis berbasis observable evidence. RCA yang baik membedakan symptom, trigger, proximate cause, root cause, contributing factor, missing guardrail, missing telemetry, corrective action, dan preventive action.

1. Core Mental Model

RCA bukan mencari siapa yang salah.

RCA adalah proses menjawab:

Kenapa sistem mengizinkan kegagalan ini terjadi, berdampak, dan tidak terdeteksi atau tidak termitigasi lebih cepat?

Observability berperan dalam RCA sebagai:

• bukti timeline;
• bukti scope impact;
• bukti causal chain;
• bukti trigger;
• bukti guardrail yang gagal atau hilang;
• bukti telemetry yang hilang;
• bukti bahwa corrective action nanti bisa diverifikasi.

RCA tanpa observability biasanya menjadi narasi, bukan analisis.

2. RCA Vocabulary

Gunakan istilah dengan disiplin.

Bad RCA collapses these into one line:

Root cause: pricing service timeout.

Better RCA separates layers:

Symptom: quote-order-api POST /quotes 5xx increased.
Proximate cause: pricing-service HTTP calls timed out.
Root cause: new pricing enrichment path made synchronous dependency calls without bounded retry/circuit breaker and saturated request threads when pricing latency degraded.
Contributing factor: alert fired on 5xx only after 15 minutes; no burn-rate alert for route-level quote creation failure.
Missing telemetry: retry count and client connection pool wait were not visible on the service dashboard.

3. RCA Is About Systems, Not Blame

A production system fails because multiple conditions align.

Avoid person-centered RCA:

Engineer forgot to add a timeout.

Prefer system-centered RCA:

The outbound HTTP client wrapper did not enforce default timeout/circuit breaker settings, and PR review did not include an observability/resilience checklist for new dependencies.

The second statement leads to scalable improvement.

4. Evidence Standard for RCA

Each RCA claim should point to evidence.

If evidence cannot support the claim, mark it as hypothesis.

5. Timeline as RCA Backbone

Incident timeline should survive the heat of the incident and become RCA evidence.

RCA timeline should include:

• first customer impact;
• first technical symptom;
• first alert;
• first human acknowledgement;
• first mitigation;
• recovery time;
• deployment/config/audit events;
• dependency/platform events;
• key investigative discoveries;
• false leads when important;
• telemetry gaps encountered.

Example:

Timeline prevents “we think it started around…” ambiguity.

6. Symptom vs Cause

Symptoms are what you observe.

Causes are what generated those symptoms.

Examples:

RCA must not call the alert itself the root cause.

Bad:

Root cause: high latency.

Better:

Symptom: high latency.
Proximate cause: DB connection pool wait time increased.
Root cause: new endpoint code opened long transactions while making outbound calls, holding DB connections longer than expected.

7. Trigger vs Root Cause

Trigger is the event that exposed the failure.

Root cause is the deeper reason the system failed.

If you say “deployment caused it”, ask:

What about the deployment caused it, and why did our system not catch or limit it?

8. Proximate Cause vs Root Cause

Proximate cause is close to the symptom.

Root cause is deeper and more structural.

Example chain:

Symptom:
POST /orders p95 latency increased to 8s.

Proximate cause:
Database queries waited for connections.

Root cause:
A new code path held DB transactions open while waiting on RabbitMQ publish confirms, causing connection pool exhaustion under normal traffic.

Contributing factors:
- No metric for transaction duration by operation.
- No alert on connection pool pending count.
- Trace spans did not show transaction boundary.
- PR review did not flag outbound call inside transaction.

The deeper cause points to better prevention.

9. Contributing Factors

Contributing factors are conditions that made the incident worse, longer, harder to detect, or harder to mitigate.

Common contributing factors:

• missing dashboard;
• missing alert;
• noisy alert causing delayed response;
• no runbook;
• missing owner;
• weak deployment marker;
• missing trace/log correlation;
• insufficient sampling;
• high-cardinality metric dropped by backend;
• no business impact metric;
• no feature flag rollback path;
• unbounded retry;
• no circuit breaker;
• stale dependency documentation;
• unclear escalation path.

Contributing factors are not excuses. They are improvement opportunities.

10. Missing Guardrails

Guardrails are controls that prevent, detect, limit, or recover from failures.

RCA should ask:

1. Which guardrail failed?
2. Which guardrail was missing?
3. Which guardrail existed but was not visible?
4. Which guardrail was ignored because it was noisy?
5. Which guardrail should be added without creating excessive complexity?

11. Missing Telemetry as RCA Finding

Missing telemetry may not be the root cause of the incident, but it can be a root cause of slow diagnosis.

Examples:

Telemetry gaps should become tracked action items.

Do not write RCA like:

Need better monitoring.

Write:

Add route-level POST /quotes business success metric and alert when quote creation success rate burns SLO for 5m/30m windows. Current dashboard only shows HTTP 5xx and misses domain-level failures returned as 200 with business error code.

12. RCA Signal Matrix

Use this matrix to connect RCA dimensions to observability signals.

RCA quality improves when multiple signal types agree.

13. Java/JAX-RS RCA Patterns

Common RCA patterns in Java/JAX-RS services:

13.1 Exception mapping gap

Symptom:

500 rate increased for validation-like failures.

Root pattern:

New domain exception was not mapped in ExceptionMapper, causing expected business validation errors to become 500 responses.

Evidence:

• error logs with exception class;
• route-level 5xx metric;
• traces with span status ERROR;
• response body/error code analysis;
• PR diff adding new exception.

Preventive action:

• exception taxonomy test;
• ExceptionMapper coverage test;
• stable error code convention;
• metric by error category.

13.2 Missing timeout

Symptom:

Latency and active request count increased.

Root pattern:

Outbound HTTP client had no bounded timeout, causing request threads to wait under dependency slowness.

Evidence:

• HTTP client spans long duration;
• active request/thread pool metrics increased;
• dependency latency dashboard;
• timeout config absent in code/config.

Preventive action:

• default timeout wrapper;
• PR checklist for new dependencies;
• client latency/timeout metrics;
• circuit breaker dashboard.

13.3 Transaction boundary mistake

Symptom:

DB pool exhausted after release.

Root pattern:

Code held DB transaction open while calling downstream service or broker, increasing connection hold time.

Evidence:

• pool pending count;
• transaction duration metric;
• traces showing DB transaction around outbound call;
• code diff;
• slow query/lock data.

Preventive action:

• transaction boundary review;
• integration test;
• transaction duration instrumentation;
• architecture guideline.

14. PostgreSQL/MyBatis/JPA RCA Patterns

RCA should include both application behavior and database behavior.

15. Messaging RCA Patterns

15.1 Kafka consumer lag

Possible root causes:

• consumer processing slowed;
• downstream dependency degraded;
• partition skew;
• rebalance storm;
• poison message retry loop;
• consumer replicas insufficient;
• batch size/config changed.

Evidence:

• lag by partition/group;
• processing latency;
• consumer error logs;
• retry/DLQ metrics;
• trace spans for consumer handler;
• deployment markers.

15.2 RabbitMQ redelivery loop

Possible root causes:

• message handler fails before ack;
• nack/requeue policy causes infinite loop;
• downstream call fails;
• poison message not routed to DLQ;
• consumer prefetch too high;
• idempotency failure.

Evidence:

• redelivery count;
• unacked messages;
• queue depth;
• DLQ growth;
• ack/nack logs;
• message age;
• consumer traces.

15.3 Duplicate event side effects

Possible root causes:

• producer retry without idempotency;
• consumer not idempotent;
• message dedup key missing;
• event ID not persisted;
• transaction/outbox boundary issue.

Evidence:

• duplicate event IDs;
• business audit records;
• idempotency store metrics;
• outbox logs;
• consumer processing logs.

16. Redis RCA Patterns

17. Camunda / Workflow RCA Patterns

Workflow RCA must consider both technical execution and business lifecycle.

Common patterns:

RCA should include business state, not only engine error.

18. Kubernetes / Platform RCA Patterns

Platform root causes often require platform/SRE collaboration.

19. Alerting and Detection RCA

Every RCA should ask:

1. Did alert fire?
2. Did it fire early enough?
3. Was it routed to the right owner?
4. Was the severity correct?
5. Was the alert actionable?
6. Did it link to dashboard/runbook?
7. Was it noisy before the incident?
8. Did alert suppression hide it?
9. Was the SLO/SLI definition appropriate?
10. Did business impact appear before technical alert?

Detection failure examples:

20. Corrective vs Preventive Actions

Corrective action fixes this incident's concrete issue.

Preventive action reduces future class of incidents.

Avoid weak actions:

Be more careful.
Improve monitoring.
Investigate further.

Prefer testable actions:

Add alert: quote_creation_success_rate burn over 5m/30m for prod routes, with runbook link and owner label. Validate using replayed incident data before enabling paging.

21. Verification Signals for Actions

Every action should have a verification signal.

No verification signal means the action may not actually reduce risk.

22. RCA Template

Use a structured RCA template:

# RCA: <incident title>

## Summary
- What happened:
- Customer/business impact:
- Duration:
- Severity:
- Services affected:

## Timeline
| Time | Event | Evidence | Notes |
|---|---|---|---|

## Impact
- Affected operations:
- Affected tenants/customers/regions:
- Failed/delayed transaction count:
- SLO/SLA impact:

## Symptoms
- Observable symptoms:
- First symptom timestamp:
- Alert timestamp:

## Trigger
- Deployment/config/platform/business event:
- Evidence:

## Proximate cause
- Immediate technical cause:
- Evidence:

## Root cause
- System-level cause:
- Evidence:

## Contributing factors
- Factor:
- Evidence:
- Impact on detection/mitigation:

## Missing guardrails
- Prevention:
- Detection:
- Limitation:
- Recovery:
- Evidence:

## Missing telemetry
- Missing signal:
- How it affected debugging:
- Proposed telemetry improvement:

## What worked well
- Signals/tools/processes that helped:

## What did not work well
- Signals/tools/processes that delayed response:

## Actions
| Action | Type | Owner | Due date | Verification signal |
|---|---|---|---|---|

23. RCA Quality Checklist

A good RCA:

• separates symptom, impact, trigger, proximate cause, root cause, and contributing factors;
• cites concrete evidence;
• includes timeline with timestamps;
• explains why detection/mitigation took as long as it did;
• identifies missing guardrails;
• identifies missing telemetry;
• avoids blaming individuals;
• avoids vague actions;
• assigns owners and due dates;
• defines verification signals;
• checks privacy/security implications;
• checks cost/cardinality implications of new telemetry;
• updates runbooks/dashboards/alerts if needed.

A weak RCA:

• says “root cause unknown” without evidence gaps;
• says “human error”;
• says “monitoring should improve”;
• has no timeline;
• has no customer impact estimate;
• has actions with no owner;
• has actions that cannot be verified;
• ignores telemetry failures discovered during incident.

24. Internal Verification Checklist

Use this in the CSG/team context without inventing internal standards.

24.1 RCA process

• Where are RCAs documented?
• What incidents require RCA?
• Who approves RCA quality?
• Are RCA actions tracked in backlog?
• Are action owners and due dates required?
• Are completed actions verified?
• Are recurring incident patterns reviewed?

24.2 Evidence sources

• Are incident timelines preserved?
• Are dashboard snapshots or links stored?
• Are log queries recorded?
• Are trace IDs recorded?
• Are deployment/config/audit markers recorded?
• Are customer impact estimates recorded?
• Are mitigation actions timestamped?

24.3 Observability gaps

• Are missing telemetry items tracked?
• Are missing dashboards tracked?
• Are missing/late/noisy alerts tracked?
• Are trace breaks tracked?
• Are correlation ID gaps tracked?
• Are business metric gaps tracked?
• Are privacy/cost risks of new telemetry reviewed?

24.4 Governance

• Does PR review include observability readiness?
• Does ADR include observability impact?
• Are runbooks updated after RCA?
• Are alerts reviewed after incident?
• Are SLOs adjusted when they do not reflect user impact?
• Are platform/SRE/security/backend teams included when needed?

25. Senior Engineer RCA Review Questions

When reviewing an RCA, ask:

1. What was the first user-impacting symptom?
2. What was the first internal technical symptom?
3. What evidence supports the stated root cause?
4. What evidence disproves alternative hypotheses?
5. Did recent change analysis include code, config, data, platform, cloud, and traffic?
6. Did the incident expose missing telemetry?
7. Did detection fail, delay, or route incorrectly?
8. Was mitigation safe and validated?
9. Could this happen again in another service?
10. Are actions specific, owned, due, and verifiable?
11. Do actions increase telemetry cost/cardinality/privacy risk?
12. Should this become a standard, test, dashboard, alert, or runbook update?

Senior-level RCA review is less about finding a clever cause and more about improving the operating system of engineering.

26. Practical Exercises

1. Take one historical incident and classify all statements into symptom, impact, trigger, proximate cause, root cause, contributing factor, missing guardrail, and missing telemetry.
2. Rewrite a vague action like “improve monitoring” into three specific actions with verification signals.
3. Pick one service and identify three RCA findings that would be hard to prove because telemetry is missing.
4. Review one alert and determine whether it would support RCA evidence or only incident detection.
5. Simulate an incident where Kafka consumer lag grows and write a causal timeline with metrics, logs, traces, and business impact.
6. Review one PR and ask what RCA evidence would exist if the new code failed in production.

27. Key Takeaways

• RCA is not blame; it is system learning.
• Symptom, trigger, proximate cause, root cause, and contributing factors must be separated.
• Every RCA claim should be tied to evidence.
• Missing telemetry is a legitimate RCA finding when it delays diagnosis or hides impact.
• Corrective actions fix the immediate defect; preventive actions reduce the future class of failures.
• Every action needs a verification signal.
• The best RCAs improve code, architecture, telemetry, alerting, runbooks, process, and engineering judgment.