Cheatsheet Observability Part 059 — PR Review and Architecture Decision Checklist

Fokus part ini: menjadikan observability sebagai bagian dari engineering review, bukan pekerjaan tambahan setelah production incident. Setiap perubahan yang mengubah behavior production harus direview dari sisi logs, metrics, traces, auditability, dashboards, alerts, SLO, privacy, cost, dan operational readiness.

1. Core Mental Model

PR review observability menjawab pertanyaan sederhana:

Jika perubahan ini gagal di production,
apakah kita punya evidence yang cukup untuk mendeteksi, memahami,
mengukur impact, mengisolasi root cause, dan memitigasi tanpa deploy ulang?

Review code hanya dari sisi functional correctness belum cukup.

Untuk enterprise Java/JAX-RS system, perubahan kecil bisa mengubah:

• request latency;
• error taxonomy;
• audit evidence;
• database query behavior;
• Kafka/RabbitMQ processing behavior;
• Redis cache correctness;
• Camunda workflow state;
• Kubernetes runtime behavior;
• SLO burn rate;
• log volume;
• metric cardinality;
• trace continuity;
• privacy exposure.

Observability review adalah guardrail agar production tidak menjadi black box.

2. Why Observability Belongs in PR Review

Observability yang buruk biasanya tidak terlihat saat unit test hijau.

Contoh umum:

Functional behavior: OK
Production evidence: broken

Misalnya:

• endpoint baru tidak punya route template metric;
• error baru dipetakan ke HTTP 500 tanpa error code;
• retry loop tidak punya metric;
• Kafka consumer tidak propagate trace context;
• audit log tidak mencatat actor;
• log menulis raw request body berisi data sensitif;
• metric memakai order_id sebagai label;
• dashboard tidak mencakup dependency baru;
• alert tidak punya runbook;
• SLO query tidak menangkap endpoint baru.

PR review adalah titik termurah untuk menangkap observability gap.

Setelah incident, biayanya lebih mahal:

missing telemetry -> slower triage -> longer mitigation -> larger customer impact -> weaker RCA -> repeated incident

3. Observability Review Surface

Tidak semua PR perlu observability review mendalam.

Tetapi PR berikut hampir selalu perlu:

Senior reviewer harus bertanya:

Apa pertanyaan production baru yang akan muncul karena perubahan ini?
Signal apa yang menjawabnya?

4. Change Classification Before Review

Sebelum masuk checklist detail, klasifikasikan perubahan.

A. User-facing synchronous path

Biasanya JAX-RS endpoint, API gateway path, downstream HTTP call, database read/write.

Review utama:

• request latency;
• status code;
• error mapping;
• request/response correlation;
• dependency span;
• SLO impact;
• access log consistency.

B. Asynchronous messaging path

Kafka/RabbitMQ producer/consumer, DLQ, retry, replay, event handlers.

Review utama:

• event identity;
• correlation/causation ID;
• trace propagation;
• consumer lag;
• processing latency;
• idempotency;
• duplicate handling;
• DLQ evidence.

C. Stateful domain lifecycle path

Quote/order lifecycle, approval, fulfillment, fallout, amendment, cancellation.

Review utama:

• state transition log;
• transition metric;
• audit record;
• invalid transition evidence;
• state aging;
• stuck state detection.

D. Platform/deployment path

Kubernetes manifest, Helm values, Argo CD app, collector config, alerts, dashboards.

Review utama:

• labels/annotations;
• version metadata;
• scrape config;
• resource limits;
• probe behavior;
• alert/runbook link;
• rollout/rollback signal.

5. Logging Review Checklist

Logging review memastikan log event berguna untuk debugging, bukan hanya ada.

Check the event purpose

Tanyakan:

Log ini menjawab pertanyaan apa?
Siapa yang akan membaca log ini saat incident?
Apakah log ini event penting atau noise?

Good log event:

• punya event name jelas;
• punya outcome;
• punya context;
• punya business key jika aman dan dibutuhkan;
• punya error code jika failure;
• tidak memuat payload besar;
• tidak memuat secret/PII;
• bisa dikorelasikan ke trace/request.

Bad log event:

Processing data...
Done.
Failed.
Something went wrong.
Exception occurred.

Log seperti itu tidak cukup untuk incident.

Logging review questions

• Apakah log event berada di boundary yang tepat?
• Apakah start/end log diperlukan, atau cukup metric/trace?
• Apakah log level benar?
• Apakah error expected dilog sebagai WARN/INFO, bukan ERROR?
• Apakah stack trace hanya dilog satu kali di boundary yang tepat?
• Apakah retry failure dibedakan dari final failure?
• Apakah log punya event, outcome, duration_ms, error_code, correlation_id?
• Apakah log bisa dicari berdasarkan request ID/trace ID/business key?
• Apakah log tidak mencetak raw request/response body?
• Apakah log tidak menulis token, cookie, authorization header, password, API key?

6. Log Level Review Checklist

Log level adalah kontrak operational severity.

Reviewer questions

Apakah ERROR benar-benar actionable?
Apakah WARN terlalu noisy?
Apakah INFO terlalu banyak di hot path?
Apakah DEBUG bisa meledakkan cost jika dinyalakan?
Apakah expected validation error salah dilog sebagai ERROR?
Apakah dependency timeout final berbeda dari retry attempt?

Common PR smell

catch (Exception e) {
    log.error("Failed", e);
    throw e;
}

Masalah:

• message terlalu generik;
• stack trace mungkin diduplikasi di layer atas;
• tidak ada operation name;
• tidak ada error code;
• tidak ada dependency name;
• tidak jelas retriable atau final;
• tidak jelas customer/business impact.

Lebih baik secara konsep:

event=quote.pricing.dependency_failed
level=ERROR
outcome=failure
error_code=PRICING_TIMEOUT
dependency=pricing-service
retry_exhausted=true
duration_ms=3000
correlation_id=...
trace_id=...

7. Structured Field Review Checklist

Structured field review memastikan log bisa di-query dan di-aggregate.

Required field classes:

Reviewer questions

• Apakah field memakai naming convention internal?
• Apakah route memakai template, bukan raw path?
• Apakah business key aman untuk dilog?
• Apakah user/tenant key boleh ada di log sesuai policy?
• Apakah field value bounded?
• Apakah error message tidak menjadi field cardinality tinggi?
• Apakah timestamp timezone konsisten?
• Apakah field tidak berubah nama antar service?

8. Correlation and Context Review Checklist

Observability gagal jika signal tidak bisa dikorelasikan.

Review context untuk:

• inbound HTTP;
• outbound HTTP;
• Kafka producer;
• Kafka consumer;
• RabbitMQ publisher;
• RabbitMQ consumer;
• Redis operation;
• JDBC operation;
• background job;
• Camunda worker;
• executor/thread pool;
• scheduled task.

Required questions

Dari log ini, bisa lompat ke trace?
Dari trace ini, bisa cari log?
Dari event ini, bisa tahu request asalnya?
Dari consumer failure ini, bisa tahu producer causation-nya?
Dari business key ini, bisa lihat seluruh lifecycle?

Specific checks

• request ID generated if missing;
• correlation ID propagated to downstream;
• traceparent propagated across HTTP;
• message headers carry trace/correlation context;
• causation ID set for emitted event;
• idempotency key logged safely;
• MDC cleaned after request;
• context copied to executor tasks;
• background jobs create fresh execution context;
• inbound untrusted headers validated at trust boundary.

9. Metrics Review Checklist

Metrics review memastikan perubahan bisa dimonitor dan di-alert.

Metric purpose

Setiap metric harus punya purpose.

Is it for dashboard?
Is it for alerting?
Is it for SLO?
Is it for capacity planning?
Is it for debugging?
Is it for business reporting?

Metric tanpa purpose cenderung menjadi cost.

Metric type review

Reviewer questions

• Apakah metric type benar?
• Apakah unit eksplisit?
• Apakah label low-cardinality?
• Apakah route label memakai template?
• Apakah status code label aman?
• Apakah dependency name label bounded?
• Apakah error code label bounded?
• Apakah metric akan dipakai dashboard/alert/SLO?
• Apakah metric memiliki owner?
• Apakah metric duplicate dengan metric existing?

10. Cardinality Review Checklist

Cardinality review adalah mandatory untuk metric PR.

High-cardinality labels yang hampir selalu berbahaya:

• request ID;
• trace ID;
• span ID;
• user ID;
• session ID;
• raw order ID;
• raw quote ID;
• raw URL path;
• raw query string;
• raw error message;
• SQL statement with parameters;
• message key;
• stack trace;
• timestamp;
• UUID.

Safer alternatives

Reviewer questions

Berapa kemungkinan jumlah unique value label ini per jam?
Apakah label ini bounded secara natural?
Apakah label ini bisa bertambah seiring customer/request/event?
Apakah label ini dibutuhkan untuk aggregation?
Apakah log lebih cocok daripada metric label?

11. Tracing Review Checklist

Tracing review memastikan execution path bisa dibaca.

Span boundary review

Span biasanya dibutuhkan untuk:

• inbound HTTP request;
• outbound HTTP dependency;
• database query/transaction;
• Redis command/group operation;
• Kafka/RabbitMQ publish;
• Kafka/RabbitMQ consume/process;
• background job execution;
• Camunda worker/task handler;
• important business operation.

Span quality review

Good span has:

• meaningful name;
• parent/child relation correct;
• service/resource attributes correct;
• route template, not raw path;
• dependency name;
• duration;
• status;
• error attribute if failed;
• bounded attributes;
• no PII/secrets.

Reviewer questions

• Apakah trace root dimulai di boundary yang benar?
• Apakah outbound call menjadi child span?
• Apakah consumer span terhubung ke producer context?
• Apakah retry attempts terlihat?
• Apakah span status diset saat ExceptionMapper mengubah response?
• Apakah manual span diperlukan untuk business operation?
• Apakah span terlalu granular dan mahal?
• Apakah sampling bisa menyembunyikan error penting?

12. Audit Review Checklist

Audit review berbeda dari logging review.

Audit event harus menjawab:

Who did what, to which entity, when, where, why, and what changed?

Audit review fields

• actor;
• actor type;
• tenant/customer;
• action;
• target entity;
• target ID;
• before value;
• after value;
• reason/comment;
• source channel;
• request/correlation ID;
• timestamp;
• authorization context;
• impersonation/delegation indicator;
• outcome.

Reviewer questions

• Apakah aksi ini perlu audit menurut domain/security/compliance?
• Apakah audit event dibuat pada successful action only atau also failed security attempt?
• Apakah before/after cukup untuk evidence?
• Apakah audit log immutable?
• Apakah audit storage berbeda dari application log?
• Apakah audit event tidak kehilangan actor saat async processing?
• Apakah manual/system actor dibedakan dari human actor?

13. Security and Privacy Review Checklist

Observability tidak boleh menjadi data breach.

Must check

• request body logging;
• response body logging;
• Authorization header;
• Cookie header;
• session ID;
• API key;
• OAuth/JWT token;
• password/secret;
• PII;
• commercial/contract/pricing data;
• SQL parameter logging;
• Redis key exposure;
• Kafka/RabbitMQ message payload logging;
• heap dump/thread dump access;
• trace baggage content.

Reviewer questions

Apakah field ini boleh masuk log?
Apakah field ini boleh masuk metric label?
Apakah field ini boleh masuk trace attribute?
Apakah field ini boleh masuk baggage?
Apakah field ini butuh masking/redaction?
Apakah telemetry backend access-nya cukup terbatas?

Strong rule

Do not use telemetry as a payload warehouse.

Telemetry harus membawa evidence yang cukup, bukan data mentah sebanyak mungkin.

14. Dashboard Review Checklist

Perubahan penting harus terlihat di dashboard yang relevan.

Review dashboard impact jika PR:

• menambah endpoint;
• menambah dependency;
• mengubah error behavior;
• mengubah queue/workflow behavior;
• mengubah SLO path;
• mengubah deployment metadata;
• menambah business lifecycle state;
• mengubah retry/DLQ behavior.

Reviewer questions

• Apakah existing dashboard otomatis menangkap metric baru?
• Apakah route template muncul di API dashboard?
• Apakah dependency baru muncul di dependency dashboard?
• Apakah queue/topic baru muncul di messaging dashboard?
• Apakah new state muncul di lifecycle dashboard?
• Apakah panel punya drilldown ke logs/traces?
• Apakah dashboard membedakan old/new version saat rollout?
• Apakah dashboard bisa dipakai saat incident tanpa membaca code?

15. Alert Review Checklist

Tidak semua metric perlu alert.

Alert harus actionable.

Reviewer questions

Jika alert ini fire jam 03:00, apa aksi engineer?
Apakah alert ini symptom-based atau cause-based?
Apakah severity-nya sesuai customer impact?
Apakah threshold/window-nya tahan noise?
Apakah ada runbook?
Apakah ada dashboard link?
Apakah ada owner?
Apakah ada escalation path?
Apakah alert duplicate dengan alert lain?
Apakah alert akan fire selama deployment normal?

Good alert has

• clear title;
• affected service;
• severity;
• symptom;
• likely impact;
• threshold/window;
• dashboard link;
• runbook link;
• owner/escalation;
• labels for routing;
• suppression/deduplication strategy.

16. SLO Review Checklist

Perubahan yang memengaruhi user-visible reliability harus direview terhadap SLO.

Check if PR affects

• availability;
• latency;
• correctness;
• freshness;
• durability;
• queue lag;
• workflow completion;
• fulfillment completion;
• approval aging;
• reconciliation accuracy.

Reviewer questions

• Apakah endpoint/path ini termasuk SLO?
• Apakah error baru dihitung sebagai SLI failure?
• Apakah fallback/degraded response dihitung success atau failure?
• Apakah latency histogram bucket cukup untuk target SLO?
• Apakah async backlog punya SLO/SLI?
• Apakah workflow completion punya SLO?
• Apakah burn-rate alert perlu diperbarui?

17. Operational Readiness Review Checklist

Operational readiness berarti engineer bisa support feature ini di production.

Required evidence

• logs answer important failure questions;
• metrics show rate/error/duration/saturation;
• traces show dependency path;
• audit covers business/security action;
• dashboard exists or existing dashboard covers it;
• alert exists if customer-impacting failure needs action;
• runbook exists for non-obvious failure;
• SLO impact understood;
• rollback/disable strategy exists;
• config/feature flag visible;
• privacy/cost reviewed.

Reviewer questions

Bagaimana kita tahu feature ini rusak?
Bagaimana kita tahu siapa yang terdampak?
Bagaimana kita tahu sejak kapan?
Bagaimana kita tahu dependency mana penyebabnya?
Bagaimana kita mitigate?
Bagaimana kita rollback atau disable?
Apa evidence untuk RCA?

18. ADR for Observability Decisions

Tidak semua observability decision perlu ADR.

ADR layak dibuat ketika keputusan:

• berlaku lintas service;
• memengaruhi cost besar;
• memengaruhi privacy/security;
• mengubah telemetry standard;
• mengubah sampling/retention;
• menambah audit requirement;
• memilih backend/vendor/collector topology;
• mengubah SLO definition;
• mengubah alerting/paging standard.

Minimal ADR structure

# ADR: <Decision Title>

## Context
Apa masalah production/observability yang ingin diselesaikan?

## Decision
Apa keputusan yang diambil?

## Options Considered
Apa alternatifnya?

## Consequences
Apa trade-off correctness, performance, cost, privacy, operation?

## Signal Impact
Logs, metrics, traces, audit, dashboard, alert, SLO apa yang berubah?

## Rollout Plan
Bagaimana perubahan diterapkan dan divalidasi?

## Internal Verification
Apa yang perlu diverifikasi di codebase/platform/team?

19. PR Template Add-on for Observability

Tambahkan section ini ke PR template jika belum ada.

## Observability Impact

- [ ] No observability impact
- [ ] Logs added/changed
- [ ] Metrics added/changed
- [ ] Traces/spans added/changed
- [ ] Context propagation changed
- [ ] Audit logging added/changed
- [ ] Dashboard update needed
- [ ] Alert update needed
- [ ] SLO/SLI impact reviewed
- [ ] Privacy/security logging reviewed
- [ ] Cost/cardinality reviewed
- [ ] Runbook update needed

## Production Debugging Questions

1. How do we know this is failing in production?
2. How do we correlate this with a request/trace/business key?
3. How do we measure customer/business impact?
4. How do we distinguish dependency failure from application failure?
5. What dashboard or query should on-call use?
6. What alert, if any, should fire?
7. What data must not be logged or attached to telemetry?

20. Senior Engineer Review Questions

Use these questions during design review or PR review.

Signal design

What question does this signal answer?
What decision does it support?
Who owns it?
How long must it be retained?
What does it cost?

Failure modeling

What happens if dependency times out?
What happens if retry succeeds after partial failure?
What happens if consumer receives duplicate event?
What happens if DB transaction commits but event publish fails?
What happens if workflow is stuck in non-terminal state?
What happens if telemetry pipeline is down?

Debugging

Can we reconstruct the timeline?
Can we identify blast radius?
Can we find customer impact?
Can we correlate logs, metrics, traces, audit, and deployment marker?
Can we prove root cause with evidence?

Governance

Does this follow naming standards?
Does this introduce high cardinality?
Does this leak sensitive data?
Does this require dashboard/alert/SLO update?
Does this need an ADR?

21. Common Observability PR Anti-patterns

Anti-pattern: logging everything

More logs != better observability.

Risk:

• noise;
• cost;
• PII leakage;
• slower queries;
• unclear signal.

Anti-pattern: metrics with unbounded labels

request_id, order_id, user_id, raw_path as metric labels

Risk:

• backend cost explosion;
• query timeouts;
• dashboard instability.

Anti-pattern: trace without useful attributes

A trace that only says GET /api without route, status, dependency, error, or business context is weak evidence.

Anti-pattern: alert without runbook

An alert that cannot be acted upon is noise.

Anti-pattern: audit hidden inside application log

Audit evidence should not depend on best-effort debug logs.

Anti-pattern: feature complete but support incomplete

Feature is not production-ready if on-call cannot detect and debug it.

22. Example Review: New Quote Pricing Endpoint

Change:

POST /quotes/{quoteId}/price

Review checklist:

Logs

• request accepted;
• pricing started/completed/failed;
• pricing dependency failure;
• error code;
• duration;
• quote ID policy checked;
• no raw pricing payload if sensitive.

Metrics

• request count by route/status;
• latency histogram by route;
• pricing success/failure counter;
• dependency latency;
• retry count;
• validation failure count.

Tracing

• inbound HTTP span;
• pricing service internal span if useful;
• DB read/write spans;
• downstream pricing engine span;
• Kafka/RabbitMQ publish span if event emitted.

Audit

• quote priced action;
• actor;
• quote ID;
• before/after pricing status;
• reason/source;
• timestamp.

Dashboard/Alert/SLO

• endpoint appears on API dashboard;
• dependency health visible;
• latency/error SLO impact reviewed;
• alert only if customer-impacting and actionable.

23. Example Review: New Kafka Consumer

Change:

Consumer handles OrderValidated event.

Review checklist:

Logs

• event received;
• processing started/completed/failed;
• duplicate detected;
• invalid event rejected;
• DLQ published;
• event age;
• correlation/causation ID.

Metrics

• messages consumed;
• processing duration;
• success/failure count;
• retry count;
• DLQ count;
• event age;
• consumer lag.

Tracing

• consumer span;
• producer context extraction;
• child spans for DB/Redis/downstream calls;
• retry/DLQ span.

Correctness

• idempotency key;
• offset commit behavior;
• transactional boundary;
• duplicate event behavior;
• poison message behavior.

Operational

• replay process documented;
• DLQ runbook exists;
• dashboard shows lag/failure/DLQ;
• alert threshold avoids noise.

24. Definition of Observability Ready

A change is observability-ready when:

1. Its important runtime behavior is visible.
2. Its important failure modes are detectable.
3. Its telemetry is correlated across boundaries.
4. Its metrics are low-cardinality and meaningful.
5. Its traces explain dependency latency/failure.
6. Its logs are structured, safe, and useful.
7. Its audit evidence is complete where required.
8. Its dashboards answer triage questions.
9. Its alerts are actionable and owned.
10. Its SLO impact is understood.
11. Its telemetry cost is acceptable.
12. Its privacy/security risk is reviewed.
13. Its runbook/rollback path is clear.

25. Internal Verification Checklist

Use this checklist inside the actual team/codebase.

Codebase

• Logging framework and config.
• Structured log encoder/schema.
• MDC/context helper.
• JAX-RS filters/interceptors.
• ExceptionMapper behavior.
• Metric registry/instrumentation library.
• OpenTelemetry API/SDK/agent usage.
• Kafka/RabbitMQ header propagation code.
• JDBC/DataSource instrumentation.
• Redis client instrumentation.
• Audit event publisher/storage.
• Background job instrumentation.

Platform

• Log backend and retention.
• Metric backend and retention.
• Trace backend and retention.
• OpenTelemetry Collector config.
• Kubernetes labels/annotations.
• Prometheus scrape config or equivalent.
• CloudWatch/Azure Monitor integration.
• Dashboard ownership.
• Alert ownership.
• Runbook location.

Process

• PR template observability section.
• ADR template.
• SLO document.
• Incident review process.
• RCA template.
• Alert review cadence.
• Privacy/security review path.
• Cost review path.
• SRE/platform escalation path.

CSG/team-specific verification

Do not assume internal standards.

Verify with senior engineer/SRE/platform/security team:

• approved log field names;
• correlation ID convention;
• trace propagation standard;
• metric naming convention;
• allowed metric labels;
• audit policy;
• PII/secrets logging policy;
• dashboard standard;
• alert severity policy;
• SLO ownership;
• incident response process;
• telemetry retention/cost guardrails.

26. Final Takeaway

Observability review is not about adding more telemetry.

It is about ensuring production behavior is explainable.

A senior backend engineer should review every meaningful change with this question:

When this fails under real production conditions,
will the next engineer have enough evidence to act quickly and correctly?

If the answer is no, the change is not fully production-ready.