Cheatsheet Observability Part 060 — 30/60/90-Day Observability Onboarding Plan

Fokus part ini: membuat rencana praktis agar engineer baru di service/team enterprise bisa cepat memahami observability nyata: log format, dashboards, alerts, traces, incident history, instrumentation conventions, runbooks, SLO, privacy, cost, dan production support workflow.

1. Core Mental Model

Onboarding observability bukan sekadar belajar tool.

Tujuannya adalah memahami:

Bagaimana sistem terlihat saat sehat?
Bagaimana sistem terlihat saat rusak?
Signal apa yang dipercaya team?
Signal apa yang sering misleading?
Dashboard mana yang dipakai saat incident?
Alert mana yang actionable?
Runbook mana yang benar-benar membantu?
Telemetry gap apa yang sudah diketahui?

Dalam enterprise Java/JAX-RS system, observability onboarding berarti memetakan realitas production dari beberapa lapisan:

• API/JAX-RS request flow;
• service layer;
• PostgreSQL/JDBC/MyBatis/JPA;
• Redis;
• Kafka/RabbitMQ;
• Camunda/workflow;
• NGINX/ingress/API gateway;
• Kubernetes;
• AWS/Azure/on-prem infrastructure;
• CI/CD and deployment metadata;
• incident process;
• security/privacy policy;
• SLO and alerting ownership.

Target 90 hari bukan menjadi owner semua observability.

Target realistis:

Mampu membaca signal production,
mengikuti incident debugging,
menemukan gap evidence,
dan memperbaiki satu area observability yang nyata.

2. Onboarding Principle: Observe Before Changing

Kesalahan umum engineer baru:

Langsung menambah log/metric/alert tanpa memahami standard internal.

Risikonya:

• field tidak sesuai convention;
• metric high-cardinality;
• alert noisy;
• log bocor data sensitif;
• dashboard duplicate;
• trace context tidak sesuai platform;
• signal tidak dipakai siapa pun.

Prinsip yang lebih baik:

First map.
Then validate.
Then improve.

Urutan aman:

1. pahami existing telemetry;
2. pahami incident history;
3. pahami ownership;
4. temukan gap berdasarkan real problem;
5. usulkan improvement kecil tetapi tajam;
6. validasi dengan senior/SRE/platform/security;
7. dokumentasikan runbook/checklist.

3. First 30 Days — Learn the Production Reality

Fokus 30 hari pertama:

Build situational awareness.
Do not optimize prematurely.
Do not invent internal standards.

Output utama:

• tahu dashboard utama;
• tahu log format;
• tahu cara mencari request/trace/correlation ID;
• tahu alert utama;
• tahu service dependency;
• tahu incident besar terakhir;
• tahu siapa owner observability;
• tahu dokumen/runbook yang dipakai.

4. Days 1–7 — Find the Map

Pada minggu pertama, cari peta sistem dan peta telemetry.

Documents to find

• service architecture overview;
• API contract/OpenAPI if available;
• service ownership doc;
• deployment/GitOps doc;
• logging standard;
• metric naming convention;
• tracing/OpenTelemetry standard;
• dashboard catalog;
• alert catalog;
• runbook index;
• SLO/SLA document;
• incident/RCA repository;
• audit logging policy;
• privacy/security logging policy.

Repositories/configs to inspect

• JAX-RS filters;
• exception mappers;
• logging config;
• metric registry setup;
• OpenTelemetry setup;
• collector config;
• Kubernetes manifests;
• Helm values;
• Argo CD app config;
• CI/CD pipeline;
• alert rule repository;
• dashboard-as-code repository if exists.

Questions to answer

What are the main services?
What are the main APIs?
What are the main dependencies?
What is the runtime environment?
Where do logs go?
Where do metrics go?
Where do traces go?
Where are dashboards?
Where are alerts defined?
Where are runbooks?
Who is on-call or production support owner?

5. Days 8–14 — Learn Logs and Correlation

Logs are usually the fastest entry point into production reality.

What to inspect

• structured log format;
• required fields;
• timestamp and timezone;
• service/environment/version fields;
• request ID;
• correlation ID;
• trace ID;
• span ID;
• tenant/customer fields;
• actor/user fields;
• business keys such as quote/order/process instance;
• error code fields;
• stack trace format;
• redaction/masking behavior.

Practical exercises

Pick one successful request and one failed request.

For each:

Find ingress/access log.
Find application request start/end log.
Find service-layer business log.
Find database/dependency error if any.
Find trace ID from log.
Find correlation ID across services.
Find quote/order/process ID if available.

Red flags to note

• logs are plain text only;
• no correlation ID;
• trace ID missing;
• inconsistent field names;
• raw payload logged;
• expected validation errors logged as ERROR;
• stack traces duplicated;
• logs cannot be joined across service boundaries;
• timezone mismatch.

Internal verification checklist

Ask team:

• What is the canonical correlation ID field?
• Is trace ID expected in every app log?
• Are quote/order IDs allowed in logs?
• Which fields are forbidden?
• Is there a standard event naming convention?
• How long are logs retained?

6. Days 15–21 — Learn Dashboards and Metrics

Dashboards show what the team considers important.

Dashboards to study

• service health dashboard;
• API latency/error dashboard;
• dependency dashboard;
• PostgreSQL dashboard;
• Redis dashboard;
• Kafka/RabbitMQ dashboard;
• Camunda/workflow dashboard;
• Kubernetes dashboard;
• deployment/release dashboard;
• business lifecycle dashboard;
• SLO/error budget dashboard;
• executive or customer-impact dashboard if available.

Metrics to identify

For each owned service, find:

• request rate;
• error rate;
• latency histogram/percentile;
• active requests;
• thread pool usage;
• connection pool usage;
• JVM memory and GC;
• CPU and memory container usage;
• pod restarts;
• dependency latency/error;
• queue lag/depth;
• retry/DLQ counts;
• workflow incidents;
• business state transitions.

Practical exercise

Pick one service dashboard.

Answer:

How do I know the service is healthy?
How do I know latency is degraded?
How do I know errors are customer-impacting?
How do I know DB is the bottleneck?
How do I know Kafka/RabbitMQ backlog is growing?
How do I know a deployment caused the issue?
Where do I click next for logs/traces?

Red flags to note

• dashboard has many charts but no clear question;
• panels use raw path instead of route template;
• no deployment marker;
• no dependency drilldown;
• no SLO view;
• no owner;
• stale metrics;
• misleading percentile queries;
• high-cardinality labels.

7. Days 22–30 — Learn Alerts, Incidents, Runbooks, and SLOs

By the end of first month, understand how the team reacts to production issues.

Alert inventory review

For top alerts, identify:

• alert name;
• service;
• severity;
• symptom;
• threshold/window;
• owner;
• routing;
• dashboard link;
• runbook link;
• recent false positives;
• recent missed incidents;
• relation to SLO.

Incident history review

Read 3–5 recent incidents or RCAs.

For each:

What was the initial symptom?
Which alert fired?
What dashboard was used?
What logs/traces were useful?
What telemetry was missing?
What was the proximate cause?
What was the root cause?
What corrective actions touched observability?

SLO review

Find:

• availability SLO;
• latency SLO;
• correctness SLO;
• freshness SLO;
• queue lag SLO;
• workflow completion SLO;
• error budget policy;
• burn-rate alerts.

First 30-day deliverable

Create a private/team note:

# Observability Onboarding Notes — First 30 Days

## Services studied
## Dashboards studied
## Key log fields
## Key correlation conventions
## Key alerts
## SLOs found
## Recent incidents studied
## Known telemetry gaps
## Questions for senior/SRE/platform/security

8. Days 31–60 — Validate and Build Working Understanding

Fokus 31–60 hari:

Move from reading dashboards to reasoning with evidence.

Output utama:

• bisa follow request end-to-end;
• bisa explain normal vs abnormal signal;
• bisa identify one or more telemetry gaps;
• bisa review PR observability impact;
• bisa reproduce telemetry locally or in lower environment;
• bisa discuss alert quality and SLO implications.

9. Days 31–40 — Trace Real Request and Event Flows

Pick 2–3 critical flows.

Example flows:

• quote creation;
• quote pricing;
• quote approval;
• order creation;
• order validation;
• order decomposition;
• fulfillment start;
• fallout creation;
• cancellation/amendment;
• reconciliation job.

For each flow, map:

HTTP ingress
JAX-RS resource
service layer
database read/write
Redis cache/lock/idempotency
Kafka/RabbitMQ publish
Kafka/RabbitMQ consume
Camunda process/job/task
outbound HTTP dependency
background job if any
audit event
business state transition

Flow map template

# Flow: <name>

## Entry point
## Main service methods
## Dependencies
## Database tables/entities touched
## Redis usage
## Kafka/RabbitMQ topics/queues
## Camunda process/tasks
## Business state transitions
## Logs emitted
## Metrics emitted
## Spans emitted
## Audit records emitted
## Dashboards covering this flow
## Alerts covering this flow
## Known gaps

Validation questions

• Can trace show full flow?
• Do logs contain the same correlation ID?
• Do async events preserve causation?
• Are business state transitions visible?
• Can failures be separated by validation/dependency/timeout/domain rule?
• Can stuck state be detected?

10. Days 41–50 — Understand Instrumentation Conventions

Now inspect the instrumentation design.

Logging convention

Understand:

• event naming;
• field naming;
• MDC setup;
• error taxonomy;
• exception mapper logging;
• log level policy;
• redaction/masking utilities.

Metrics convention

Understand:

• metric prefix/naming;
• units;
• labels;
• disallowed labels;
• histogram buckets;
• route template handling;
• status/error code labels;
• business metrics ownership.

Tracing convention

Understand:

• OpenTelemetry agent vs SDK;
• auto vs manual instrumentation;
• resource attributes;
• semantic conventions;
• span naming;
• sampling;
• collector topology;
• propagation policy;
• baggage policy.

Audit convention

Understand:

• what actions require audit;
• where audit events are stored;
• actor representation;
• target entity representation;
• before/after policy;
• immutability;
• retention;
• access control.

11. Days 51–60 — Review Alert Quality and Operational Readiness

Study operational readiness at service level.

Pick one service and answer

What are the top 5 failure modes?
Which alerts detect them?
Which dashboards triage them?
Which logs/traces explain them?
Which runbooks mitigate them?
Which SLOs measure customer impact?

Failure modes to test mentally

• JAX-RS endpoint latency spike;
• database pool exhaustion;
• PostgreSQL lock wait/deadlock;
• Redis latency/eviction;
• Kafka consumer lag;
• RabbitMQ unacked buildup;
• Camunda failed job spike;
• dependency timeout;
• Kubernetes OOMKilled;
• CPU throttling;
• bad deployment/config;
• PII leakage in logs;
• telemetry collector outage.

Alert quality checks

• Is alert symptom-based?
• Is severity correct?
• Is threshold/window reasonable?
• Does alert have owner?
• Does alert link to dashboard/runbook?
• Is alert noisy?
• Has alert missed real incidents?
• Is alert tied to SLO burn rate?

60-day deliverable

Create a gap analysis:

# Observability Gap Analysis — 60 Days

## Flow studied
## Existing signal coverage
## Missing logs
## Missing metrics
## Broken trace/context propagation
## Dashboard gaps
## Alert quality issues
## SLO/SLI ambiguity
## Runbook gaps
## Privacy/cost risks
## Recommended improvement candidates

12. Days 61–90 — Improve One Meaningful Gap

Fokus 61–90 hari:

Make one concrete observability improvement that helps production support.

Pilih improvement yang:

• kecil cukup untuk selesai;
• nyata berdasarkan evidence;
• disetujui senior/SRE/platform/security;
• punya measurable impact;
• tidak menambah noise/cost berlebihan;
• memperbaiki debugging/incident readiness.

13. Good 61–90 Day Improvement Options

Option A — Fix missing correlation in logs

Good when:

• logs sulit di-join;
• request ID/trace ID tidak konsisten;
• async processing kehilangan context.

Deliverables:

• fix MDC propagation;
• add tests;
• verify in lower environment;
• update logging checklist;
• document query examples.

Option B — Add missing dependency dashboard panel

Good when:

• service sering degrade karena dependency;
• dashboard tidak menunjukkan root bottleneck;
• on-call harus query manual.

Deliverables:

• add latency/error/retry/timeout panels;
• add drilldown links;
• add owner labels;
• validate with incident history.

Option C — Reduce noisy alert

Good when:

• alert sering false positive;
• team mengalami alert fatigue;
• alert tidak actionable.

Deliverables:

• analyze alert history;
• adjust threshold/window;
• add burn-rate logic if appropriate;
• improve annotation/runbook;
• document before/after noise.

Option D — Add runbook for recurring incident

Good when:

• incident sering sama;
• mitigation hanya ada di kepala senior engineer;
• new engineer tidak tahu langkah triage.

Deliverables:

• symptom checklist;
• dashboard links;
• log queries;
• trace query examples;
• mitigation steps;
• escalation path;
• validation steps.

Option E — Add audit evidence for critical action

Good when:

• business/security action kurang evidence;
• actor/target/before-after tidak lengkap;
• support/compliance sering butuh manual DB query.

Deliverables:

• audit event mapping;
• tests;
• retention/access review;
• documentation;
• privacy/security review.

Option F — Fix high-cardinality metric

Good when:

• metric backend cost tinggi;
• dashboard lambat;
• label memakai raw ID/path/error message.

Deliverables:

• replace unsafe labels;
• migrate dashboard/alert queries;
• compare cardinality before/after;
• update metric design checklist.

14. Days 61–70 — Design the Improvement

Before changing code/config, write a small proposal.

# Observability Improvement Proposal

## Problem
What production/debugging problem exists?

## Evidence
What incident, dashboard, alert, or log sample proves it?

## Proposed Change
What log/metric/trace/dashboard/alert/runbook will change?

## Impact
How does this improve detection, triage, RCA, or support?

## Risks
Privacy, cost, cardinality, noise, performance, compatibility.

## Validation Plan
How will we know it works?

## Rollback Plan
How can this be reverted or disabled?

## Internal Review Needed
Senior engineer, SRE/platform, security, domain owner.

Review before implementation

Validate with:

• service owner;
• senior engineer;
• SRE/platform;
• security/privacy if telemetry contains sensitive context;
• domain/product owner if business/audit signal changes.

15. Days 71–80 — Implement and Validate

Implementation should include tests where possible.

Validation checklist

• log fields emitted as expected;
• metric type/unit/labels correct;
• trace span attributes correct;
• context propagation works;
• redaction/masking works;
• dashboard query returns expected data;
• alert rule tested if changed;
• runbook reviewed;
• lower environment verified;
• production rollout monitored.

Evidence to capture

Before screenshot/query result.
After screenshot/query result.
Sample log event.
Sample trace.
Sample metric query.
Dashboard panel link.
Alert simulation or validation evidence.

16. Days 81–90 — Document and Socialize

The improvement is not done until others can use it.

Finalize docs

Update:

• service runbook;
• dashboard notes;
• alert annotation;
• PR checklist;
• troubleshooting playbook;
• onboarding notes;
• ADR if decision is broader.

Share with team

Prepare short demo:

Problem observed.
Evidence before.
Change made.
Evidence after.
How to use it during incident.
What still remains as future work.

90-day deliverable

# 90-Day Observability Contribution Summary

## What I learned
## Production flows studied
## Incidents reviewed
## Observability gap selected
## Improvement implemented
## Evidence before/after
## Runbook/dashboard/alert updates
## Remaining risks
## Recommended next improvements

17. Questions to Ask Senior Backend Engineers

Ask precise questions.

System behavior

Which flows are most business-critical?
Which endpoints are most sensitive to latency?
Which failures are hardest to debug?
Which dependencies cause the most incidents?
Which domain states commonly get stuck?
Which retries are safe vs dangerous?

Observability

Which logs do you trust during incident?
Which dashboard do you open first?
Which metrics are misleading?
Where do traces usually break?
Which alerts are noisy?
Which telemetry gap has hurt us before?

Code review

What observability issues do you usually catch in PRs?
What fields must every log include?
What metric labels are forbidden?
When do we add manual spans?
When do we create audit events?

18. Questions to Ask SRE/Platform Team

SRE/platform owns much of the telemetry pipeline and operational standards.

Ask:

What is the official logging platform?
What is the official metrics backend?
What is the official tracing backend?
Do we use OpenTelemetry agent, SDK, collector, or mixed setup?
What resource attributes are mandatory?
What collector topology is used?
What sampling strategy is used?
What retention policies exist?
What are the cost guardrails?
How are dashboards and alerts managed?
What is the alert severity/routing policy?
How are incidents escalated?
What telemetry pipeline failures have occurred before?

Also ask for examples:

Show me a good dashboard.
Show me a good alert.
Show me a good runbook.
Show me a recent incident where observability helped.
Show me a recent incident where observability was missing.

19. Questions to Ask Security/Privacy Team

Observability often touches sensitive data.

Ask:

What data is forbidden in logs?
What data is forbidden in trace attributes?
What data is forbidden in metric labels?
What headers must be redacted?
Can tenant/customer/quote/order identifiers be logged?
Are pricing/commercial details sensitive?
What is the audit logging requirement?
What retention applies to audit logs?
Who can access telemetry backends?
What is the incident process for telemetry data leakage?

Specific for Java/JAX-RS:

Can request/response body ever be logged?
Are query parameters considered sensitive?
How should Authorization/Cookie headers be handled?
Are exception messages reviewed for sensitive content?
Are heap dumps/thread dumps allowed in production support?

20. Questions to Ask DB/Messaging/Workflow Owners

PostgreSQL / persistence

Where are slow queries visible?
How do we correlate app request to DB query?
What connection pool metrics are monitored?
What lock/deadlock alerts exist?
How are migrations observed?

Kafka/RabbitMQ

Where is consumer lag visible?
Where are DLQ counts visible?
How do we trace event causation?
What is the replay process?
How do we detect duplicate events?
How do we measure event age?

Redis

Where is cache hit/miss visible?
How do we detect eviction or hot key issues?
Are Redis keys safe to expose in telemetry?
How are locks/idempotency keys observed?

Camunda/workflow

Where are failed jobs/incidents visible?
How do we detect stuck process instances?
How do we monitor human task aging?
How are process variables handled safely?
How do we correlate process instance to quote/order?

21. Dashboards to Study Deeply

Do not just browse dashboards. Learn what each dashboard can and cannot answer.

For each dashboard, document:

Purpose
Audience
Owner
Primary questions answered
Top panels
Important labels/filters
Drilldown path to logs/traces
Known misleading panels
Recent incident usage
Missing panels

Minimum dashboard set

• service health;
• API health;
• dependency health;
• database health;
• messaging health;
• Redis health;
• workflow/Camunda health;
• Kubernetes workload health;
• release/deployment health;
• SLO/error budget;
• business lifecycle;
• incident dashboard.

22. Incidents to Study

Choose incidents that reveal different failure modes.

Study at least one incident from each category if available:

• latency degradation;
• error spike;
• bad deployment;
• bad configuration;
• database lock/pool exhaustion;
• Kafka/RabbitMQ backlog;
• Redis degradation;
• Camunda stuck process/failed job;
• Kubernetes OOM/restart/throttling;
• cloud service degradation;
• audit/compliance issue;
• missing alert;
• noisy alert;
• missing logs/traces;
• PII/logging incident.

For each incident, ask:

What signal detected it?
What signal confirmed it?
What signal was missing?
How was customer impact measured?
How was blast radius determined?
How was root cause proven?
What observability improvement followed?

23. Personal Skill Milestones

By day 30, you should be able to

• find logs for a request;
• find trace from log;
• find service health dashboard;
• identify top alerts;
• explain basic request flow;
• name key dependencies;
• identify incident/runbook location;
• explain correlation ID convention at high level.

By day 60, you should be able to

• trace one business flow end-to-end;
• identify missing telemetry;
• review basic observability impact in PR;
• explain key dashboards;
• explain top alerts and their action;
• identify SLO/SLI coverage;
• discuss telemetry cost/privacy risks.

By day 90, you should be able to

• improve one observability gap;
• write or improve a runbook;
• reduce one alert/dashboard/logging problem;
• contribute to PR/ADR observability discussion;
• support incident debugging with evidence;
• propose next observability improvements.

24. Common Onboarding Pitfalls

Pitfall 1 — Learning tool UI but not system behavior

Knowing how to click dashboards is not the same as understanding production flow.

Pitfall 2 — Trusting all dashboards equally

Some dashboards are stale, noisy, misleading, or unused.

Ask which dashboards senior engineers actually use during incidents.

Pitfall 3 — Adding telemetry without ownership

Telemetry without owner becomes abandoned cost.

Pitfall 4 — Ignoring privacy and cost

A useful log that leaks sensitive data is not acceptable.

A useful metric that destroys backend cardinality is not acceptable.

Pitfall 5 — Confusing audit logs with debug logs

Audit evidence must be deliberate, complete, and governed.

Pitfall 6 — Treating async flow like HTTP flow

Kafka/RabbitMQ/Camunda/background jobs need causation, event age, retry, idempotency, and stuck-state thinking.

Pitfall 7 — Not studying past incidents

Incident history reveals real system behavior better than architecture diagrams alone.

25. 30/60/90 Checklist

First 30 days

• Found service architecture docs.
• Found logging standard.
• Found metrics standard.
• Found tracing/OpenTelemetry standard.
• Found audit logging policy.
• Found privacy/security logging policy.
• Found main dashboards.
• Found alert catalog.
• Found runbook location.
• Found SLO/SLA docs.
• Read recent incidents/RCAs.
• Traced at least one request from log to trace.
• Identified correlation ID convention.
• Identified telemetry owners.

Days 31–60

• Mapped one critical request flow.
• Mapped one async event flow.
• Mapped one business lifecycle flow.
• Reviewed JAX-RS logging/instrumentation.
• Reviewed dependency metrics.
• Reviewed top alerts.
• Reviewed one dashboard for correctness.
• Reviewed trace propagation across one boundary.
• Identified at least three telemetry gaps.
• Discussed findings with senior/SRE/platform.

Days 61–90

• Selected one meaningful improvement.
• Wrote short proposal.
• Reviewed privacy/cost/cardinality impact.
• Implemented or contributed change.
• Added tests where possible.
• Validated in lower environment.
• Captured before/after evidence.
• Updated dashboard/alert/runbook/docs.
• Shared improvement with team.
• Proposed next improvements.

26. Internal Verification Checklist

Because internal stack/process may vary, verify these explicitly.

Observability stack

• Actual log backend.
• Actual metric backend.
• Actual tracing backend.
• OpenTelemetry usage pattern.
• Collector topology.
• Sampling strategy.
• Retention policy.
• Dashboard ownership.
• Alert ownership.
• SLO ownership.

Codebase conventions

• Logging framework.
• Structured log schema.
• MDC fields.
• Correlation/request ID rules.
• Metric naming rules.
• Allowed labels.
• Span naming rules.
• Audit event model.
• Redaction/masking utilities.
• Exception mapping policy.

Operations process

• On-call process.
• Escalation path.
• Incident channel/process.
• RCA template.
• Runbook template.
• Alert review cadence.
• Dashboard review cadence.
• SLO review cadence.
• Security incident process.
• Cost review process.

Domain-specific

• Quote lifecycle states.
• Order lifecycle states.
• Approval flow.
• Pricing flow.
• Fulfillment flow.
• Fallout process.
• Cancellation/amendment process.
• Reconciliation process.
• Audit requirements per business action.
• Business dashboards used by support/operations.

27. Final Takeaway

A strong observability onboarding plan turns a new engineer from code reader into production reasoner.

The end state is not:

I know which tools we use.

The end state is:

I can follow a customer-impacting problem across logs, metrics, traces,
audit records, dashboards, alerts, deployments, dependencies, and domain state —
and I know where the evidence is weak enough to improve.

That is the production-readiness mindset expected from a senior backend engineer.