Cheatsheet Observability

Fondasi observability, monitoring, telemetry, logging, metrics, tracing, alerting, auditing, dan production debugging untuk Java/JAX-RS enterprise systems.

Distributed Systems Mental Model

24 min

Cara melihat request lifecycle, distributed transaction, microservice boundary, dependency call, async event, background job, human workflow, partial failure, timeout, retry, duplicate event, stale read, queue lag, service degradation, blast radius, dan signal design sebagai fondasi observability.

Telemetry Signals

28 min

Pemetaan jenis signal observability, ownership, quality, retention, dan cost untuk enterprise Java/JAX-RS systems.

Logging Foundation

25 min

Dasar log event, log line, structured log, JSON log, log level, timestamp, context field, dan stack trace untuk Java/JAX-RS production systems.

Java Logging Ecosystem

Ekosistem logging Java untuk aplikasi enterprise JAX-RS: SLF4J, Logback, Log4j2, JUL, bridge libraries, async appender, JSON encoder, MDC, Marker, parameterized logging, exception logging, konfigurasi, dan runtime log level.

Structured Logging

11 min

Desain structured logging untuk Java/JAX-RS enterprise systems: JSON log schema, standard fields, timestamp, level, logger, service, environment, version, request, actor, tenant, business key, error fields, dependency fields, privacy, queryability, dan review checklist.

Log Level Discipline

Disiplin penggunaan TRACE, DEBUG, INFO, WARN, ERROR, expected error, unexpected error, retriable error, non-retriable error, log spam, missing log, dan review checklist untuk Java/JAX-RS production systems.

Log Context and MDC

13 min

Cara membawa request context, correlation ID, trace ID, span ID, tenant ID, actor ID, request ID, dan business key ke log menggunakan MDC/ThreadLocal secara aman di Java/JAX-RS, thread pool, async job, virtual thread, messaging, dan production systems.

Correlation, Causation, and Request Identity

Standar identitas teknis dan bisnis untuk korelasi HTTP, messaging, workflow, background job, audit, dan production debugging.

Logging in JAX-RS/Jakarta REST

Request/response logging, JAX-RS filters, ExceptionMapper, latency log, status code log, endpoint template, header/body risk, and production-safe REST observability.

HTTP Access Logging and Edge Logs

NGINX, ingress, API gateway, load balancer, JAX-RS access log, status code, request time, upstream response time, client IP, forwarded headers, and 499/502/503/504 analysis.

Error Logging and Exception Observability

Exception taxonomy, domain errors, validation/auth/dependency/timeout errors, retryability, stack traces, root cause, error codes, response mapping, and deduplication.

Audit Logging

Audit log purpose, business/security/compliance audit, who/what/when/where/why, before-after values, actor, target entity, immutability, retention, and production review checklist.

Security and Privacy in Logging

PII, secrets, tokens, authorization headers, cookies, personal data, commercial data, redaction, masking, sampling, access control, and secure logging review checklist.

Metrics Foundation

Counter, gauge, histogram, summary, timer, rate, percentile, bucket, label/tag, cardinality, time series, scrape/push, metric naming, dan metric lifecycle untuk enterprise Java/JAX-RS systems.

Metric Design Discipline

Purpose, owner, unit, type, labels, cardinality, business dimension, technical dimension, aggregation, retention, cost, dan PR review discipline untuk metrics di enterprise Java/JAX-RS systems.

RED, USE, and Golden Signals

Rate, Errors, Duration, Utilization, Saturation, Errors, latency, traffic, errors, saturation, service health, dependency health, dan cara memilih metric framework untuk enterprise Java/JAX-RS systems.

Application Metrics for Java/JAX-RS

Request count, request latency, error count, status code, endpoint template, payload size, active request, thread pool usage, queue size, JVM memory, GC pause, CPU, file descriptor, connection pool, dan application metric review discipline.

JVM Metrics

Heap usage, non-heap usage, metaspace, direct memory, GC count, GC pause, thread count, deadlock detection, class loading, CPU usage, safepoint awareness, allocation rate, container memory, container CPU throttling, dan JVM metric review discipline.

Database Observability

Connection pool metrics, active/idle/pending connections, pool exhaustion, query latency, slow query, lock wait, deadlock, transaction duration, rows affected, PostgreSQL metrics, MyBatis/JPA SQL visibility, dan database observability checklist.

Messaging Observability

Kafka producer/consumer metrics, RabbitMQ queue depth, consumer lag, retry count, DLQ count, redelivery, publish/consume rate, message processing latency, event age, duplicate event count, dan messaging observability checklist.

Redis Observability

Cache hit/miss ratio, latency, command rate, slowlog, memory usage, eviction, expiration, keyspace size, connected/blocked clients, replication lag, stream pending entries, dan Redis observability checklist.

Workflow/Camunda Observability

Observability untuk workflow engine, process instance, failed job, incident, worker latency, task aging, timer backlog, message correlation failure, stuck process, dan human task aging dalam enterprise Java/JAX-RS systems.

Kubernetes Observability

Observability Kubernetes untuk pod status, restarts, CPU, memory, throttling, OOMKilled, readiness/liveness, node pressure, deployment availability, HPA, Kubernetes events, ingress metrics, dan production debugging Java/JAX-RS services.

Cloud Observability: AWS and Azure

Cloud observability untuk AWS/Azure dalam konteks Java/JAX-RS enterprise service: CloudWatch, Azure Monitor, Log Analytics, Application Insights, load balancer, API gateway/APIM, managed database, managed broker, cloud service health, dan hybrid/on-prem boundary.

Distributed Tracing Foundation

Fondasi distributed tracing untuk enterprise Java/JAX-RS systems: trace, span, parent/child/root span, span context, trace ID, span ID, attributes, events, status, baggage, context propagation, sampling, dan trace reasoning untuk production debugging.

OpenTelemetry Foundation

Fondasi OpenTelemetry untuk enterprise Java/JAX-RS systems: API, SDK, auto-instrumentation, manual instrumentation, resource, span, metric, log bridge, exporter, collector, propagator, semantic conventions, privacy, cost, dan production readiness.

OpenTelemetry Collector

OpenTelemetry Collector sebagai pipeline telemetry production: receiver, processor, exporter, pipeline, batch, memory limiter, attributes/resource processor, sampling, retry, deployment mode, security, cost, dan failure debugging.

Context Propagation

Context propagation untuk enterprise Java/JAX-RS systems: W3C traceparent, tracestate, baggage, HTTP headers, Kafka/RabbitMQ headers, executor propagation, ThreadLocal, MDC, trace continuity, context loss, context spoofing, privacy, dan production debugging.

JAX-RS and Servlet Instrumentation

Instrumentation untuk Java/JAX-RS dan Servlet lifecycle: servlet filter, JAX-RS filter, Jersey instrumentation, resource span, ExceptionMapper span status, route template, HTTP method/status, client metadata, async request, streaming response, privacy, cardinality, dan production readiness.

HTTP Client Instrumentation

Instrumentation untuk outbound HTTP dependency calls dari Java/JAX-RS service: Jersey Client, OpenFeign, Retrofit, Java HTTP Client, timeout, retry, DNS/TLS latency, connection pool, downstream status code, circuit breaker, context propagation, metrics, logs, traces, dashboard, alerting, privacy, dan production debugging.

Database Instrumentation

13 min

Instrumentation untuk JDBC, DataSource, connection pool, PostgreSQL, MyBatis, JPA/Hibernate, query span, SQL sanitization, SQL parameter privacy, transaction span, slow query correlation, lock/deadlock evidence, metrics, logs, traces, dashboard, alerting, dan production debugging.

Kafka Instrumentation

20 min

Instrumentation Kafka untuk Java/JAX-RS enterprise systems: producer span, consumer span, publish latency, processing latency, consumer lag, topic, partition, offset, message key privacy, header propagation, batch processing, retry/DLQ span, event age, metrics, logs, tracing, dashboard, alerting, dan production debugging.

RabbitMQ Instrumentation

Instrumentation RabbitMQ untuk Java/JAX-RS enterprise systems: publish span, consume span, exchange, queue, routing key, delivery tag, redelivery, ack/nack, retry/DLQ, header propagation, queue latency, message age, metrics, logs, tracing, dashboard, alerting, dan production debugging.

Redis Instrumentation

Instrumentation Redis untuk Java/JAX-RS enterprise systems: command span, key privacy, command latency, cache hit/miss, TTL, pipeline, Lua script, rate limiter, lock, idempotency, stream metrics, logs, tracing, dashboard, alerting, dan production debugging.

Background Job and Scheduler Instrumentation

Instrumentation background job dan scheduler untuk Java/JAX-RS enterprise systems: job start/end/duration/status/retry/failure/lock/batch size/processed count/reconciliation result/Kubernetes CronJob/scheduler drift/logs/metrics/traces/audit/runbook/production debugging.

Business Observability for CPQ/Order Management

Business observability untuk CPQ, quote management, order management, quote-to-cash, approval, pricing, fulfillment, fallout, amendment, cancellation, reconciliation, SLA bisnis, dan production debugging berbasis lifecycle evidence.

Domain State and Lifecycle Observability

Observability untuk domain state machine dan lifecycle: state transition metric, invalid transition, transition latency, state aging, terminal/retryable/manual intervention/stuck state, lifecycle funnel, business SLA, transition audit, dan production debugging.

Dashboard Design

Desain dashboard observability yang purpose-driven, audience-aware, incident-oriented, signal-quality-aware, cost-aware, dan cocok untuk Java/JAX-RS enterprise systems.

Service Health Dashboard

Desain service health dashboard untuk Java/JAX-RS service: request rate, error rate, latency, saturation, dependency errors, thread pool, connection pool, JVM, GC, pod restart, deployment version, deployment marker, dan incident drilldown.

Dependency Health Dashboard

Desain dependency health dashboard untuk PostgreSQL, Redis, Kafka, RabbitMQ, Camunda, downstream HTTP, cloud service, timeout, retry, circuit breaker, latency, error budget, dan incident drilldown.

Alerting Foundation

Fondasi alerting untuk production systems: alert purpose, symptom-based alert, cause-based alert, paging/ticket/info alert, severity, threshold, burn rate, alert window, noise, fatigue, ownership, dan runbook readiness.

Alert Design and Runbooks

Desain alert yang actionable: alert message, labels, annotations, runbook link, dashboard link, owner, escalation path, customer impact hint, triage steps, false positive handling, suppression, deduplication, privacy, cost, dan review checklist.

SLI, SLO, SLA, and Error Budget

Reliability target untuk engineering: SLI, SLO, SLA, error budget, availability, latency, correctness, freshness, durability, queue lag, workflow completion, burn-rate alerting, SLO review, dan design checklist untuk Java/JAX-RS enterprise systems.

Incident Debugging with Observability

20 min

Practical incident debugging using logs, metrics, traces, audit logs, events, dashboards, deployment markers, and platform signals for Java/JAX-RS enterprise systems. Covers evidence collection, timeline reconstruction, recent change analysis, blast radius, customer impact, mitigation, and internal verification.

RCA Signals

How to use observability evidence for RCA: symptom, proximate cause, root cause, contributing factor, trigger, missing guardrails, missing telemetry, corrective action, preventive action, and production learning loops for Java/JAX-RS enterprise systems.

Release and Deployment Observability

How to make releases observable: deployment markers, version labels, commit SHA, build ID, image digest, feature flags, config version, migration version, canary metrics, rollback signals, release health dashboards, and production-safe deployment verification for Java/JAX-RS enterprise systems.

Observability Cost Management

How to manage observability cost without destroying production visibility: log volume, metric cardinality, trace volume, sampling, retention, storage, ingestion, query cost, dashboard cost, debug log cost, audit retention cost, and cost-quality trade-offs for Java/JAX-RS enterprise systems.

Sampling Strategy

How to design log and trace sampling without hiding critical production evidence: log sampling, trace sampling, head sampling, tail sampling, error-biased sampling, latency-biased sampling, business-critical sampling, per-service sampling, per-endpoint sampling, debug sampling, and sampling risk for Java/JAX-RS enterprise systems.

High Cardinality and Label Governance

How to prevent metric and telemetry cardinality explosions: labels, tags, dimensions, user ID risk, request ID risk, order ID risk, tenant ID trade-offs, raw path explosions, error message label risk, backend cost, cardinality detection, and label governance for Java/JAX-RS enterprise systems.

Health Checks vs Observability

How to design health endpoints, liveness probes, readiness probes, startup probes, shallow and deep health checks, dependency health, synthetic checks, and telemetry separation for Java/JAX-RS services running in Kubernetes and enterprise hybrid environments.

Synthetic Monitoring

How to design synthetic monitoring for APIs and business journeys: synthetic requests, smoke tests, user journey probes, regional and private endpoint probes, authenticated checks, dependency checks, false positives, frequency, alerting, and observability integration.

Profiling and Continuous Profiling

CPU profile, memory profile, allocation profile, thread dump, heap dump, flame graph, Java Flight Recorder, async profiler awareness, continuous profiling, production-safe profiling, and privacy risk in dumps.

Telemetry Storage and Retention

Telemetry backend, retention policy, hot/warm/cold storage, archive, audit retention, compliance retention, query performance, storage cost, access control, and lifecycle governance for logs, metrics, and traces.

Kubernetes and GitOps Observability

Manifest labels, pod labels, service labels, deployment version, Helm release, Argo CD application, Git commit annotation, config version annotation, Prometheus scrape annotation, OpenTelemetry sidecar/agent, collector deployment, GitOps drift signal, and Kubernetes/GitOps observability checklist.

AWS/Azure Cloud Observability

CloudWatch metrics, CloudWatch logs, CloudWatch alarms, Azure Monitor metrics, Log Analytics query, Application Insights, managed service metrics, load balancer logs, WAF logs, private endpoint logs, cloud audit logs, cross-cloud dashboards, and cloud observability review checklist.

Observability Governance

Signal ownership, naming convention, logging standard, metric standard, trace standard, dashboard standard, alert standard, SLO standard, runbook standard, privacy standard, cost guardrail, review process, and governance checklist.

Testing Observability

Unit test for log context, integration test for trace propagation, metric assertion, structured log assertion, error log test, audit log test, OpenTelemetry test exporter, local collector, Testcontainers observability stack, synthetic smoke test, dashboard validation, alert rule test, and observability testing checklist.

PR Review and Architecture Decision Checklist

Logging review, log level review, structured field review, correlation ID review, metric review, cardinality review, trace review, context propagation review, audit review, privacy review, dashboard review, alert review, SLO review, cost review, operational readiness review, ADR for observability decision, and questions senior engineers should ask.

30/60/90-Day Observability Onboarding Plan

First 30 days: learn log format, dashboards, alerts, trace flow, and incident history. Days 31-60: understand instrumentation conventions, service metrics, dependency signals, and alert quality. Days 61-90: improve one observability gap, add runbook, reduce alert noise, improve incident debugging evidence, and ask the right questions to senior engineers, SRE/platform, and security teams.

Troubleshooting Playbook

23 min

Playbook production-safe untuk mendiagnosis missing logs, wrong log level, missing correlation ID, broken trace, missing metric, metric cardinality explosion, misleading dashboard, noisy alert, missing alert, unexplained SLO burn, expensive logs, PII leakage, timezone mismatch, and sampling blind spots in enterprise Java/JAX-RS systems.

Final Part: Observability Mastery Map for Senior Backend Engineer