RED, USE, and Golden Signals

RED, USE, dan golden signals adalah cara disiplin untuk memilih metric yang benar.

Tanpa framework, team biasanya expose metric berdasarkan apa yang mudah tersedia dari library. Hasilnya dashboard penuh angka, tetapi tidak bisa menjawab pertanyaan production yang paling penting:

• Apakah service menerima traffic normal?
• Apakah service menghasilkan error abnormal?
• Apakah latency memburuk?
• Apakah service atau dependency sedang saturated?
• Apakah bottleneck ada di aplikasi, JVM, database, messaging, Redis, Kubernetes, atau cloud platform?
• Apakah user/customer/business flow terdampak?

Part ini membahas tiga kerangka metric yang paling penting untuk senior backend engineer:

• RED method untuk request-serving services.
• USE method untuk resources dan infrastructure.
• Golden signals untuk reliability dan SLO awareness.

Konteks utama tetap enterprise Java/JAX-RS service yang berinteraksi dengan PostgreSQL, MyBatis/JPA/JDBC, Kafka, RabbitMQ, Redis, Camunda, NGINX, Docker/Kubernetes, AWS/Azure, dan deployment cloud/on-prem/hybrid.

1. Why Metric Framework Exists

Metric framework ada karena metric mentah tidak otomatis berguna.

Contoh metric mentah:

jvm_memory_used_bytes
http_requests_total
process_cpu_usage
postgres_connections_active
kafka_consumer_records_lag_max
redis_command_duration_seconds

Metric tersebut berguna, tetapi belum menjawab urutan diagnosis.

Saat incident, engineer tidak butuh semua angka. Engineer butuh jalur berpikir:

1. Apakah request masuk?
2. Apakah error naik?
3. Apakah latency naik?
4. Apakah saturation naik?
5. Dependency mana yang berubah?
6. Apakah ini setelah deployment/config change?
7. Apakah blast radius satu endpoint, satu tenant, satu region, satu pod, atau seluruh service?

RED, USE, dan golden signals menyediakan struktur untuk menjawab pertanyaan itu.

2. RED Method

RED adalah framework untuk service yang melayani request.

RED berarti:

Untuk Java/JAX-RS service, RED biasanya diterapkan pada HTTP server endpoints.

Contoh metric:

http.server.requests.total
http.server.request.duration.seconds
http.server.errors.total

Dengan label yang aman:

service
method
route
status_code
status_class
error_type
environment

Hindari label ber-cardinality tinggi:

request_id
user_id
quote_id
order_id
raw_path
full_error_message

3. Rate

Rate menjawab apakah service sedang menerima traffic.

Contoh pertanyaan:

Apakah order submission turun ke nol karena upstream gateway gagal?

Apakah pricing endpoint menerima spike traffic setelah batch job berjalan?

Apakah consumer service memproses event lebih lambat daripada publish rate?

Rate untuk HTTP biasanya berasal dari counter:

http.server.requests.total

Lalu query backend menghitung rate:

rate(http.server.requests.total[5m])

Metric rate harus bisa dipotong berdasarkan route template, bukan raw path.

Baik:

route="/quotes/{quoteId}/price"

Buruk:

path="/quotes/Q-928381/price"

Raw path bisa menciptakan cardinality explosion.

4. Errors

Errors menjawab apakah request gagal.

Tetapi error tidak boleh hanya dilihat sebagai HTTP 500.

Untuk Java/JAX-RS service, error bisa muncul sebagai:

• HTTP 5xx.
• HTTP 4xx yang abnormal.
• Exception sebelum response mapping.
• Timeout ke dependency.
• Circuit breaker open.
• Database transaction rollback.
• Kafka/RabbitMQ publish failure.
• Redis operation failure.
• Camunda job failure.
• Business failure seperti invalid lifecycle transition.

Error metric harus membedakan minimal:

status_class="4xx" | "5xx"
error_type="validation" | "auth" | "authorization" | "domain" | "dependency" | "timeout" | "unexpected"
outcome="success" | "failure"

Namun jangan membuat label terlalu detail.

Buruk:

error_message="Could not price quote Q-928381 because product PROD-773..."

Baik:

error_type="pricing_dependency_timeout"

5. Duration

Duration menjawab seberapa lama request diproses.

Duration sebaiknya direpresentasikan sebagai histogram/timer, bukan gauge.

Contoh:

http.server.request.duration.seconds

Duration harus bisa dianalisis dengan percentile:

p50
p90
p95
p99

Mean latency sering menipu.

Contoh:

Average terlihat aman, tetapi tail latency menunjukkan sebagian user mengalami response lambat.

Untuk enterprise CPQ/order system, tail latency sering lebih penting karena operasi seperti pricing, validation, decomposition, approval lookup, dan fulfillment orchestration dapat punya dependency chain panjang.

6. RED for JAX-RS Endpoint

Untuk JAX-RS endpoint, RED harus berbasis route/resource template.

Contoh endpoint:

@Path("/quotes/{quoteId}/price")
public class QuotePricingResource {
    @POST
    public Response priceQuote(@PathParam("quoteId") String quoteId) {
        // pricing orchestration
    }
}

Metric yang diharapkan:

http.server.requests.total{
  service="quote-api",
  method="POST",
  route="/quotes/{quoteId}/price",
  status_class="2xx",
  environment="prod"
}

http.server.request.duration.seconds{
  service="quote-api",
  method="POST",
  route="/quotes/{quoteId}/price",
  environment="prod"
}

Yang tidak boleh:

route="/quotes/Q-123456/price"

Karena Q-123456 adalah business identifier yang akan menciptakan time series baru untuk setiap quote.

7. RED for Async Consumers

RED tidak hanya untuk HTTP.

Untuk Kafka/RabbitMQ consumer, analoginya:

Contoh metric:

messaging.consumer.messages.processed.total
messaging.consumer.processing.duration.seconds
messaging.consumer.errors.total
messaging.consumer.retries.total
messaging.consumer.dlq.total

Label aman:

service
topic_or_queue
consumer_group
message_type
outcome
error_type

Label berbahaya:

message_id
order_id
quote_id
raw_payload_type_with_version_per_message
exception_message

8. RED for Background Jobs

Untuk scheduled job atau reconciliation job, RED perlu diadaptasi.

Contoh:

job.executions.total{job="order-reconciliation", outcome="success"}
job.duration.seconds{job="order-reconciliation"}
job.items.processed.total{job="order-reconciliation", outcome="success"}
job.items.failed.total{job="order-reconciliation", error_type="dependency_timeout"}

Job observability harus menjawab:

• Apakah job berjalan sesuai jadwal?
• Apakah job selesai?
• Berapa lama dibanding baseline?
• Berapa item diproses?
• Berapa item gagal?
• Apakah kegagalan partial atau total?
• Apakah job overlap dengan run sebelumnya?

9. USE Method

USE adalah framework untuk resource.

USE berarti:

USE cocok untuk:

• CPU.
• Memory.
• Disk.
• Network.
• Thread pool.
• Connection pool.
• Queue.
• Kafka consumer.
• RabbitMQ queue.
• Redis connection pool.
• Database connection pool.
• Kubernetes node/pod/container.

10. Utilization

Utilization menjawab penggunaan resource saat ini.

Contoh:

CPU usage 85%
Heap usage 70%
Connection pool active 45/50
Redis memory used 80%
Kafka partition consumption rate 500 msg/s

Utilization tinggi belum tentu buruk.

CPU 80% bisa sehat jika latency stabil, error rendah, dan tidak ada throttling.

Heap 75% bisa normal jika GC pause rendah.

Connection pool 90% bisa berbahaya jika pending thread naik.

Utilization harus dibaca bersama saturation dan errors.

11. Saturation

Saturation adalah signal bahwa demand melebihi kapasitas langsung.

Contoh saturation:

Thread pool queue size meningkat
Connection pool pending threads meningkat
Kafka consumer lag naik
RabbitMQ queue depth naik
CPU throttling naik
GC pause makin panjang
Database lock wait meningkat
NGINX upstream queueing meningkat

Saturation sering lebih penting daripada utilization.

Contoh:

CPU usage hanya 55%, tetapi request latency p99 naik.

Kemungkinan:

• Thread pool blocked menunggu database.
• Connection pool exhausted.
• Lock contention di PostgreSQL.
• External dependency timeout.
• Container CPU throttling walaupun average CPU tidak tinggi.
• GC safepoint pause.

Saturation membantu menemukan bottleneck aktual.

12. Errors in USE

Errors pada resource berbeda dari HTTP errors.

Contoh resource errors:

Resource error harus dihubungkan dengan service-level RED.

Contoh diagnosis:

HTTP p99 naik + DB connection pool pending naik + pool acquire timeout naik
= latency kemungkinan akibat DB pool saturation.

13. USE for JVM

Untuk JVM, USE dapat diterapkan seperti ini:

Metric penting:

jvm.memory.used.bytes
jvm.memory.committed.bytes
jvm.gc.duration.seconds
jvm.threads.count
process.cpu.usage
container.cpu.throttled.periods.total
container.memory.working_set.bytes

JVM metric harus dibaca bersama application latency.

GC pause tidak penting hanya karena ada GC. GC menjadi penting ketika:

• p99 latency ikut naik.
• request timeout meningkat.
• CPU usage naik.
• heap occupancy tidak turun setelah GC.
• pod restart karena OOMKilled.

14. USE for Thread Pools

Thread pool adalah resource kritis di Java backend.

Metric yang perlu diperhatikan:

executor.active.threads
executor.pool.size
executor.queue.size
executor.completed.tasks.total
executor.rejected.tasks.total
executor.task.duration.seconds

Failure mode:

• Thread pool penuh.
• Queue tumbuh terus.
• Task ditolak.
• Worker blocked menunggu DB/HTTP/Redis.
• Scheduler job overlap.
• Async processing kehilangan MDC/trace context.

USE mapping:

15. USE for Database Connection Pool

Connection pool adalah salah satu bottleneck paling umum di Java enterprise systems.

Metric:

db.pool.connections.active
db.pool.connections.idle
db.pool.connections.pending
db.pool.connections.max
db.pool.acquire.duration.seconds
db.pool.timeout.total

USE mapping:

Diagnosis umum:

HTTP p95 naik
+ active connection mendekati max
+ pending connection naik
+ query latency tidak naik banyak
= kemungkinan pool terlalu kecil, connection leak, atau transaction terlalu lama.

HTTP p95 naik
+ active connection naik
+ query latency naik
+ PostgreSQL lock wait naik
= kemungkinan database contention, bukan sekadar pool sizing.

16. USE for Kafka and RabbitMQ

Messaging resource punya saturation yang sangat jelas: lag atau queue depth.

Kafka:

kafka.consumer.lag
kafka.consumer.records.consumed.rate
kafka.consumer.processing.duration.seconds
kafka.producer.request.latency.seconds
kafka.producer.errors.total

RabbitMQ:

rabbitmq.queue.messages.ready
rabbitmq.queue.messages.unacked
rabbitmq.queue.publish.rate
rabbitmq.queue.deliver.rate
rabbitmq.consumer.redelivered.total
rabbitmq.dlq.messages.total

USE mapping:

Important distinction:

High publish rate + stable lag = healthy high traffic.
High publish rate + growing lag = consumer cannot keep up.
Normal publish rate + growing lag = consumer degraded.
Low publish rate + high lag = consumer previously down or poison messages.

17. USE for Redis

Redis observability sering disalahpahami sebagai hanya hit ratio.

USE mapping:

Metric penting:

redis.memory.used.bytes
redis.commands.processed.total
redis.command.duration.seconds
redis.evicted.keys.total
redis.expired.keys.total
redis.connected.clients
redis.blocked.clients
redis.replication.lag.seconds

Cache hit ratio harus dibaca bersama latency dan fallback cost.

Contoh:

Cache hit ratio turun dari 95% ke 60%
+ PostgreSQL query latency naik
+ HTTP p99 naik
= cache miss menyebabkan DB overload.

18. Golden Signals

Golden signals biasanya terdiri dari:

Golden signals mirip RED, tetapi eksplisit menambahkan saturation.

Untuk service Java/JAX-RS, golden signals dapat dirumuskan sebagai:

Traffic    = HTTP request rate + async message rate + job execution rate
Errors     = HTTP 5xx/abnormal 4xx + dependency failure + business failure
Latency    = HTTP duration + dependency duration + async processing duration
Saturation = thread pool + connection pool + CPU/memory + queue/lag + GC + throttling

19. Golden Signals for CPQ/Order Service

Untuk CPQ/order management, golden signals teknis perlu ditambah business-aware signals.

Contoh untuk quote pricing service:

Contoh untuk order fulfillment flow:

20. Choosing RED vs USE vs Golden Signals

Gunakan RED untuk:

• HTTP APIs.
• RPC-like service calls.
• Consumer processing.
• Background job execution.
• Workflow worker execution.

Gunakan USE untuk:

• JVM.
• Thread pool.
• Connection pool.
• Kubernetes pod/node.
• Database.
• Redis.
• Kafka/RabbitMQ queues.
• Disk/network/CPU/memory.

Gunakan golden signals untuk:

• Service health dashboard.
• Executive reliability summary.
• SLO discussion.
• Incident triage landing page.
• Cross-service comparison.

Prinsip praktis:

RED tells whether the service is serving users correctly.
USE tells whether underlying resources are becoming bottlenecks.
Golden signals combine both into reliability view.

21. Service Health Metric Minimum

Untuk satu Java/JAX-RS service, minimum metric sebaiknya mencakup:

http.server.requests.total
http.server.request.duration.seconds
http.server.errors.total
jvm.memory.used.bytes
jvm.gc.duration.seconds
process.cpu.usage
executor.active.threads
executor.queue.size
db.pool.connections.active
db.pool.connections.pending
db.pool.acquire.duration.seconds
kubernetes.pod.restart.total

Jika service memakai dependency tambahan:

http.client.request.duration.seconds
http.client.errors.total
redis.command.duration.seconds
kafka.consumer.lag
rabbitmq.queue.messages.ready
camunda.failed.jobs.total

Jika service punya business critical flow:

quote.pricing.requests.total
quote.pricing.duration.seconds
order.lifecycle.transitions.total
order.stuck.current
approval.tasks.aging.seconds

22. Dependency Health Metric Minimum

Dependency health harus menjawab:

• Apakah dependency reachable?
• Apakah dependency latency naik?
• Apakah dependency error naik?
• Apakah retry meningkat?
• Apakah circuit breaker open?
• Apakah saturation terjadi di client side atau server side?

Contoh outbound HTTP dependency:

http.client.requests.total
http.client.request.duration.seconds
http.client.errors.total
http.client.timeouts.total
http.client.retries.total
circuitbreaker.state
circuitbreaker.calls.total

Contoh database dependency:

db.query.duration.seconds
db.pool.connections.active
db.pool.connections.pending
db.pool.acquire.duration.seconds
db.errors.total

Contoh messaging dependency:

messaging.producer.publish.duration.seconds
messaging.producer.errors.total
messaging.consumer.lag
messaging.consumer.processing.duration.seconds
messaging.dlq.messages.total

23. Common Anti-Patterns

Anti-pattern 1: Dashboard only shows CPU and memory

CPU dan memory tidak cukup untuk service health.

Service bisa error 80% dengan CPU normal.

Service bisa latency p99 tinggi karena DB lock walaupun memory normal.

Anti-pattern 2: Only average latency

Average latency menyembunyikan tail latency.

Gunakan histogram dan percentile.

Anti-pattern 3: Error count without traffic context

100 errors per minute buruk jika traffic 200/minute.

100 errors per minute mungkin kecil jika traffic 2 million/minute, meskipun tetap perlu dilihat berdasarkan SLO.

Gunakan error rate.

Anti-pattern 4: Raw path as metric label

Raw path seperti /orders/12345 menyebabkan cardinality explosion.

Gunakan route template seperti /orders/{orderId}.

Anti-pattern 5: Per-user or per-order metric label

User ID, request ID, quote ID, order ID tidak boleh menjadi metric label default.

Gunakan log/trace/audit untuk high-cardinality lookup.

Anti-pattern 6: No dependency metrics

Tanpa dependency metrics, semua latency terlihat seperti latency aplikasi.

Anti-pattern 7: Alerts based only on causes

Alert CPU > 80% sering noisy.

Lebih baik paging berdasarkan symptom/SLO burn, lalu gunakan CPU sebagai diagnostic signal.

24. Failure Mode Mapping

25. Correctness Concerns

Metric correctness problems include:

• Counter reset not handled.
• Duration unit inconsistent.
• Histogram bucket too coarse.
• Route label uses raw path.
• Error metric excludes mapped exceptions.
• 4xx business failures incorrectly mixed with 5xx system failures.
• Retry attempts counted as user requests.
• Async processing duration measured from consume start, not event creation time.
• Queue lag metric missing for some consumer groups.
• Multi-pod aggregation double counts gauge-like values incorrectly.

Senior engineer should always ask:

Is this metric measuring what the dashboard thinks it is measuring?

26. Performance Concerns

Metrics instrumentation can harm performance if careless.

Risk examples:

• High-cardinality labels explode memory and backend storage.
• Heavy label computation on hot path.
• Synchronous metric export blocking request thread.
• Per-item metric emission in huge batch.
• Excessive histogram buckets.
• Metric collection lock contention.

Mitigation:

• Keep labels low-cardinality.
• Use route templates.
• Avoid request/user/entity IDs as labels.
• Use batching/exporter designed for production.
• Prefer aggregate counters for batch item results.
• Review metric volume before rollout.

27. Security and Privacy Concerns

Metrics often seem safe because they are numeric, but labels can leak sensitive data.

Do not label metrics with:

user_id
email
phone_number
access_token
session_id
quote_id
order_id
raw_url
raw_query
full_error_message
customer_name
account_number

Tenant ID may be allowed only if internal policy permits it and cardinality is controlled.

For CPQ/order systems, quote/order identifiers are usually better kept in logs/traces/audit, not metrics.

28. Cost Concerns

The cost of metrics is dominated by time series cardinality.

Time series count roughly equals:

metric_name x label_value_combinations x environments x instances

Example:

http.server.request.duration.seconds
labels: method(5) x route(80) x status_code(20) x pod(30)
= 240,000 potential series before buckets

If histogram has many buckets, series multiply further.

Adding user_id or order_id can destroy the backend.

Metric design must treat labels as cost-bearing schema, not casual metadata.

29. Operational Concerns

A good service health metric set should support:

• On-call triage.
• SLO burn alert.
• Deployment comparison.
• Capacity planning.
• Dependency blame avoidance.
• Regression detection.
• Business impact estimation.
• Incident RCA.

Metrics should not require the original author to interpret them.

If only one engineer understands a dashboard, the dashboard is operationally weak.

30. PR Review Checklist

When reviewing metrics in a PR, ask:

• What production question does this metric answer?
• Is the type correct: counter, gauge, histogram, timer?
• Is the unit explicit?
• Are labels low-cardinality?
• Are route/path labels templated?
• Are business identifiers excluded from labels?
• Does this metric support RED, USE, or golden signals?
• Is error classification meaningful but bounded?
• Does it duplicate existing metrics?
• Will it be used in dashboard, alert, SLO, or debugging?
• Is it safe for privacy/security?
• What is the expected volume and cost?
• How will it behave in batch/async/retry scenarios?
• Is aggregation across pods/services correct?

31. Internal Verification Checklist

For CSG/team context, verify rather than assume:

• Which metric backend is used internally.
• Whether Prometheus, OpenTelemetry metrics, Micrometer, vendor agent, or custom metric library is standard.
• Naming convention for service metrics.
• Required labels for service, environment, region, namespace, pod, version, and owner.
• Whether route labels must use JAX-RS template paths.
• Standard RED dashboard template.
• Standard USE dashboard template.
• Existing service health dashboard.
• Existing dependency health dashboard.
• Existing golden signal dashboard.
• Alert policy for error rate and latency.
• SLO/error budget conventions.
• Allowed and disallowed metric labels.
• Cardinality limits per service/team.
• How metric cost is reviewed.
• Whether tenant ID is allowed as label.
• Whether quote ID/order ID must never be metric labels.
• Existing incident notes where missing metrics slowed triage.
• Existing runbooks for high latency, high error, DB pool exhaustion, queue lag, Redis degradation, and pod restarts.

32. Senior Engineer Mental Model

Use this mental model:

RED tells me whether the service is failing users.
USE tells me whether a resource is the bottleneck.
Golden signals tell me whether reliability is degrading.
Business metrics tell me whether the domain flow is affected.

During incident:

Start from symptom.
Check traffic.
Check errors.
Check latency.
Check saturation.
Check recent changes.
Check dependencies.
Check business impact.
Then drill into logs/traces/audit.

Do not start with random logs.

Metrics provide the map. Logs and traces provide the evidence trail.

33. Key Takeaways

• RED is the default framework for request-serving services.
• USE is the default framework for resources and saturation analysis.
• Golden signals combine latency, traffic, errors, and saturation for reliability view.
• Java/JAX-RS observability must include HTTP, JVM, thread pool, DB pool, dependency, Kubernetes, and business metrics.
• Async consumers and background jobs also need RED-like metrics.
• Metric labels are schema and cost decisions.
• High-cardinality labels are one of the fastest ways to damage metric systems.
• Senior engineers review metrics for correctness, actionability, privacy, cost, and operational usefulness.