Application Metrics for Java/JAX-RS

Application metrics adalah metric yang menjelaskan apakah aplikasi Java/JAX-RS berjalan sehat dari sudut request processing, runtime, resource usage, dan dependency interaction.

Part ini fokus pada metric di dalam service backend, bukan metric platform secara umum.

Target mental model:

Can this Java/JAX-RS service explain its own health under production traffic?

Untuk enterprise CPQ/order management system, application metrics harus membantu menjawab:

• Endpoint mana yang menerima traffic?
• Endpoint mana yang error?
• Endpoint mana yang lambat?
• Apakah latency berasal dari aplikasi, database, Redis, Kafka/RabbitMQ, Camunda, downstream service, atau Kubernetes resource pressure?
• Apakah thread pool atau connection pool saturated?
• Apakah JVM sedang mengalami GC pressure?
• Apakah deployment baru mengubah error rate, latency, atau throughput?
• Apakah business operation seperti pricing, quote approval, order validation, atau fulfillment mulai degrade?

1. Application Metrics Are Not Just Library Defaults

Banyak Java service sudah expose metric dari framework atau agent.

Contoh:

http.server.requests
jvm.memory.used
jvm.gc.pause
process.cpu.usage
hikaricp.connections.active

Itu berguna, tetapi belum cukup.

Application metrics harus menggabungkan:

• Framework-provided metrics.
• Runtime metrics.
• Resource pool metrics.
• Dependency metrics.
• Business operation metrics.
• Error classification metrics.
• Deployment/version metadata.

Jika hanya mengandalkan defaults, blind spot umum tetap muncul:

• Semua endpoint digabung menjadi satu.
• Route label tidak konsisten.
• Error taxonomy tidak jelas.
• Business failure tidak terlihat.
• Dependency latency tidak dipisah.
• Retry dan timeout tidak dihitung.
• Thread pool saturation tidak terlihat.
• Metric tidak bisa dikaitkan ke version/deployment.

2. Application Metric Layers

Pikirkan application metrics sebagai beberapa layer.

Layer 1: HTTP/JAX-RS request metrics
Layer 2: Error and exception metrics
Layer 3: Dependency metrics
Layer 4: Runtime/JVM metrics
Layer 5: Pool and concurrency metrics
Layer 6: Business operation metrics
Layer 7: Deployment/version metrics

Setiap layer menjawab pertanyaan berbeda.

3. Request Count

Request count adalah counter untuk jumlah request masuk.

Contoh metric:

http.server.requests.total

Label umum:

service
method
route
status_code
status_class
environment
version

Contoh:

http.server.requests.total{
  service="quote-api",
  method="POST",
  route="/quotes/{quoteId}/price",
  status_code="200",
  status_class="2xx",
  environment="prod",
  version="2026.07.11-1"
}

Request count harus dipakai sebagai rate:

requests per second
requests per minute
requests per route
requests per status class

Request count tanpa label route terlalu kasar.

Request count dengan raw path terlalu mahal.

4. Endpoint Template Discipline

Untuk JAX-RS, route label harus memakai endpoint template.

Baik:

/quotes/{quoteId}/price
/orders/{orderId}/submit
/accounts/{accountId}/orders

Buruk:

/quotes/Q-12345/price
/orders/O-88481/submit
/accounts/A-99821/orders

Endpoint template penting untuk:

• Aggregation benar.
• Cardinality terkendali.
• Dashboard readable.
• Alert rule stabil.
• SLO per endpoint memungkinkan.

JAX-RS instrumentation harus bisa mendapatkan matched resource path, bukan hanya raw URI.

Internal detail implementasi bisa berbeda tergantung Jersey/RESTEasy/Servlet integration. Verifikasi di codebase dan platform standard.

5. Request Latency

Request latency mengukur durasi request dari entry point sampai response selesai.

Metric:

http.server.request.duration.seconds

Gunakan histogram/timer, bukan gauge.

Label umum:

service
method
route
status_class
environment
version

Latency perlu dilihat sebagai distribusi:

p50
p90
p95
p99
max, if available and meaningful

Untuk production debugging, p95/p99 sering lebih penting daripada average.

Contoh interpretasi:

p50 stable, p99 naik

Kemungkinan:

• Sebagian request terkena dependency slow.
• Tail latency dari DB lock.
• Cache miss path lambat.
• Specific endpoint degrade.
• Thread pool queueing.
• Retry menambah delay.

p50, p95, p99 semua naik

Kemungkinan:

• Service-wide saturation.
• Downstream dependency global degrade.
• CPU throttling.
• Database pool issue.
• Deployment regression.

6. Request Latency Boundaries

Jelaskan boundary latency.

Untuk HTTP server metric, durasi bisa mencakup:

• Request accepted by servlet container.
• JAX-RS filter execution.
• Authentication/authorization processing.
• Resource method.
• Service layer.
• Database/Redis/downstream calls.
• Serialization.
• Response filter.
• Response flush.

Namun beberapa instrumentation mungkin tidak mencakup full response streaming.

Untuk streaming response atau async request, verifikasi:

• Apakah metric selesai saat resource method return?
• Apakah metric selesai saat response body benar-benar selesai dikirim?
• Apakah exception saat streaming tercatat?
• Apakah async timeout tercatat sebagai failure?

7. Error Count

Error count harus mengukur request atau operation yang gagal.

Metric:

http.server.errors.total

Atau error berasal dari request counter dengan status label:

http.server.requests.total{status_class="5xx"}

Namun untuk diagnosis, status code saja sering kurang.

Tambahkan taxonomy error yang bounded:

error_type="validation"
error_type="authentication"
error_type="authorization"
error_type="domain_rule"
error_type="dependency_timeout"
error_type="database_error"
error_type="messaging_publish_error"
error_type="unexpected"

Jangan gunakan full exception class secara bebas jika jumlahnya tidak terkendali.

Jangan gunakan exception message sebagai label.

8. Status Code Metrics

Status code penting untuk HTTP observability.

Minimal:

status_code
status_class

Interpretasi umum:

Tetapi di enterprise APIs, 4xx tidak selalu noise.

Contoh 4xx yang operationally important:

• 401 spike: auth token issue.
• 403 spike: authorization config issue.
• 404 spike: route/gateway/client version mismatch.
• 409 spike: optimistic lock/idempotency conflict.
• 422 spike: validation contract or upstream payload issue.
• 429 spike: rate limit or traffic spike.

4xx alerting harus hati-hati, tetapi dashboard harus tetap menampilkannya.

9. Active Requests

Active request gauge menunjukkan request yang sedang diproses.

Metric:

http.server.active_requests

Useful untuk mendeteksi:

• In-flight request buildup.
• Thread saturation.
• Slow dependency.
• Backpressure.
• Requests stuck.

Interpretasi:

active_requests naik
+ request rate stabil
+ latency naik
= request processing makin lambat.

active_requests turun ke nol
+ traffic biasanya ada
= upstream/gateway/client traffic berhenti atau service tidak menerima request.

Label active request harus sangat terbatas.

Biasanya cukup:

service
environment
method
route

Jika terlalu banyak label, gauge menjadi mahal dan membingungkan.

10. Payload Size Metrics

Payload size bisa penting untuk enterprise APIs.

Metric:

http.server.request.body.size.bytes
http.server.response.body.size.bytes

Gunakan histogram.

Payload size membantu mendiagnosis:

• Large quote/order payload.
• Serialization overhead.
• Unexpected client behavior.
• NGINX/gateway limit errors.
• Memory pressure.
• Network latency.
• Large response regression.

Privacy warning:

Do not log payload body as metric label.

Hanya ukur size, bukan isi.

11. Serialization and Deserialization Metrics

Untuk JAX-RS service dengan JSON payload besar, serialization/deserialization bisa menjadi bottleneck.

Potential metrics:

http.server.request.deserialize.duration.seconds
http.server.response.serialize.duration.seconds
json.parse.errors.total
json.mapping.errors.total

Metric ini tidak harus ada di semua service. Gunakan jika:

• Payload besar.
• Mapping kompleks.
• Latency p99 tidak dijelaskan DB/dependency.
• Banyak 400/422 karena payload contract.
• Service melakukan heavy DTO transformation.

12. Exception Mapper Metrics

JAX-RS ExceptionMapper adalah tempat penting untuk error classification.

Metric yang berguna:

application.exceptions.total{
  exception_category="validation",
  mapped_status="422"
}

Atau:

http.server.errors.total{
  error_type="dependency_timeout",
  status_code="504"
}

Tujuan:

• Membedakan expected vs unexpected error.
• Mengetahui error taxonomy.
• Menghubungkan exception dengan status code.
• Mengurangi dependency pada log search untuk trend error.

Anti-pattern:

exception_message as label
stacktrace_hash as label without governance
raw_exception_class with unlimited custom exceptions

13. Dependency Call Metrics

Aplikasi Java/JAX-RS jarang berdiri sendiri.

Metric outbound dependency harus menjawab:

• Dependency mana yang dipanggil?
• Berapa latency?
• Berapa error?
• Berapa timeout?
• Berapa retry?
• Apakah circuit breaker open?
• Apakah fallback dipakai?

Contoh HTTP client metrics:

http.client.requests.total
http.client.request.duration.seconds
http.client.errors.total
http.client.timeouts.total
http.client.retries.total

Label aman:

service
dependency
method
route_template_or_operation
status_class
error_type

Jangan label dengan full URL jika mengandung ID atau query parameter.

14. Database Access Metrics

Database metric di application side berbeda dari database server metric.

Application side harus melihat:

db.client.operations.total
db.client.operation.duration.seconds
db.client.errors.total
db.transaction.duration.seconds
db.rows.affected

Connection pool metrics:

db.pool.connections.active
db.pool.connections.idle
db.pool.connections.pending
db.pool.connections.max
db.pool.acquire.duration.seconds
db.pool.timeouts.total

Untuk MyBatis/JPA/JDBC, operation label harus bounded.

Baik:

operation="QuoteMapper.findById"
operation="OrderRepository.save"
operation="pricing_rule_lookup"

Berisiko:

sql="select * from quote where quote_id = 'Q-12345' ..."

SQL statement perlu sanitization jika dipakai sebagai span attribute/log, dan biasanya tidak cocok sebagai metric label.

15. Redis Metrics from Application Perspective

Redis server dashboard penting, tetapi application juga perlu Redis client metrics.

Metric:

redis.client.commands.total
redis.client.command.duration.seconds
redis.client.errors.total
redis.client.timeouts.total
cache.requests.total
cache.hit.total
cache.miss.total

Label aman:

service
redis_operation
cache_name
outcome
error_type

Hindari:

redis_key
user_id
session_id
quote_id
order_id

Redis key sering mengandung sensitive/business identifiers dan high cardinality.

16. Kafka Producer Metrics

Jika JAX-RS request menghasilkan event Kafka, service harus punya producer metrics.

Metric:

kafka.producer.records.sent.total
kafka.producer.publish.duration.seconds
kafka.producer.errors.total
kafka.producer.retries.total

Label aman:

topic
message_type
outcome
error_type

Important production questions:

• Apakah request sukses tetapi event publish gagal?
• Apakah publish latency membuat HTTP request lambat?
• Apakah retry producer meningkat?
• Apakah topic tertentu gagal?
• Apakah idempotency/outbox pattern diperlukan?

Untuk mission-critical order system, event publish failure bisa menjadi data consistency risk, bukan hanya technical error.

17. Kafka Consumer Metrics

Jika service mengonsumsi Kafka:

kafka.consumer.records.consumed.total
kafka.consumer.processing.duration.seconds
kafka.consumer.errors.total
kafka.consumer.retries.total
kafka.consumer.dlq.total
kafka.consumer.lag
kafka.consumer.event.age.seconds

Event age penting:

event age = now - event created timestamp

Processing duration hanya mengukur waktu setelah consume dimulai.

Event age mengukur backlog/freshness dari perspektif business flow.

Untuk order fulfillment, event age sering lebih relevan daripada pure processing duration.

18. RabbitMQ Metrics

Application metrics untuk RabbitMQ:

rabbitmq.publisher.messages.published.total
rabbitmq.publisher.publish.duration.seconds
rabbitmq.publisher.errors.total
rabbitmq.consumer.messages.consumed.total
rabbitmq.consumer.processing.duration.seconds
rabbitmq.consumer.errors.total
rabbitmq.consumer.redeliveries.total
rabbitmq.consumer.acks.total
rabbitmq.consumer.nacks.total
rabbitmq.consumer.dlq.total

Application metrics harus melengkapi broker metrics seperti queue depth dan unacked messages.

Kombinasi penting:

consumer.processing.duration naik
+ queue depth naik
+ unacked naik
= consumer bottleneck.

redelivery naik
+ DLQ naik
= poison message atau retry loop.

19. Camunda/Workflow Worker Metrics

Jika service menjalankan worker atau external task:

workflow.worker.jobs.acquired.total
workflow.worker.jobs.completed.total
workflow.worker.jobs.failed.total
workflow.worker.job.duration.seconds
workflow.worker.retries.total
workflow.worker.lock.failures.total
workflow.incidents.total
workflow.task.aging.seconds

Business questions:

• Apakah worker mengambil job?
• Apakah job selesai?
• Apakah failed job meningkat?
• Apakah incident bertambah?
• Apakah human task aging?
• Apakah process instance stuck?

Workflow metrics harus dikaitkan dengan lifecycle state dan business SLA.

20. Thread Pool Metrics

Thread pool usage adalah salah satu application metrics paling penting.

Metric:

executor.pool.size
executor.active.threads
executor.queue.size
executor.completed.tasks.total
executor.rejected.tasks.total
executor.task.duration.seconds

Untuk server thread pool, executor worker, scheduler, async job, dan consumer worker, berikan label bounded:

executor_name="http-worker"
executor_name="pricing-worker"
executor_name="order-consumer"
executor_name="reconciliation-job"

Failure mode:

• Active threads selalu mendekati max.
• Queue size naik terus.
• Rejected task naik.
• Task duration naik.
• Thread blocked karena dependency.

Thread pool metrics harus dikorelasikan dengan traces/thread dumps saat incident berat.

21. Queue Size Metrics Inside Application

Selain Kafka/RabbitMQ, aplikasi bisa punya internal queue.

Contoh:

• Executor queue.
• In-memory buffer.
• Batch queue.
• Retry queue.
• Local work queue.
• Async dispatch queue.

Metric:

application.queue.size
application.queue.enqueue.total
application.queue.dequeue.total
application.queue.oldest.item.age.seconds
application.queue.dropped.total

Queue size saja tidak cukup.

Oldest item age sering lebih meaningful untuk user/business impact.

queue size 1,000

Belum tentu buruk jika throughput tinggi.

oldest item age 45 minutes

Lebih jelas menunjukkan freshness/SLA risk.

22. JVM Memory Metrics in Application Dashboard

Walaupun Part 019 akan membahas JVM metrics lebih dalam, application dashboard minimal perlu memory signal.

Metric:

jvm.memory.used.bytes
jvm.memory.committed.bytes
jvm.memory.max.bytes
jvm.memory.usage.after.gc
jvm.buffer.memory.used.bytes

Breakdown:

heap
non_heap
metaspace
direct_buffer

Interpretasi dasar:

heap used naik terus setelah GC
= possible memory leak or cache growth.

direct memory naik
= possible NIO/client/buffer issue.

memory usage dekat container limit
= OOMKilled risk.

23. GC Pause Metrics

GC pause mempengaruhi latency.

Metric:

jvm.gc.pause.duration.seconds
jvm.gc.pause.total
jvm.gc.memory.allocated.bytes
jvm.gc.memory.promoted.bytes

Useful dashboard views:

• GC pause p95/p99.
• Total GC time per minute.
• Allocation rate.
• Heap after GC.
• Correlation with HTTP p99.

GC tidak otomatis masalah.

GC menjadi masalah jika:

• Pause meningkat.
• Request latency meningkat.
• Timeouts meningkat.
• CPU naik.
• Heap tidak turun.
• Pod restart karena OOMKilled.

24. CPU Metrics

CPU metric di containerized Java harus dibaca hati-hati.

Metric:

process.cpu.usage
system.cpu.usage
container.cpu.usage
container.cpu.throttled.periods.total
container.cpu.throttled.time.seconds

CPU throttling sangat penting di Kubernetes.

Service bisa terlihat memakai CPU sedang, tetapi throttling menyebabkan latency spike.

Contoh:

HTTP p99 naik
+ CPU usage 60%
+ CPU throttling naik
= container CPU limit terlalu ketat atau burst traffic.

Jangan hanya lihat CPU average.

25. File Descriptor and Socket Metrics

File descriptor sering dilupakan.

Metric:

process.open.file_descriptors
process.max.file_descriptors
network.connections.active
network.connections.errors.total

Failure mode:

• Too many open files.
• Socket leak.
• HTTP client connection leak.
• DB connection leak.
• Log file handler leak.
• High inbound/outbound connection churn.

Symptoms:

• Random connection failures.
• Dependency call failures.
• Accept failure.
• Service health unstable.

26. Connection Pool Metrics

Connection pool metrics tidak hanya untuk database.

Types:

• DB pool.
• HTTP client pool.
• Redis client pool.
• Messaging channel/connection pool.

Common metrics:

pool.connections.active
pool.connections.idle
pool.connections.pending
pool.connections.max
pool.acquire.duration.seconds
pool.acquire.timeout.total

Questions:

• Apakah pool terlalu kecil?
• Apakah connection leak?
• Apakah dependency lambat sehingga connection tertahan?
• Apakah retry memperbanyak pressure?
• Apakah pool timeout menyebabkan user error?

Pool saturation sering menjadi root cause latency.

27. Version and Deployment Labels

Application metrics harus bisa dikaitkan dengan deployment.

Useful labels/resource attributes:

service.name
service.version
deployment.environment
k8s.namespace.name
k8s.pod.name
container.image.name
container.image.tag
git.commit.sha
build.id

Namun hati-hati: label seperti pod name bisa meningkatkan cardinality, tapi sering dibutuhkan untuk debugging platform. Biasanya pod-level labels lebih cocok di infrastructure metrics, sementara service-level dashboard agregasi harus default.

Deployment marker juga penting:

release.deployment.timestamp
release.version
release.commit_sha

Dengan deployment marker, engineer bisa melihat:

Error rate naik 3 menit setelah version X deployed.

28. Business Operation Metrics

Application metrics harus mencakup business operation yang critical.

Contoh CPQ:

quote.pricing.requests.total
quote.pricing.duration.seconds
quote.pricing.errors.total
quote.approval.submissions.total
quote.approval.duration.seconds

Contoh order management:

order.submissions.total
order.validation.duration.seconds
order.fulfillment.started.total
order.fulfillment.failed.total
order.fallout.created.total
order.lifecycle.transition.total
order.lifecycle.transition.duration.seconds

Business metrics harus bounded dan privacy-aware.

Label aman:

operation
outcome
error_category
channel
product_family_if_low_cardinality

Label berisiko:

quote_id
order_id
customer_id
account_id
product_id_if_high_cardinality

29. Correctness Metrics

Tidak semua problem terlihat sebagai 5xx.

Correctness metrics mendeteksi system menghasilkan hasil yang salah, tidak lengkap, stale, duplicate, atau inconsistent.

Contoh:

order.duplicate_event.detected.total
order.invalid_state_transition.total
quote.pricing.incomplete_result.total
reconciliation.mismatch.total
idempotency.duplicate_request.total
workflow.message_correlation.failure.total

Correctness metrics sangat penting di CPQ/order systems karena business failure bisa terjadi walaupun HTTP status 200.

Contoh:

API returns 200, but fulfillment event not published.

Itu bukan HTTP error biasa. Itu consistency failure.

30. Rate Limit and Backpressure Metrics

Enterprise APIs sering punya rate limiting/backpressure.

Metric:

rate_limiter.requests.total
rate_limiter.rejected.total
rate_limiter.tokens.available
backpressure.rejections.total
backpressure.queue.size

Useful questions:

• Apakah client mengirim traffic berlebihan?
• Apakah rejection normal atau regression?
• Apakah rate limit melindungi dependency?
• Apakah backpressure menyebabkan business flow delay?

HTTP 429 harus ditampilkan di dashboard, tetapi alert-nya tergantung policy.

31. Retry Metrics

Retry bisa membantu resilience, tetapi juga bisa memperparah incident.

Metric:

retry.attempts.total
retry.exhausted.total
retry.duration.seconds

Label:

dependency
operation
error_type
outcome

Important patterns:

retry attempts naik
+ dependency latency naik
= dependency degradation.

retry attempts naik
+ request latency p99 naik
= retry adds user-visible delay.

retry exhausted naik
= resilience policy no longer masking failure.

Retry metrics harus dikorelasikan dengan circuit breaker metrics.

32. Circuit Breaker Metrics

Jika service memakai circuit breaker:

circuitbreaker.state
circuitbreaker.calls.total
circuitbreaker.failures.total
circuitbreaker.slow.calls.total
circuitbreaker.not_permitted.calls.total

Labels:

circuitbreaker_name
dependency
state
outcome

Questions:

• Apakah circuit open?
• Dependency mana yang dilindungi?
• Apakah fallback dipakai?
• Apakah not permitted calls menyebabkan user impact?
• Apakah threshold terlalu sensitif?

Circuit breaker open sebaiknya terlihat di dependency health dashboard.

33. Timeout Metrics

Timeout adalah signal penting.

Metric:

operation.timeouts.total
http.client.timeouts.total
db.pool.acquire.timeouts.total
redis.timeouts.total
messaging.publish.timeouts.total

Timeout harus diklasifikasikan:

timeout_type="connect"
timeout_type="read"
timeout_type="write"
timeout_type="pool_acquire"
timeout_type="lock_wait"
timeout_type="processing"

Timeout metric menjawab:

• Apakah dependency unreachable?
• Apakah dependency lambat?
• Apakah pool exhausted?
• Apakah lock contention?
• Apakah network issue?
• Apakah timeout value terlalu rendah/tinggi?

34. Application Metric Naming

Gunakan naming yang konsisten.

Prinsip:

• Nama menjelaskan domain dan measurement.
• Unit jelas.
• Type tidak ambigu.
• Labels menjelaskan dimension rendah cardinality.

Contoh:

http.server.requests.total
http.server.request.duration.seconds
db.pool.acquire.duration.seconds
quote.pricing.duration.seconds
order.lifecycle.transitions.total
workflow.worker.job.duration.seconds

Hindari:

pricing_metric
order_time
error_count
latency
process_data

Nama metric adalah API jangka panjang.

Mengubah nama metric bisa merusak dashboard, alert, SLO, dan runbook.

35. Metric Cardinality in Application Metrics

Application metrics paling rentan cardinality explosion karena engineer tergoda memasukkan business context.

Disallowed by default:

request_id
trace_id
span_id
user_id
actor_id
quote_id
order_id
account_id
customer_id
session_id
email
raw_path
raw_query
exception_message
sql_statement
redis_key
message_id

Potentially allowed only with governance:

tenant_id
region
channel
product_family
pricing_mode
order_type
workflow_type

Rule of thumb:

If the value can grow with users, requests, orders, quotes, or messages, it is probably not a safe metric label.

Use logs/traces/audit for high-cardinality lookup.

Use metrics for aggregate trend.

36. Metric Aggregation Correctness

Aggregation can lie.

Examples:

Average latency across all endpoints hides one slow endpoint.

Error rate across all tenants hides one tenant outage.

Summing gauges across pods may be meaningful for active requests, but not for ratios.

Averaging percentiles across pods is mathematically unsafe in many backends.

Prefer histograms that can aggregate correctly across instances.

Review whether dashboard query is semantically correct.

37. Application Metrics and Logs

Metrics show what changed.

Logs explain individual events.

Good metric-to-log relationship:

Metric: http.server.errors.total{error_type="dependency_timeout"} spikes.
Log: structured error log contains request_id, trace_id, dependency, timeout_ms, route, business_key.
Trace: shows downstream call duration and span status.

Metric alone should not need high-cardinality label.

Instead:

• Use metric to detect trend.
• Use trace/log to inspect examples.
• Use business/audit data to determine impact.

38. Application Metrics and Traces

Traces explain duration breakdown.

Metric says:

POST /quotes/{quoteId}/price p99 latency increased.

Trace should show:

JAX-RS resource span
service layer span
PostgreSQL query span
Redis cache span
pricing dependency HTTP span
Kafka publish span

Application metric design should align with span operation names.

If metric route is /quotes/{quoteId}/price, trace span should also show route template, not raw path only.

39. Dashboard Minimum for Application Metrics

A useful application dashboard should show:

• Request rate by route.
• Error rate by route/status/error type.
• Latency p50/p95/p99 by route.
• Active requests.
• JVM heap and GC pause.
• CPU and CPU throttling.
• Thread pool active/queue/rejected.
• DB pool active/pending/acquire duration.
• Dependency latency/error/timeout/retry.
• Kafka/RabbitMQ lag/queue depth if applicable.
• Redis latency/hit/miss if applicable.
• Pod restart/OOMKilled/probe failure.
• Deployment version/marker.
• Business critical operation metrics.

Dashboard should start broad, then allow drilldown.

40. Alert Candidates from Application Metrics

Not every metric needs alert.

Good alert candidates:

• High error rate for user-facing endpoint.
• SLO burn for availability/latency.
• p99 latency above threshold for sustained window.
• DB pool acquire timeout.
• Thread pool rejected task.
• Consumer lag beyond freshness SLO.
• DLQ growth.
• Pod restart loop.
• OOMKilled.
• Circuit breaker open for critical dependency.
• Business correctness failure spike.

Bad paging candidates by default:

• CPU high without user impact.
• Memory high without OOM/GC/latency signal.
• One-off warning count.
• Debug metric anomaly with no action.

Alert should be symptom-first, diagnostic metrics should support triage.

41. Failure Mode Mapping

42. Correctness Concerns

Application metrics can be wrong.

Common problems:

• Request counted before authentication only, but dashboard assumes business request.
• ExceptionMapper returns 200/4xx but metric does not count business failure.
• Retry attempts counted as separate user requests.
• Async event success counted before transaction commits.
• Kafka publish success counted before broker ack.
• DB operation duration excludes pool wait.
• Latency metric ends before streaming response completes.
• Route template missing for unmatched exception path.
• Version label missing after deployment.
• Metric emitted per item with unbounded labels.

Metric correctness must be tested and reviewed.

43. Performance Concerns

Application metric instrumentation must be lightweight.

Risks:

• Metrics emitted in extremely hot loops.
• Label values computed from expensive serialization.
• Dynamic metric names per business entity.
• Too many histograms.
• Synchronous export blocking request threads.
• Metric lock contention.
• Large batch emits per-row metrics.

Guidance:

• Use bounded labels.
• Use aggregate counters.
• Avoid per-entity metric names.
• Use histograms only where distribution matters.
• Batch exporter properly.
• Test under load.

44. Security and Privacy Concerns

Application metrics must not leak sensitive data.

Unsafe labels:

email
phone
customer_name
account_number
quote_id
order_id
token
api_key
session_id
authorization_header
cookie
raw_query
raw_url
sql_statement_with_literals
redis_key
message_payload

Even if backend access is restricted, telemetry should follow least exposure.

For audit/compliance needs, use audit log with access control and retention policy, not arbitrary metric labels.

45. Cost Concerns

Every label multiplies time series.

High-cardinality application metrics can be extremely expensive because they are emitted at high request volume.

Before adding a label, ask:

• How many possible values exist?
• Does value count grow with users/orders/quotes/messages?
• Is it needed for dashboard aggregation?
• Is it needed for alert routing?
• Can the same question be answered with logs/traces?
• What is retention requirement?

Metric cost is architecture concern, not platform-only concern.

46. Internal Verification Checklist

For CSG/team context, verify rather than assume:

• Standard Java metrics library used internally.
• Whether Micrometer, OpenTelemetry metrics, MicroProfile Metrics, vendor agent, or custom wrappers are used.
• Whether JAX-RS route template is available in metric labels.
• Standard HTTP server metric names.
• Standard HTTP client metric names.
• Standard labels/resource attributes.
• Whether service version/build/commit is attached to metrics.
• Whether endpoint dashboard already exists.
• Whether request duration includes full response lifecycle.
• Whether async request and streaming response are measured correctly.
• Whether ExceptionMapper updates error classification.
• Whether mapped domain errors appear in metrics.
• Whether DB pool metrics are exposed.
• Whether MyBatis/JPA/JDBC operation metrics exist.
• Whether Redis/Kafka/RabbitMQ/Camunda client metrics exist.
• Whether thread pool/executor metrics are exposed.
• Whether CPU throttling and pod restart are visible near app dashboard.
• Whether tenant/channel/product labels are allowed.
• Whether quote/order/customer IDs are disallowed as metric labels.
• Whether metric cardinality reports are available.
• Whether application metrics feed dashboards, alerts, or SLOs.
• Whether incidents revealed missing application metrics.

47. Application Metric Review Checklist

Use this checklist when reviewing PRs or architecture changes:

• Does the service expose request count by method, route, status?
• Does it expose request latency as histogram/timer?
• Are route labels templated?
• Are raw paths excluded?
• Are request/user/order/quote IDs excluded from labels?
• Are error types bounded and useful?
• Are expected and unexpected errors distinguishable?
• Are dependency calls measured separately?
• Are timeout/retry/circuit breaker metrics available?
• Are DB pool metrics exposed?
• Are executor/thread pool metrics exposed?
• Are JVM memory/GC/CPU metrics available?
• Are Kubernetes resource signals visible in dashboard?
• Are business critical operations measured?
• Are correctness failures measured?
• Are metrics connected to dashboards/alerts/SLOs?
• Is cardinality controlled?
• Is privacy protected?
• Is cost acceptable?
• Are metric semantics documented?

48. Senior Engineer Mental Model

Think of application metrics as the service's self-reporting contract.

A production-grade Java/JAX-RS service should be able to say:

How much work am I receiving?
How fast am I doing it?
How often am I failing?
What kind of failures are happening?
Which dependencies are involved?
Am I saturated internally?
Is my runtime healthy?
Did the last deployment change behavior?
Are business operations completing correctly?

If the service cannot answer those questions through metrics, incident response will depend too much on log archaeology and individual engineer memory.

49. Key Takeaways

• Application metrics are the operational contract of a Java/JAX-RS service.
• HTTP request metrics must include count, latency, status, error classification, and route template.
• Dependency metrics must separate database, Redis, HTTP, Kafka, RabbitMQ, and workflow behavior.
• Thread pool, connection pool, JVM, GC, CPU, and file descriptor metrics explain saturation.
• Business operation metrics reveal domain impact invisible to HTTP status alone.
• Correctness metrics matter in CPQ/order systems because success responses can still hide business inconsistency.
• Metric labels must be bounded, privacy-safe, and cost-aware.
• Metrics detect the shape of failure; logs, traces, and audit provide event-level evidence.