Cheatsheet Observability Part 031 — HTTP Client Instrumentation

Fokus part ini: memahami bagaimana outbound HTTP call dari service Java/JAX-RS/Jakarta REST harus diinstrumentasi agar dependency latency, error, timeout, retry, connection pool, DNS/TLS, circuit breaker, dan propagation context terlihat jelas di production. Tujuannya bukan hanya tahu bahwa request gagal, tetapi tahu downstream mana yang gagal, tahap mana yang lambat, apakah retry memperparah beban, apakah circuit breaker melindungi service, dan apakah trace/log/metric cukup untuk incident debugging.

1. Core Mental Model

HTTP client instrumentation adalah observability di sisi caller ketika service memanggil service lain.

Dalam request lifecycle Java/JAX-RS, outbound HTTP call biasanya terjadi setelah resource method masuk ke service layer:

Client / Gateway
  ↓
JAX-RS resource method
  ↓
Service layer
  ↓
HTTP client call to downstream service
  ↓
Downstream service / API gateway / external API

Tanpa instrumentation yang benar, production incident sering berhenti di kalimat:

Endpoint lambat karena downstream lambat.

Itu belum cukup.

Instrumentation yang baik harus menjawab:

• downstream service apa yang dipanggil;
• endpoint downstream apa yang dipanggil;
• request gagal di connect, TLS, request write, server processing, response read, timeout, atau response mapping;
• status code downstream apa;
• berapa latency total call;
• berapa latency per attempt jika ada retry;
• apakah request memakai connection pool atau membuat connection baru;
• apakah connection pool exhausted;
• apakah circuit breaker open/half-open/closed;
• apakah context trace/correlation diteruskan;
• apakah timeout budget sesuai dengan upstream SLO;
• apakah log/metric/trace aman dari PII, token, dan payload leakage.

Rule utama:

Treat every outbound HTTP call as a dependency boundary, failure boundary, latency boundary, and cost boundary.

2. Why HTTP Client Observability Matters

Outbound HTTP dependency adalah salah satu sumber incident paling umum di microservices.

Contoh failure:

• downstream service down;
• API gateway overload;
• DNS resolution lambat;
• TLS handshake lambat;
• connection pool habis;
• timeout terlalu panjang;
• retry storm;
• circuit breaker tidak aktif;
• downstream mengembalikan 429/500/502/503/504;
• response body besar;
• schema response berubah;
• authentication token expired;
• request body invalid;
• network policy memblokir traffic;
• Kubernetes service endpoint kosong;
• cloud load balancer mengalami degradation;
• on-prem network route bermasalah.

Tanpa outbound instrumentation, symptom yang terlihat hanya:

Our API latency p95 naik.
Our error rate naik.
Thread pool penuh.

Dengan outbound instrumentation, kita bisa melihat:

POST /quotes/{id}/submit p95 naik karena 78% latency di downstream pricing-service POST /price/calculate.
Retries naik 5x setelah pricing-service mulai return 503.
Circuit breaker belum open karena failure predicate tidak memasukkan timeout exception.

3. Caller View vs Callee View

HTTP dependency harus dilihat dari dua sisi.

Saat incident, jangan hanya cek dashboard service sendiri. Cek juga:

• dependency dashboard;
• downstream service dashboard;
• edge logs;
• Kubernetes/network events;
• cloud service health;
• deployment marker kedua service.

4. Supported HTTP Client Libraries

Di Java enterprise system, outbound HTTP client bisa memakai banyak library.

Common examples:

• Jersey Client;
• Jakarta REST Client;
• OpenFeign;
• Retrofit;
• Java 11+ java.net.http.HttpClient;
• Apache HttpClient;
• OkHttp;
• Spring WebClient/RestTemplate jika ada hybrid stack;
• vendor SDK yang memakai HTTP client internal.

Internal stack harus diverifikasi. Jangan mengasumsikan satu library.

Internal verification checklist

Cek di codebase:

• dependency pom.xml / build.gradle;
• wrapper HTTP client internal;
• generated client;
• SDK client;
• Feign/Jersey/Retrofit usage;
• shared client factory;
• timeout config;
• retry config;
• connection pool config;
• TLS config;
• proxy config;
• tracing auto-instrumentation coverage;
• custom filters/interceptors.

5. Minimum Signals for Outbound HTTP Calls

Setiap outbound HTTP call production-grade minimal punya:

Trace

• client span;
• downstream host/service name;
• HTTP method;
• route/template jika tersedia;
• status code;
• error status;
• exception type;
• retry attempt if applicable;
• duration;
• propagated trace context.

Metrics

• request count by downstream, method, route, status class;
• latency histogram;
• error count;
• timeout count;
• retry count;
• circuit breaker state;
• connection pool usage;
• active/pending calls;
• request/response size if useful and safe.

Logs

• dependency call failure;
• timeout event;
• retry exhausted;
• circuit breaker opened;
• authentication/authorization failure;
• unexpected response mapping failure;
• sanitized downstream error code.

Do not log every successful dependency call at INFO if volume is high. Use metrics/traces for success path, logs for meaningful events.

6. Span Design for HTTP Client Calls

Recommended client span name:

HTTP {method} {downstream_route_template}

Example:

HTTP POST /pricing/calculate

If route template is not available, use a stable logical operation name:

pricing-service.calculate-price
inventory-service.reserve-resource
customer-service.get-account

Avoid span names like:

HTTP GET /customers/123456789/contracts/987654321

Why:

• high-cardinality;
• leaks business identifiers;
• makes aggregation poor;
• increases backend storage/index cost.

Good attributes:

http.request.method=POST
http.response.status_code=503
server.address=pricing-service
server.port=443
url.scheme=https
service.dependency.name=pricing-service
service.dependency.operation=calculatePrice
net.peer.name=pricing-service.default.svc.cluster.local
retry.attempt=2
circuit_breaker.name=pricingClient

Be careful with:

url.full
http.request.header.authorization
http.request.body
http.response.body
customer.id
order.id
quote.id

Business IDs should usually be logs/audit fields with access control, not metric labels or raw span names.

7. Context Propagation

Every outbound HTTP call should propagate trace context.

Common headers:

traceparent
tracestate
baggage
x-correlation-id
x-request-id

The exact internal convention must be verified.

Propagation rule:

Inbound context -> MDC/request context -> outbound HTTP headers -> downstream server span

Failure modes:

Internal verification checklist

Cek:

• inbound trace extraction;
• outbound trace injection;
• correlation ID standard;
• trusted boundary policy;
• baggage allowlist;
• HTTP client interceptor/filter;
• propagation tests;
• trace continuity across real service call.

8. Timeout Observability

Timeout harus terlihat sebagai signal, bukan hanya exception.

Common timeout types:

Production anti-pattern:

Semua timeout dilog sebagai "Downstream error" tanpa type.

Better:

{
  "event": "downstream_http_timeout",
  "dependency": "pricing-service",
  "operation": "calculatePrice",
  "timeout_type": "read_timeout",
  "timeout_ms": 1500,
  "attempt": 1,
  "trace_id": "...",
  "correlation_id": "..."
}

Timeout budget must align with upstream latency objective.

Example:

Endpoint p95 target: 800 ms
Pricing call timeout: 5000 ms

This is inconsistent. One slow dependency call can violate the endpoint SLO before timeout triggers.

9. Retry Observability

Retry can improve resilience or destroy the system.

Observe:

• retry count;
• retry attempt latency;
• retry reason;
• retry exhausted count;
• retry success after failure;
• downstream status code that triggers retry;
• exception type that triggers retry;
• total elapsed time including retry;
• jitter/backoff behavior.

Important distinction:

Single attempt latency != user-perceived latency when retry is enabled.

If one request performs 3 attempts:

attempt 1: 800 ms timeout
attempt 2: 800 ms timeout
attempt 3: 200 ms success

Dependency span model should make this visible. Options:

• one parent span for logical dependency call;
• child spans per attempt;
• retry attempt attributes/events.

Do not hide retries behind a single success metric. A call that succeeds after 3 attempts is still a degradation signal.

10. Circuit Breaker Observability

Circuit breaker is only useful if its state and decisions are visible.

Observe:

• circuit breaker state: closed/open/half-open;
• calls permitted;
• calls rejected;
• failure rate;
• slow call rate;
• state transition count;
• fallback count;
• recovery time;
• exception predicate;
• downstream dependency name.

Useful log events:

circuit_breaker_opened
circuit_breaker_half_opened
circuit_breaker_closed
circuit_breaker_call_rejected
fallback_executed

Failure modes:

11. Connection Pool Observability

HTTP client connection pool issues often look like downstream latency.

Observe:

• active connections;
• idle connections;
• max connections;
• pending acquisition count;
• pool acquisition latency;
• pool acquisition timeout;
• connection reuse rate;
• connection creation rate;
• connection eviction;
• keep-alive errors.

Symptoms of pool exhaustion:

Request latency rises before downstream server latency rises.
Thread pool starts blocking.
Pool acquisition timeout appears.
CPU may remain normal.
Downstream access logs may not show matching traffic increase.

This is why caller-side metrics matter.

12. DNS and TLS Latency Awareness

Most application traces do not break down DNS/TLS by default.

Still, engineers should know these can dominate latency:

• DNS lookup delay;
• DNS cache miss;
• Kubernetes DNS/CoreDNS issue;
• TLS handshake cost;
• certificate validation issue;
• expired certificate;
• proxy/TLS termination problem;
• cloud private endpoint route issue.

When suspecting DNS/TLS:

• compare caller latency vs downstream server latency;
• check connection reuse;
• inspect client connection pool;
• inspect ingress/load balancer metrics;
• check CoreDNS or platform DNS metrics;
• check TLS/certificate errors;
• check recent network policy or certificate changes.

13. Metrics Design for HTTP Clients

Recommended metric set:

http_client_request_duration_seconds
http_client_requests_total
http_client_errors_total
http_client_timeouts_total
http_client_retries_total
http_client_retry_exhausted_total
http_client_active_requests
http_client_connection_pool_active
http_client_connection_pool_idle
http_client_connection_pool_pending
http_client_circuit_breaker_state
http_client_circuit_breaker_rejected_total

Recommended labels:

service
environment
downstream_service
operation
method
status_code or status_class
error_type

Avoid labels:

url.full
raw_path
query_string
request_id
correlation_id
user_id
order_id
quote_id
exception_message

status_code can be acceptable if bounded. status_class is cheaper and often enough for high-level dashboards.

14. Logging Strategy for HTTP Client Calls

Do log:

• timeout;
• retry exhausted;
• circuit breaker open/reject;
• unexpected 5xx;
• unexpected 4xx from dependency;
• authentication failure;
• schema/response mapping failure;
• dependency unavailable;
• fallback used for important business flow.

Avoid logging:

• every successful call at INFO in high-volume path;
• full request/response bodies;
• Authorization header;
• cookies;
• API keys;
• access tokens;
• full URL with sensitive query params;
• raw external error body without sanitization.

Good dependency failure log:

{
  "event": "downstream_http_failure",
  "dependency": "pricing-service",
  "operation": "calculatePrice",
  "method": "POST",
  "route": "/pricing/calculate",
  "status_code": 503,
  "attempt": 2,
  "retry_exhausted": false,
  "latency_ms": 742,
  "error_type": "downstream_unavailable",
  "trace_id": "...",
  "correlation_id": "...",
  "quote_id": "Q-..."
}

Only include quote_id or order_id if allowed by internal logging policy.

15. HTTP Status Code Semantics

Caller instrumentation must classify response correctly.

Do not classify all 4xx as harmless. In service-to-service communication, many 4xx responses indicate contract, data, auth, or state-management bugs.

16. Dashboard Design for HTTP Dependencies

A useful dependency dashboard shows:

• dependency request rate;
• dependency error rate;
• dependency latency p50/p95/p99;
• timeout rate;
• retry rate;
• retry exhausted rate;
• circuit breaker state;
• rejected calls;
• connection pool active/idle/pending;
• status code distribution;
• top slow downstream operations;
• comparison with service endpoint latency;
• deployment markers;
• downstream service health link.

Key question:

Is our service unhealthy because our code is slow, or because a dependency is slow/unavailable?

17. Alerting for HTTP Dependencies

Good alerts are symptom-oriented and actionable.

Possible alerts:

• high timeout rate to critical dependency;
• high 5xx/503/504 rate from dependency;
• dependency latency SLO burn;
• circuit breaker open for critical dependency;
• retry exhausted spike;
• connection pool pending acquisition high;
• auth failure spike;
• rate limit 429 spike.

Bad alerts:

• alert on one isolated 500;
• alert on every circuit breaker state change without severity;
• alert on all 4xx without classifying expected domain errors;
• alert with no downstream owner/runbook/dashboard link.

Alert message should include:

service
critical dependency
operation
symptom
start time
severity
impact hint
dashboard link
runbook link
recent deployment info

18. Production Debugging Playbook

When API latency/error rises and HTTP dependency is suspected:

1. Check service RED metrics.
2. Identify impacted endpoint template.
3. Open traces for slow/error requests.
4. Find dependency span dominating latency.
5. Check dependency metrics by downstream service/operation.
6. Compare caller-side latency with downstream server-side latency.
7. Check timeout/retry/circuit breaker metrics.
8. Check connection pool pending/active metrics.
9. Check status code distribution.
10. Check edge/gateway logs if 502/503/504 appears.
11. Check recent deployment/config/certificate/network changes.
12. Determine customer/business impact.
13. Mitigate: disable feature, reduce traffic, open circuit, increase capacity, rollback, or coordinate with dependency owner.

Important diagnostic distinction:

Caller sees timeout, but downstream sees no request

Possible causes:

• DNS issue;
• connection pool exhaustion;
• network policy;
• TLS handshake issue;
• gateway issue before downstream;
• client-side timeout before request sent.

Caller sees 504, downstream completed successfully

Possible causes:

• gateway timeout;
• response path issue;
• timeout mismatch;
• client cancelled;
• load balancer idle timeout.

19. Implementation Pattern: Interceptor/Filter

A shared HTTP client wrapper or interceptor is preferred over ad hoc instrumentation per call.

Pseudo pattern:

public final class InstrumentedHttpClient {
  public HttpResponse execute(HttpRequest request, DependencyContext dependency) {
    long start = clock.nanoTime();

    try (Scope scope = tracing.startClientSpan(dependency.operation())) {
      propagation.injectCurrentContext(request.headers());
      metrics.activeRequests().increment();

      HttpResponse response = delegate.execute(request);

      metrics.recordDuration(dependency, response.statusCode(), elapsed(start));
      tracing.setHttpStatus(response.statusCode());

      if (isUnexpected(response)) {
        log.warn("downstream_http_unexpected_response", fields(dependency, response));
      }

      return response;
    } catch (TimeoutException e) {
      metrics.timeout(dependency, timeoutType(e)).increment();
      log.warn("downstream_http_timeout", fields(dependency, e));
      tracing.recordException(e);
      throw e;
    } catch (Exception e) {
      metrics.error(dependency, classify(e)).increment();
      log.error("downstream_http_failure", fields(dependency, e));
      tracing.recordException(e);
      throw e;
    } finally {
      metrics.activeRequests().decrement();
    }
  }
}

The real implementation depends on internal stack.

For Jersey Client, this may be a ClientRequestFilter and ClientResponseFilter. For Feign, it may be RequestInterceptor, Client, ErrorDecoder, and metrics capability. For Java HttpClient, it may require wrapper code around send/sendAsync.

20. Privacy and Security Concerns

Outbound calls often contain sensitive data.

Never expose blindly:

• Authorization;
• Cookie;
• Set-Cookie;
• API keys;
• bearer tokens;
• session IDs;
• customer personal data;
• payment/commercial details;
• full query string;
• request/response body;
• raw error body from external system.

Safer alternatives:

• log operation name instead of full URL;
• log route template instead of raw path;
• log status class instead of full error body;
• log sanitized error code;
• log payload size instead of payload content;
• log hash only if policy allows and use case is clear;
• store audit/compliance evidence separately with access control.

21. Failure Mode Table

22. PR Review Checklist

Ask these questions in PR review:

• Does the outbound call have timeout?
• Is timeout aligned with caller SLO?
• Is retry bounded and backoff/jitter configured?
• Are retryable errors explicit?
• Does it emit dependency metrics?
• Does it create or continue client spans?
• Does it propagate trace/correlation context?
• Are route/operation labels low-cardinality?
• Are secrets/headers/body protected?
• Is circuit breaker needed?
• Is fallback behavior observable?
• Is connection pool configured and monitored?
• Is downstream status/error classification correct?
• Does dashboard/alert/runbook need update?
• Does the change affect dependency ownership or incident response?

23. Internal Verification Checklist

Verify in internal CSG/team context:

• HTTP client libraries used per service;
• shared client abstraction or wrapper;
• OpenTelemetry auto-instrumentation coverage;
• custom client interceptors/filters;
• standard downstream dependency labels;
• route template availability;
• timeout defaults;
• retry defaults;
• circuit breaker library and metrics;
• connection pool metrics;
• propagation headers;
• baggage policy;
• privacy policy for headers/body/query parameters;
• dashboard for dependency health;
• alert rules for critical dependencies;
• runbooks for timeout/503/504/retry storm;
• incident examples involving downstream HTTP calls.

24. Senior Engineer Heuristics

Use these heuristics:

• If a service calls another service, the dependency must be visible as a first-class signal.
• Timeout without retry may fail fast; retry without budget may amplify incident.
• Circuit breaker without dashboard is operationally ambiguous.
• Dependency metric without route/operation grouping is too coarse.
• Dependency metric with raw URL label is too expensive and unsafe.
• Trace without logs may show shape but not business meaning.
• Logs without trace may show event but not causal path.
• Caller latency and callee latency must be compared before blaming downstream.

25. Summary

HTTP client instrumentation is the observability discipline for outbound dependency boundaries.

A production-ready Java/JAX-RS service should make every critical downstream HTTP call visible through:

• client spans;
• low-cardinality dependency metrics;
• safe structured logs;
• timeout/retry/circuit breaker evidence;
• connection pool metrics;
• trace/correlation propagation;
• dependency dashboards;
• actionable alerts;
• runbooks.

The goal is not to collect more telemetry. The goal is to answer production questions quickly:

What dependency is hurting us?
How badly?
Since when?
For which endpoint/business flow?
Is it timeout, retry, pool, network, auth, or downstream failure?
What can we safely do now?