Part 082 — Outbound Resilience, Idempotency Key, and Request Signing

Fokus part ini: membuat outbound call aman terhadap latency, duplicate execution, downstream failure, unauthorized tampering, dan retry amplification.

Outbound HTTP call adalah dependency boundary.

Dependency boundary selalu punya tiga pertanyaan produksi:

1. Berapa lama kita boleh menunggu?
2. Apa yang kita lakukan saat dependency gagal?
3. Bagaimana kita memastikan request yang dikirim aman dan tidak dieksekusi ganda secara berbahaya?

Part ini mengikat beberapa konsep:

• timeout
• retry
• retry budget
• circuit breaker
• bulkhead
• rate limiting
• fallback
• idempotency key
• request signing
• error mapping
• observability
• failure amplification

1. Core Mental Model

Outbound call bukan hanya function call.

Application service
  -> integration adapter
  -> resilience policy
  -> auth/signing/interceptor
  -> HTTP client
  -> network/platform
  -> downstream service

Setiap layer bisa gagal.

Before request leaves process:
  config error
  serialization error
  missing credentials
  circuit open
  bulkhead full

During transport:
  DNS failure
  connection timeout
  TLS handshake failure
  write timeout
  network reset

After request reaches downstream:
  slow response
  4xx/5xx response
  partial processing
  timeout after side effect
  malformed response

Senior engineer harus memisahkan:

request not sent
request sent but not processed
request processed but response lost
request processed and error returned
request status unknown

Retry decision bergantung pada klasifikasi itu.

2. Timeout Hierarchy

Timeout adalah batas waktu untuk melindungi resource.

Timeout tidak boleh dianggap hanya sebagai setting teknis.

Ia adalah bagian dari contract.

Timeout hierarchy:

client/browser timeout
  -> API gateway timeout
  -> ingress/service mesh timeout
  -> JAX-RS inbound timeout/deadline
  -> application use case budget
  -> outbound client total timeout
  -> connect timeout
  -> write timeout
  -> read timeout
  -> DB/Kafka/cache/cloud SDK timeout

Rule:

Lower layer timeout must fit inside upper layer budget.

Bad:

Gateway timeout: 30s
JAX-RS handler: no deadline
Outbound HTTP read timeout: 60s

This causes zombie work.

Better:

Gateway timeout: 30s
Application deadline: 25s
Outbound call A: 3s
Outbound call B: 5s
DB call: 2s
Buffer for cleanup/error mapping: 1s

Timeouts to verify:

• connect timeout
• read timeout
• write timeout
• call/total timeout
• pool acquisition timeout
• DNS/TLS timeout where available
• resilience timeout wrapper
• gateway/mesh timeout

3. Timeout Failure Semantics

Timeout does not always mean no side effect.

Example:

Service A sends POST /orders/submit
Service B receives request
Service B creates order
Service B response is delayed
Service A times out

From Service A perspective:

result unknown

Dangerous wrong assumption:

Timeout means downstream did nothing.

Correct assumption:

Timeout after request was sent may mean the operation succeeded, failed, or is still running.

That is why mutating outbound calls need:

• idempotency key
• operation ID
• retry-safe contract
• status lookup endpoint
• reconciliation job
• clear timeout mapping

4. Retry Decision Model

Retry is useful only when it improves probability of success without causing unsafe duplication or overload.

Retry candidates:

connection refused before request is sent
connection timeout before write
HTTP 429 with backoff
HTTP 503 with backoff
transient 502/503/504 depending on platform contract
idempotent GET with bounded retry
PUT/DELETE with understood idempotency
POST with idempotency key

Retry danger zone:

POST without idempotency key
financial/pricing/order mutation
workflow transition
file upload
long-running operation
unknown timeout outcome
non-repeatable stream body

Retry policy should define:

• max attempts
• total retry budget
• exponential backoff
• jitter
• retryable exception/status
• non-retryable exception/status
• idempotency requirement
• observability labels
• kill switch

Bad retry:

retry 3 times on all exceptions

Better retry:

retry only known transient failures
with bounded budget
with jitter
only if operation is idempotent or has idempotency key

5. Retry Budget

Retry budget limits total retry pressure.

Without retry budget:

normal traffic: 100 rps
failure begins
retry x3
load becomes 300 rps
service gets slower
more timeout
more retry
cascading failure

Retry budget can be expressed as:

retries <= 10% of original request volume

or:

per-client retry tokens per time window

Senior principle:

Retries are a scarce resource, not a default behavior.

Internal verification checklist:

• Is retry budget implemented?
• Is retry visible in metrics?
• Can retry be disabled quickly?
• Are retries coordinated with gateway/service mesh/SDK retries?
• Are retry storms detectable?

6. Circuit Breaker

Circuit breaker prevents repeated calls to a dependency that is likely failing.

States:

closed
  calls allowed

open
  calls rejected fast

half-open
  limited trial calls allowed

Circuit breaker protects:

• request threads
• connection pools
• downstream service
• user latency
• upstream callers

But it also changes behavior.

If circuit is open, your integration adapter must decide:

• return fallback?
• return dependency unavailable?
• use cached data?
• enqueue for async processing?
• reject request?

Circuit breaker must be per dependency and often per operation class.

Bad:

one global circuit breaker for all downstreams

Better:

pricing-read-circuit
catalog-read-circuit
order-submit-circuit

because failure semantics differ.

7. Bulkhead

Bulkhead isolates capacity.

Without bulkhead:

slow pricing service consumes all request threads
catalog endpoint also fails
health checks slow down
entire service becomes unhealthy

Bulkhead types:

• thread pool bulkhead
• semaphore bulkhead
• connection pool per downstream
• queue size limit
• per-tenant limit
• per-operation concurrency limit

Bulkhead policy should answer:

How many concurrent calls may this dependency consume?
What happens when the limit is reached?
Do we fail fast or queue?
How long may queue wait?

Fail-fast is often better than unbounded queue.

Unbounded queue hides overload until latency explodes.

8. Rate Limiting and Load Shedding

Rate limiting controls request rate.

Load shedding rejects work when the system is overloaded.

They are related but not identical.

rate limiting = policy-based admission control
load shedding = survival behavior under overload

Outbound rate limiting may be needed when:

• downstream has quota
• partner API enforces limit
• cloud API has throttling
• internal service is fragile
• tenant-level fairness is required

Load shedding may be needed when:

• queue is full
• latency budget is already exhausted
• CPU/memory pressure is high
• circuit breaker is open
• dependency failure is causing backlog

Return shape should be explicit:

429 Too Many Requests
503 Service Unavailable
problem detail with retry guidance

9. Fallback and Graceful Degradation

Fallback is not always good.

Good fallback:

• cached product catalog for read-only browsing
• default non-critical UI hints
• stale-but-marked reference data
• skip optional enrichment
• return partial response if API contract supports it

Bad fallback:

• fake successful order submission
• fake successful payment
• silently use wrong pricing rules
• silently ignore failed authorization
• return stale data without marking freshness

Fallback must preserve domain correctness.

For CPQ/order-style systems, be very careful with:

• pricing
• tax
• discounts
• contract terms
• entitlement
• inventory/availability
• approval workflow
• order submission

Senior rule:

Graceful degradation must be explicit to caller or provably safe for the business invariant.

10. Idempotency Key

Idempotency key lets the client safely retry a mutating operation.

Example:

POST /orders
Idempotency-Key: 8b1a-...

Server behavior:

first request with key
  -> process operation
  -> store result for key

same key again with same request fingerprint
  -> return same result

same key with different request fingerprint
  -> reject as conflict

Outbound integration should use idempotency key when calling downstream mutating APIs.

Use cases:

• create order
• submit quote
• reserve inventory
• initiate payment
• start workflow
• create document
• publish side-effecting command

Key properties:

• unique per logical operation
• stable across retry
• propagated through layers
• stored with request fingerprint
• expires after safe retention window
• associated with tenant/account/security context

Anti-pattern:

generate new idempotency key on each retry

That defeats idempotency.

11. Idempotency Key vs Business ID

Idempotency key is not always the same as business ID.

Business ID:

orderId = ORD-123

Idempotency key:

requestId/operationId = unique execution attempt group

Sometimes they can be the same if the API contract is designed that way.

But often they differ:

Create order request does not know orderId yet.
It uses idempotency key to safely create exactly one order.

For enterprise systems, consider storing:

tenant_id
operation_type
idempotency_key
request_fingerprint
status
response_reference
created_at
expires_at

12. Idempotency Failure Modes

Internal verification checklist:

• Is idempotency implemented locally, downstream, or both?
• Which operations require idempotency key?
• Is key generated at operation boundary?
• Is request fingerprint checked?
• Is idempotency tenant-scoped?
• What is retention/TTL?
• Is there a reconciliation path for unknown outcome?

13. Request Signing

Request signing protects integrity and authenticity of outbound requests.

It is common for:

• partner APIs
• internal high-trust service calls
• payment/financial APIs
• cloud APIs
• webhook delivery
• cross-network integration

Basic model:

canonical request
  -> method
  -> path
  -> query
  -> selected headers
  -> body hash
  -> timestamp
  -> nonce/idempotency key
  -> HMAC/private key signature

Receiver verifies:

• sender identity
• timestamp freshness
• body not tampered
• headers/path/query not tampered
• replay protection
• key validity

Important:

Sign exactly what the receiver verifies.

If proxy modifies headers or path after signing, signature may fail.

14. Request Signing Failure Modes

Internal verification checklist:

• Is request signing required for any downstream?
• Which algorithm is used?
• What is the canonical request format?
• Are signed headers allowed through gateway/proxy?
• Is timestamp/nonce checked?
• How is key rotation handled?
• Are signing secrets redacted?
• Is there a signature verification test suite?

15. Authentication, Signing, and mTLS Are Different

Do not collapse these concepts.

OAuth/JWT token
  proves caller identity/authorization claims

mTLS
  proves transport-level peer identity

Request signature
  proves message integrity and freshness

They can be used together.

Example:

mTLS between services
JWT for service/user claims
HMAC signature for partner API request integrity
Idempotency key for retry-safe mutation

Review each separately:

• Who is the caller?
• Is the transport peer trusted?
• Was the message altered?
• Is the request fresh?
• Can duplicate execution happen?

16. Error Mapping for Outbound Calls

Outbound errors should be mapped into internal categories.

Suggested taxonomy:

DownstreamClientError
  400, 404 depending on domain, 409 contract/domain conflict

DownstreamAuthError
  401, 403, token/signature/mTLS failure

DownstreamRateLimitError
  429, quota exceeded

DownstreamUnavailableError
  502, 503, connection refused, DNS failure

DownstreamTimeoutError
  timeout with unknown outcome

DownstreamProtocolError
  malformed response, invalid content type, invalid schema

DownstreamDataError
  semantically invalid response data

Mapping matters because:

• retry policy uses it
• alerting uses it
• fallback uses it
• API error response uses it
• RCA uses it

Bad:

catch (Exception e) {
    throw new RuntimeException("Downstream failed", e);
}

Better:

catch (SocketTimeoutException e) {
    throw new DownstreamTimeoutException(clientName, operation, UNKNOWN_OUTCOME, e);
}

17. Failure Amplification

One inbound request can amplify into many outbound calls.

1 inbound request
  -> 5 downstream calls
  -> each retries 3 times
  -> each has 2s timeout
  -> each logs full error body

During outage:

traffic x fanout x retry x timeout = incident size

Failure amplification sources:

• nested retries
• fan-out loops
• unbounded async concurrency
• no bulkhead
• no rate limit
• slow fallback
• large logging volume
• high-cardinality metrics
• retry from gateway + service + SDK

Mitigation:

• retry budget
• bounded concurrency
• per-client timeout
• per-client bulkhead
• circuit breaker
• request coalescing
• cache stable reference data
• load shedding
• disable retry via config/kill switch

18. Resilience4j or Equivalent Library

Many Java services use a resilience library such as Resilience4j or an internal equivalent.

Do not assume which one is used.

Concepts to verify:

• timeout/time limiter
• retry
• circuit breaker
• bulkhead
• rate limiter
• fallback decorator
• metrics integration
• annotation-based vs programmatic usage
• configuration source
• per-client/per-operation policies

Annotation-based resilience can be convenient but can hide control flow.

Example risk:

@Retry + @CircuitBreaker + @TimeLimiter order may not be what developer assumes.

Senior review should check:

What is the decoration order?
Which exception types are recorded?
Which statuses are retryable?
What happens when fallback throws?
Are metrics emitted with safe labels?

19. Policy Placement

Good placement:

Resource method
  -> use case
  -> port
  -> integration adapter
       -> resilience policy
       -> idempotency/signing/auth interceptor
       -> HTTP client

Avoid:

Resource method manually retries downstream call.

Avoid:

Retry hidden in generated client with unknown defaults.

Avoid:

Business code catches raw HTTP exceptions.

Policy should be close to dependency boundary, not scattered across business logic.

20. Observability Requirements

Outbound resilience must be observable.

Metrics:

outbound.request.count{client,operation,status_class}
outbound.duration{client,operation,status_class}
outbound.timeout.count{client,operation}
outbound.retry.count{client,operation,reason}
outbound.circuit.state{client,operation}
outbound.bulkhead.rejected.count{client,operation}
outbound.rate_limited.count{client,operation}
outbound.fallback.count{client,operation}
outbound.idempotency.reuse.count{client,operation}
outbound.signing.failure.count{client,operation}

Traces:

• one span per outbound call
• sanitized route/template, not raw path with IDs
• retry count attribute
• circuit breaker state if available
• downstream service identity
• error type

Logs:

• structured failure event
• no secrets/tokens/signature raw material
• no full PII payload
• include correlation ID and causation ID
• include idempotency key only if policy allows and it is non-sensitive

21. Testing Strategy

Minimum tests for outbound integration:

• success response
• 400 error mapping
• 401/403 auth/signature mapping
• 404 behavior based on domain meaning
• 409 conflict mapping
• 429 retry/backoff behavior
• 500/503 mapping
• connection timeout
• read timeout
• malformed JSON
• unexpected content type
• empty body
• retry stops after max attempts
• retry not attempted for unsafe mutation without idempotency key
• idempotency key stable across retry
• request signature includes expected canonical fields
• redaction prevents secret logging
• trace/correlation headers propagated

Use fake/mock HTTP server rather than mocking the client interface only.

Why:

Mocking the interface does not test annotation mapping, headers, encoding, decoding, or error decoder.

22. Internal Verification Checklist

For CSG Quote & Order or any enterprise environment, verify:

• What resilience library is standard?
• Is Resilience4j used, or an internal/platform equivalent?
• Are timeouts mandatory per outbound client?
• Are retry policies centrally governed?
• Are retries disabled for unsafe operations by default?
• Is retry budget implemented?
• Are circuit breakers per downstream/operation?
• Are bulkheads configured?
• Is rate limiting done in service, gateway, mesh, or downstream?
• Is fallback allowed for pricing/catalog/order flows?
• Which operations require idempotency key?
• Is idempotency key generated at operation boundary?
• Does downstream honor idempotency key?
• Is request signing required?
• How are signing keys stored and rotated?
• Are request signatures tested against canonical examples?
• Are outbound errors mapped consistently?
• Are metrics/traces/logs emitted per downstream?
• Can retry/circuit/fallback be changed by config safely?
• Is there a kill switch for dangerous integration paths?

23. PR Review Checklist

Review outbound resilience changes for:

• dependency name and operation name
• timeout budget
• retryable/non-retryable errors
• max attempts and backoff/jitter
• retry budget
• circuit breaker threshold/window
• bulkhead capacity
• rate limit policy
• fallback correctness
• idempotency key requirement
• idempotency key lifecycle
• request signing canonicalization
• token/mTLS/signing separation
• error mapping taxonomy
• observability metrics/tracing/logging
• high-cardinality label risk
• redaction of secrets/PII
• tests for timeout/retry/signing/idempotency
• runbook impact

24. Senior Takeaway

Outbound resilience is not a library setting.

It is an architectural contract between:

• caller
• downstream service
• network/platform
• business operation
• retry/idempotency semantics
• security model
• observability system
• incident response process

The key senior questions are:

Can this call hang?
Can it be safely retried?
Can it duplicate side effects?
Can it overload downstream?
Can it leak tenant/security context?
Can it be tampered with?
Can we debug it during an incident?
Can we change the policy safely without redeploy?

If the answer is unclear, the integration is not production-ready.