slow dependency -> blocked thread -> queue grows -> memory rises -> latency rises
               -> retries increase -> downstream gets worse -> service collapses

- dependency bisa lambat
- client bisa disconnect
- request bisa duplicate
- thread pool bisa penuh
- queue bisa menumpuk
- Kubernetes bisa kill pod
- retry bisa memperbesar traffic
- timeout di satu layer bisa bertabrakan dengan timeout layer lain

Every unit of work must have a deadline, a cancellation path, and a bounded resource footprint.

How long is this work still worth doing?

How long before the library throws exception?

- client expectation
- gateway timeout
- server/container timeout
- application deadline
- DB statement timeout
- HTTP client connect/read timeout
- Kafka send timeout
- executor queue wait timeout
- Kubernetes probe and termination settings

client timeout > gateway timeout > service deadline > dependency timeout

client timeout = 5s
service keeps working for 60s
DB query runs for 120s

Which timeout fires first, and what state is left behind?

DB timeout: 5s
HTTP client timeout: 5s
Kafka timeout: 5s
Total request might take 15s+

Request deadline: now + 8s
Every dependency call consumes remaining budget.

public record Deadline(Instant expiresAt) {
    public Duration remaining(Clock clock) {
        Duration remaining = Duration.between(clock.instant(), expiresAt);
        return remaining.isNegative() ? Duration.ZERO : remaining;
    }

    public boolean expired(Clock clock) {
        return !clock.instant().isBefore(expiresAt);
    }
}

Duration remaining = ctx.deadline().remaining(clock);
if (remaining.compareTo(Duration.ofMillis(200)) < 0) {
    throw new RequestDeadlineExceededException();
}

- client disconnect
- gateway timeout
- service request deadline exceeded
- parent task cancelled
- pod shutdown
- circuit breaker open
- queue rejection

Client no longer waiting does not automatically mean server stopped working.

- response write may fail
- server may still run business logic
- downstream calls may continue
- database transaction may continue
- logs may show success even though client saw failure

- Did mutation already commit?
- Did event already publish?
- Can the command be safely retried?
- Is idempotency key used?
- Should server finish command even if client disconnects?

Cancellation policy must be operation-specific, especially for mutating commands.

incoming rate > processing capacity
queue grows
latency grows
timeouts grow
retries grow
system collapses

incoming rate > capacity
system rejects early or slows intake
latency remains bounded
failure is explicit

- bounded thread pools
- bounded queues
- connection pool limits
- rate limiting
- semaphore bulkhead
- circuit breaker
- load shedding
- max request body size
- streaming flow control
- Kafka consumer pause/resume
- DB statement timeout

Executors.newFixedThreadPool(50); // often backed by unbounded queue in common patterns

- memory grows
- stale tasks run long after caller timed out
- latency becomes unbounded
- shutdown takes too long

- bounded pool
- bounded queue
- named threads
- rejection policy
- metrics
- timeout/deadline before enqueue
- context propagation wrapper

ThreadPoolExecutor executor = new ThreadPoolExecutor(
    20,
    20,
    0L,
    TimeUnit.MILLISECONDS,
    new ArrayBlockingQueue<>(500),
    namedThreadFactory("quote-worker"),
    new ThreadPoolExecutor.AbortPolicy()
);

try {
    executor.execute(task);
} catch (RejectedExecutionException e) {
    throw new ServiceOverloadedException("quote-worker queue is full", e);
}

- request latency spikes
- CPU not necessarily high
- threads stuck in WAITING/BLOCKED/TIMED_WAITING
- health checks fail
- queue length increases
- container still alive but unresponsive

- slow DB query
- missing HTTP client timeout
- connection pool starvation
- blocking I/O on request threads
- large file buffering
- unbounded async queue
- lock contention
- retry storm

- thread dump
- pool metrics
- DB pool metrics
- HTTP client pool metrics
- request latency histogram
- dependency latency
- garbage collection logs
- Kubernetes CPU throttling metrics

[ ] What are most request threads doing?
[ ] Are they waiting on DB connection?
[ ] Are they blocked on socket read?
[ ] Are they waiting on lock?
[ ] Are they doing CPU work?
[ ] Are they writing large response?

- downstream is overloaded
- DB has too many active sessions
- latency increases globally

- service queues internally
- throughput limited
- request threads wait for connections

No acquisition timeout -> thread can wait too long
Short acquisition timeout -> fail fast during saturation

[ ] Is connection pool size justified?
[ ] Is acquisition timeout configured?
[ ] Are pool metrics exported?
[ ] Are slow queries visible?
[ ] Does service fail fast when pool is exhausted?

attempt 1 hangs
attempt 2 never starts
request deadline exceeded
resources remain held

3 attempts x 5s each = 15s
but caller timeout = 8s

Request deadline: 8s
Attempt 1: 1.5s
Backoff: 100ms
Attempt 2: 1.5s
Backoff: 200ms
Attempt 3: 1s
Remaining time for response handling

- retry only retryable failures
- retry only idempotent or idempotency-protected operations
- retry with bounded attempts
- retry with jittered backoff
- retry within deadline
- do not retry overload blindly

- stable error code
- human-readable message safe for client
- correlation ID
- retryability hint, if part of API standard
- no stack trace
- no secret/internal topology

- server reads faster than downstream can write
- client reads slowly
- proxy buffers full response
- heap fills with byte arrays
- timeout kills long stream
- partial file not cleaned up

[ ] Is data streamed or fully buffered?
[ ] Is response flush behavior understood?
[ ] Does proxy/gateway buffer response?
[ ] Is max duration configured?
[ ] Is client disconnect detected?
[ ] Are partial files cleaned up?
[ ] Are byte counts logged/metricized?

[ ] max body size
[ ] streaming parser
[ ] temp file location
[ ] disk quota
[ ] virus/security scanning boundary
[ ] checksum
[ ] cleanup on failure

- connection acquisition timeout
- statement timeout
- transaction timeout
- lock timeout
- idle-in-transaction timeout
- query plan/index review

HTTP request timeout fires at gateway after 30s.
DB query continues for 5 minutes.
Connection remains occupied.
Other requests wait for pool.
Service collapses gradually.

- configure DB statement timeout below request deadline
- avoid long transaction around remote calls
- log slow query with correlation ID
- export pool metrics
- use lock timeout for lock-heavy operations

- buffer memory
- max block time
- delivery timeout
- linger/batch settings
- acks
- retry config

Kafka broker unavailable
producer buffer fills
send blocks
request threads block
HTTP latency spikes

[ ] Is event publish part of request critical path?
[ ] Is outbox pattern used instead?
[ ] What happens if Kafka is down?
[ ] Does request fail, degrade, or persist command for later publish?
[ ] Are producer send failures observable?

- pause/resume can apply backpressure
- max poll records controls batch size
- processing time must fit poll interval or use correct pattern
- DLQ/retry must not create infinite poison loop

- readiness probe
- liveness probe
- startup probe
- terminationGracePeriodSeconds
- preStop hook
- resource requests/limits
- HPA signals
- ingress timeout annotations

- application started
- critical dependency optional/required status
- worker queue saturation
- shutdown/draining mode

- return cached catalog when pricing service is slow, if allowed
- disable non-critical enrichment
- skip optional recommendation lookup
- queue background notification instead of synchronous send
- use read-only mode during partial outage

- using stale pricing when quote requires exact current price
- skipping authorization check
- hiding failed order submission as success
- silently dropping event

Never degrade correctness, security, or auditability silently.

- request queue too deep
- CPU throttling severe
- DB pool exhausted
- executor saturated
- memory pressure
- circuit breaker open
- dependency latency above threshold

503 Service Unavailable
Retry-After: <optional>
X-Correlation-ID: <id>

- shed request count
- reason
- endpoint
- tenant/class of traffic, if allowed
- dependency causing pressure

- short timeout
- no retry unless dependency call is safe
- cache allowed if data semantics permit
- fail fast on pool exhaustion

- idempotency key recommended
- transaction boundary explicit
- downstream call avoided inside transaction
- event publish via outbox if consistency needed
- response semantics clear: 200/201/202/409

- avoid holding HTTP request open
- return 202 Accepted
- provide operation status endpoint
- persist operation state
- support idempotency and reconciliation

- stream payload
- apply size limits
- cleanup partial data
- monitor byte count and duration
- avoid request-thread starvation

- request duration histogram
- dependency duration histogram
- timeout count by operation/dependency
- cancellation count
- rejected task count
- executor active/queued/completed
- DB pool active/idle/pending
- HTTP client pool metrics
- Kafka producer buffer/latency/error
- load shedding count

- timeout event with operation and remaining deadline
- cancellation reason
- rejected execution reason
- dependency failure classification
- correlation ID and trace ID

- span status/error for timeout
- dependency span duration
- queue wait span, if available
- cancellation attribute, if standard internally

1. Check whether latency is endpoint-wide or dependency-specific.
2. Compare p50/p95/p99 latency.
3. Check timeout count and where timeout fires.
4. Check thread pool and request thread state.
5. Check DB/HTTP/Kafka pool saturation.
6. Check retry volume.
7. Check queue length and rejection count.
8. Check CPU throttling and memory pressure.
9. Check recent deployment/config change.
10. Identify whether work continues after caller timeout.

- thread dump
- heap/native memory metrics
- application logs by correlation ID
- distributed trace
- DB slow query log
- connection pool metrics
- Kubernetes pod events
- ingress/gateway logs

Anti-pattern: no timeout on outbound HTTP client
Impact: stuck request threads

Anti-pattern: DB query timeout longer than gateway timeout
Impact: zombie DB work after client failure

Anti-pattern: unbounded executor queue
Impact: memory growth and stale work

Anti-pattern: retrying non-idempotent command
Impact: duplicate mutation

Anti-pattern: retrying overload response aggressively
Impact: retry storm

Anti-pattern: holding DB transaction while calling remote service
Impact: long locks and pool exhaustion

Anti-pattern: async work without cancellation/error handling
Impact: orphaned tasks and invisible failures

Anti-pattern: fallback that hides correctness failure
Impact: data corruption or false success

[ ] Does every outbound HTTP client have connect/read/request timeout?
[ ] Does DB access have statement/lock/acquisition timeout where appropriate?
[ ] Is request deadline defined or implied clearly?
[ ] Are dependency timeouts shorter than caller/gateway timeout?
[ ] Is retry bounded and within deadline?
[ ] Is retry limited to idempotent or idempotency-protected operations?
[ ] Is async queue bounded?
[ ] Is rejection handled explicitly?
[ ] Are thread pools named and metricized?
[ ] Does code handle cancellation/interruption where relevant?
[ ] Does mutating endpoint define behavior after client disconnect?
[ ] Does endpoint fail fast under saturation?
[ ] Are overload responses mapped consistently?
[ ] Are timeout and rejection metrics/logs available?
[ ] Is fallback safe and documented?

HTTP/server runtime
[ ] What is inbound request timeout?
[ ] What timeout is configured at gateway/load balancer/ingress?
[ ] What timeout is configured in Servlet/Jersey/embedded runtime?
[ ] Are client disconnects detected/logged?

Outbound HTTP
[ ] Which HTTP client library is used?
[ ] Are connect/read/request timeouts configured centrally?
[ ] Are retries configured centrally?
[ ] Is there a standard resilience wrapper?

Resilience library/platform
[ ] Is Resilience4j or equivalent used internally?
[ ] Are circuit breaker, retry, bulkhead, rate limiter standards documented?
[ ] Are configs per dependency or global?
[ ] Are retry budgets enforced?

Thread pools/executors
[ ] Which executors exist?
[ ] Are queues bounded?
[ ] Are rejection policies explicit?
[ ] Are executor metrics exported?
[ ] Is context propagation integrated?

Database
[ ] What connection pool is used?
[ ] What are pool size and acquisition timeout?
[ ] Are statement timeout and lock timeout configured?
[ ] Are slow queries logged with correlation ID?

Kafka/eventing
[ ] Is event publishing synchronous or outbox-based?
[ ] What producer timeout/buffer settings exist?
[ ] What happens when Kafka is unavailable?
[ ] Are consumer retry/DLQ policies bounded?

Kubernetes/platform
[ ] What are pod CPU/memory limits?
[ ] Are pods CPU-throttled during load?
[ ] Are readiness/liveness/startup probes aligned with startup and overload?
[ ] Is termination graceful?
[ ] Are ingress timeout annotations configured?

Observability
[ ] Are timeout metrics emitted?
[ ] Are rejection/load-shedding metrics emitted?
[ ] Are dependency latency histograms available?
[ ] Are traces showing dependency duration and errors?

[ ] inbound timeout understood
[ ] gateway timeout understood
[ ] outbound HTTP timeouts configured
[ ] DB statement/acquisition timeout configured
[ ] bounded executor queues
[ ] explicit rejection handling
[ ] retry budget and idempotency rules
[ ] cancellation behavior for long work
[ ] backpressure/load shedding strategy
[ ] graceful shutdown and draining
[ ] timeout/rejection/dependency metrics
[ ] runbook for latency and saturation incidents

- Timeout is a business and operational contract.
- Cancellation stops work that no longer matters, but mutating commands need explicit policy.
- Backpressure is controlled failure that prevents collapse.
- Queues must be bounded and observable.
- Retry must fit inside deadline and idempotency rules.
- Client timeout does not automatically stop server-side work.
- DB, HTTP, Kafka, executor, gateway, and Kubernetes timeouts must be aligned.
- Senior engineers review resource control, not only functional correctness.