Order order = orderService.create(command);

timeout  = batas waktu untuk satu operasi lokal
deadline = batas waktu total untuk seluruh workflow/request

User request deadline: 2s
Service A timeout ke B: 2s
Service B timeout ke C: 2s
Service C timeout ke D: 2s

Request deadline: now + 2s
A menerima deadline
A memberi B sisa budget: 1.7s
B memberi C sisa budget: 1.1s
C memberi D sisa budget: 500ms

API Gateway -> Order Service -> Payment Service -> Fraud Service -> DB

3 attempts * 350ms = 1050ms

public final class Deadline {
    private final long deadlineNanos;

    private Deadline(long deadlineNanos) {
        this.deadlineNanos = deadlineNanos;
    }

    public static Deadline after(Duration duration) {
        return new Deadline(System.nanoTime() + duration.toNanos());
    }

    public Duration remaining() {
        long remaining = deadlineNanos - System.nanoTime();
        return remaining <= 0 ? Duration.ZERO : Duration.ofNanos(remaining);
    }

    public boolean expired() {
        return remaining().isZero();
    }
}

public <T> T executeWithDeadline(
        Supplier<T> operation,
        Deadline deadline,
        RetryPolicy policy
) {
    int attempt = 1;
    Throwable lastFailure = null;

    while (!deadline.expired() && attempt <= policy.maxAttempts()) {
        try {
            return operation.get();
        } catch (Throwable failure) {
            lastFailure = failure;

            if (!policy.isRetryable(failure)) {
                throw failure;
            }

            Duration delay = policy.nextDelay(attempt);

            if (delay.compareTo(deadline.remaining()) >= 0) {
                break;
            }

            sleepInterruptibly(delay);
            attempt++;
        }
    }

    throw new DependencyTimeoutException(
            "Operation did not complete within deadline",
            lastFailure
    );
}

3^5 = 243 attempts

t=0ms failure
t=1ms retry
t=2ms retry
t=3ms retry

delay = min(base * 2^(attempt - 1), maxDelay)

delay = random(0, cappedExponentialDelay)

public final class Backoff {
    private final Duration base;
    private final Duration max;

    public Backoff(Duration base, Duration max) {
        this.base = base;
        this.max = max;
    }

    public Duration fullJitterDelay(int attempt) {
        long exponential = base.toMillis() * (1L << Math.max(0, attempt - 1));
        long capped = Math.min(exponential, max.toMillis());
        long jittered = ThreadLocalRandom.current().nextLong(capped + 1);
        return Duration.ofMillis(jittered);
    }
}

// Bad: all clients retry at the same rhythm
Thread.sleep(100);

Duration delay = backoff.fullJitterDelay(attempt);
sleepInterruptibly(delay);

public enum FailureKind {
    TRANSIENT,
    THROTTLED,
    TIMEOUT_UNKNOWN_OUTCOME,
    PERMANENT,
    DOMAIN_REJECTION,
    PROGRAMMER_ERROR
}

public record ClassifiedFailure(
        FailureKind kind,
        boolean retryable,
        boolean requiresIdempotency,
        String reason
) {}

public final class DependencyFailureClassifier {

    public ClassifiedFailure classify(Throwable failure) {
        if (failure instanceof java.net.SocketTimeoutException) {
            return new ClassifiedFailure(
                    FailureKind.TIMEOUT_UNKNOWN_OUTCOME,
                    true,
                    true,
                    "socket timeout; server outcome unknown"
            );
        }

        if (failure instanceof java.net.ConnectException) {
            return new ClassifiedFailure(
                    FailureKind.TRANSIENT,
                    true,
                    false,
                    "connection could not be established"
            );
        }

        if (failure instanceof IllegalArgumentException) {
            return new ClassifiedFailure(
                    FailureKind.PROGRAMMER_ERROR,
                    false,
                    false,
                    "invalid caller argument"
            );
        }

        return new ClassifiedFailure(
                FailureKind.PERMANENT,
                false,
                false,
                "unclassified failure is not retryable by default"
        );
    }
}

unknown failure => not retryable
unknown outcome => retry only with idempotency

POST /payments
Idempotency-Key: 01J2A4P9Y6W7K3C8K9M4Q6R2ZS
Content-Type: application/json

{
  "invoiceId": "INV-2026-0001",
  "amount": 100000,
  "currency": "IDR"
}

CREATE TABLE idempotency_record (
    scope              VARCHAR(128) NOT NULL,
    idempotency_key    VARCHAR(128) NOT NULL,
    payload_hash       VARCHAR(128) NOT NULL,
    status             VARCHAR(32)  NOT NULL,
    resource_type      VARCHAR(64),
    resource_id        VARCHAR(128),
    response_status    INTEGER,
    response_body      TEXT,
    created_at         TIMESTAMP NOT NULL,
    updated_at         TIMESTAMP NOT NULL,
    expires_at         TIMESTAMP NOT NULL,
    PRIMARY KEY (scope, idempotency_key)
);

1. create payment committed
2. application crashes before saving idempotency result
3. client retries
4. second payment created

@Transactional
public PaymentResponse createPayment(CreatePaymentCommand command, IdempotencyKey key) {
    IdempotencyRecord record = idempotencyRepository.tryStart(
            command.scope(),
            key.value(),
            command.payloadHash()
    );

    if (record.isReplayable()) {
        return record.replayAs(PaymentResponse.class);
    }

    if (record.isPayloadMismatch()) {
        throw new IdempotencyConflictException("Idempotency key reused with different payload");
    }

    Payment payment = paymentRepository.save(Payment.create(command));

    PaymentResponse response = PaymentResponse.from(payment);

    idempotencyRepository.markSucceeded(
            command.scope(),
            key.value(),
            payment.id(),
            response
    );

    return response;
}

For one idempotency scope + key + payload hash,
there must be at most one committed business effect.

tenantId + clientId + operationName

User A dan User B bisa sama-sama generate UUID yang sama walau probabilitas kecil.
Client internal dan external tidak boleh saling overwrite.
Operation create-payment dan create-refund tidak boleh berbagi key namespace.

public final class PayloadHasher {

    public String sha256CanonicalJson(String canonicalJson) {
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(canonicalJson.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(hash);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException("SHA-256 not available", e);
        }
    }
}

{"amount":100000,"currency":"IDR"}

{"currency":"IDR","amount":100000}

@Transactional
public void process() {
    retry.execute(() -> {
        repository.updateSomething();
        externalClient.call();
        return null;
    });
}

public Case assignInvestigator(CaseId caseId, InvestigatorId investigatorId) {
    return retryOptimisticLock(() -> {
        Case caze = caseRepository.findById(caseId);
        caze.assignInvestigator(investigatorId);
        return caseRepository.save(caze);
    });
}

"Approve if amount <= previous balance"

public void handle(MessageEnvelope envelope) {
    if (processedMessageRepository.exists(envelope.messageId())) {
        return;
    }

    processBusinessCommand(envelope);

    processedMessageRepository.markProcessed(envelope.messageId());
}

Normal requests per minute: 10,000
Retry budget: 10%
Max retry attempts per minute: 1,000

public final class RetryBudget {
    private final AtomicLong primaryCalls = new AtomicLong();
    private final AtomicLong retryCalls = new AtomicLong();

    public void recordPrimary() {
        primaryCalls.incrementAndGet();
    }

    public boolean tryAcquireRetry() {
        long primary = Math.max(1, primaryCalls.get());
        long retry = retryCalls.get();

        if (retry * 10 >= primary) {
            return false;
        }

        retryCalls.incrementAndGet();
        return true;
    }
}

{
  "event": "dependency_retry_scheduled",
  "dependency": "payment-service",
  "operation": "createPayment",
  "attempt": 2,
  "maxAttempts": 3,
  "delayMs": 180,
  "failureKind": "TIMEOUT_UNKNOWN_OUTCOME",
  "idempotencyKeyPresent": true,
  "correlationId": "corr-123",
  "traceId": "trace-abc"
}

Retrying...

Span: POST /orders
  Event: dependency.attempt attempt=1
  Event: dependency.timeout timeout_type=request
  Event: dependency.retry_scheduled delay_ms=180
  Event: dependency.attempt attempt=2
  Event: dependency.success

Dari semua request yang semula gagal, berapa persen diselamatkan retry?
Berapa latency tambahan karena retry?
Berapa duplicate request yang ditahan idempotency?
Berapa retry yang memperburuk overload?

HttpClient client = HttpClient.newBuilder()
        .connectTimeout(Duration.ofMillis(200))
        .build();

HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://payment.example.internal/payments"))
        .timeout(Duration.ofMillis(500))
        .header("Idempotency-Key", idempotencyKey)
        .POST(HttpRequest.BodyPublishers.ofString(payload))
        .build();

HttpResponse<String> response = client.send(
        request,
        HttpResponse.BodyHandlers.ofString()
);

public record DependencyRetryPolicy(
        String dependencyName,
        int maxAttempts,
        Duration baseDelay,
        Duration maxDelay,
        Duration totalDeadline,
        boolean requiresIdempotencyForUnknownOutcome
) {
    public void validate() {
        if (maxAttempts < 1) {
            throw new IllegalArgumentException("maxAttempts must be >= 1");
        }
        if (baseDelay.isNegative() || baseDelay.isZero()) {
            throw new IllegalArgumentException("baseDelay must be positive");
        }
        if (maxDelay.compareTo(baseDelay) < 0) {
            throw new IllegalArgumentException("maxDelay must be >= baseDelay");
        }
    }
}

catch (Exception e) {
    return retry();
}

Client gives up at 500ms
Server continues processing for 30s
Retry arrives
Duplicate processing starts

[ ] Apakah semua call punya timeout?
[ ] Apakah timeout dibedakan antara connect, request, dan total deadline?
[ ] Apakah retryable failure sudah diklasifikasikan?
[ ] Apakah non-retryable failure tidak diulang?
[ ] Apakah retry memakai capped exponential backoff + jitter?
[ ] Apakah max attempts jelas?
[ ] Apakah ada retry budget?
[ ] Apakah unknown outcome dilindungi idempotency?
[ ] Apakah idempotency key punya scope?
[ ] Apakah payload hash dicek?
[ ] Apakah response sukses bisa di-replay?
[ ] Apakah duplicate in-progress ditangani?
[ ] Apakah business effect dan idempotency record atomic?
[ ] Apakah logs/metrics/traces menampilkan attempts?
[ ] Apakah alert bisa mendeteksi retry storm?

dependency | operation | failure | retry? | timeout? | idempotency? | fallback?

case_create_attempts_total
case_create_retries_total
case_create_idempotency_duplicate_total
case_create_timeout_total

Apakah retry meningkatkan success rate?
Berapa latency cost retry?

"Berapa kali retry?"

"Failure mana yang retryable?"
"Apakah outcome unknown?"
"Apakah operation idempotent?"
"Siapa owner retry?"
"Apakah retry punya deadline?"
"Apakah retry memperburuk overload?"
"Bagaimana telemetry membuktikan policy ini benar?"