Part 036 — Idempotency as a First-Class Design Rule

In distributed systems, duplicate requests are not an edge case. They are a normal consequence of retry, timeout, redelivery, crash recovery, and human impatience. If duplicate execution breaks the business, the design is incomplete.

The previous part covered outbox/inbox. This part zooms into the rule that makes retry safe: idempotency.

Most engineers first meet idempotency through HTTP:

GET, PUT, DELETE are idempotent by method semantics.
POST is not automatically idempotent.

That is useful, but incomplete.

For microservices, idempotency is not mainly about HTTP verbs. It is about business side effects.

The real question is:

If the same intent is received more than once, can the service produce one correct business outcome?

1. Idempotency: Precise Meaning

A operation is idempotent when applying it multiple times has the same intended effect as applying it once.

Mathematically:

f(f(x)) = f(x)

In systems design:

same request intent + same idempotency scope -> one business effect

Examples:

Idempotency is not automatic. It is designed.

2. Why Duplicates Happen

Duplicates happen even when nobody made a mistake.

Sources:

1. Client timeout after server committed.
2. Load balancer retry.
3. API gateway retry.
4. Service mesh retry.
5. Application retry.
6. Broker at-least-once delivery.
7. Consumer crash before ack.
8. Outbox publisher retry after uncertain send result.
9. User double-click.
10. Mobile client reconnect.
11. Browser refresh/resubmit.
12. Batch replay.
13. Disaster recovery replay.
14. Manual operator replay.

If the server cannot distinguish retry from new intent, it may repeat the side effect.

3. Idempotency Is About Intent, Not Payload Only

Two requests can have the same payload but represent different business intent.

Example:

{
  "caseId": "CASE-2026-000481",
  "reason": "REPEAT_VIOLATION"
}

If this is sent twice with the same idempotency key, it is probably a retry.

If sent twice with different idempotency keys, it might be two separate escalation attempts, or it might still violate domain rule.

Therefore idempotency requires scope:

idempotency scope = actor/client + operation + resource + intent key + time window

Bad key:

random UUID only, never stored with operation context

Better key:

tenant:client:operation:resource:idempotencyKey

Example:

tenant-a:portal-ui:escalate-case:CASE-2026-000481:2a8634e9-8658-4d66-9349-f3c9a2d7f739

4. Idempotency Strategy Types

There is no single strategy. Use the cheapest correct one.

The best systems combine several.

Example:

API idempotency key + domain invariant + database unique constraint + outbox event ID + consumer inbox

5. Natural Idempotency

Natural idempotency means repeating the operation naturally reaches the same state.

Example:

public void close(CloseReason reason, Actor actor) {
    if (this.status == CaseStatus.CLOSED) {
        return;
    }
    this.status = CaseStatus.CLOSED;
    this.closedReason = reason;
    this.closedBy = actor.id();
    this.closedAt = clock.now();
}

But be careful.

This version is not strictly idempotent if repeated with a different reason or actor:

first call: close(reason=A, actor=X)
second call: close(reason=B, actor=Y)

A better model:

public CloseCaseResult close(CloseReason reason, Actor actor) {
    if (this.status == CaseStatus.CLOSED) {
        if (!Objects.equals(this.closedReason, reason)) {
            return CloseCaseResult.conflict("Case already closed with different reason");
        }
        return CloseCaseResult.alreadyClosed(this.id);
    }

    this.status = CaseStatus.CLOSED;
    this.closedReason = reason;
    this.closedBy = actor.id();
    this.closedAt = clock.now();
    return CloseCaseResult.closed(this.id);
}

Idempotency is not “ignore duplicate blindly”. It is “handle same intent safely and detect conflicting intent.”

6. Business Unique Constraint

If a business rule says there must be only one thing, encode that in the database.

Example:

There can be only one active supervisor review task per case escalation version.

SQL:

CREATE UNIQUE INDEX uq_supervisor_review_task
    ON review_task (case_id, source_case_version, task_type)
    WHERE task_type = 'SUPERVISOR_REVIEW';

Java:

public ReviewTask createSupervisorReview(CaseId caseId, long caseVersion) {
    try {
        return reviewTaskRepository.insert(
                ReviewTask.supervisorReview(caseId, caseVersion)
        );
    } catch (DuplicateKeyException duplicate) {
        return reviewTaskRepository.findSupervisorReview(caseId, caseVersion);
    }
}

This is stronger than checking first:

if (!exists()) insert(); // race condition

Use the database as the final guard for uniqueness.

7. Idempotency Key Store

For commands that create side effects, clients should send an idempotency key.

Request:

POST /cases/CASE-2026-000481/escalations
Idempotency-Key: 2a8634e9-8658-4d66-9349-f3c9a2d7f739
Content-Type: application/json

{
  "reasonCode": "REPEAT_VIOLATION",
  "comment": "Repeated breach within review window"
}

The server stores the key and result.

Table:

CREATE TABLE idempotency_record (
    scope              VARCHAR(200) NOT NULL,
    idempotency_key    VARCHAR(200) NOT NULL,
    request_hash       VARCHAR(128) NOT NULL,
    status             VARCHAR(30) NOT NULL,
    response_status    INT NULL,
    response_body      JSONB NULL,
    resource_type      VARCHAR(100) NULL,
    resource_id        VARCHAR(100) NULL,
    error_code         VARCHAR(100) NULL,
    created_at         TIMESTAMPTZ NOT NULL DEFAULT now(),
    locked_until       TIMESTAMPTZ NULL,
    completed_at       TIMESTAMPTZ NULL,
    expires_at         TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (scope, idempotency_key)
);

CREATE INDEX idx_idempotency_record_expires
    ON idempotency_record (expires_at);

Status model:

STARTED -> COMPLETED
       └-> FAILED_RETRYABLE
       └-> FAILED_TERMINAL

8. Request Hash: Prevent Key Reuse Bugs

A client must not reuse the same idempotency key with a different payload.

Store a request fingerprint:

hash(method + path + canonical-json-body + tenant + actor + operation)

If key exists and hash differs:

409 Conflict
Content-Type: application/problem+json

{
  "type": "https://api.example.com/problems/idempotency-key-reused",
  "title": "Idempotency key reused with different request",
  "status": 409,
  "detail": "The provided Idempotency-Key was already used for a different request."
}

This prevents a dangerous class of bugs:

same idempotency key accidentally reused for different command

9. Response Replay

When the same idempotency key is retried with the same request hash, the server can replay the original response.

First call:

202 Accepted
Location: /cases/CASE-2026-000481/escalations/ESC-991

Retry:

202 Accepted
Location: /cases/CASE-2026-000481/escalations/ESC-991
Idempotent-Replayed: true

Should you replay errors?

Decision:

Stripe-style APIs commonly store the result of the first request for a key and return the same result on retries. The broader principle is: once the server accepts responsibility for an idempotency key, it must make retries deterministic enough for clients to recover safely.

10. Idempotency Around Transaction Boundaries

A robust command handler flow:

Key rule:

idempotency record + domain side effect + outbox event must commit atomically.

If idempotency record commits but domain effect does not, retries may get stuck.

If domain effect commits but idempotency record does not, retries may duplicate.

11. Java Idempotency Boundary

Define a small abstraction:

public interface IdempotencyStore {
    IdempotencyDecision startOrReplay(IdempotencyRequest request);
    void complete(IdempotencyCompletion completion);
    void failTerminal(IdempotencyFailure failure);
}

Decision model:

public sealed interface IdempotencyDecision {
    record Start(UUID recordId) implements IdempotencyDecision {}
    record Replay(int status, String body) implements IdempotencyDecision {}
    record Conflict(String reason) implements IdempotencyDecision {}
    record InProgress(Duration retryAfter) implements IdempotencyDecision {}
}

Command handler:

@Transactional
public EscalateCaseResponse handle(EscalateCaseHttpRequest request) {
    IdempotencyRequest idem = IdempotencyRequest.from(
            request.tenantId(),
            request.actorId(),
            "EscalateCase",
            request.caseId(),
            request.idempotencyKey(),
            request.canonicalBodyHash()
    );

    IdempotencyDecision decision = idempotencyStore.startOrReplay(idem);

    if (decision instanceof IdempotencyDecision.Replay replay) {
        return EscalateCaseResponse.fromStored(replay.status(), replay.body());
    }

    if (decision instanceof IdempotencyDecision.Conflict conflict) {
        throw new IdempotencyConflictException(conflict.reason());
    }

    if (decision instanceof IdempotencyDecision.InProgress inProgress) {
        throw new OperationStillProcessingException(inProgress.retryAfter());
    }

    CaseFile caseFile = caseRepository.getForUpdate(request.caseId());
    Escalation escalation = caseFile.escalate(request.reasonCode(), request.actorId());

    caseRepository.save(caseFile);
    outboxRepository.append(eventMapper.caseEscalated(caseFile, escalation, request));

    EscalateCaseResponse response = EscalateCaseResponse.accepted(escalation.id());
    idempotencyStore.complete(IdempotencyCompletion.from(idem, response));

    return response;
}

This makes idempotency part of the application boundary, not a random filter.

12. Should Idempotency Be Middleware?

Sometimes, but not always.

Middleware can handle:

• extracting idempotency key,
• checking required header,
• canonical request hash,
• generic response replay.

Application service must handle:

• operation scope,
• business resource identity,
• transaction boundary,
• side-effect classification,
• conflict semantics,
• response meaning.

Bad design:

Generic HTTP filter stores response after controller returns, but domain transaction already committed separately.

Better design:

HTTP layer parses key, application command handler owns idempotency transaction.

Idempotency is too important to hide entirely in infrastructure.

13. Handling In-Progress Requests

What if the first request is still processing and retry arrives?

Options:

For microservices, prefer status URL for long operations:

202 Accepted
Location: /operations/op-2026-000991
Retry-After: 3

Do not hold server threads for long-running idempotency waits.

14. TTL and Retention

Idempotency records cannot live forever in high-volume systems.

But TTL is a business decision, not just storage cleanup.

Questions:

1. How long can clients retry?
2. How long can network failures last?
3. How long can users reasonably resubmit?
4. Is the operation financially/regulatorily sensitive?
5. Do we need audit evidence of duplicate attempts?
6. Can the same business operation naturally happen again later?

Examples:

For audit-heavy systems, separate operational TTL from audit retention:

hot idempotency table: 7-30 days
archive/audit evidence: retention policy period

15. HTTP Method Idempotency Is Not Enough

RFC-style HTTP semantics say PUT and DELETE are idempotent, while POST is not automatically idempotent.

But method semantics do not solve your business problem.

Example:

DELETE /cases/CASE-2026-000481/tasks/TASK-123

Repeating DELETE may be HTTP-idempotent because final state is task deleted.

But if each delete appends an audit record like:

Actor deleted task

then the externally observable audit side effect may not be idempotent.

You need to define idempotency at the business effect layer:

one task deletion decision,
one audit fact,
one notification,
one projection change.

16. Messaging Idempotency

For message consumers, idempotency key is usually the message/event ID.

@Transactional
public void consume(MessageEnvelope<CaseEscalatedV1> message) {
    if (inbox.alreadyProcessed(CONSUMER_NAME, message.eventId())) {
        return;
    }

    inbox.recordProcessing(CONSUMER_NAME, message.eventId());
    handler.apply(message.payload());
    inbox.recordProcessed(CONSUMER_NAME, message.eventId());
}

But event ID dedupe alone may not be enough.

Scenario:

Bug causes source service to emit two different event IDs for the same business fact.

Consumer should also guard business uniqueness:

CREATE UNIQUE INDEX uq_notification_once_per_case_escalation
    ON notification (case_id, notification_type, source_case_version);

Dedupe by event ID protects against broker duplicates.

Business unique constraints protect against semantic duplicates.

You need both when the side effect matters.

17. Version Guards for Projections

Projection updates should often be idempotent by version.

Event:

{
  "caseId": "CASE-2026-000481",
  "caseVersion": 17,
  "status": "ESCALATED"
}

Projection handler:

@Transactional
public void apply(CaseEscalatedV1 event) {
    CaseOverview overview = repository.findOrCreate(event.caseId());

    if (overview.version() >= event.caseVersion()) {
        return;
    }

    if (overview.version() + 1 != event.caseVersion()) {
        throw new ProjectionGapException(event.caseId(), overview.version(), event.caseVersion());
    }

    overview.markEscalated(event.caseVersion(), event.riskLevel(), event.occurredAt());
    repository.save(overview);
}

Three outcomes:

This is idempotency plus ordering correctness.

18. Idempotency for External Side Effects

External side effects are hardest because your database cannot enforce what another system does.

Examples:

• sending email,
• SMS,
• payment charge,
• document submission,
• regulator API call,
• legacy case system update.

Pattern:

Local outbox command + stable external idempotency key + stored external result

Example:

public void sendSupervisorNotification(NotificationCommand command) {
    String externalKey = "supervisor-notification:%s:%s".formatted(
            command.caseId(),
            command.caseVersion()
    );

    ExternalResponse response = notificationClient.send(
            externalKey,
            command.recipient(),
            command.message()
    );

    notificationAttemptRepository.recordResult(
            command.commandId(),
            externalKey,
            response.providerMessageId(),
            response.status()
    );
}

If provider supports idempotency keys, pass your key.

If provider does not, store your own send ledger and make replay a human/operational decision after unknown outcomes.

19. Unknown Outcome Problem

The hardest failure is not success or failure. It is unknown.

The worker does not know whether the external side effect happened.

Strategies:

Do not automatically retry irreversible effects without idempotency or status lookup.

20. Idempotency and State Machines

State machines make idempotency explicit.

Example:

A state transition command should define:

State machine + idempotency gives you deterministic duplicate handling.

21. Audit and Idempotency

Audit systems need special care.

Bad audit behavior:

Duplicate API retry creates duplicate audit facts, making it appear that actor performed action twice.

Better audit behavior:

First accepted command creates business audit event.
Duplicate retry creates technical access/retry log if needed, but not another business decision event.

Distinguish:

This matters for regulatory defensibility.

22. Canonical Request Hash

Hashing JSON is trickier than it looks.

These two payloads are semantically identical:

{"reasonCode":"REPEAT_VIOLATION","comment":"x"}

{
  "comment": "x",
  "reasonCode": "REPEAT_VIOLATION"
}

Naive string hash differs.

Use canonicalization:

• normalize JSON field order,
• remove insignificant whitespace,
• normalize number representation,
• normalize Unicode if needed,
• include method/path/tenant/actor/operation,
• avoid hashing volatile headers.

Pseudo:

public String canonicalHash(CommandRequest request) {
    CanonicalJson canonical = jsonCanonicalizer.canonicalize(request.body());
    String fingerprint = String.join("\n",
            request.method(),
            request.pathTemplate(),
            request.tenantId(),
            request.actorId(),
            request.operationName(),
            canonical.value()
    );
    return sha256(fingerprint);
}

Never use only the raw request body as the idempotency fingerprint.

23. Concurrency Control

Two identical retries may arrive at the same time.

The idempotency store must be race-safe.

Anti-pattern:

if (!idempotencyStore.exists(key)) {
    idempotencyStore.insert(key);
    executeSideEffect();
}

Race:

Thread A checks: absent
Thread B checks: absent
Both insert/execute or one fails after effect starts

Correct pattern:

INSERT INTO idempotency_record(scope, idempotency_key, request_hash, status, expires_at)
VALUES (?, ?, ?, 'STARTED', ?)
ON CONFLICT (scope, idempotency_key) DO NOTHING;

Then inspect whether insert succeeded.

If insert did not succeed, load existing record and decide replay/conflict/in-progress.

Use database uniqueness as the arbiter.

24. Idempotency Response Semantics

Possible responses:

For async commands, a clean response is:

202 Accepted
Location: /operations/op-2026-00991

Then idempotency retry returns the same operation URL.

25. Idempotency Key Generation

Client-generated key is usually better for API retries because the client knows retry intent.

Rules for clients:

Generate a new key for each new business intent.
Reuse the same key for retries of the same intent.
Do not reuse keys across different operations/resources.
Use high-entropy random keys or deterministic operation keys where appropriate.

Server-generated deterministic keys are useful for internal workflows:

create-review-task:CASE-2026-000481:caseVersion:17
send-supervisor-notification:CASE-2026-000481:caseVersion:17:supervisor:USR-1901

Do not use timestamps alone.

Do not use user ID alone.

Do not use request body hash alone.

26. Interaction with Retries

Retry without idempotency is gambling.

Retry-safe classification:

A resilient service defines retry policy and idempotency policy together.

27. Idempotency Smells

28. Design Checklist

For every command endpoint or message consumer, ask:

1. What is the business side effect?
2. What duplicate inputs can occur?
3. What identifies same intent?
4. What identifies conflicting intent?
5. What is the idempotency scope?
6. Is there a durable idempotency record?
7. Is the dedupe guard in the same transaction as the side effect?
8. Is there a database unique constraint for the business effect?
9. What response is replayed?
10. What happens if the request is still in progress?
11. What is the TTL/retention policy?
12. Does retry produce duplicate audit events?
13. Does outbox event duplicate cause duplicate consumer side effect?
14. Does external API support idempotency key?
15. How are unknown outcomes reconciled?
16. What metrics show duplicate rate?
17. What alert fires when idempotency conflicts spike?
18. Can operators safely replay the command/message?

29. Metrics

Idempotency metrics:

idempotency.started.count
idempotency.replayed.count
idempotency.conflict.count
idempotency.in_progress.count
idempotency.expired.count
idempotency.unknown_outcome.count
idempotency.store.latency.ms

Business duplicate protection metrics:

review_task.duplicate_prevented.count
notification.duplicate_suppressed.count
projection.stale_event_ignored.count
message.duplicate_consumed.count

High duplicate rate can indicate:

• client timeout too short,
• slow service,
• gateway retry misconfiguration,
• consumer ack failure,
• broker redelivery storm,
• user double-submit issue,
• external provider instability.

Duplicate rate is a reliability signal.

30. Worked Example: Retry-Safe Case Escalation API

Endpoint:

POST /cases/{caseId}/escalations
Idempotency-Key: <client-generated-key>

Business rule:

A case version can have at most one escalation action.
Duplicate retry returns same escalation ID.
Different reason for same idempotency key returns conflict.
Escalating already closed case returns business conflict.

Database guards:

CREATE UNIQUE INDEX uq_escalation_per_case_version
    ON case_escalation (case_id, source_case_version);

CREATE UNIQUE INDEX uq_idempotency
    ON idempotency_record (scope, idempotency_key);

Flow:

This design is retry-safe at the API layer and event-safe at the messaging layer.

31. Mental Model Summary

Idempotency is not a decorator. It is a correctness property.

Use this mental model:

Retry is inevitable.
Duplicate delivery is normal.
Timeout means unknown, not failed.
Idempotency turns unknown outcomes into recoverable outcomes.

A production-grade Java microservice should treat idempotency as part of:

• API design,
• command handling,
• database constraints,
• outbox/inbox messaging,
• external integration,
• audit design,
• retry policy,
• observability.

The goal is not to prevent duplicate requests.

The goal is to ensure duplicate requests cannot corrupt business state.

32. Exercises

Exercise 1 — Classify Operations

Classify these operations:

Create case
Escalate case
Close case
Assign reviewer
Send notification
Generate regulatory report
Increment risk score
Import evidence file

For each, decide:

• naturally idempotent,
• requires idempotency key,
• requires unique constraint,
• requires inbox,
• requires external idempotency.

Exercise 2 — Design Idempotency Scope

Design an idempotency scope for:

POST /cases/{caseId}/decisions

Include tenant, actor, operation, resource, and key.

Exercise 3 — Unknown Outcome

An external notification provider times out after receiving your request.

Define:

• retry rule,
• idempotency key,
• status lookup strategy,
• reconciliation job,
• operator playbook.

Exercise 4 — Audit Safety

Design audit behavior for duplicate retries of:

Approve enforcement action

Separate business audit event from technical request log.

Exercise 5 — Concurrency Test

Write a test scenario where two identical requests with the same idempotency key arrive concurrently.

Expected result:

one side effect,
one completed idempotency record,
second request replays or waits,
no duplicate audit event,
no duplicate outbox event.

References

• RFC 9110 — HTTP method semantics and idempotent methods.
• Stripe API Documentation — Idempotent requests and idempotency keys.
• Stripe Engineering — Designing robust and predictable APIs with idempotency.
• Chris Richardson, Microservices Patterns — Idempotent Consumer pattern.
• Debezium documentation — Outbox pattern and reliable data exchange.