Part 027 — Distributed Idempotency Foundation

Idempotency is not a Redis feature.

Idempotency is a system behavior.

Redis is only one possible storage backend for enforcing that behavior.

In enterprise Java/JAX-RS systems, idempotency usually exists because clients, gateways, networks, load balancers, service meshes, brokers, workers, or users can repeat the same operation.

The core problem is simple:

The same logical request may arrive more than once.
The system must not apply the business effect more than once.

The hard part is that the duplicates may arrive:

• immediately
• after a timeout
• while the first request is still processing
• after the database commit succeeded
• after the response failed to reach the client
• after a service restart
• after a Kafka/RabbitMQ redelivery
• from another pod
• from another region or deployment segment

Idempotency is about controlling those ambiguity windows.

1. Core Definition

An operation is idempotent when repeating the same logical request has the same externally meaningful effect as executing it once.

Example:

Create quote with idempotency key K
Request 1 -> creates quote Q-1001
Request 2 with same K and same fingerprint -> returns Q-1001, does not create Q-1002

The second request may still perform reads, logging, metrics, or response replay.

But it must not repeat the business mutation.

2. Idempotency Is Not the Same as HTTP Idempotence

HTTP semantics help, but they are not enough.

Do not assume that a PUT endpoint is safe because the method name says so.

Idempotency is implemented in domain logic, storage, and response semantics.

3. Why Idempotency Exists

Idempotency exists because distributed systems cannot reliably distinguish:

The request was not processed.
The request was processed but the response was lost.
The request is still processing.
The request partially processed.
The request completed but the client timed out.

From the client's perspective, many failures look identical.

Example:

Client -> POST /quotes/submit
Service -> commits order transition
Service -> response blocked by network timeout
Client -> retries POST /quotes/submit

Without idempotency, the retry may submit the quote twice.

With idempotency, the retry should discover the previous result.

4. Common Enterprise Use Cases

Idempotency is most important for commands that create or advance business state.

Examples:

• create quote
• submit quote
• approve quote
• convert quote to order
• create order
• reserve inventory
• capture payment
• apply discount
• create customer account
• activate subscription
• trigger fulfillment
• send irreversible external request
• enqueue domain event
• process broker message
• retry long-running job

For CPQ, quote management, order management, and quote-to-cash systems, idempotency matters because many operations are expensive, user-visible, externally integrated, or financially meaningful.

5. The Exactly-Once Illusion

A common mistake is saying:

We need exactly-once processing.

A better statement is:

We need at-least-once delivery with idempotent effects.

Exactly-once across HTTP, Java service, Redis, PostgreSQL, Kafka, RabbitMQ, and external systems is usually an illusion.

What you can realistically build is:

• unique business constraints
• idempotency keys
• request fingerprints
• deterministic state transitions
• transactional database writes
• deduplication stores
• response replay
• idempotent message consumers
• outbox/inbox patterns
• retry-safe workers
• compensating actions when required

The goal is not magical exactly-once.

The goal is controlled duplicate effects.

6. Idempotency Boundary

Before implementing idempotency, define the boundary.

Ask:

What exactly must not happen twice?

Possible boundaries:

A weak design says:

This endpoint is idempotent.

A strong design says:

For the same tenant, actor, operation, idempotency key, and request fingerprint,
we create at most one quote submission result within 24 hours.
Retries receive the original result or a clear in-progress/conflict response.

7. Idempotency Key

An idempotency key is a client-supplied or system-generated identifier for a logical operation.

Example header:

Idempotency-Key: 01J0ZK5N9N6E0Q8V7D4Z2AZP9K

A good idempotency key is:

• unique per logical operation
• stable across retries
• scoped to tenant/user/client/operation
• not reused for different payloads
• not raw PII
• long enough to avoid collision
• logged only safely
• retained only as long as needed

The key alone is not enough.

You also need a request fingerprint.

8. Request Fingerprint

A request fingerprint is a canonical hash of the meaningful request payload.

It prevents accidental or malicious reuse of the same idempotency key for a different operation.

Example:

fingerprint = SHA-256(
  method + path + tenantId + normalizedBody + relevantHeaders + actorId
)

Do not hash unstable fields such as:

• correlation ID
• request timestamp
• trace ID
• random nonce
• header ordering
• JSON field ordering
• whitespace differences

Canonicalization matters.

Bad fingerprinting causes false conflicts or missed duplicates.

9. Idempotency Scope

Idempotency key scope determines what counts as duplicate.

Possible scope components:

tenant
actor/user
client/application
operation name
endpoint
resource id
business command type
idempotency key
request fingerprint

Example Redis key shape:

idem:{env}:{service}:{tenantHash}:{operation}:{idempotencyKeyHash}

For Redis Cluster, you may need hash tags:

idem:{tenant:t-123}:quote-submit:{keyHash}

But hash tags can create slot hotspots.

Use them only when a multi-key atomic operation requires colocation.

10. Idempotency State Machine

A practical idempotency system usually needs states.

Minimal state machine:

ABSENT
  -> PROCESSING
  -> COMPLETED
  -> EXPIRED

More realistic state machine:

ABSENT
  -> PROCESSING
  -> COMPLETED
  -> FAILED_RETRYABLE
  -> FAILED_FINAL
  -> UNKNOWN
  -> EXPIRED

Where:

The dangerous state is UNKNOWN.

It appears when the system loses certainty across Redis, database, broker, or external calls.

11. Processing State

PROCESSING means:

Some worker/service instance has accepted responsibility for the first execution.

It does not mean the business operation is complete.

Duplicate requests during PROCESSING may return:

• 409 Conflict
• 202 Accepted
• 425 Too Early
• 429 Too Many Requests
• a retryable domain error
• a wait/poll response

The choice depends on API contract.

Avoid making duplicate requests wait indefinitely inside the request thread.

That can consume servlet/JAX-RS worker threads and amplify incidents.

12. Completed State

COMPLETED means:

The business effect has been applied and the result can be replayed or referenced.

Stored result options:

Full response replay is convenient but risky if response includes sensitive data or schema-evolving payloads.

Resource reference is often safer:

{
  "state": "COMPLETED",
  "resourceType": "quoteSubmission",
  "resourceId": "QS-1001",
  "statusCode": 201
}

13. Failed State

A failed request may or may not be safe to retry.

Important distinction:

Failure before business effect -> retry may be safe
Failure after business effect -> retry must replay/discover result
Failure during commit boundary -> outcome may be unknown

Examples:

A mature design distinguishes final failure, retryable failure, and unknown outcome.

14. Unknown State

UNKNOWN is the state most systems ignore until production incidents happen.

Unknown appears when the system cannot prove whether the side effect happened.

Examples:

Payment provider timeout after capture request sent
Database commit result lost due to connection failure
Service crashes after DB commit but before Redis COMPLETED update
Kafka publish ambiguous after broker timeout
Worker crashes after external API call but before recording result

Unknown must be handled deliberately.

Options:

• query source of truth
• query external provider by idempotency key/reference
• use database unique constraint to discover prior result
• put operation into reconciliation workflow
• return conflict/in-progress and require polling
• mark for manual investigation if financially/security sensitive

Do not convert unknown into retry blindly.

That is how duplicates happen.

15. TTL and Retention Window

Idempotency records are not meant to live forever unless business/compliance requires it.

TTL depends on retry window and business risk.

Examples:

TTL is a correctness boundary.

After TTL expires, the system may treat the same idempotency key as new.

If that is unacceptable, Redis-only idempotency is insufficient.

Use PostgreSQL unique constraints or durable idempotency tables.

16. Duplicate Request Types

Not all duplicates are the same.

The dangerous one is cross-channel duplicate.

Example:

HTTP request creates order.
Kafka redelivery also triggers order creation.
Both use different dedupe mechanisms.

Idempotency must align across channels when they touch the same business effect.

17. Concurrent Same Request

Two identical requests may arrive at the same time.

Example:

Pod A receives POST /orders with key K
Pod B receives POST /orders with key K
Both check Redis
Both see nothing if check+set is not atomic
Both process
Duplicate order created

The first transition from ABSENT to PROCESSING must be atomic.

Common approaches:

• Redis SET key value NX PX ttl
• PostgreSQL unique constraint insert
• database idempotency table
• Lua script with state check
• external provider idempotency key

A non-atomic GET then SET is not idempotency.

It is a race condition.

18. Response Replay

A retry should receive a predictable response.

Possible API behaviors:

Decide whether repeated successful create returns 201 Created or 200 OK.

Both can be valid if documented.

Do not return a newly generated response that implies a second mutation.

19. Idempotency and PostgreSQL

PostgreSQL is often the stronger correctness anchor.

Use database constraints for hard uniqueness.

Examples:

unique (tenant_id, idempotency_key)
unique (tenant_id, external_request_id)
unique (tenant_id, business_command_id)
unique (order_id, transition_name)

Redis can reduce latency and protect concurrent retries, but PostgreSQL should often enforce final correctness for durable business effects.

A strong design often uses both:

Redis -> fast duplicate gate / response cache
PostgreSQL -> durable uniqueness / source of truth

Do not rely on Redis alone for irreversible financial or contractual effects unless durability requirements are explicitly satisfied.

20. Idempotency and MyBatis/JDBC

In Java services using MyBatis/JDBC, transaction boundary matters.

Bad pattern:

SET Redis PROCESSING
insert business rows
update Redis COMPLETED before DB commit
commit fails
retry sees COMPLETED incorrectly

Better pattern:

SET Redis PROCESSING
begin DB transaction
insert/update business rows with unique constraint
commit DB transaction
update Redis COMPLETED with resource reference

Still not perfect.

If the service crashes after DB commit but before Redis update, Redis remains PROCESSING or expires.

So the retry path must be able to query PostgreSQL to discover the durable result.

21. Idempotency and Kafka/RabbitMQ

Message brokers are commonly at-least-once.

This means duplicate delivery is normal.

Consumer idempotency options:

• processed message table in PostgreSQL
• Redis dedupe key with TTL
• domain unique constraints
• inbox pattern
• idempotent state transition
• deterministic projection update

Kafka example:

message key: quote-123
message id: evt-789
consumer receives evt-789 twice
consumer must not apply quote transition twice

RabbitMQ example:

worker processes message
worker commits DB
worker crashes before ack
message redelivered
worker must detect already processed effect

Redis dedupe can help, but broker redelivery windows, retention, and durability requirements must be considered.

22. Idempotent State Transitions

Sometimes the best idempotency design is not an external idempotency store.

It is a domain state machine that naturally rejects duplicate transitions.

Example:

DRAFT -> SUBMITTED -> APPROVED -> ORDERED

A duplicate submit command sees the quote already in SUBMITTED and returns the existing submitted state.

This is stronger than only storing request keys.

A good enterprise system combines:

request idempotency
+ domain state machine invariants
+ database constraints
+ event deduplication

23. Java/JAX-RS Placement

Possible placement:

Avoid putting all logic in a generic HTTP filter if idempotency depends on domain state.

The filter can normalize and validate.

The service must decide business semantics.

24. Example Request Lifecycle

1. Client sends POST with Idempotency-Key.
2. JAX-RS filter validates key format.
3. Service computes request fingerprint.
4. Service attempts to acquire PROCESSING record.
5. If acquired, service performs domain command.
6. PostgreSQL transaction commits business result.
7. Service stores COMPLETED result/reference.
8. Service returns response.
9. Duplicate retry reads COMPLETED and replays result.

Failure-aware lifecycle:

1. Acquire PROCESSING.
2. Execute command.
3. Commit DB.
4. Crash before COMPLETED.
5. Retry sees PROCESSING or expired key.
6. Retry queries DB by idempotency key/business unique reference.
7. Retry repairs Redis result or returns discovered result.

The repair path is part of the design, not an afterthought.

25. Failure Modes

Most bugs are not Redis bugs.

They are boundary bugs.

26. Timeout Ambiguity

Timeouts are not failures.

Timeouts are uncertainty.

Example:

Service calls external provider.
Provider processes request.
Network times out before response.

The caller does not know the outcome.

Safe handling requires:

• external idempotency key if supported
• provider status lookup
• durable request reference
• reconciliation job
• unknown state handling

Do not simply retry external side effects unless the provider guarantees idempotency for the same key.

27. Security and Privacy Concerns

Idempotency data may contain sensitive information.

Risks:

• raw idempotency keys in logs
• PII in Redis key names
• full response body cached in Redis
• token/session-like data stored without TTL
• long retention of payload fingerprints
• snapshot/backup exposure
• cross-tenant key collision

Safer practices:

• hash external keys in Redis key names
• store request fingerprint, not raw body
• avoid caching sensitive full response unless required
• apply TTL
• redact logs
• scope by tenant/client/operation
• use ACL/TLS/network isolation

28. Performance Concerns

Idempotency sits on the write path.

Common performance risks:

• high-cardinality keys
• large cached responses
• expensive fingerprinting
• Redis timeout on critical endpoint
• repeated polling during PROCESSING
• hot tenant/operation key design
• unbounded idempotency retention
• too many Redis round trips

The design should keep the common path small:

acquire -> execute -> complete

And keep duplicate path fast:

read -> compare fingerprint -> replay/reference

29. Observability Concerns

Track idempotency as a first-class behavior.

Useful metrics:

idempotency.acquire.success
idempotency.acquire.duplicate_processing
idempotency.acquire.duplicate_completed
idempotency.fingerprint_mismatch
idempotency.replay.success
idempotency.unknown_state
idempotency.redis.timeout
idempotency.db_discovery.success
idempotency.record.expired

Useful log fields:

operation
tenantHash
idempotencyKeyHash
fingerprintHash
state
resourceReference
correlationId
outcome

Avoid raw payload, raw PII, and raw secret-like keys.

30. Debugging Questions

When duplicate processing happens, ask:

Was an idempotency key required?
Was the same key reused?
Was fingerprint computed consistently?
Was acquire atomic?
Was TTL still active?
Did Redis timeout or fail?
Did DB commit succeed?
Was there a crash between DB commit and Redis update?
Was the duplicate from HTTP, Kafka, RabbitMQ, or a worker retry?
Was there a durable uniqueness constraint?
Was response replay implemented?

Do not start by blaming Redis.

Start by reconstructing the lifecycle.

31. Design Trade-Offs

There is no universal best choice.

The right choice depends on business impact of duplicates.

32. Internal Verification Checklist

For CSG/team verification, check:

• Which APIs require idempotency keys.
• Whether idempotency is enforced in JAX-RS filter, resource, service layer, or repository.
• Whether idempotency key scope includes tenant/client/operation.
• Whether raw PII appears in idempotency keys or logs.
• Whether request fingerprinting exists.
• Whether fingerprint canonicalization is stable.
• Whether duplicate same-key different-payload requests are rejected.
• Whether concurrent duplicates are protected by an atomic acquire.
• Whether PROCESSING, COMPLETED, FAILED, and UNKNOWN states are modeled.
• Whether TTL aligns with retry and business risk window.
• Whether response replay stores full response or resource reference.
• Whether PostgreSQL unique constraints backstop durable correctness.
• Whether MyBatis/JDBC transaction boundaries are safe.
• Whether Kafka/RabbitMQ consumers are idempotent.
• Whether external calls use provider-side idempotency keys where available.
• Whether Redis outage behavior is fail-open or fail-closed.
• Whether idempotency metrics and logs exist.
• Whether security/privacy review covers stored response and key naming.

33. PR Review Checklist

When reviewing idempotency design, ask:

• What duplicate effect are we preventing?
• What is the idempotency boundary?
• Who generates the idempotency key?
• Is the key scoped by tenant/client/operation?
• Is there a request fingerprint?
• Is fingerprinting canonical and stable?
• What happens on same key but different payload?
• Is acquire atomic?
• What happens during concurrent duplicate request?
• What state is returned while first request is processing?
• What is stored for response replay?
• What is the TTL and why?
• What happens after TTL expiry?
• What happens if Redis is unavailable?
• What happens if PostgreSQL commit succeeds but Redis update fails?
• What happens if Redis says completed but DB commit failed?
• Is there a durable uniqueness constraint?
• Are broker consumers idempotent too?
• Are external side effects idempotent?
• Is unknown outcome handled?
• Are metrics and logs sufficient?
• Are PII and sensitive response data protected?

A PR that says "we use Redis SETNX" is not enough.

The review must cover the whole lifecycle.

34. Summary

Distributed idempotency is about controlling duplicate effects under retries, concurrency, timeouts, crashes, and redelivery.

The main concepts are:

idempotency key
request fingerprint
scope
state machine
atomic acquire
processing state
completed state
response replay
TTL
unknown outcome
source-of-truth discovery

The most important production truth:

Idempotency is not exactly-once.
It is disciplined duplicate-effect control.

Redis is useful because it is fast and atomic for simple primitives.

But durable correctness often still belongs in PostgreSQL constraints, domain state machines, and broker-consumer idempotency.