Part 028 — Redis Idempotency Store

Part 027 explained the concept.

This part focuses on designing a Redis-backed idempotency store.

A Redis idempotency store must answer four questions quickly and safely:

Have we seen this logical request before?
Is the original request still processing?
Did it already complete?
Is this retry actually the same request?

A weak implementation stores only a marker.

A production implementation stores a small state machine.

1. Core Responsibilities

A Redis idempotency store should provide operations like:

acquire(key, fingerprint, ttl) -> ACQUIRED | DUPLICATE | CONFLICT
complete(key, result, ttl) -> OK | NOT_OWNER | MISSING
fail(key, failure, ttl) -> OK | NOT_OWNER | MISSING
get(key) -> state record

It should centralize:

• key naming
• TTL policy
• request fingerprint comparison
• state transitions
• response replay data
• failure behavior
• metrics
• logs
• Redis timeout handling

Do not scatter raw SETNX, GET, and DEL calls across business services.

2. Basic Redis Key Shape

A common key shape:

idem:{env}:{service}:{tenantHash}:{operation}:{idempotencyKeyHash}

Example:

idem:prod:quote-api:tenant_7e3b:quote-submit:key_19af

Rules:

• include environment only if shared Redis spans environments
• include service or bounded context
• include tenant scope for multi-tenant systems
• include operation name
• hash external idempotency key if it may contain sensitive data
• avoid raw email/user/customer identifiers
• keep key length reasonable
• always use TTL

For Redis Cluster multi-key scripts, hash tags may be needed:

idem:{tenant_7e3b}:quote-submit:key_19af

Do not overuse hash tags.

A single hot tenant can become a hot slot.

3. Idempotency Record Shape

Store a compact JSON value or Redis hash.

Example JSON record:

{
  "state": "PROCESSING",
  "fingerprint": "sha256:6f1d...",
  "owner": "pod-a:thread-42:01J0...",
  "operation": "quote-submit",
  "tenantHash": "tenant_7e3b",
  "createdAt": 1783740000000,
  "updatedAt": 1783740000000,
  "expiresAt": 1783826400000,
  "result": null,
  "error": null
}

Completed record:

{
  "state": "COMPLETED",
  "fingerprint": "sha256:6f1d...",
  "operation": "quote-submit",
  "resourceType": "quoteSubmission",
  "resourceId": "QS-1001",
  "statusCode": 201,
  "createdAt": 1783740000000,
  "updatedAt": 1783740003120,
  "expiresAt": 1783826400000
}

Prefer compact records.

Redis is not a document database.

4. String vs Hash for Idempotency Records

For most idempotency stores, a JSON string is enough.

Use a version field if payload can evolve:

{
  "schemaVersion": 1,
  "state": "COMPLETED"
}

Avoid Java native serialization.

It is brittle across class changes and rolling deployments.

5. Atomic Acquire with SET NX PX

The first request must atomically create a PROCESSING record.

Redis primitive:

SET key processingRecord NX PX processingTtlMillis

Meaning:

Create the record only if absent.
Apply TTL immediately.

This avoids the classic race:

GET key
if missing:
  SET key

That pattern is unsafe under concurrent requests.

6. Processing TTL vs Completed TTL

Use separate TTL concepts.

Example:

processing TTL: 2 minutes
completed TTL: 24 hours
failed validation TTL: 24 hours
unknown TTL: 10 minutes + reconciliation

Processing TTL should be long enough for the operation under normal conditions.

Completed TTL should match API contract and business duplicate risk.

7. Acquire Result Semantics

Acquire can return:

Do not collapse all duplicates into one response.

State matters.

8. Java Interface

Create a small port/interface.

public interface IdempotencyStore {
    AcquireResult acquire(IdempotencyCommand command);
    CompletionResult complete(IdempotencyCompletion completion);
    FailureResult fail(IdempotencyFailure failure);
    Optional<IdempotencyRecord> get(IdempotencyKey key);
}

Example records:

public record IdempotencyCommand(
    String tenantId,
    String operation,
    String idempotencyKey,
    String requestFingerprint,
    Duration processingTtl,
    String owner
) {}

public record IdempotencyCompletion(
    IdempotencyKey key,
    String owner,
    String requestFingerprint,
    IdempotencyResult result,
    Duration completedTtl
) {}

Keep this abstraction independent from JAX-RS.

JAX-RS is an entrypoint.

Idempotency is domain infrastructure.

9. JAX-RS Request Flow

Example flow:

JAX-RS resource
  -> validate Idempotency-Key header
  -> normalize request payload
  -> compute fingerprint
  -> service.submitQuote(command)
  -> idempotencyStore.acquire(...)
  -> process or replay based on acquire result

Possible response mapping:

For business-critical create/submit endpoints, fail-open may create duplicates.

For low-risk operations, fail-open may be acceptable.

Document the decision.

10. Request Fingerprint Storage

On acquire, store the fingerprint.

On duplicate, compare fingerprint.

Pseudo-flow:

record = GET key
if record missing:
  SET key PROCESSING NX PX ttl
  return ACQUIRED
if record.fingerprint != incomingFingerprint:
  return CONFLICT
if record.state == COMPLETED:
  return DUPLICATE_COMPLETED
if record.state == PROCESSING:
  return DUPLICATE_PROCESSING

The first branch must be atomic.

The duplicate inspection can use GET, but careful designs combine check and create in Lua for consistent semantics.

11. Why Lua May Be Useful

SET NX PX is enough for simple acquire.

Lua becomes useful when acquire must:

• create if absent
• read existing state if present
• compare fingerprint
• return state-specific result
• update stale processing marker
• preserve TTL rules
• handle owner token

Example conceptual script behavior:

if key does not exist:
  set PROCESSING record with TTL
  return ACQUIRED
else:
  decode existing record
  if fingerprint mismatch:
    return CONFLICT
  else:
    return existing state

Keep scripts short.

Do not put business logic inside Redis Lua.

12. Owner Token

A processing record should include an owner token.

Example:

owner = serviceInstanceId + ':' + requestId + ':' + randomToken

The owner prevents a stale or unrelated worker from completing someone else's record.

Completion should check:

state == PROCESSING
fingerprint == expected
owner == expected

If owner does not match, completion should not overwrite the record.

This matters when:

• processing TTL expires
• retry acquires a new processing record
• old request resumes after GC pause
• pod was slow but not dead

Owner token is similar in spirit to lock value safety.

13. Completion Update

After business success, update Redis to COMPLETED.

Important rule:

Do not write COMPLETED before the durable business effect is committed.

Typical flow:

acquire PROCESSING
begin DB transaction
apply domain mutation
commit DB transaction
set Redis COMPLETED with completed TTL
return response

Completion update should be conditional if possible:

only complete if current state is PROCESSING and owner matches

Use Lua or compare-and-set-like logic when needed.

14. Result Storage Options

For enterprise systems, resource reference is often the best default.

Example:

{
  "state": "COMPLETED",
  "statusCode": 201,
  "resourceType": "order",
  "resourceId": "ORD-9001"
}

The retry path can rebuild the response from PostgreSQL.

15. Full Response Cache Risk

Caching the full response can be useful.

But check:

• does response include PII?
• does response include token/security data?
• does response include volatile timestamps?
• can response schema change during rolling deployment?
• is response too large?
• will Redis snapshots/backups now contain sensitive data?
• is encryption/network isolation sufficient?
• can support safely inspect the record?

If the response is sensitive or large, store only a reference.

16. PostgreSQL Backstop Pattern

Redis protects fast path concurrency.

PostgreSQL protects durable correctness.

Recommended pattern for important commands:

Redis acquire PROCESSING
PostgreSQL insert idempotency/business row with unique constraint
PostgreSQL commit
Redis complete with reference

Example constraint:

create unique index uq_idem_quote_submit
on quote_submission_request (tenant_id, idempotency_key);

If Redis loses state, PostgreSQL can still discover the existing result.

This is essential when Redis is cache-like or has limited persistence guarantees.

17. Redis Failure Before Processing

Scenario:

Client sends request.
Service cannot reach Redis during acquire.

Options:

For high-impact operations, DB fallback is often better than fail-open.

Fail-open must be explicitly approved.

18. Redis Failure After DB Commit

Scenario:

Redis PROCESSING acquired.
DB commit succeeds.
Redis COMPLETED update fails.
Response may or may not reach client.
Client retries.

If retry sees PROCESSING, expired marker, or no key, the service must not repeat the mutation blindly.

Correct behavior:

Query PostgreSQL by idempotency key or business unique reference.
If durable result exists, repair Redis COMPLETED and replay/reference result.
If no result exists, decide whether safe to retry.

This is one of the most important failure windows.

19. Redis Completed But DB Commit Failed

Scenario:

Service writes Redis COMPLETED.
DB commit fails.
Retry sees completed result.
But durable resource does not exist.

This is a design bug.

Prevent it by ordering:

DB commit first
Redis completed second

If response replay depends on DB, this bug becomes visible quickly.

If full response is cached in Redis, it can hide the corruption.

That is another reason to prefer resource references.

20. Processing TTL Expired While Work Continues

Scenario:

Request A acquires PROCESSING with 60s TTL.
GC pause or slow DB/external call lasts 90s.
Key expires.
Request B acquires same key.
A resumes and completes.
B also processes.

Mitigations:

• set realistic processing TTL
• use owner token
• conditionally complete only if owner matches
• use DB uniqueness constraint
• avoid long external calls inside weak idempotency lease
• renew processing marker only if safe and owner matches

Processing TTL is not just cleanup.

It is a lease.

21. Lock + Response Pattern

An idempotency store often resembles a lock plus result cache.

But it is not identical to a distributed lock.

Do not implement idempotency by only acquiring and deleting a lock.

If the record is deleted after completion, late retries cannot replay/discover the result.

22. Expired Idempotency Key

After TTL expiry, Redis cannot prove prior processing.

Possible behaviors:

• treat as new request
• query PostgreSQL before processing
• reject as expired key if key timestamp encoded
• require client to generate a new key
• use durable idempotency table for longer retention

For high-value business operations, never rely only on a short Redis TTL.

TTL expiry should be a conscious API contract.

23. Request Hash Mismatch

Same idempotency key with different fingerprint should usually return conflict.

Example response:

HTTP/1.1 409 Conflict
Content-Type: application/json

{
  "error": "idempotency_key_conflict",
  "message": "The supplied idempotency key was already used for a different request."
}

Do not overwrite the existing record.

Do not process as new.

Do not expose the original fingerprint or payload.

24. Redis and Kafka/RabbitMQ Integration

For broker consumers, Redis idempotency store can dedupe messages.

Example key:

idem:prod:quote-worker:tenant_7e3b:event:evtHash

But broker semantics matter.

Kafka:

Duplicate can come from rebalance, retry, replay, offset reset.

RabbitMQ:

Duplicate can come from nack, consumer crash before ack, redelivery.

Redis TTL must cover the redelivery/replay window.

If historical replay is possible after TTL, Redis-only dedupe is insufficient.

Use PostgreSQL inbox/processed-message table for durable dedupe.

25. Idempotency with Outbox/Inbox

For strong event-driven consistency, combine Redis with durable patterns.

HTTP command
  -> DB transaction writes business rows + outbox event + idempotency row
  -> outbox publisher sends Kafka/RabbitMQ event
  -> consumer writes inbox/processed-message row
  -> Redis optionally caches recent dedupe/result

Redis is useful for fast recent duplicate detection.

The database provides durable proof.

This is especially important when message replay is part of normal operations.

26. Redis Cluster Considerations

Single-key idempotency operations are easy in Redis Cluster.

Multi-key operations require same hash slot.

Be careful with:

• Lua scripts touching multiple keys
• idempotency record + response body in separate keys
• index key + record key
• tenant-level lookup structures

If using multiple keys, hash tags may be required:

idem:{tenant_7e3b}:quote-submit:key_19af:state
idem:{tenant_7e3b}:quote-submit:key_19af:response

But grouping too much by tenant can create hot slots.

Prefer single-key record when possible.

27. Serialization and Schema Evolution

Idempotency records may outlive a deployment.

Therefore:

• include schema version
• use stable JSON field names
• ignore unknown fields
• tolerate missing optional fields
• avoid Java class names in payload
• avoid enum ordinal serialization
• store timestamps in epoch millis or ISO format consistently
• keep result references stable

Rolling deployment issue:

Pod v1 writes idempotency record.
Pod v2 reads it after deployment.
Deserialization must still work.

A failed deserialization can become an idempotency failure.

28. Security and Privacy

Idempotency store may hold:

• operation names
• tenant references
• user/client references
• request fingerprints
• response references
• status codes
• error summaries
• cached response body

Controls:

• hash sensitive identifiers in key names
• avoid full payload storage
• avoid raw PII in records unless required and approved
• enforce TTL
• redact logs
• protect Redis with ACL/TLS/network policy
• review snapshots/backups
• restrict support access to sensitive keyspace

Security review must include both key name and value.

29. Observability

Metrics:

redis_idempotency_acquire_total{operation,result}
redis_idempotency_complete_total{operation,result}
redis_idempotency_conflict_total{operation}
redis_idempotency_replay_total{operation}
redis_idempotency_processing_duplicate_total{operation}
redis_idempotency_unknown_total{operation}
redis_idempotency_redis_error_total{operation,error}
redis_idempotency_db_discovery_total{operation,result}

Avoid labels like raw tenant ID, raw user ID, or idempotency key.

Logs should include hashes:

operation=quote-submit
idempotencyKeyHash=key_19af
tenantHash=tenant_7e3b
fingerprintHash=fp_9d21
state=COMPLETED
outcome=REPLAYED
correlationId=...

30. Production-Safe Debugging

When debugging duplicate or stuck idempotency behavior:

1. Identify operation and tenant scope.
2. Find idempotency key hash from logs.
3. Inspect Redis record safely using exact key, not KEYS scan.
4. Check TTL.
5. Check state and owner.
6. Check PostgreSQL durable record.
7. Check application logs around acquire/complete.
8. Check Redis timeout/error metrics.
9. Check pod restarts/GC pauses/rolling deploys.
10. Check Kafka/RabbitMQ redelivery if async path involved.

Do not run broad production scans without platform/SRE approval.

31. Common Implementation Bugs

Most bugs are design bugs, not client-library bugs.

32. Testing Strategy

Test with real Redis behavior, not only mocks.

Important tests:

first acquire succeeds
duplicate same fingerprint returns processing/completed
duplicate different fingerprint returns conflict
concurrent acquire allows only one owner
complete requires matching owner
completed response can be replayed
processing TTL expiry behavior is understood
completed TTL expiry behavior is understood
Redis timeout follows explicit policy
DB commit success + Redis complete failure is recovered
Redis completed is never written before DB commit
Kafka/RabbitMQ redelivery is deduped
serialization works across schema versions

Use Testcontainers Redis for integration tests where possible.

Use concurrency tests for acquire.

Mocks are not enough for idempotency race behavior.

33. Internal Verification Checklist

For CSG/team verification, check:

• Whether a shared idempotency store abstraction exists.
• Whether Redis idempotency key naming is documented.
• Whether keys include tenant/client/operation scope.
• Whether external idempotency keys are hashed before Redis key usage.
• Whether request fingerprinting is implemented and canonicalized.
• Whether same-key different-payload conflict is handled.
• Whether acquire uses SET NX PX, Lua, or another atomic mechanism.
• Whether processing TTL and completed TTL are separate.
• Whether owner token is stored and checked on completion.
• Whether completed record stores full response or resource reference.
• Whether sensitive response data is stored in Redis.
• Whether DB commit happens before Redis completed update.
• Whether PostgreSQL has durable uniqueness/idempotency backstop.
• Whether retry can repair Redis from PostgreSQL after DB commit + Redis failure.
• Whether Kafka/RabbitMQ consumers use compatible dedupe semantics.
• Whether Redis Cluster constraints affect key design/scripts.
• Whether metrics/logs exist for acquire, replay, conflict, unknown, and Redis error.
• Whether tests cover concurrency and failure windows.
• Whether security/privacy reviewed Redis idempotency records and snapshots.

34. PR Review Checklist

When reviewing a Redis idempotency store PR, ask:

• Is idempotency implemented as a shared component?
• What is the exact Redis key shape?
• Are key names free of PII?
• Is TTL always set on acquire?
• Are processing and completed TTLs different where needed?
• Is acquire atomic?
• Does duplicate handling compare request fingerprint?
• Does conflict return a clear response?
• Is there an owner token?
• Can a stale owner overwrite completion?
• Is Redis completed written only after durable DB commit?
• What happens if Redis complete fails after DB commit?
• Can retry discover result from PostgreSQL?
• Is Redis-only acceptable for this business operation?
• Does the design handle concurrent duplicates?
• Does it handle timeout ambiguity?
• Does it handle expired keys?
• Is full response caching safe?
• Does serialization survive rolling deployments?
• Does it work in Redis Cluster if applicable?
• Are metrics/logs production-safe?
• Are tests based on real Redis?

The implementation is acceptable only when the failure windows are explicit.

35. Minimal Production Design

A reasonable default design for important HTTP commands:

1. Require Idempotency-Key header.
2. Scope key by tenant + operation + key hash.
3. Compute canonical request fingerprint.
4. Atomic Redis acquire with PROCESSING + owner + TTL.
5. Reject same key different fingerprint.
6. Return processing response for concurrent duplicate.
7. Execute domain command inside DB transaction.
8. Enforce PostgreSQL unique constraint.
9. Commit DB.
10. Store COMPLETED in Redis with resource reference and longer TTL.
11. On retry, replay from Redis or rebuild from PostgreSQL.
12. On Redis failure after DB commit, repair from PostgreSQL.

This is not the only design.

But it has the right shape.

36. Summary

A Redis idempotency store is a small state machine backed by atomic Redis operations.

The essential rules are:

• atomic acquire
• stable key scope
• request fingerprint comparison
• explicit states
• separate processing/completed TTL
• owner token for safe completion
• durable PostgreSQL backstop for important effects
• response replay or resource reference
• repair path for Redis/database failure windows
• safe broker redelivery handling
• security/privacy-conscious records
• metrics and production-safe debugging

Redis gives speed and atomic primitives.

Correctness comes from the full design.