Part 023 — Redis Streams for Job Queue and Event Stream

Redis Streams can be used as a lightweight durable queue or event log-lite.

That does not mean Redis Streams should replace Kafka or RabbitMQ everywhere.

The correct mental model is narrower:

Redis Streams = retained Redis-native records + consumer-group work distribution + explicit ACK/retry state

Streams are useful when the system needs better reliability than Pub/Sub, but does not need the full operational and semantic weight of a broker platform.

In an enterprise Java/JAX-RS system, Redis Streams commonly appear around:

• background task dispatch
• local asynchronous processing
• lightweight work queues
• retryable cache projection
• operational event fan-out inside one bounded context
• delayed post-processing when combined with sorted sets
• transient integration glue between services
• stream-backed job workers where Kafka/RabbitMQ would be excessive

The dangerous mistake is treating Redis Streams as if they automatically provide the same semantics as Kafka, RabbitMQ, a workflow engine, or a relational outbox.

They do not.

Streams give you tools.

You still own the correctness model.

1. When Redis Streams Are a Good Fit

Redis Streams are a good fit when the queue or stream is close to the application boundary and the operational scope is intentionally bounded.

Good examples:

• process quote recalculation jobs inside one service domain
• update a Redis projection after receiving a database or broker event
• fan out small internal tasks to workers
• retry short-lived asynchronous work
• process cache invalidation work where loss is unacceptable but long-term audit is not required
• maintain lightweight worker groups with explicit acknowledgment
• create a bounded stream of operational events for short replay windows

The core advantage is that the producer and worker can share Redis as a fast coordination substrate.

The core risk is that teams may accidentally turn Redis into an enterprise event backbone.

That is usually the wrong direction.

2. When Redis Streams Are the Wrong Fit

Redis Streams are usually the wrong fit when the system needs:

• long-term event retention
• regulatory audit trail
• cross-team event ownership
• complex routing semantics
• enterprise-wide topic governance
• high-throughput partitioned event streaming
• long replay windows
• schema registry governance
• exactly-once business processing
• strong delivery guarantees across regions
• broker-level dead-letter and routing controls

For those cases, Kafka, RabbitMQ, a workflow engine, or a database-backed outbox/inbox pattern may be more appropriate.

Redis Streams are powerful, but they are still Redis data structures.

They live inside Redis memory, persistence, replication, failover, eviction, and operational limits.

3. Stream as Durable Queue Mental Model

A stream-backed queue has four important states:

Added       -> XADD wrote entry to stream
Delivered   -> XREADGROUP delivered entry to a consumer
Pending     -> entry is in the Pending Entry List until ACK
Acknowledged -> XACK confirms processing completed

This is better than Redis Pub/Sub because messages can remain in the stream.

It is also better than a naive Redis List queue because Redis tracks pending delivery for consumer groups.

But the system is only reliable if workers implement the lifecycle correctly.

A stream entry that is delivered but never acknowledged remains pending.

Pending is not the same as failed.

Pending means Redis is waiting for the consumer group to decide what to do.

4. Basic Lifecycle

A common lifecycle looks like this:

JAX-RS request
  -> service validates command
  -> service commits primary state in PostgreSQL if needed
  -> service appends stream entry with XADD
  -> worker reads with XREADGROUP
  -> worker processes job
  -> worker writes side effect / updates DB / updates cache
  -> worker acknowledges with XACK

The subtle part is the boundary between database commit and stream append.

If the business state is stored in PostgreSQL, Redis Streams should usually not become the only source of truth for that state.

If a request updates PostgreSQL and then appends to Redis, the following partial failures exist:

For critical workflows, use a transactional outbox in PostgreSQL or a broker designed for the durability requirement.

Redis Streams are excellent for bounded asynchronous work.

They are not a magic transaction bridge.

5. Stream as Event Log-Lite

Redis Streams can behave like a short-lived event log.

This means entries may be retained for a configured window or capped length, and consumers can replay from earlier IDs.

Useful cases:

• rebuild a small projection
• recover after worker downtime
• debug recent event flow
• replay recent cache-update events
• support multiple internal consumers over a bounded history

But event log-lite has boundaries.

It does not mean:

• permanent history
• schema governance
• unlimited replay
• independent team ownership
• durable audit evidence
• broker-grade partition management

A Redis Stream can support replay.

It should not be assumed to support indefinite replay.

6. Consumer Group Model

Consumer groups allow multiple workers to divide work from the same stream.

Conceptually:

Stream: quote:events
Group:  quote-workers
Consumers:
  - quote-worker-pod-1
  - quote-worker-pod-2
  - quote-worker-pod-3

Each entry is delivered to one consumer in the group.

That makes consumer groups suitable for queue-like processing.

A different consumer group can process the same stream independently.

This is useful when one stream drives multiple independent projections or worker types.

However, every additional consumer group creates its own pending state and operational responsibility.

Unused or forgotten groups can accumulate pending entries and confuse operational diagnosis.

7. Consumer Naming Discipline

In Kubernetes, consumer names should be stable enough for debugging but unique enough to avoid collisions.

Common naming pattern:

<service-name>-<pod-name>

Example:

quote-worker-quote-worker-7d9c8f9d8b-vx2ra

Avoid anonymous or random-only names that cannot be traced to a pod.

Also avoid reusing the same consumer name across all pods.

If every pod uses the same consumer name, pending ownership becomes ambiguous and debugging gets harder.

Internal platforms may already enforce a naming convention.

If not, treat consumer naming as part of the worker design.

8. ACK Discipline

The rule is simple:

ACK only after the side effect is safe to consider completed.

Do not acknowledge immediately after reading.

That turns the stream into at-most-once processing.

Do not delay ACK forever after success.

That creates duplicate retry pressure.

The correct ACK point depends on the job:

ACK is not just a Redis command.

It is a correctness boundary.

9. Pending Entry List

The Pending Entry List, or PEL, tracks entries delivered to consumers but not acknowledged.

PEL is what makes stream consumer groups much safer than plain Pub/Sub.

The PEL answers questions such as:

• which messages are in flight?
• which consumer owns them?
• how long have they been idle?
• how many delivery attempts happened?
• which consumers are stuck?

Operationally, the PEL is one of the most important parts of Redis Streams.

A growing PEL usually means one of these:

• workers are crashing
• workers are too slow
• workers are not ACKing
• downstream dependency is slow or down
• poison messages are repeatedly failing
• claim/retry logic is missing
• scaling is insufficient

Never deploy Redis Streams without a PEL monitoring strategy.

10. Retry and Claim Stale Messages

If a worker receives an entry and dies before ACK, the entry remains pending.

Another worker can claim it after an idle threshold.

Typical pattern:

1. Worker A reads entry.
2. Worker A crashes.
3. Entry remains pending for Worker A.
4. Worker B scans pending entries.
5. Worker B claims stale entry after idle threshold.
6. Worker B processes entry.
7. Worker B ACKs entry.

Redis provides XPENDING, XCLAIM, and XAUTOCLAIM for this flow.

The idle threshold must be longer than normal processing time.

If the threshold is too short, healthy long-running jobs can be claimed by another worker and processed twice.

If the threshold is too long, recovery after worker crash is slow.

This threshold is a business and operational decision, not merely a Redis setting.

11. Retry Count and Poison Messages

A poison message is an entry that repeatedly fails processing.

Examples:

• malformed payload
• missing required DB row
• incompatible schema version
• external dependency always rejects the job
• business invariant violation
• serialization bug
• tenant configuration issue

A stream worker should not retry poison messages forever.

A safe pattern:

read entry
  -> process
  -> if success: XACK
  -> if transient failure: leave pending or retry later
  -> if retry count exceeded: copy to DLQ-like stream and XACK original

Redis Streams do not automatically provide DLQ behavior.

You design it.

A DLQ-like stream can contain:

• original stream key
• original stream ID
• consumer group
• payload
• failure reason
• stack/error code
• retry count
• first failure timestamp
• last failure timestamp
• correlation ID
• tenant ID if safe and allowed

Do not store sensitive payloads in DLQ without privacy review.

12. DLQ-like Stream Pattern

A DLQ-like Redis Stream is a separate stream for failed messages.

Example naming pattern:

stream:{service}:{event}:dlq

Example:

stream:quote-order:quote-recalculation:dlq

The worker can write failed entry metadata to the DLQ stream before acknowledging the original entry.

Critical sequence:

1. Worker determines message is no longer retryable.
2. Worker writes failure record to DLQ stream.
3. Worker ACKs original message.

Partial failure still exists.

If DLQ write succeeds but ACK fails, message may be retried and duplicated in DLQ.

Therefore DLQ entries should include original stream ID and group so duplicate DLQ records can be identified.

For critical workflows, use database-backed failure tracking.

13. Backpressure

Redis Streams can accumulate entries faster than workers process them.

Backpressure symptoms:

• stream length grows
• consumer lag grows
• PEL grows
• worker CPU increases
• Redis memory increases
• processing latency increases
• downstream DB/API receives too much traffic

Backpressure strategy should answer:

• Can producers slow down?
• Can workers scale horizontally?
• Is downstream dependency the bottleneck?
• Should jobs be dropped, delayed, or rejected?
• Is queue length bounded?
• What is the customer impact if processing lags?

Redis itself will accept writes until memory or policy limits are reached.

That does not mean the system is healthy.

Queue growth is deferred pain.

14. Stream Trimming and Retention

Streams grow unless trimmed.

Trimming can be length-based or strategy-based.

Example command concept:

XADD stream:quote:events MAXLEN ~ 100000 * type QuoteUpdated quoteId Q-123

The ~ form is approximate trimming.

Approximate trimming is often faster and sufficient for many operational streams.

Retention must align with replay and retry needs.

Bad retention examples:

• trim so aggressively that pending/retry diagnostics become impossible
• retain unbounded streams until Redis memory pressure occurs
• use the same retention policy for all tenants and workloads without traffic analysis
• store sensitive data longer than privacy policy allows

The stream retention question is:

How long must this entry remain available, and why?

If nobody can answer, the retention policy is not designed.

15. Replay Semantics

Streams can support replay by reading from an earlier ID.

But replay is not free.

Replay can:

• duplicate side effects
• reapply old cache updates
• overwrite newer projections with stale data
• overload downstream systems
• violate idempotency assumptions
• expose old payloads longer than expected

A replay-safe consumer should be idempotent.

For cache projection, include version checks.

For database updates, include compare-and-set or unique constraints where appropriate.

For external side effects, record idempotency keys.

Replay is useful only if processing is safe to repeat.

16. Ordering

Redis Streams preserve order within a stream.

Consumer groups distribute entries across consumers.

That means processing completion order may differ from stream order.

If strict per-entity ordering is required, design explicitly.

Options:

• one stream per ordering key category
• route same entity to same stream/key
• use entity version checks
• process conflicting entities serially
• use DB constraints to reject stale updates
• avoid parallel workers for strict ordering workloads

Do not assume that multiple consumers preserve business ordering.

They preserve work distribution.

Business ordering is your responsibility.

17. Consumer Scaling

Adding more consumers can increase throughput only when the bottleneck is worker capacity.

It will not help if the bottleneck is:

• Redis command latency
• PostgreSQL contention
• external API rate limit
• serialization cost
• hot key updates
• stream single-key bottleneck
• network saturation
• poison messages blocking progress

Scaling workers also increases:

• Redis connections
• downstream DB/API pressure
• duplicate-processing risk under retries
• operational noise
• concurrency bugs

Worker scaling must be paired with downstream capacity and idempotency design.

18. Java Worker Implementation Notes

In Java services, stream consumers are usually implemented as background workers, scheduled loops, or managed executor tasks.

Important design points:

• separate HTTP request handling from worker thread pools
• use bounded executor queues
• set Redis command timeouts
• set processing timeouts
• expose worker metrics
• handle graceful shutdown
• do not block application startup forever waiting for Redis
• avoid unbounded per-message thread creation
• preserve correlation IDs where possible
• deserialize payload defensively
• validate schema version before processing

A simple worker loop is easy to write.

A production worker loop is a reliability component.

19. Graceful Shutdown

In Kubernetes, pods are terminated during rollout, scaling, node drain, or failure.

Stream workers should handle shutdown explicitly.

Good shutdown behavior:

1. Stop reading new entries.
2. Finish in-flight jobs if possible within grace period.
3. ACK completed jobs.
4. Leave unfinished jobs pending for later claim.
5. Close Redis client cleanly.

Bad shutdown behavior:

• keep reading until process is killed
• ACK before processing completes
• drop in-memory jobs
• kill worker thread without visibility
• extend job beyond termination grace period without recovery plan

Graceful shutdown affects duplicate processing and recovery latency.

20. PostgreSQL/MyBatis/JDBC Interaction

A Redis Stream worker often reads a message and then accesses PostgreSQL.

Key concerns:

• message may reference a row that was rolled back or deleted
• message may arrive before database replica catches up
• worker may process stale entity version
• DB transaction may commit but XACK may fail
• XACK may succeed but DB update may fail if ordered incorrectly
• retry may repeat DB writes
• lock contention may appear when worker concurrency increases

For MyBatis/JDBC code:

• define transaction boundary clearly
• use idempotent writes where possible
• use unique constraints for deduplication when needed
• avoid long transactions inside worker loops
• expose DB error category to retry logic
• distinguish transient DB failure from permanent business failure

Redis Streams coordinate work.

PostgreSQL should remain the durable source of truth for durable business state.

21. Kafka/RabbitMQ Interaction

Redis Streams may coexist with Kafka and RabbitMQ.

Common patterns:

If Kafka/RabbitMQ is already the source of events, do not duplicate semantics casually in Redis.

Redis may be a projection or local work queue.

It should not silently become a second broker with different guarantees.

22. Kubernetes and Cloud Deployment Concerns

Redis Streams in Kubernetes/cloud-managed Redis depend on deployment characteristics.

Review:

• Redis persistence configuration
• failover behavior
• cluster mode compatibility
• memory limit and stream growth
• network latency between workers and Redis
• pod restart behavior
• worker replica count
• total Redis client connections
• CPU throttling on workers
• node drain and termination grace period
• cloud maintenance window
• backup and restore behavior

A stream queue is not just application code.

It is application code plus Redis deployment plus worker runtime plus downstream dependencies.

23. Stream Operational Checklist

Track at minimum:

• stream length
• entries added per second
• entries consumed per second
• consumer lag
• pending entry count
• oldest pending age
• delivery count distribution
• failed job count
• DLQ-like stream length
• trim rate
• Redis memory usage
• Redis command latency
• worker processing latency
• worker error rate
• worker restarts
• downstream DB/API latency

For production systems, pending age is often more important than raw stream length.

A small PEL with very old entries can be worse than a large stream with active processing.

24. Common Failure Modes

25. Production-Safe Debugging Flow

When a stream worker seems stuck:

1. Check whether producers are still writing entries.
2. Check stream length and growth rate.
3. Check consumer group existence.
4. Check PEL count and oldest pending age.
5. Check worker logs by consumer name.
6. Check downstream dependency latency/error rate.
7. Check retry and claim behavior.
8. Check DLQ-like stream.
9. Check Redis latency and memory.
10. Check recent deployment or schema changes.

Avoid using broad expensive key scans in production.

Avoid dumping large stream payloads without privacy review.

Prefer targeted inspection by stream key and group.

26. PR Review Checklist

Ask these questions in review:

• Why Redis Streams instead of Kafka, RabbitMQ, database outbox, or list queue?
• What is the source of truth?
• What is the stream key naming convention?
• What is the retention/trimming policy?
• What is the consumer group name?
• How are consumer names generated?
• When does the worker ACK?
• Is processing idempotent?
• What happens if processing succeeds but ACK fails?
• What happens if worker crashes before ACK?
• How are stale pending messages claimed?
• What is the retry limit?
• What is the poison message strategy?
• Is there a DLQ-like stream or failure table?
• Are payloads versioned?
• Are sensitive fields avoided or protected?
• Are metrics and alerts defined?
• Is graceful shutdown handled?
• Does scaling workers overload PostgreSQL or external systems?
• Is the design compatible with Redis Cluster if used?

27. Internal Verification Checklist

Verify these inside the actual team/codebase:

• Redis Streams usage exists or not.
• Stream key naming convention.
• Consumer group naming convention.
• Consumer naming strategy in Kubernetes.
• Java client API used for XADD, XREADGROUP, XACK, XPENDING, XCLAIM, or XAUTOCLAIM.
• Worker thread pool configuration.
• Worker graceful shutdown behavior.
• ACK point in code.
• Retry count policy.
• Claim stale message threshold.
• Poison message and DLQ-like handling.
• Stream trimming/retention policy.
• Payload schema/versioning.
• PII/sensitive data inside stream payload.
• Metrics for stream length, pending count, lag, and oldest pending age.
• Alerting thresholds.
• Redis persistence/replication/failover configuration.
• Known incident notes involving stream workers.
• Ownership boundary between backend team, platform team, and SRE.

28. Key Takeaways

Redis Streams are useful for bounded, Redis-native asynchronous processing.

They are stronger than Pub/Sub because entries can be retained and acknowledged.

They are stronger than simple list queues because consumer groups track pending work.

They are not a complete replacement for Kafka, RabbitMQ, database outbox, or workflow engines.

For production use, the hard parts are not XADD and XREADGROUP.

The hard parts are:

• ACK discipline
• retry policy
• poison message handling
• retention
• replay safety
• idempotency
• ordering
• backpressure
• observability
• failover behavior
• privacy review
• operational ownership

A Redis Stream design is not complete until it explains what happens when workers crash, Redis fails over, messages are duplicated, payloads are invalid, and downstream systems are slow.