Part 024 — Redis as Job Queue

Redis can be used as a job queue.

But Redis is not one queue abstraction.

It offers primitives that can be assembled into queue-like behavior.

The three most common building blocks are:

List       -> simple FIFO/LIFO queue
Stream     -> retained queue with consumer groups and ACK
Sorted Set -> delayed/scheduled queue using score as time or priority

A production job queue is not just a data structure.

It is a system with producers, workers, retries, idempotency, visibility, observability, failure handling, and operational ownership.

The key question is not:

Can Redis enqueue a job?

The real question is:

Can this Redis-based queue preserve the business correctness we need when workers crash, retries happen, Redis fails over, and downstream systems are slow?

1. Job Queue Mental Model

A job queue separates request acceptance from work execution.

Typical flow:

HTTP request
  -> validate command
  -> persist durable state if needed
  -> enqueue job
  -> return response
  -> worker processes job asynchronously

This helps when work is:

• slow
• retryable
• non-interactive
• bursty
• dependent on external systems
• safe to perform asynchronously
• better handled by worker pools

It is risky when the work is:

• part of a transaction that must complete before response
• impossible to retry safely
• externally visible and non-idempotent
• legally/audit critical without durable tracking
• dependent on strict global ordering

A job queue trades synchronous latency for asynchronous complexity.

Redis makes enqueue/dequeue fast.

It does not eliminate correctness design.

2. Choosing the Redis Queue Primitive

Use the primitive based on behavior needed.

A mature Redis job queue may combine primitives.

Example:

Sorted Set stores delayed jobs by due time.
Scheduler moves due jobs to Stream.
Workers process Stream with consumer group.
Failures go to DLQ-like Stream or DB table.

That is already a small queueing system.

Treat it as production infrastructure.

3. Job Message Design

A job message should contain enough information to process safely, but not so much that Redis becomes a sensitive data dump.

Typical fields:

jobId
jobType
tenantId
entityId
entityVersion
requestedBy
correlationId
createdAt
notBefore
attempt
maxAttempts
payloadVersion
payloadRef or small payload
idempotencyKey

Prefer storing large or sensitive payloads in PostgreSQL/object storage and putting a reference in Redis.

Bad job payloads:

• full customer profile with PII
• huge serialized Java object
• unversioned JSON with no schema discipline
• payload requiring a specific Java class name
• sensitive token or credential
• entire quote/order object when only ID is needed

Good job payloads are small, versioned, traceable, and safe to retry.

4. Producer Lifecycle in Java/JAX-RS

In a Java/JAX-RS service, a producer often runs inside an HTTP request path.

Example lifecycle:

JAX-RS resource
  -> parse request
  -> validate input
  -> authorize actor
  -> call service layer
  -> commit PostgreSQL transaction if needed
  -> enqueue Redis job
  -> return 202 Accepted or domain response

Important questions:

• Is the job enqueue part of a DB transaction?
• What happens if DB commit succeeds but enqueue fails?
• What happens if enqueue succeeds but response times out?
• Is the HTTP response allowed before job completion?
• Does the client need a job status endpoint?
• Is the operation idempotent if the client retries?

For business-critical workflows, do not rely on "DB commit then Redis enqueue" without thinking through partial failure.

A PostgreSQL outbox may be safer when enqueue must not be lost.

5. Worker Lifecycle

A worker is a long-running component that repeatedly claims work, processes it, and records completion.

General lifecycle:

start worker
  -> initialize Redis client
  -> register metrics
  -> read job
  -> validate payload
  -> check cancellation/idempotency
  -> process job
  -> update status/result
  -> ACK/remove job
  -> repeat

A worker must distinguish:

• success
• transient failure
• permanent failure
• duplicate job
• cancelled job
• stale job
• malformed job
• dependency unavailable
• shutdown requested

If all exceptions are treated the same, retry behavior will be wrong.

6. List-Based Queue

Redis Lists can implement simple queues.

Basic FIFO pattern:

LPUSH queue:quote:jobs <job-json>
BRPOP queue:quote:jobs 5

This is simple and fast.

But with BRPOP, once a worker pops a job, Redis no longer tracks it.

If the worker crashes after popping and before processing completes, the job is lost unless the application implements a reliability pattern.

List queues are acceptable for:

• low-criticality work
• best-effort background tasks
• jobs that can be recreated
• jobs whose source of truth is elsewhere
• simple internal workloads with known tolerance

They are dangerous for critical work without additional design.

7. Reliable List Queue Pattern

A more reliable list pattern moves a job from a ready queue to a processing queue atomically.

Conceptually:

LMOVE queue:ready queue:processing RIGHT LEFT

Or blocking equivalent:

BLMOVE queue:ready queue:processing RIGHT LEFT 5

Lifecycle:

1. Producer pushes job to ready queue.
2. Worker atomically moves job to processing queue.
3. Worker processes job.
4. Worker removes job from processing queue after success.
5. Reaper finds old processing jobs and retries them.

This mimics visibility timeout behavior.

But Redis Lists do not track attempt count, owner, idle time, or pending state automatically.

You must add metadata elsewhere.

If you need this complexity, Redis Streams may be a better fit.

8. Stream-Based Queue

Redis Streams are usually better than Lists when the job queue needs:

• consumer groups
• pending tracking
• ACK
• claim stale work
• delivery count
• replay of recent entries
• multiple independent consumers
• operational visibility into in-flight messages

Stream lifecycle:

XADD job stream
XREADGROUP by worker
process job
XACK after success
claim stale pending entries if worker dies
move poison entries to DLQ-like stream
trim stream based on retention policy

Streams are not automatically correct.

They simply expose more state so the application can be more correct.

9. Sorted-Set Delayed Queue

Sorted Sets are useful for delayed jobs because score can represent due time.

Example mental model:

key:   zset:quote:delayed-jobs
score: due timestamp in milliseconds
value: job id or compact job envelope

Scheduler loop:

1. Read due jobs where score <= now.
2. Atomically claim due jobs.
3. Move claimed jobs to ready queue or stream.
4. Remove from delayed set.

This pattern needs careful atomicity.

If multiple scheduler pods run concurrently, the same delayed job can be moved twice unless claiming is atomic.

Lua scripting is often used for safe claim-and-remove.

Delayed queue design must answer:

• what clock is used?
• how precise must scheduling be?
• can jobs run late?
• can jobs run early?
• can duplicate dispatch happen?
• how are cancelled delayed jobs handled?
• how is sorted-set growth controlled?

10. Priority Jobs

Priority can be implemented in several ways.

Options:

Priority should not be added casually.

It changes fairness and operational behavior.

Questions to ask:

• Can low-priority jobs starve?
• Is priority tenant-aware?
• Can one tenant flood high-priority queue?
• Are priority changes audited?
• Can priority be abused by clients?

For enterprise systems, priority often intersects with customer tiers and operational policy.

That needs governance.

11. Retry Jobs

Retry is where many job queues become dangerous.

A retry policy should define:

• retryable error categories
• non-retryable error categories
• max attempts
• backoff strategy
• jitter
• retry delay cap
• poison job behavior
• idempotency requirement
• observability

Example classification:

Retry without classification becomes an incident amplifier.

A permanently bad job can consume worker capacity forever.

12. Poison Jobs

A poison job repeatedly fails and cannot complete through retry.

A good poison strategy includes:

• max attempt count
• failure reason capture
• DLQ-like destination
• alerting
• manual inspection path
• replay/requeue procedure
• privacy-safe payload handling
• owner assignment

Do not silently drop poison jobs unless the business explicitly accepts loss.

Do not retry poison jobs forever.

Do not keep poison jobs only in logs.

Logs are not a queue state store.

13. Idempotent Jobs

Every Redis job that can be retried should be idempotent or protected by an idempotency record.

Duplicate processing can happen because:

• worker crashes after side effect before ACK
• claim threshold is too low
• Redis failover loses recent ACK/write
• producer retries enqueue
• HTTP client retries request
• scheduled job is inserted twice
• manual replay occurs

Strategies:

• job ID uniqueness
• idempotency key in Redis or PostgreSQL
• database unique constraint
• compare-and-set on entity version
• external idempotency key for external API calls
• status machine with terminal state

Idempotency should be designed before retry is enabled.

Retry without idempotency is gambling.

14. Job Timeout

A job timeout defines how long a job may run before it is considered unhealthy or reclaimable.

Timeout types:

Timeouts must align.

If Redis claim timeout is shorter than normal processing time, jobs duplicate.

If dependency timeout is longer than Kubernetes termination grace period, shutdown is messy.

If business deadline is ignored, stale work may apply after it is no longer valid.

15. Job Progress and Status

For user-visible asynchronous work, a job status model may be required.

Example status machine:

SUBMITTED
QUEUED
RUNNING
SUCCEEDED
FAILED_RETRYABLE
FAILED_TERMINAL
CANCELLED
EXPIRED
UNKNOWN

Where should status live?

• Redis if status is ephemeral and operational
• PostgreSQL if status is user-visible, auditable, or durable
• both if Redis accelerates status reads but DB is source of truth

Do not store critical job status only in Redis unless data loss is acceptable.

For quote/order workflows, durable business status usually belongs in PostgreSQL or the domain system of record.

Redis can accelerate or coordinate.

16. Job Cancellation

Cancellation is not automatic.

A worker must check cancellation at safe points.

Pattern:

cancel flag stored by jobId
worker checks before start
worker checks between major processing steps
worker stops if cancellation is safe
worker marks cancelled or leaves current side effect consistent

Cancellation is hard when:

• external calls are already made
• DB transaction is already committed
• job is inside non-interruptible computation
• worker crashes mid-cancel
• cancellation races with success

Redis can store a cancellation flag with TTL.

But durable cancellation for user-visible workflows should usually be reflected in PostgreSQL.

17. Queue Visibility Pattern

Broker systems often have visibility timeout: once a worker receives a message, it is hidden for a period; if not ACKed, it becomes visible again.

Redis does not provide one universal visibility abstraction.

Approximate options:

• List ready + processing queue + reaper
• Stream pending entries + claim stale messages
• Sorted set processing index by deadline
• DB-backed job status with worker lease

Streams provide the closest Redis-native mechanism because PEL tracks pending work.

Lists require more custom machinery.

Sorted sets are useful for deadline tracking.

The visibility model must be explicit in design docs.

18. Observability for Redis Job Queues

Track metrics at producer, Redis, worker, and downstream layers.

Producer metrics:

• enqueue success/failure
• enqueue latency
• jobs created by type
• duplicate enqueue attempts
• idempotency conflict count

Redis metrics:

• ready queue length
• stream length
• delayed set size
• processing queue length
• pending entry count
• oldest pending age
• memory usage
• command latency
• evictions

Worker metrics:

• jobs processed
• success rate
• failure rate by category
• retry count
• processing duration
• queue wait time
• active workers
• worker restarts
• graceful shutdown completion

Downstream metrics:

• PostgreSQL latency/error
• external API latency/error
• Kafka/RabbitMQ publish/consume latency if involved
• cache update latency

A queue without metrics is a hidden backlog.

19. Java/JAX-RS Response Strategy

When an HTTP request enqueues a job, the response should communicate the lifecycle honestly.

Common options:

Do not return success if the job was not durably accepted according to the business requirement.

If Redis enqueue failure means work is not accepted, respond accordingly.

If DB state already committed but Redis enqueue failed, the system needs reconciliation.

20. PostgreSQL Integration

Redis job queues often depend on PostgreSQL for durable state.

Patterns:

For critical workflows, prefer durable job state in PostgreSQL and use Redis as acceleration/coordination.

Redis-only queues can be acceptable for ephemeral operational jobs.

21. Kafka/RabbitMQ Comparison

Redis queue should be compared honestly with broker alternatives.

Choosing Redis because it is already available is not an architecture justification.

Choose Redis because the semantics match.

22. Kubernetes Worker Scaling

Redis job workers in Kubernetes need workload-level review.

Check:

• replica count
• connection pool per pod
• max concurrent jobs per pod
• CPU/memory requests and limits
• termination grace period
• readiness behavior
• liveness behavior
• startup behavior if Redis unavailable
• rolling update duplicate processing
• pod disruption budget
• horizontal pod autoscaling signal
• downstream dependency capacity

Scaling worker pods can overload PostgreSQL, external APIs, or Redis itself.

HPA based only on CPU may not reflect queue lag.

Queue-aware scaling may need backlog or oldest-job-age metrics.

23. Security and Privacy

Job payloads often contain business context.

Review carefully:

• PII in job payload
• PII in job key
• secrets in payload
• access tokens in payload
• tenant identifiers
• customer identifiers
• quote/order identifiers
• snapshot/backup exposure
• DLQ payload retention
• logs containing job payload
• worker error logs with sensitive data

For sensitive jobs, store only references in Redis and load details from the authorized source at processing time.

Redis speed is not a reason to duplicate sensitive data everywhere.

24. Common Failure Modes

25. Debugging Flow

When a Redis job queue misbehaves:

1. Identify queue primitive: list, stream, sorted set, or hybrid.
2. Identify job type and job ID.
3. Check producer enqueue logs and metrics.
4. Check Redis queue length or stream pending state.
5. Check worker logs by job ID/correlation ID.
6. Check retry count and failure category.
7. Check downstream dependency metrics.
8. Check idempotency record or DB status.
9. Check DLQ-like destination.
10. Check recent deployments and schema changes.
11. Check Redis memory/latency/eviction.
12. Check Kubernetes worker restarts and termination events.

Debug from lifecycle state, not from random Redis commands.

The key question is always:

Where is this job in its lifecycle right now?

26. Redis Job Queue PR Review Checklist

Ask these questions before approving:

• Why Redis queue instead of RabbitMQ, Kafka, PostgreSQL outbox, or workflow engine?
• Is the job critical or best-effort?
• What is the source of truth for job state?
• Which Redis primitive is used and why?
• Is the job payload small and versioned?
• Does payload contain PII or secrets?
• Is enqueue atomic enough relative to DB commit?
• What happens if enqueue fails?
• What happens if worker crashes after claiming job?
• What happens if processing succeeds but ACK/remove fails?
• Is job processing idempotent?
• How are retries classified?
• What is max attempt count?
• What is the poison job strategy?
• Is there a DLQ-like stream/table?
• How are delayed jobs claimed atomically?
• How is cancellation handled?
• How is progress/status exposed?
• What metrics and alerts exist?
• What is the retention/cleanup policy?
• Does worker scaling overload dependencies?
• Is graceful shutdown implemented?
• Is the design compatible with Redis Cluster if used?

27. Internal Verification Checklist

Verify these in the actual system:

• Whether Redis is used as job queue.
• Queue primitive: List, Stream, Sorted Set, or hybrid.
• Queue key naming convention.
• Producer code path in JAX-RS/service layer.
• DB transaction boundary relative to enqueue.
• Job payload schema and versioning.
• Payload size and sensitive data review.
• Worker implementation and thread pool.
• Worker concurrency per pod.
• Retry policy and max attempts.
• Poison job/DLQ-like handling.
• Idempotency key or DB uniqueness guard.
• Delayed job scheduler if any.
• Cancellation/status model if any.
• Queue length/backlog metrics.
• Oldest job age alert.
• Worker success/failure metrics.
• Redis memory/latency alerts.
• Kubernetes termination grace period.
• Runbook for stuck job, growing queue, and poison job.
• Ownership boundary between backend, platform, SRE, and security.

28. Key Takeaways

Redis can implement useful job queues, but the queue semantics depend on the chosen data structure.

Lists are simple but weak for reliability unless extra processing queues and reapers are added.

Streams are better for worker groups, ACK, retry visibility, and pending state.

Sorted Sets are useful for delayed or scheduled jobs.

A production Redis job queue needs:

• clear source of truth
• small versioned payloads
• idempotency
• retry classification
• poison job handling
• visibility/reclaim strategy
• metrics and alerts
• graceful shutdown
• retention cleanup
• privacy review
• downstream capacity control

The senior-engineer review question is not whether the Redis commands work.

The review question is whether the system still behaves correctly when the happy path breaks.