Part 050 — Redis On-Prem and Hybrid Deployment

On-prem Redis is not “Redis without cloud.”

It is Redis with more operational ownership.

In a cloud-managed service, the provider absorbs part of the infrastructure burden.

In on-prem or hybrid deployment, internal teams own far more of the failure surface:

• OS configuration.
• Kernel tuning.
• Filesystem behavior.
• Disk performance.
• Network path.
• Firewall rules.
• TLS certificates.
• Process supervision.
• Persistence configuration.
• Replication.
• Sentinel or Cluster topology.
• Backup and restore.
• Monitoring and alerting.
• Patch and upgrade lifecycle.
• Capacity planning.
• Incident response.
• Security hardening.

For enterprise Java/JAX-RS systems, this means Redis design must include both application semantics and operational readiness.

A correct Redis command is not enough.

A production-ready Redis deployment needs a credible operating model.

1. Mental Model: Self-Managed Redis Means Self-Managed Failure

Redis is simple to start.

redis-server

That simplicity is deceptive.

Production Redis requires decisions about:

• Memory sizing.
• Persistence durability.
• Eviction policy.
• Replication lag.
• Failover authority.
• Client reconnection.
• Network isolation.
• Backup safety.
• Access control.
• Command restriction.
• Upgrade compatibility.
• Observability.
• Runbook quality.

Self-managed Redis is viable only when these responsibilities have clear owners.

If no team owns them, the deployment is not production-ready.

2. On-Prem vs Hybrid: Different Risk Shapes

Hybrid Redis is often harder than either pure cloud or pure on-prem.

The network becomes part of the consistency model.

3. Responsibility Boundary

Before approving on-prem Redis, define ownership.

If all rows say “someone,” the system is unmanaged.

4. Reference Architecture: On-Prem Redis for Java Services

This is only one shape.

The actual architecture may be:

• Standalone Redis.
• Primary/replica without Sentinel.
• Sentinel-managed HA.
• Redis Cluster.
• Redis Enterprise or Redis-compatible platform.
• Kubernetes StatefulSet/operator.
• VM-based deployment.

The topology must be explicit.

5. OS and Kernel Considerations

Redis performance is tightly coupled to OS behavior.

On self-managed Redis, verify OS-level settings through platform/SRE standards.

Common areas:

• Overcommit memory behavior.
• Transparent Huge Pages.
• File descriptor limits.
• TCP backlog.
• TCP keepalive.
• Ephemeral port behavior.
• Swap policy.
• NUMA behavior on large machines.
• Time synchronization.
• Process limits.
• Disk scheduler for persistence-heavy workloads.

Why Backend Engineers Should Care

A Java service incident may surface as Redis timeout.

The actual cause may be:

• OS swapping.
• Kernel memory pressure.
• File descriptor exhaustion.
• Slow disk during AOF fsync.
• Network queue drops.
• CPU steal/noisy neighbor.

Even if platform owns the OS, backend engineers need enough model to ask the right questions.

6. Filesystem and Disk

Redis is in-memory, but persistence and recovery depend on disk.

Disk matters when using:

• RDB snapshots.
• AOF.
• AOF rewrite.
• Backups.
• Redis Streams or queues where recovery expectation exists.
• Large data sets with slow restart times.

Review:

• Is persistence enabled?
• Is disk sized for dataset plus rewrite overhead?
• Is disk latency monitored?
• Is filesystem full alert configured?
• Is backup path separate from primary disk?
• Is restore time acceptable?
• Does persistence create latency spikes?

Persistence Failure Chain

AOF enabled
  -> disk latency rises
  -> fsync slows
  -> Redis event loop stalls or latency spikes
  -> Java Redis commands timeout
  -> service retries
  -> Redis load increases
  -> customer-facing latency escalates

Do not debug this only at the Java layer.

7. Network and Firewall

On-prem Redis is often inside complex network zones.

Verify:

• Which subnets can access Redis.
• Which ports are open.
• Whether Sentinel/Cluster ports are reachable.
• Whether replicas can communicate.
• Whether client-to-node connectivity works after failover.
• Whether DNS/VIP points to the correct role.
• Whether TLS termination is direct or proxied.
• Whether cross-datacenter latency is acceptable.
• Whether firewall changes are change-controlled.

Cluster-Specific Network Warning

Redis Cluster clients need to reach the actual nodes returned by cluster topology discovery.

A single load-balanced endpoint is not enough if clients are redirected to node-specific addresses that are unreachable.

Symptoms:

• MOVED errors.
• Connection timeouts to private node IPs.
• Works from bastion but not application subnet.
• Works in one datacenter but not another.

Sentinel-Specific Network Warning

Sentinel-aware clients must reach Sentinel nodes and the Redis nodes they announce.

If Sentinel announces hostnames/IPs not reachable from Java pods, failover discovery breaks.

8. TLS and Certificate Management

On-prem TLS is often harder than cloud TLS.

Verify:

• Is Redis TLS enabled end-to-end?
• Is TLS terminated by Redis, proxy, or load balancer?
• Is mTLS required?
• Which CA signs the certificate?
• Does the Java trust store trust the CA?
• How are certificates rotated?
• Is certificate expiry monitored?
• Are hostnames stable across failover?
• Are Sentinel/Cluster endpoints TLS-compatible?
• Are non-TLS ports disabled?

Certificate Rotation Failure Mode

Certificate rotates
  -> JVM trust store not updated
  -> Redis SSL handshake fails
  -> all Redis commands fail
  -> cache fallback overloads DB
  -> application incident occurs

Certificate rotation must be tested like a production failover.

9. Authentication and ACL

Self-managed Redis must explicitly enforce access control.

Review:

• Is authentication enabled?
• Are ACL users used instead of one shared password?
• Are permissions scoped by command category?
• Are permissions scoped by key pattern?
• Are dangerous commands disabled or restricted?
• Are credentials rotated?
• Are credentials stored in Kubernetes Secret, Vault, or approved secret manager?
• Are credentials logged accidentally?
• Is break-glass access audited?

Dangerous Commands

Restrict access to commands such as:

FLUSHALL
FLUSHDB
CONFIG
SHUTDOWN
MODULE
EVAL if not approved
KEYS in production contexts
MONITOR in production contexts

The exact ACL policy depends on operational needs.

The anti-pattern is giving every application full administrative Redis access.

10. Sentinel Deployment

Sentinel provides monitoring and failover for primary/replica Redis.

It does not turn Redis into a strongly consistent database.

Review:

• Number of Sentinel nodes.
• Quorum.
• Placement across failure domains.
• Network reachability.
• Announced hostnames/IPs.
• Failover timeout.
• Client Sentinel support.
• Split-brain prevention.
• Runbook for manual intervention.

Sentinel Failure Modes

Java Client Checklist

• Is Lettuce/Jedis/Redisson configured for Sentinel?
• Are all Sentinel endpoints configured?
• Is master name correct?
• Are timeouts bounded?
• Does client refresh topology after failover?
• Has failover been tested from application pods?

11. Redis Cluster Deployment

Redis Cluster is for horizontal partitioning and HA per slot.

It adds application-visible constraints.

Review:

• Number of masters/shards.
• Replica per shard.
• Slot distribution.
• Node placement across failure domains.
• Client cluster mode.
• MOVED / ASK handling.
• Multi-key command compatibility.
• Hash tag strategy.
• Lua script compatibility.
• Resharding procedure.
• Hot slot monitoring.

Cluster Anti-Pattern

Enable Redis Cluster for scale
without reviewing key design
without reviewing multi-key commands
without testing Java client redirection

This creates production failures that look like random Redis errors but are actually architecture mismatch.

12. Redis on On-Prem Kubernetes

Running Redis in Kubernetes is possible.

It is not automatically easier.

Review:

• StatefulSet design.
• PersistentVolume and StorageClass.
• Pod anti-affinity.
• Pod disruption budget.
• Resource requests/limits.
• Liveness and readiness probes.
• Graceful termination.
• Backup sidecars/jobs.
• Operator maturity.
• Helm chart ownership.
• Network policy.
• Secret management.
• Upgrade procedure.
• Disaster recovery.

Kubernetes-Specific Failure Modes

For stateful Redis, Kubernetes primitives must be designed deliberately.

13. Backup and Restore

Backup is not successful until restore is tested.

Review:

• Backup frequency.
• Backup location.
• Encryption.
• Access control.
• Retention.
• Restore procedure.
• Restore time objective.
• Restore point objective.
• Privacy review.
• Data classification.
• Recovery testing cadence.

Redis Restore Questions

• Is Redis restored into the same environment or isolated environment first?
• Are old sessions/tokens safe to restore?
• Are old idempotency records safe to restore?
• Are queue jobs safe to replay?
• Are cached objects compatible with current Java class/schema version?
• Are TTLs preserved in a way that matches expectation?
• Is restore coordinated with PostgreSQL restore point?
• Is Kafka/RabbitMQ replay needed after restore?

Restoring Redis without coordinating source-of-truth state can create correctness incidents.

14. Monitoring Stack

On-prem Redis needs an explicit observability stack.

Common components:

• Redis exporter.
• Prometheus.
• Grafana.
• Log aggregation.
• Alertmanager or equivalent.
• Synthetic Redis checks.
• Java client metrics.
• Incident dashboard.

Track:

• Memory used.
• RSS and fragmentation.
• Evicted keys.
• Expired keys.
• Ops/sec.
• Command latency.
• Slowlog.
• Connected clients.
• Blocked clients.
• Rejected connections.
• Replication lag.
• Persistence status.
• AOF/RDB errors.
• Sentinel state.
• Cluster slot state.
• Keyspace hit/miss.
• Stream pending entries.

Alerting Principle

Alert on customer-impacting risk, not every interesting number.

Good alerts answer:

• Is Redis unavailable?
• Is Redis becoming slow?
• Is Redis evicting keys unexpectedly?
• Is memory near unsafe level?
• Is replication broken?
• Is failover happening?
• Are clients blocked or rejected?
• Are queues/streams stuck?

15. Patch Management and Upgrades

Self-managed Redis needs lifecycle management.

Review:

• Redis engine version.
• Security patch cadence.
• Upgrade path.
• Compatibility notes.
• Command behavior changes.
• Persistence format compatibility.
• Client compatibility.
• Module compatibility if any.
• Rollback plan.
• Test environment parity.
• Maintenance window.
• Communication plan.

Upgrade Risk Areas

• Java client protocol compatibility.
• ACL behavior.
• TLS defaults.
• Cluster behavior.
• Persistence loading.
• Lua script behavior.
• Module availability.
• Memory encoding changes.
• Deprecated commands.

An upgrade should include application integration tests, not only Redis process tests.

16. Air-Gapped Deployment

Air-gapped or restricted environments add extra constraints.

Verify:

• How Redis packages/images are obtained.
• How vulnerabilities are scanned.
• How patches are imported.
• How certificates are issued and rotated.
• How backups leave or stay inside the environment.
• How monitoring data is collected.
• How logs are reviewed.
• How support bundles are exported.
• How disaster recovery is tested.
• How Java dependencies are mirrored.

Air-Gapped Anti-Pattern

Redis image copied once
never patched
no vulnerability feed
no restore test
no certificate rotation process

Air-gapped does not mean safe.

It often means slow to patch unless the process is deliberate.

17. Hybrid Connectivity

Hybrid Redis requires network reliability to be part of the design.

Review:

• Connectivity path: VPN, ExpressRoute, private link, leased line, internal WAN, or other.
• Round-trip latency.
• Packet loss.
• Bandwidth.
• DNS resolution.
• Firewall statefulness.
• Failover route.
• Regional dependency.
• Maintenance ownership.
• Security inspection devices.

Hybrid Failure Pattern

Cloud Java service depends on on-prem Redis
hybrid link degrades
Redis commands timeout
service retries
thread pool saturates
DB fallback over cloud-to-on-prem also slows
customer APIs fail

Hybrid Redis should be treated as a high-risk synchronous dependency unless latency and partition behavior are proven.

18. Java/JAX-RS Client Design for On-Prem Redis

In self-managed environments, Java client behavior must compensate for less provider abstraction.

Verify:

• Explicit topology mode: standalone, Sentinel, Cluster.
• Bounded connect timeout.
• Bounded command timeout.
• Bounded retry.
• Jittered reconnect.
• Pool sizing.
• TLS trust store.
• ACL credentials.
• Metrics export.
• Graceful shutdown.
• Circuit breaker.
• Bulkhead for Redis-heavy paths.
• Fallback semantics.

Bad Pattern

// Anti-pattern: per-request Redis client construction
public Response getQuote(String id) {
    RedisClient client = RedisClient.create(redisUri);
    // use client
}

Why bad:

• Connection storm.
• TLS/auth overhead per request.
• Resource leak risk.
• Latency amplification.
• Hard to observe.

Better:

• Create managed singleton/client pool.
• Inject into service layer.
• Export metrics.
• Close during graceful shutdown.

19. PostgreSQL, Kafka, and RabbitMQ in Hybrid Context

Redis rarely fails alone.

In enterprise systems, Redis interacts with:

• PostgreSQL for source-of-truth state.
• MyBatis/JDBC transaction boundaries.
• Kafka/RabbitMQ for invalidation or projection.
• Java service retries and fallback logic.

Hybrid topology complicates these interactions.

Review:

• Are Redis and PostgreSQL in the same datacenter?
• Is Redis close to Java service but far from DB?
• Are invalidation consumers in cloud or on-prem?
• Can broker lag create stale Redis data?
• Does Redis outage redirect load across network boundary?
• Are transaction/outbox patterns region-aware?
• Does cache warming cross hybrid links?

Consistency Risk Example

Order update commits in on-prem PostgreSQL
Kafka event travels to cloud consumer
consumer updates cloud-side Redis
hybrid link delays event
cloud JAX-RS service reads stale Redis projection
customer sees old order status

This is not a Redis bug.

It is a distributed consistency design issue.

20. Capacity Planning

Self-managed Redis capacity planning must include more than dataset size.

Estimate:

• Key count.
• Value size.
• Object overhead.
• TTL distribution.
• Expiration rate.
• Eviction policy.
• Peak QPS.
• Command mix.
• Pipeline size.
• Connection count.
• Replication buffer.
• Persistence rewrite overhead.
• Fragmentation headroom.
• Growth per tenant/customer.
• Failover capacity.

Memory Headroom Rule

Do not size Redis at 95% memory usage.

Leave room for:

• Fragmentation.
• Replication buffers.
• AOF rewrite.
• Traffic bursts.
• Hot tenant growth.
• Temporary key spikes.
• Operational diagnostics.

Exact headroom depends on workload and platform standards.

The important point is that maxmemory is not the same as safe usable memory.

21. Performance Engineering On-Prem

On-prem Redis performance depends on the full path:

Java thread
  -> client pool
  -> TLS/auth
  -> network
  -> Redis event loop
  -> memory allocator
  -> persistence/replication
  -> response network path
  -> Java deserialization

Review performance at each layer:

• Command complexity.
• Payload size.
• Serialization cost.
• Pipeline/batching.
• Connection reuse.
• Network RTT.
• CPU saturation.
• Memory fragmentation.
• Disk latency for persistence.
• Replication lag.
• Slowlog.
• Client-side timeout.

Production Debugging Rule

If Java sees Redis timeout, do not immediately increase timeout.

First determine whether the cause is:

• Slow command.
• Big key.
• Hot key.
• Network latency.
• TLS handshake.
• Pool exhaustion.
• Redis CPU.
• Persistence disk stall.
• Replication/failover.
• Packet loss.
• Firewall/proxy issue.

Increasing timeout can hide a worsening incident.

22. Security and Compliance

On-prem Redis often sits closer to regulated data.

Review:

• Network segmentation.
• Firewall allowlist.
• TLS/mTLS.
• ACL per service.
• Dangerous command restriction.
• Secret rotation.
• Break-glass audit.
• Backup encryption.
• Snapshot access control.
• PII in keys/values.
• Token/session data.
• Log redaction.
• Data retention.
• Data deletion.
• Multi-tenant isolation.

Key Name Privacy

Bad:

customer-cache:john.doe@example.com:quote:Q-1001

Better:

quote-cache:tenant:t-001:customer-id:c-9f31:quote:Q-1001

Even better if customer identifiers are internal opaque IDs and documented as non-sensitive.

Redis key names are often visible in logs, diagnostics, metrics, and backups.

Treat them as data.

23. Disaster Recovery

DR for Redis depends on Redis use case.

For cache-only Redis:

• Rebuild may be enough.
• Backup may be unnecessary.
• Cache warming may be needed.
• DB protection is critical during rebuild.

For session Redis:

• Users may be logged out.
• Security team must approve behavior.
• Token/session consistency matters.

For idempotency Redis:

• Duplicate processing risk exists.
• PostgreSQL reconciliation may be required.

For queue/stream Redis:

• Job replay/loss risk exists.
• DLQ/retry model must be explicit.
• Kafka/RabbitMQ may be better source of durable work.

DR Checklist

• RTO and RPO defined.
• Restore tested.
• Cache warming plan exists.
• DB fallback capacity validated.
• Keyspace privacy reviewed.
• Runbook tested.
• Business impact documented.
• Application behavior after Redis loss is known.

24. Operational Runbook Requirements

An on-prem Redis deployment should have runbooks for:

• Redis unavailable.
• High latency.
• Memory pressure.
• Eviction spike.
• Expired key spike.
• Big key.
• Hot key.
• Connection storm.
• Blocked clients.
• Slowlog spike.
• Replication lag.
• Sentinel failover.
• Cluster slot issue.
• Disk full.
• AOF/RDB failure.
• TLS certificate expiry.
• Auth failure.
• Backup failure.
• Restore procedure.
• Patch/upgrade rollback.

Minimum Runbook Shape

Each runbook should include:

• Symptoms.
• Customer impact.
• Metrics to inspect.
• Safe commands.
• Dangerous commands to avoid.
• Escalation path.
• Mitigation steps.
• Recovery validation.
• Post-incident evidence.

25. Internal Verification Checklist

Deployment

• Redis engine/version.
• Redis-compatible distribution if not Redis OSS.
• VM, bare metal, Kubernetes, or appliance/platform.
• Standalone, Sentinel, Cluster, or vendor HA.
• Node placement and failure domains.
• Resource sizing.
• Persistence mode.
• Backup location.
• Restore test result.

OS / Runtime

• File descriptors.
• Memory overcommit.
• THP setting.
• Swap policy.
• CPU allocation.
• Disk latency.
• Time sync.
• Process supervisor.
• Log rotation.

Network / Security

• Firewall rules.
• DNS/VIP behavior.
• TLS/mTLS.
• Certificate rotation.
• ACL users.
• Secret storage.
• Dangerous command restrictions.
• Break-glass audit.

Java Application

• Client topology mode.
• Timeout settings.
• Retry settings.
• Pool settings.
• TLS trust store.
• Auth credentials.
• Metrics.
• Circuit breaker.
• Fallback behavior.
• Graceful shutdown.

Redis Usage

• Key naming.
• TTL policy.
• Eviction policy.
• Cache invalidation.
• Rate limiting.
• Idempotency.
• Distributed lock.
• Streams/queues.
• Pub/Sub.
• Session/token data.
• Privacy classification.

Operations

• Dashboard.
• Alerts.
• Runbook.
• Incident history.
• Patch schedule.
• Upgrade procedure.
• DR test.
• Owner/escalation matrix.

26. PR Review Checklist for On-Prem/Hybrid Redis Usage

When a PR introduces or changes Redis usage in on-prem/hybrid environments, ask:

1. Is the target Redis topology known?
2. Does the key design match Cluster/Sentinel/standalone behavior?
3. Are TTLs explicit?
4. Is eviction acceptable?
5. Does Redis store sensitive data?
6. Is the Java client configured for TLS/auth/topology?
7. Are timeouts shorter than request budget?
8. Are retries bounded?
9. Does fallback overload PostgreSQL?
10. Does cache invalidation cross hybrid boundaries?
11. Are broker events idempotent and ordered enough?
12. Does lock usage require fencing?
13. Is idempotency state durable enough?
14. Are metrics and alerts added?
15. Is there a runbook update?
16. Are platform/SRE/security teams required reviewers?

A Redis PR is not only a code change.

It is often a distributed-systems contract change.

27. Architecture Decision Questions

Before approving self-managed or hybrid Redis, ask:

1. Why self-managed Redis instead of managed Redis?
2. Who owns Redis operations?
3. What is the topology?
4. What is the HA/failover mechanism?
5. What is the persistence policy?
6. What is the backup/restore plan?
7. What is the patch/upgrade plan?
8. What is the network path from Java service to Redis?
9. What happens during network partition?
10. What happens during Redis failover?
11. What happens during certificate expiry/rotation?
12. What happens if Redis loses data?
13. Can the Java service degrade safely?
14. Can PostgreSQL absorb fallback traffic?
15. Are Kafka/RabbitMQ interactions replay-safe?
16. Is privacy/compliance satisfied?
17. Are observability and runbooks production-ready?

If the answers are vague, the deployment is not ready.

28. Senior Engineer Heuristics

Self-managed Redis is reasonable when:

• There is a clear operational owner.
• The team has Redis production experience.
• Topology and failover are tested.
• Monitoring and runbooks are mature.
• Backup/restore is tested if needed.
• Security hardening is enforced.
• Java clients are configured for the topology.
• Application semantics tolerate Redis failure modes.

Self-managed Redis is risky when:

• It exists because “Redis is easy to install.”
• Nobody owns patching.
• Sentinel/Cluster behavior is untested.
• Backups exist but restore is untested.
• TLS/cert rotation is manual and untested.
• Java clients use default timeouts/retries.
• Redis is treated as durable source of truth without design.
• Hybrid latency is assumed, not measured.
• No one knows what happens if Redis is down.

29. Summary

On-prem and hybrid Redis deployment is fundamentally an ownership problem.

The Redis command model is the same, but the production burden is larger:

• OS.
• Network.
• Disk.
• TLS.
• ACL.
• HA.
• Failover.
• Backup.
• Restore.
• Patch.
• Upgrade.
• Monitoring.
• Incident response.

For Java/JAX-RS enterprise systems, the application must still handle Redis as a fallible dependency with bounded timeouts, explicit fallback behavior, clear TTL policy, and observable failure modes.

A senior engineer should approve self-managed or hybrid Redis only when the team can explain both:

• How Redis is used by the application.
• How Redis is operated when things go wrong.

If either side is missing, Redis is not production-ready.