cache latency naik -> app thread menunggu -> connection pool penuh
cache miss naik    -> database QPS naik -> DB latency naik
cache failover     -> client reconnect storm -> CPU/network spike
cache eviction     -> hit rate turun -> source of truth dihantam
hot key            -> satu shard/node bottleneck meskipun cluster besar

A cache incident is often a database incident in disguise.

CPUUtilization high      -> host is busy.
EngineCPUUtilization high -> Redis/Valkey command execution is the bottleneck.

1. User-facing latency/error
2. Cache latency and command error
3. Hit rate / miss rate
4. Evictions and memory usage
5. Engine CPU and host CPU
6. Connections and new connections
7. Network throughput
8. Replication lag / failover events
9. DB QPS and DB latency
10. Top tenant/key family approximation from app telemetry

Cache data can be evicted.
Coordination state should be protected or moved to durable store.

Is memory growth proportional to business growth, traffic spike, deployment bug, or missing TTL?

1. Did deployment change key schema or TTL?
2. Did hit rate fall before memory rose?
3. Did evictions start after a traffic event?
4. Is memory concentrated in one shard/node?
5. Did value size change?
6. Did background warming/backfill run?

long baseTtlSeconds = 300;
long jitter = ThreadLocalRandom.current().nextLong(0, 60);
long ttl = baseTtlSeconds + jitter;

{
  "payload": { "caseId": "C-123", "status": "OPEN" },
  "freshUntil": "2026-07-07T10:05:00Z",
  "serveUntil": "2026-07-07T10:10:00Z",
  "version": 17
}

now < freshUntil -> serve
freshUntil <= now < serveUntil -> serve stale and trigger async refresh
now >= serveUntil -> block/rebuild or fail based on route policy

authorization decision
case finalization state
financial balance after transaction
legal deadline calculation

catalog display
dashboard counters
non-critical profile rendering
search suggestion metadata

cluster has capacity but one node/shard is overloaded
EngineCPUUtilization high on one node
network out high on one node
latency spikes for one route/tenant
DB is fine but cache latency high

counter:{name}:shard:0
counter:{name}:shard:1
...
counter:{name}:shard:N

more network bytes
longer serialization/deserialization
higher memory fragmentation risk
slower replication
larger blast radius per miss

Cache the view you need, not the entire aggregate graph.
Prefer stable schema and compact payload.
Avoid storing huge arrays that change partially.
Consider compression only after measuring CPU trade-off.
Split hot fields from cold fields.

cache key: case:{id}
value: entire case graph + documents + comments + permissions + history

case:{id}:summary
case:{id}:timeline:page:{n}
case:{id}:permission-epoch
case:{id}:latest-action

[ ] Use bounded connection pool.
[ ] Set connect timeout and command timeout.
[ ] Retry only safe commands.
[ ] Add exponential backoff with jitter.
[ ] Avoid unbounded async concurrency.
[ ] Handle MOVED/ASK/topology change if cluster mode enabled.
[ ] Avoid DNS cache settings that prevent failover discovery.
[ ] Emit client-side metrics, not only server metrics.

cache node failover -> clients disconnect -> all clients reconnect immediately -> new connections spike -> engine/host pressure -> command latency rises -> app retries -> worse storm

bounded pool + jittered reconnect + circuit breaker + local degraded mode

short connection errors
timeouts
retries
possible topology refresh
some in-flight commands ambiguous
read replica promotion behavior
client reconnect storm

CacheHitRate drops
DB QPS rises
API latency rises
Evictions may or may not rise

1. Check deployment timeline for key/TTL/schema change.
2. Check evictions and memory pressure.
3. Check invalidation event volume.
4. Check whether cache warming/backfill was disabled.
5. Check whether one tenant/key family caused churn.
6. Apply DB protection: throttle expensive routes, reduce worker concurrency, enable degraded cache path.
7. Restore previous key format or rollback deployment if needed.

Is this cache empty, invalidated, evicted, bypassed, or using the wrong key?

Evictions > 0 or sharply increasing
memory usage high
hit rate slowly or suddenly drops

1. Identify whether eviction policy is expected.
2. Check memory usage per node/shard, not only cluster aggregate.
3. Check key cardinality and value size change after deployment.
4. Reduce TTL for low-value keys only if memory is dominated by stale data.
5. Disable or slow warming/backfill jobs.
6. Scale node memory or shard count if growth is legitimate.
7. For session/coordination keys, move critical state out of eviction-prone cache or isolate cluster.

Eviction of cache-aside data is a performance problem.
Eviction of session/lock/rate state can be correctness or security problem.

one shard/node high CPU/network
route latency skewed
tenant-specific latency spike
command latency high despite normal aggregate capacity

1. Use app telemetry to identify route + cache key family.
2. Compare per-node EngineCPUUtilization and network metrics.
3. Check whether key is read-hot or write-hot.
4. For read-hot: local cache, key replication, CDN, precompute.
5. For write-hot: sharded counter, queue aggregation, rate limit, redesign key.
6. Add tenant-level protection if one tenant dominates.

miss spike followed by DB spike
same expensive query repeated
traffic aligned with TTL expiration
latency wave every N minutes

1. Add TTL jitter.
2. Add request coalescing for hot keys.
3. Enable stale-while-revalidate where safe.
4. Protect DB with bounded concurrency.
5. Warm critical keys gradually, not all at once.
6. Add per-key rebuild metrics.

CompletableFuture<Value> existing = inFlight.get(key);
if (existing != null) return existing;

CompletableFuture<Value> created = new CompletableFuture<>();
CompletableFuture<Value> previous = inFlight.putIfAbsent(key, created);
if (previous != null) return previous;

try {
  Value value = loadFromDb(key);
  cache.set(key, value, ttlWithJitter());
  created.complete(value);
  return created;
} catch (Throwable t) {
  created.completeExceptionally(t);
  throw t;
} finally {
  inFlight.remove(key);
}

user sees old state
cache value version older than DB version
invalidation worker lag
event replay happened

1. Compare source-of-truth version vs cached version.
2. Check invalidation event was emitted after DB commit.
3. Check EventBridge/SNS/SQS/Lambda delivery and DLQ.
4. Check out-of-order invalidation if multiple updates occurred.
5. Delete exact keys or bump namespace epoch.
6. Reconcile affected aggregates.
7. Add version guard to cache set to avoid stale overwrite.

Only write cache if incoming version >= cached version.

KEYS *
large SMEMBERS
large LRANGE
large HGETALL for huge hash
unbounded Lua scripts
blocking operations without isolation

SCAN with bounded operational tooling only
bounded page sizes
small hashes
purpose-specific keys
precomputed projections

Did we accidentally introduce an O(N) command into a hot request path?

Condition: CacheHitRate drops below baseline AND DB read QPS increases AND API latency increases
Action: investigate cache miss storm, protect DB

Condition: CPU > 80% for 1 minute
Action: panic

[ ] Cold cache startup at peak traffic.
[ ] Simultaneous TTL expiry for hot keys.
[ ] One tenant generates 10x traffic.
[ ] Cache unavailable for 2 minutes.
[ ] Node failover while traffic is high.
[ ] DB slower than usual during cache miss storm.
[ ] Invalidation worker DLQ grows.
[ ] Deployment changes key schema.
[ ] Large value accidentally cached.
[ ] Connection pool maxed out.

API p50/p95/p99
cache latency
hit rate
DB QPS/latency
connection count
evictions
engine CPU
error rate
recovery time

[ ] TLS enabled where required.
[ ] Auth/ACL/secrets managed and rotated.
[ ] Key/value does not contain unnecessary PII.
[ ] Logs do not print raw cache values.
[ ] Session keys are random and non-guessable.
[ ] Authorization cache has short TTL/epoch.
[ ] Data deletion policy includes cache invalidation.
[ ] Incident runbook includes forced namespace invalidation.
[ ] Cross-tenant key names cannot collide.

[ ] Every cache namespace has owner, source of truth, TTL, and staleness budget.
[ ] Critical coordination state is not mixed with best-effort read cache.
[ ] Eviction policy is intentional.
[ ] Memory usage, evictions, hit rate, engine CPU, connections, network, and replication lag are monitored.
[ ] Client has bounded connection pool.
[ ] Client has sane connect/read timeout.
[ ] Retries use exponential backoff with jitter.
[ ] Cache failover behavior is tested.
[ ] Cache unavailable path is defined: fail-open, fail-closed, or degraded.
[ ] Stampede protection exists for hot expensive keys.
[ ] TTL jitter exists for high-volume keys.
[ ] Hot key detection uses app telemetry, not only cache metrics.
[ ] Invalidation pipeline has DLQ, retry, and reconciliation.
[ ] Stale overwrite is prevented with version guard where needed.
[ ] Runbooks exist for hit-rate collapse, eviction spike, hot key, connection storm, stale data, and failover.