Cheatsheet Observability Part 035 — Redis Instrumentation

Fokus part ini: memahami bagaimana penggunaan Redis di aplikasi Java/JAX-RS harus diinstrumentasi agar cache behavior, command latency, key privacy, cache hit/miss, TTL, lock, idempotency, rate limiter, pipeline, Lua script, dan Redis Stream dapat dianalisis saat incident production. Redis instrumentation yang baik tidak hanya menjawab “Redis lambat atau tidak”, tetapi juga Redis dipakai untuk apa, apakah hasilnya benar, apakah key aman, dan apakah kegagalan Redis memperburuk customer impact.

1. Core Mental Model

Redis sering tampak sederhana karena dipakai sebagai cache atau key-value store. Di production, Redis biasanya menjadi dependency kritikal untuk banyak behavior:

• cache read/write;
• session atau short-lived state;
• distributed lock;
• idempotency key;
• rate limiter;
• deduplication;
• lightweight queue atau Redis Stream;
• feature/config cache;
• temporary workflow state;
• cross-request coordination.

Redis instrumentation harus menjawab:

• command Redis apa yang dijalankan;
• apakah command itu bagian dari cache, lock, rate limiter, idempotency, atau stream;
• berapa latency Redis;
• apakah latency berasal dari client, network, server, pool, pipeline, atau slow command;
• apakah cache menghasilkan hit/miss ratio yang sehat;
• apakah TTL benar;
• apakah lock tidak bocor;
• apakah idempotency key mencegah duplicate processing;
• apakah Redis key aman dari PII/secrets;
• apakah metric label tidak meledak karena key mentah;
• apakah failure Redis menyebabkan fallback, degraded mode, retry storm, atau full outage.

Redis bukan hanya dependency teknis. Untuk sistem CPQ/order management, Redis dapat mempengaruhi correctness: pricing cache stale, quote lock gagal, idempotency key hilang, approval session expired, atau duplicate order processing.

2. Redis Usage Pattern Determines Observability

Jangan membuat Redis dashboard generik saja. Observability Redis harus mengikuti usage pattern.

Instrumentation yang tidak membedakan usage pattern akan sulit dipakai saat incident. redis.command.duration saja tidak cukup untuk menjawab apakah order stuck karena lock, cache stale, atau stream backlog.

3. Redis Boundary in Java/JAX-RS Request Flow

Redis call sering berada di tengah request lifecycle.

HTTP request
  ↓
JAX-RS resource
  ↓
Service layer
  ↓
Redis client
  ↓
Redis server / cluster
  ↓
Cache result / lock result / idempotency result
  ↓
Database / Kafka / RabbitMQ / downstream service

Redis instrumentation harus menjaga context dari JAX-RS request:

• trace_id;
• span_id;
• correlation_id;
• request_id;
• tenant_id bila policy mengizinkan;
• business key seperti quote_id atau order_id di log/trace, bukan metric label high-cardinality;
• endpoint template;
• operation purpose, misalnya quote-cache-read, order-idempotency-check, atau pricing-lock-acquire.

Redis span tanpa business purpose sering tidak cukup:

GET redis

Lebih berguna:

Redis GET quote-pricing-cache

atau span attributes:

redis.operation.purpose=pricing-cache
cache.result=hit
business.domain=quote-management

4. Redis Client Instrumentation Boundary

Di Java, Redis dapat diakses melalui beberapa client/library:

• Lettuce;
• Jedis;
• Redisson;
• Spring Data Redis, jika stack memakai Spring;
• custom wrapper;
• framework cache abstraction;
• Redis driver via library internal.

Hal yang harus dipastikan:

• command Redis menghasilkan span atau metric;
• connection pool/client metrics tersedia;
• timeout dan retry terlihat;
• key tidak dicatat mentah;
• pipeline dan batch tidak menyembunyikan command mahal;
• Lua script diberi nama operasi yang aman;
• exception Redis dipetakan ke log/error taxonomy;
• fallback cache terlihat, bukan diam-diam.

Jika auto-instrumentation menghasilkan span Redis terlalu rendah-level, tambahkan manual instrumentation di wrapper agar purpose bisnis terlihat.

5. Command Span Design

Redis command span harus cukup detail untuk diagnosis, tetapi aman dari cardinality dan privacy leak.

Contoh span name:

Redis GET quote-pricing-cache
Redis SET order-idempotency-key
Redis EVAL rate-limiter-check
Redis XREADGROUP fulfillment-stream

Useful span attributes:

Avoid:

• raw Redis key containing user/order/quote/account/customer data;
• full command with values;
• serialized payload;
• token/session/API key;
• raw Lua script body;
• user ID/request ID/order ID as metric label;
• high-cardinality exception message as metric label.

Good instrumentation shows what category of key was used, not the actual key.

6. Key Privacy and Key Pattern Strategy

Redis keys often contain business identifiers:

quote:Q-12345:pricing
order:O-777:idempotency
tenant:T-42:user:U-99:session

Raw keys are dangerous because they can leak:

• customer identifiers;
• tenant identifiers;
• user identifiers;
• account identifiers;
• quote/order IDs;
• session IDs;
• token-derived values;
• commercially sensitive pricing context.

Safer approach:

redis.key.pattern = quote:{quoteId}:pricing
redis.key.hash = sha256-prefix-only-if-approved
redis.operation.purpose = pricing-cache

Use redis.key.pattern for low-cardinality grouping. Use raw key only in tightly controlled debug flow, with approval, short retention, and redaction policy.

Review rule

A Redis key must not become:

• metric label;
• dashboard group-by field;
• alert label;
• unredacted log field;
• trace attribute without review.

Key pattern is usually enough for production debugging.

7. Cache Hit/Miss Instrumentation

Cache instrumentation must distinguish result and correctness.

Core metrics:

Good labels:

• cache_name=quote-pricing-cache;
• operation=get|set|delete|refresh;
• result=hit|miss|stale|bypass|error;
• reason=not_found|expired|bypass_policy|redis_error.

Bad labels:

• key=quote:Q-12345:pricing;
• quote_id=Q-12345;
• user_id=U-99;
• error_message=...full dynamic message....

Cache correctness questions

Cache hit ratio alone is not sufficient. Ask:

• Is hit ratio high because stale values are served?
• Is miss ratio high after deployment because key format changed?
• Are misses concentrated on one endpoint?
• Are refresh jobs failing?
• Are Redis errors silently falling back to database?
• Is cache stampede happening?
• Is TTL too short or too long?
• Is cached value compatible with current schema version?

8. Cache Hit/Miss Logs

Do not log every cache hit at INFO in high-throughput systems. It becomes expensive and noisy.

Useful logging pattern:

Example:

{
  "level": "WARN",
  "event.name": "cache.refresh.failed",
  "cache.name": "quote-pricing-cache",
  "cache.operation": "refresh",
  "redis.key.pattern": "quote:{quoteId}:pricing",
  "quote_id": "Q-12345",
  "correlation_id": "corr-7f3a",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "error.type": "RedisTimeoutException",
  "fallback.action": "use_database",
  "customer_impact": "possible_latency_increase"
}

quote_id may be acceptable as a log field depending internal privacy policy. It should not be a metric label unless approved.

9. TTL Observability

TTL is correctness, not only cleanup.

For CPQ/order systems, TTL can determine:

• quote price cache freshness;
• idempotency window;
• lock expiration;
• approval session expiration;
• temporary fulfillment state survival;
• dedupe memory.

Useful TTL signals:

Avoid exact per-key TTL as metric label. Use buckets:

0-10s
10s-1m
1m-5m
5m-1h
1h+
no_ttl

TTL failure modes

10. Redis Latency Instrumentation

Redis latency can come from different layers.

Metric examples:

redis_client_command_duration_seconds_bucket{operation="GET",purpose="pricing-cache",result="success"}
redis_client_errors_total{operation="GET",purpose="pricing-cache",error_type="timeout"}
redis_client_commands_total{operation="GET",purpose="pricing-cache"}

Trace analysis should answer:

Is endpoint latency dominated by Redis, DB, downstream HTTP, or application CPU?

Dashboard should separate:

• Redis server latency;
• Redis client observed latency;
• application request latency;
• Redis error/timeout rate;
• command mix.

11. Command Count Per Request

A common Redis anti-pattern is too many Redis calls per request.

Example:

GET /quotes/{id}
  Redis GET quote header
  Redis GET quote items
  Redis GET quote pricing
  Redis GET quote discount
  Redis GET quote approval
  Redis GET quote eligibility
  ... repeated per item

This can create hidden N+1 behavior similar to database N+1.

Useful signals:

• Redis command count per request;
• command count per endpoint template;
• command count per business operation;
• pipeline usage;
• batch size;
• Redis time as percentage of request latency.

Potential metric:

app_redis_commands_per_request_bucket{endpoint="/quotes/{quoteId}",purpose="pricing-cache"}

Keep endpoint templated. Never use raw path with quote/order ID.

12. Pipeline and Batch Instrumentation

Redis pipeline improves round-trip efficiency, but can hide expensive command groups.

Instrumentation should show:

• pipeline size;
• pipeline duration;
• operation purpose;
• command mix if safe;
• partial failure behavior;
• response decoding latency;
• payload size bucket.

Metric examples:

redis_pipeline_duration_seconds_bucket{purpose="quote-cache-warmup"}
redis_pipeline_size_bucket{purpose="quote-cache-warmup"}
redis_pipeline_errors_total{purpose="quote-cache-warmup",error_type="decode_failure"}

Trace span:

Redis pipeline quote-cache-warmup

Attributes:

redis.pipeline.size=50
redis.operation.purpose=quote-cache-warmup
redis.command.mix=GET,MGET

Avoid attaching every key in pipeline as span attribute.

13. Lua Script Instrumentation

Lua scripts are powerful but can become opaque.

Do not log raw Lua script body or dynamic arguments. Instead, give every script a stable operation name.

Examples:

Redis EVAL rate-limiter-check
Redis EVAL idempotency-reserve
Redis EVAL lock-release-if-owner
Redis EVAL stream-claim-stale

Useful attributes:

Failure modes:

• script too slow;
• script blocks Redis event loop;
• script has unbounded key scan;
• script changes semantics across deployment;
• script hides payload-sensitive values;
• script not compatible across Redis versions;
• script causes lock not released or idempotency state corrupted.

Internal review should include script ownership and test coverage.

14. Distributed Lock Instrumentation

Redis lock observability is correctness-critical.

Lock questions:

• Was lock acquired?
• How long did acquisition wait?
• How long was lock held?
• Was lock released by owner?
• Did lock expire before work completed?
• Did multiple workers enter critical section?
• Did lock cause throughput bottleneck?
• Did lock leak?

Metrics:

redis_lock_acquire_total{lock_name="quote-pricing-lock",result="acquired"}
redis_lock_acquire_total{lock_name="quote-pricing-lock",result="timeout"}
redis_lock_wait_duration_seconds_bucket{lock_name="quote-pricing-lock"}
redis_lock_held_duration_seconds_bucket{lock_name="quote-pricing-lock"}
redis_lock_release_total{lock_name="quote-pricing-lock",result="released"}
redis_lock_expired_before_release_total{lock_name="quote-pricing-lock"}

Good lock log:

{
  "level": "WARN",
  "event.name": "redis.lock.acquire.timeout",
  "lock.name": "quote-pricing-lock",
  "redis.key.pattern": "lock:quote:{quoteId}:pricing",
  "quote_id": "Q-12345",
  "wait_ms": 5000,
  "ttl_ms": 30000,
  "correlation_id": "corr-7f3a",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "impact": "quote_pricing_delayed"
}

Avoid logging lock value if it contains token/owner secret.

15. Idempotency Instrumentation

Idempotency is essential for HTTP retry, Kafka/RabbitMQ redelivery, and duplicate client submission.

Idempotency signals:

Metric examples:

idempotency_requests_total{operation="order-submit",result="reserved"}
idempotency_requests_total{operation="order-submit",result="duplicate"}
idempotency_conflicts_total{operation="order-submit"}
idempotency_redis_errors_total{operation="order-submit",error_type="timeout"}

Do not use raw idempotency key as metric label. In logs, treat it as sensitive unless policy allows.

Critical failure mode

Business operation succeeds, but idempotency finalization fails.

This can cause retry ambiguity:

• client retries;
• Redis says no final result;
• service may repeat side effect;
• duplicate order/event can occur.

Instrumentation must log and metric this as a high-risk correctness event.

16. Rate Limiter Instrumentation

Rate limiter Redis usage must distinguish allowed, blocked, degraded, and Redis-failed behavior.

Metrics:

rate_limiter_requests_total{limiter="quote-api",result="allowed"}
rate_limiter_requests_total{limiter="quote-api",result="blocked"}
rate_limiter_requests_total{limiter="quote-api",result="redis_error_fail_open"}
rate_limiter_requests_total{limiter="quote-api",result="redis_error_fail_closed"}
rate_limiter_duration_seconds_bucket{limiter="quote-api"}

Design question:

If Redis is unavailable, does the limiter fail open or fail closed?

Both have risk:

Logs should explicitly state degraded behavior.

17. Redis Stream Instrumentation

If Redis Streams are used, observe them like messaging systems.

Core concepts:

• stream name;
• consumer group;
• consumer name;
• pending entries list;
• message age;
• ack result;
• claim/reclaim count;
• processing latency;
• retry/dead-letter behavior if implemented.

Metrics:

redis_stream_messages_read_total{stream="fulfillment-events",consumer_group="order-workers"}
redis_stream_pending_entries{stream="fulfillment-events",consumer_group="order-workers"}
redis_stream_message_age_seconds_bucket{stream="fulfillment-events"}
redis_stream_processing_duration_seconds_bucket{stream="fulfillment-events",result="success"}
redis_stream_claim_total{stream="fulfillment-events",reason="stale_pending"}
redis_stream_ack_total{stream="fulfillment-events",result="acked"}

Trace propagation should use stream message headers/fields if the pattern supports it. Never assume Redis Stream preserves context unless implemented.

18. Redis Error Taxonomy

Redis errors should be classified.

Good metric label uses low-cardinality error_type.

Bad label:

error_message="MOVED 12345 10.0.1.2:6379 ... dynamic ..."

19. Redis Fallback Observability

Many systems treat Redis as optional cache. But fallback can create hidden load.

Fallback examples:

• cache miss fallback to PostgreSQL;
• Redis error fallback to DB;
• rate limiter fail open;
• lock failure skip operation;
• idempotency Redis failure reject request;
• stale cache fallback to old value;
• Redis Stream failure fallback to polling.

Every fallback needs signal:

fallback.action
fallback.reason
fallback.result
customer_impact
dependency_impact

Metric:

redis_fallback_total{purpose="pricing-cache",reason="timeout",action="query_database"}

Incident question:

Did Redis outage cause DB overload because fallback traffic increased?

You need Redis error metrics and DB load metrics on the same dashboard.

20. Redis Dashboard Design

A useful Redis dashboard for application engineers should include both server and app-client perspective.

Application Redis panel

• Redis client command rate by purpose;
• command latency p50/p95/p99 by purpose;
• Redis error rate by error type;
• timeout count;
• fallback count;
• cache hit/miss ratio by cache name;
• lock acquire failure/wait time;
• idempotency duplicate/conflict count;
• rate limiter allowed/blocked/degraded;
• stream pending/message age if used.

Redis server/platform panel

• CPU usage;
• memory usage;
• used memory vs maxmemory;
• eviction count;
• expired keys;
• connected clients;
• blocked clients;
• command ops/sec;
• slowlog count;
• replication lag;
• cluster slot health;
• network throughput.

Correlation panel

• application request latency;
• DB fallback traffic;
• Redis latency/error;
• pod restart/deployment marker;
• recent config changes.

21. Redis Alerting Strategy

Do not page only on raw Redis CPU or memory if application impact is unclear. Prefer symptom plus dependency alerts.

Good alerts:

• Redis timeout rate impacts service SLO;
• cache fallback to DB above threshold;
• lock acquisition timeout affects order/quote processing;
• idempotency conflict/finalization failure spike;
• Redis Stream pending age exceeds business threshold;
• Redis memory near limit with eviction spike;
• blocked clients sustained;
• replication lag exceeds threshold for read correctness.

Bad alerts:

• every single cache miss;
• one-off slow command;
• raw connected clients without context;
• high command rate without saturation/error;
• Redis CPU spike for 30 seconds with no user impact.

Alert must link to:

• Redis dashboard;
• service dashboard;
• dependency runbook;
• fallback behavior documentation;
• owner/escalation path.

22. Production Debugging Playbook

When Redis is suspected, ask in order:

1. Which user-facing symptom changed?
2. Which service and endpoint are impacted?
3. Did request latency, error rate, or saturation change?
4. Did Redis client latency increase?
5. Did Redis timeout/error rate increase?
6. Did cache hit ratio change?
7. Did fallback to PostgreSQL increase?
8. Did DB latency or pool wait increase after Redis issue?
9. Did Redis server show CPU, memory, eviction, slowlog, blocked clients, or replication lag?
10. Did a deployment/config/key format change happen recently?
11. Did trace show Redis dominating request latency?
12. Did logs show key pattern, operation purpose, and fallback result?
13. Did sampling hide traces for failed/high-latency requests?
14. Is this cache performance issue or business correctness issue?

Redis debugging is not complete until you understand whether the effect is:

• performance degradation;
• correctness risk;
• duplicate processing;
• stale data;
• customer-visible failure;
• cost/load shift to another dependency.

23. CPQ/Order Management Redis Examples

Business context should appear in logs/traces carefully:

• quote/order ID as log field if policy allows;
• business operation as low-cardinality metric label;
• tenant/customer data only if approved and redacted/masked;
• raw Redis key avoided.

24. Common Anti-Patterns

Anti-pattern: raw Redis key in metrics

redis_command_duration_seconds{key="quote:Q-12345:pricing"}

This causes cardinality explosion and may leak business data.

Anti-pattern: cache hit ratio without cache name

cache_hit_ratio=0.91

Which cache? Which endpoint? Which tenant? Which operation? It is not actionable.

Anti-pattern: Redis errors swallowed silently

try {
  return cache.get(key);
} catch (Exception e) {
  return database.load(id);
}

This hides dependency degradation and can overload the database.

Anti-pattern: logging entire cached value

Cache values may contain PII, pricing, commercial terms, token-like values, or internal state.

Anti-pattern: lock without observability

A lock that only returns true/false without wait time, hold time, TTL, or owner-safe release logs is dangerous.

Anti-pattern: treating Redis as “just cache”

If Redis backs idempotency, lock, session, rate limiter, or stream, Redis failure is a correctness concern.

25. Internal Verification Checklist

Gunakan checklist ini di codebase/team, bukan asumsi.

Redis usage discovery

• Redis dipakai untuk apa saja: cache, lock, idempotency, rate limiter, dedupe, stream, session, config, temporary workflow state?
• Redis client/library apa yang digunakan: Lettuce, Jedis, Redisson, framework abstraction, custom wrapper?
• Apakah ada satu wrapper internal untuk semua Redis access?
• Apakah Redis standalone, sentinel, cluster, managed cloud service, atau on-prem?
• Apakah Redis berada di path kritikal quote/order/approval/fulfillment?

Instrumentation

• Apakah Redis command menghasilkan span?
• Apakah Redis operation purpose terlihat?
• Apakah cache hit/miss metric tersedia?
• Apakah command latency histogram tersedia?
• Apakah timeout/retry/error metric tersedia?
• Apakah fallback ke DB atau degraded mode terlihat?
• Apakah pipeline/Lua script/lock/idempotency/rate limiter punya telemetry khusus?
• Apakah Redis Stream punya pending/message age/ack/claim metrics jika digunakan?

Key privacy

• Apakah raw Redis key muncul di log/trace/metric?
• Apakah key pattern digunakan?
• Apakah key mengandung tenant/user/order/quote/customer/session/token data?
• Apakah ada redaction/masking utility?
• Apakah Redis key dilarang menjadi metric label?
• Apakah log access control sesuai sensitivity data?

Cache correctness

• Apakah TTL terdokumentasi per cache?
• Apakah cache refresh failure terlihat?
• Apakah stale read policy ada?
• Apakah cache schema/version compatibility terlihat?
• Apakah cache stampede dicegah dan terukur?
• Apakah miss spike dikorelasikan dengan DB load?

Lock/idempotency correctness

• Apakah lock acquire wait/failure/hold/release terlihat?
• Apakah lock TTL cukup dan dimonitor?
• Apakah release hanya dilakukan oleh owner?
• Apakah idempotency reservation/finalization/duplicate/conflict terlihat?
• Apakah finalization failure diperlakukan sebagai correctness risk?

Dashboard/alert/runbook

• Apakah ada dashboard Redis app-client dan Redis server/platform?
• Apakah dashboard Redis dikaitkan dengan request latency/error dan DB fallback?
• Apakah alert Redis actionable dan symptom-aware?
• Apakah runbook menjelaskan fallback behavior?
• Apakah incident sebelumnya menyebut missing Redis telemetry?

26. PR Review Checklist

Saat mereview PR yang menyentuh Redis, tanyakan:

• Redis dipakai untuk cache, lock, idempotency, rate limiter, stream, atau state?
• Apa correctness risk jika Redis gagal?
• Apakah key pattern aman dan tidak bocor?
• Apakah raw key/value tidak masuk log/trace/metric?
• Apakah metric punya low-cardinality labels?
• Apakah command latency dan error terlihat?
• Apakah fallback behavior eksplisit dan terukur?
• Apakah TTL tepat dan terdokumentasi?
• Apakah cache hit/miss result diukur?
• Apakah lock wait/hold/release diukur?
• Apakah idempotency duplicate/conflict/finalization failure diukur?
• Apakah Redis Stream backlog/message age diukur jika digunakan?
• Apakah alert/dashboard/runbook perlu diperbarui?
• Apakah privacy/security review diperlukan?
• Apakah cost/cardinality risk sudah dicek?

27. Key Takeaways

Redis instrumentation yang baik harus menjelaskan purpose, bukan hanya command.

Prinsip utama:

• gunakan key pattern, bukan raw key;
• ukur cache hit/miss, latency, error, fallback, dan TTL;
• bedakan cache performance issue dari correctness issue;
• lock dan idempotency harus diobservasi sebagai correctness mechanism;
• pipeline dan Lua script perlu nama operasi stabil;
• Redis Stream perlu backlog/message age/ack visibility;
• fallback Redis ke DB harus terlihat karena dapat memindahkan outage;
• metric labels harus low-cardinality;
• logs/traces harus privacy-aware;
• dashboard harus menghubungkan Redis client, Redis server, app latency, dan DB fallback.

Redis yang tidak terlihat sering menjadi sumber incident yang sulit dijelaskan: latency naik, DB overload, duplicate order, stale quote, lock contention, atau workflow stuck tanpa evidence yang cukup.