Cheatsheet Kafka

Consumer lag, consumer group lag, partition lag, end-to-end latency, producer/consumer/broker metrics, rebalance rate, under-replicated partition, ISR shrink, Connect/Streams/DLQ metrics, dashboard design, dan alerting strategy untuk Kafka production operations.

Throughput, latency, batching, compression, partition count, replication factor, retention, disk IO, network IO, broker sizing, consumer parallelism, load testing, dan performance tuning checklist untuk Kafka production.

Retention by time/size, segment, cleanup policy, delete policy, compact policy, tombstone, keyed event, log compaction, compact topic use case, audit retention, disk capacity, data loss risk, dan privacy concern di Kafka production.

Transactional producer, transactional ID, producer fencing, transaction timeout, send offsets to transaction, exactly-once processing in Kafka, Kafka Streams EOS, database transaction mismatch, outbox vs Kafka transaction, dan transaction review checklist.

Dual-write problem, database write plus Kafka publish, Kafka publish plus database write, external API plus event publish, transactional outbox, CDC, saga, compensation, idempotency, reconciliation, distributed transaction avoidance, two-phase commit limitation, failure matrix, consistency trade-off, dan review checklist.

Plain Kafka Java client, producer wrapper, consumer loop abstraction, event handler, error handler, retry handler, DLQ publisher, serialization abstraction, schema validation, header propagation, metrics, graceful shutdown, threading, backpressure, unit test, integration test, dan Testcontainers.

HTTP command to event, HTTP response before event consumed, synchronous API over asynchronous processing, 202 Accepted, idempotency key, request correlation, long-running workflow, polling status endpoint, SSE/WebSocket status update, error mapping, timeout expectation, API contract dengan eventual consistency, resource boundary, service layer boundary, transaction boundary, dan review checklist.

PostgreSQL transaction boundary, MyBatis mapper transaction, insert business row + outbox row, outbox polling with SKIP LOCKED, CDC from PostgreSQL, JSONB event payload, event metadata table, idempotency table, inbox table, state transition table, locking, serialization failure after DB commit, data repair, migration ordering, dan integration checklist.

Redis cache invalidation via Kafka, Redis as idempotency store, dedup store, distributed lock around consumer, rate limiter for producer/consumer, Redis Stream vs Kafka, Redis pub/sub vs Kafka, cache staleness, event-driven cache invalidation, cache rebuild, lock expiry risk, Redis/Kafka failure interaction, dan review checklist.

Kafka mental model, RabbitMQ mental model, RabbitMQ Stream mental model, queue vs log, acknowledgement, ordering, replay, retention, routing, work distribution, fanout, backpressure, latency, throughput, operational complexity, hybrid usage, dan architecture decision checklist.

Kafka client in Kubernetes, pod lifecycle, graceful shutdown, readiness, liveness, consumer rebalance on pod restart, horizontal scaling, consumer replicas vs partition count, resource request/limit, CPU throttling impact, memory pressure, ConfigMap, Secret, NetworkPolicy, DNS, service discovery, rolling update, Pod Disruption Budget, dan deployment checklist.

Running Kafka in Kubernetes, StatefulSet, PersistentVolume, StorageClass, broker identity, pod anti-affinity, rack awareness, broker restart, rolling upgrade, operator pattern, Strimzi/Confluent Operator/Helm awareness, storage failure, node failure, network failure, dan managed Kafka trade-off.

Amazon MSK, self-managed Kafka on EC2/EKS, MSK Serverless, VPC, subnet, security group, IAM authentication, SASL/SCRAM, TLS, private connectivity, CloudWatch, MSK Connect, Schema Registry options, cross-account access, backup/replication, dan common AWS Kafka failure modes.

Self-managed Kafka on AKS/VM, Azure Event Hubs Kafka-compatible endpoint, VNet, private endpoint, NSG, managed identity, TLS/SASL, Azure Monitor, Kafka Connect, Schema Registry options, and operational failure modes.

Self-managed Kafka, broker OS tuning, disk layout, firewall, TLS, certificate management, monitoring, patching, air-gapped deployment, hybrid connectivity, cloud-to-on-prem flows, MirrorMaker 2, Cluster Linking, and operational responsibility boundaries.

Multi-AZ, multi-region, cluster replication, MirrorMaker 2, Cluster Linking, active-passive, active-active challenges, offset translation, schema replication, failover, RPO/RTO, duplicate/data-loss scenarios, and DR runbooks.

Unit testing, integration testing, contract testing, Testcontainers Kafka, Schema Registry tests, Kafka Connect tests, CDC tests, outbox/inbox tests, retry/DLQ tests, replay tests, ordering tests, load tests, chaos tests, and CI strategy.

Topic as code, schema as code, connector config as code, ACL as code, environment promotion, compatibility checks, GitOps reconciliation, drift detection, rollback strategy, and Kafka artifact review discipline.

Production runbooks for consumer lag, producer failure, DLQ, broker down, under-replicated partitions, disk full, schema incompatibility, connector failure, CDC slot lag, rebalance storm, hot partition, poison events, replay, data repair, incident severity, customer impact, and communication.

Production-safe Kafka troubleshooting for messages not produced or consumed, lag, stuck consumers, rebalance loops, serialization failures, authorization failures, broker issues, duplicates, missing events, out-of-order events, DLQ spikes, CDC failures, Streams restoration, and ksqlDB query failures.

Security, privacy, and compliance review for Kafka event data classification, PII in payloads and headers, redaction, retention, encryption, access control, least privilege, audit logs, sensitive topics, DLQ risk, replay risk, and compliance evidence.

Senior-engineer checklist for reviewing Kafka and event-driven architecture changes: event design, topic design, partition key, producer, consumer, idempotency, retry/DLQ, schema, outbox, inbox, CDC, Streams, ksqlDB, security, observability, performance, operational readiness, and ADR quality.

Final synthesis of the Kafka series: complete Kafka mental model, event-driven architecture checklist, topic design, partitioning, producer, consumer, idempotency, retry/DLQ, schema governance, outbox/inbox, CDC, Streams, ksqlDB, security, observability, production readiness, and internal verification map.