Part 077 — Kafka Streams and ksqlDB

Fokus part ini: memahami kapan Kafka Streams atau ksqlDB dibutuhkan, bagaimana lifecycle-nya, apa konsekuensi stateful stream processing, dan bagaimana mereviewnya dalam konteks Java/JAX-RS enterprise backend.

Kafka producer dan consumer biasa cukup untuk banyak kebutuhan.

Tetapi ketika sistem mulai membutuhkan:

• aggregate real-time
• materialized view
• event enrichment
• join antar stream/table
• time window computation
• deduplication berbasis state
• projection untuk read model
• continuous transformation
• fraud/rule evaluation berbasis event
• monitoring business process dari event log

maka kita masuk ke area stream processing.

Kafka Streams dan ksqlDB adalah dua cara berbeda untuk melakukan stream processing di ekosistem Kafka.

1. Core Mental Model

Kafka biasa:

producer -> topic -> consumer -> side effect

Kafka Streams:

input topics -> topology -> state store/changelog -> output topics/materialized state

ksqlDB:

SQL-like continuous query -> stream/table -> output topic/materialized view

Perbedaan penting:

Kafka consumer processes records.
Kafka Streams runs a topology.
ksqlDB runs persistent stream queries.

Kafka Streams bukan sekadar helper library untuk consumer.

Ia adalah embedded stream processing runtime yang berjalan di dalam JVM application.

2. When Stream Processing Exists

Stream processing muncul ketika sistem perlu menghitung state dari event yang terus mengalir.

Contoh enterprise:

OrderCreated
OrderUpdated
OrderCancelled
PaymentReceived
FulfillmentStarted

Dari event ini, sistem mungkin perlu membuat:

current_order_status_view
order_lifecycle_metrics
pending_fulfillment_view
customer_open_order_summary
quote_to_order_conversion_metrics

Jika dihitung via synchronous API call ke banyak service, sistem menjadi chatty dan fragile.

Jika dihitung via batch harian, data tidak real-time.

Stream processing memberi opsi:

maintain derived state continuously as events arrive

3. Kafka Streams Is a Java Library, Not a Separate Cluster

Kafka Streams berjalan sebagai library di JVM application.

Artinya:

Your application process is the stream processor.

Implikasi:

• deployment mengikuti service deployment
• scaling mengikuti jumlah instance application
• state lokal bisa ada di disk pod/node
• rebalance terjadi saat instance naik/turun
• crash recovery bergantung pada changelog topic
• observability harus dipasang di aplikasi

Berbeda dari Spark/Flink yang punya cluster execution model sendiri.

Kafka Streams lebih ringan, tetapi responsibility operations-nya masuk ke application team.

4. Basic Topology Model

Topology adalah graph processing.

source topic
  -> map/filter/transform
  -> groupBy
  -> aggregate/window/join
  -> sink topic

Contoh mental model:

order-events
  -> filter only status events
  -> group by orderId
  -> aggregate latest status
  -> write order-status-view topic

Topology terdiri dari:

• source node
• processor node
• state store
• sink node
• repartition topic
• changelog topic

PR review harus membaca topology sebagai dataflow, bukan hanya Java code.

5. KStream, KTable, and GlobalKTable

KStream

KStream merepresentasikan stream record append-only.

each record is an event/fact occurrence

Cocok untuk:

• event command/result
• click/activity stream
• order lifecycle event
• audit event
• integration event

KTable

KTable merepresentasikan changelog table.

latest value per key

Cocok untuk:

• current order state
• customer profile view
• product catalog snapshot
• entitlement state

GlobalKTable

GlobalKTable direplikasi ke semua instance.

Cocok untuk lookup kecil/menengah yang perlu tersedia lokal di semua task.

Risiko:

• memory/disk besar
• startup restore lama
• update fanout ke semua instance

6. Stream vs Table Thinking

Kesalahan umum adalah memperlakukan semua topic sebagai stream biasa.

Pertanyaan review:

Is this topic an event stream or a changelog table?
Is key stable?
Does latest value replace previous value?
Is tombstone meaningful?

Event stream:

OrderSubmitted(orderId=O1, version=1)
OrderApproved(orderId=O1, version=2)
OrderCancelled(orderId=O1, version=3)

Table/changelog:

key=O1 value={status=SUBMITTED}
key=O1 value={status=APPROVED}
key=O1 value={status=CANCELLED}

Keduanya mirip di Kafka, tapi maknanya berbeda.

7. State Store Mental Model

State store adalah local materialized state yang digunakan topology.

Contoh:

orderId -> latest status
customerId -> open order count
quoteId -> last processed event id

State store bisa dipakai untuk:

• aggregate
• join
• deduplication
• windowed computation
• lookup
• materialized read model

State store biasanya dilindungi oleh changelog topic.

local state store can be rebuilt from changelog topic

Tanpa changelog atau retention yang benar, recovery bisa gagal.

8. Local State Is a Production Concern

Stateful stream processing membawa state ke application process.

Artinya ada concern baru:

• local disk capacity
• pod restart restore time
• changelog topic retention
• state directory cleanup
• rebalance restore cost
• backup/rebuild strategy
• Kubernetes ephemeral storage limit
• state store corruption handling

Untuk Kubernetes, state lokal sering berada di ephemeral disk kecuali didesain lain.

Jika pod pindah node, local state hilang dan perlu restore dari changelog.

Itu normal, tetapi harus diperhitungkan.

9. Materialized View Pattern

Materialized view adalah state turunan yang dibangun dari event.

Contoh:

order-events -> order_current_status_view
quote-events -> quote_latest_version_view
catalog-events -> catalog_effective_view

Manfaat:

• read lebih cepat
• mengurangi synchronous call antar service
• mendukung query pattern khusus
• mendukung eventual consistency

Risiko:

• stale view
• replay bug
• schema evolution break
• partial rebuild
• duplicate side effect
• missing event menghasilkan view salah

Materialized view harus punya reconciliation strategy.

10. Windowing

Windowing dipakai saat logic bergantung pada waktu.

Tipe umum:

• tumbling window
• hopping window
• sliding window
• session window

Contoh:

count order submissions per tenant every 5 minutes

Pertanyaan penting:

Which time is used: event time, processing time, or ingestion time?
How late can events arrive?
What is grace period?
What happens to out-of-order events?

Windowing tanpa time semantics yang eksplisit akan menghasilkan metric/business state yang membingungkan.

11. Event Time vs Processing Time

Event time:

time when business event actually happened

Processing time:

time when stream processor handles the record

Ingestion time:

time when Kafka accepted the record

Untuk enterprise order/quote lifecycle, event time sering lebih benar untuk business calculation.

Tetapi processing time lebih mudah untuk operational metric.

PR review harus meminta field waktu yang dipakai secara eksplisit.

12. Repartitioning and Key Design

Kafka Streams sering perlu repartition saat key berubah.

Contoh:

input key = eventId
operation groupBy(orderId)

Kafka Streams perlu membuat repartition topic.

Risiko:

• extra network I/O
• extra topic
• extra storage
• ordering berubah relatif terhadap key lama
• latency meningkat
• topic governance menjadi kabur

Key design sangat penting.

Untuk event order:

key = orderId

sering lebih baik daripada:

key = random UUID

karena ordering per order dan grouping menjadi natural.

13. Join Semantics

Kafka Streams mendukung join antara:

• stream-stream
• stream-table
• table-table
• stream-global table

Setiap join punya semantics berbeda.

Pertanyaan review:

Is the join window bounded?
What happens if right-side record arrives late?
What happens if lookup table missing?
Is the join inner, left, or outer?
Does enrichment require latest state or historical state?

Join adalah sumber bug subtle karena menghasilkan output yang tampak benar pada happy path, tetapi salah saat event terlambat.

14. Deduplication with State Store

Deduplication sering memakai state store.

eventId -> processedAt

Flow:

if eventId exists:
  skip
else:
  process and store eventId

Concern:

• retention dedupe state
• memory/disk size
• replay behavior
• event ID uniqueness
• clock/time dependency
• changelog recovery

Deduplication tidak boleh dianggap gratis.

15. Processing Guarantees

Kafka Streams memiliki processing guarantee seperti:

• at-least-once
• exactly-once-v2 / exactly-once semantics dalam scope Kafka transactions

Namun “exactly-once” tidak berarti seluruh dunia exactly-once.

Jika topology memanggil external HTTP service, database lain, email system, atau payment service, guarantee Kafka tidak otomatis berlaku.

Mental model aman:

Exactly-once may apply to Kafka read-process-write boundary.
External side effects still need idempotency.

16. Exactly-Once Trade-Off

Exactly-once Kafka Streams bisa berguna untuk:

• aggregate ke output topic
• transactional write antar Kafka topic
• materialized changelog consistency

Trade-off:

• latency lebih tinggi
• transaction overhead
• producer fencing complexity
• config lebih sensitif
• debugging lebih sulit

Gunakan ketika correctness membutuhkan, bukan sebagai default tanpa alasan.

17. Interactive Queries

Kafka Streams dapat mengekspos state store melalui interactive queries.

Model:

JAX-RS endpoint -> local state store lookup

Masalahnya: state untuk key tertentu mungkin berada di instance lain.

Maka service perlu:

• metadata lookup
• remote routing
• key ownership awareness
• rebalance handling
• fallback saat store restoring

Dalam enterprise production, interactive query perlu design hati-hati.

Sering lebih sederhana menulis materialized view ke PostgreSQL/Redis/Elastic/read store.

18. JAX-RS Boundary with Kafka Streams

Ada beberapa pola integrasi:

Pattern A — JAX-RS service also runs stream topology

same JVM:
  HTTP API
  Kafka Streams topology

Kelebihan:

• deployment sederhana
• shared code/config
• low-latency local access

Risiko:

• API latency dan stream workload saling mempengaruhi
• scaling HTTP dan stream tidak independen
• shutdown lebih kompleks
• memory/disk lebih berat

Pattern B — Separate stream processor service

api-service
stream-processor-service

Kelebihan:

• scaling terpisah
• operational isolation
• failure domain lebih jelas

Risiko:

• lebih banyak deployment
• contract antar service perlu jelas
• duplicated libraries/config mungkin muncul

Untuk sistem enterprise, Pattern B sering lebih bersih jika stream workload berat.

19. ksqlDB Mental Model

ksqlDB menyediakan SQL-like interface untuk stream processing.

Contoh konsep:

CREATE STREAM order_events (...)
  WITH (KAFKA_TOPIC='order-events', VALUE_FORMAT='AVRO');

CREATE TABLE order_status AS
SELECT orderId, LATEST_BY_OFFSET(status) AS status
FROM order_events
GROUP BY orderId;

Mental model:

persistent query continuously reads input topic and writes output topic/table

ksqlDB cocok ketika transformation bisa diekspresikan sebagai continuous SQL.

20. ksqlDB Strengths

ksqlDB kuat untuk:

• quick stream transformation
• filtering/enrichment sederhana
• real-time aggregate
• materialized view sederhana
• operational query terhadap stream
• reducing custom Java processor code

ksqlDB membantu ketika logic declarative lebih mudah direview daripada Java code.

21. ksqlDB Weaknesses

ksqlDB kurang cocok untuk:

• complex domain logic
• heavy custom validation
• integration side effect ke banyak system
• workflow orchestration
• logic yang butuh rich Java library
• complex error recovery per record
• per-tenant custom behavior yang rumit

Jika logic butuh banyak imperative branching, Kafka Streams atau normal consumer service lebih cocok.

22. Kafka Streams vs ksqlDB vs Plain Consumer

Rule of thumb:

Use plain consumer for side-effect-heavy processing.
Use Kafka Streams for Java-based stateful stream processing.
Use ksqlDB for declarative continuous transformations and views.

23. Deployment and Scaling

Kafka Streams scaling mengikuti jumlah application instances dan jumlah stream tasks.

Batas utama:

maximum parallelism often bounded by partition count

Jika topic punya 6 partitions, menaikkan instance menjadi 20 tidak membuat 20 instance aktif memproses keyspace utama.

Scaling harus melihat:

• partition count
• task assignment
• state store size
• restore time
• CPU
• disk I/O
• network I/O
• downstream output throughput

24. Startup and Restore Lifecycle

Saat aplikasi Kafka Streams start:

load config
build topology
join consumer group
assign tasks
restore state stores from changelog
start processing

Selama restore, aplikasi mungkin belum ready.

Untuk Kubernetes:

readiness probe must not say ready before topology can serve required behavior

Jika API dan stream topology dalam JVM yang sama, readiness menjadi lebih kompleks.

25. Shutdown Lifecycle

Shutdown harus:

• stop menerima HTTP traffic jika service gabungan
• stop stream processing gracefully
• commit/flush state sesuai guarantee
• close state stores
• release resources
• expose clear logs

Killing stream processor secara kasar bisa menyebabkan:

• longer rebalance
• duplicate processing
• state restore ulang
• partially flushed output

26. Failure Modes

26.1 Topology Misconfiguration

Gejala:

• app gagal start
• missing SerDe
• incompatible topic format
• invalid state store config

26.2 Rebalance Storm

Gejala:

• throughput drop
• consumer group unstable
• repeated partition revoke/assign
• high latency

26.3 State Restore Too Slow

Gejala:

• pod lama ready
• restart recovery lambat
• changelog topic besar
• local disk sering hilang

26.4 Repartition Topic Explosion

Gejala:

• banyak internal topic
• storage meningkat
• unexpected latency
• governance topic memburuk

26.5 Late and Out-of-Order Events

Gejala:

• aggregate salah
• window result berubah terlambat
• business metric tidak cocok

26.6 Cardinality Explosion

Gejala:

• state store membesar cepat
• RocksDB/disk pressure
• GC/native memory pressure
• restore time meningkat

26.7 Schema Evolution Failure

Gejala:

• deserialization error
• query gagal saat replay
• old events tidak bisa diproses

27. Observability Checklist

Kafka Streams metrics perlu mencakup:

• records processed rate
• process latency
• poll latency
• commit latency
• skipped records
• deserialization errors
• rebalance count
• task created/closed
• state restore progress
• state store size
• changelog lag
• output rate
• error rate per topology node

Logs perlu mencakup:

• topology version
• application.id
• topic input/output
• state directory
• processing guarantee
• internal topic names
• rebalance events
• state restore start/end

28. Topic and Application ID Governance

application.id sangat penting.

Ia menentukan:

• consumer group
• internal topic prefix
• state store namespace

Mengubah application.id berarti stream app terlihat seperti consumer baru.

Dampaknya:

• offset mulai ulang sesuai reset policy
• state store baru
• internal topics baru
• replay tidak sengaja mungkin terjadi

PR review harus memperlakukan perubahan application.id sebagai breaking operational change.

29. Serialization and SerDe

Kafka Streams sangat sensitif terhadap SerDe.

SerDe harus jelas untuk:

• key
• value
• repartition topic
• state store changelog
• output topic

Failure umum:

key serde mismatch
value schema incompatible
null/tombstone not handled
old schema cannot be read during restore

SerDe harus align dengan schema governance dari Part 074.

30. Data Correctness Review Questions

Untuk setiap topology, tanyakan:

What is the input fact?
What is the derived state?
What is the key?
What is the time semantics?
What happens on duplicate?
What happens on late event?
What happens on missing event?
What happens on schema evolution?
Can the output be rebuilt?
Can the topology be replayed safely?

Jika jawaban tidak jelas, topology belum production-ready.

31. Performance Review Questions

How many input records per second?
How many output records per second?
What is expected state store size?
What is maximum key cardinality?
How many repartition steps exist?
What is restore time objective?
What is acceptable end-to-end lag?
What is disk I/O budget?

Stream processing sering gagal bukan karena logic salah, tetapi karena state dan throughput tidak dihitung.

32. Security and Compliance Considerations

Stream processors sering memindahkan data sensitif.

Review:

• PII in topics
• PII in state store
• encrypted disk requirement
• retention policy
• audit trail
• access control to internal topics
• access control to ksqlDB server
• query exposure
• logs redaction

Materialized view bisa menjadi data copy baru yang ikut tunduk pada retention dan privacy policy.

33. ksqlDB Operational Checklist

Jika ksqlDB dipakai, cek:

• siapa owner ksqlDB query
• query disimpan di repo atau dibuat manual
• CI/CD untuk query
• rollback query
• schema compatibility
• output topic ownership
• access control
• monitoring query lag
• resource limit server
• disaster recovery

ksqlDB query yang dibuat manual di console adalah configuration drift.

34. Internal Verification Checklist

Untuk konteks CSG Quote & Order atau service enterprise sejenis, jangan mengasumsikan Kafka Streams/ksqlDB dipakai.

Verifikasi:

• apakah ada dependency kafka-streams
• apakah ada package topology/processor/state-store
• apakah ada StreamsBuilder
• apakah ada KafkaStreams lifecycle management
• apakah ada ksqlDB cluster/server
• apakah ada SQL file untuk persistent query
• apakah ada internal topic dengan prefix application id
• apakah ada state directory config
• apakah topology satu JVM dengan JAX-RS API
• apakah ada separate stream processor deployment
• apakah ada schema registry integration
• apakah ada replay/rebuild runbook
• apakah ada alert untuk stream lag/state restore
• apakah ada policy untuk repartition/internal topics
• apakah ada ownership untuk materialized view

35. PR Review Checklist

Saat mereview Kafka Streams/ksqlDB change:

• input/output topic jelas
• key strategy benar
• schema compatibility dicek
• SerDe eksplisit
• state store size diperkirakan
• changelog retention cukup
• time semantics eksplisit
• late event behavior jelas
• duplicate behavior jelas
• repartition topics dipahami
• application.id tidak berubah tanpa migration plan
• processing guarantee dipilih dengan alasan
• metrics/logging tersedia
• replay/rebuild strategy tersedia
• ksqlDB query versioned di repo
• materialized view punya owner
• security/PII/retention diperiksa

36. Senior Engineer Heuristics

Gunakan stream processing ketika:

state can be derived from events
latency matters
join/aggregate can be expressed clearly
replay/rebuild is possible
operations team can support it

Hindari stream processing ketika:

logic is mostly external side effects
state cannot be rebuilt
ordering requirements are unclear
schema evolution is unmanaged
team cannot operate stateful processors

Stream processing yang baik membuat system lebih decoupled.

Stream processing yang buruk membuat system lebih sulit dipahami daripada synchronous call graph.

37. Key Takeaways

• Kafka Streams adalah embedded Java stream processing runtime.
• ksqlDB adalah SQL-like stream processing platform.
• Stateful processing membawa state, changelog, restore, disk, dan rebalance concern.
• Materialized view harus punya replay dan reconciliation strategy.
• Exactly-once Kafka tidak menghapus kebutuhan idempotency untuk external side effects.
• Key design menentukan ordering, grouping, repartitioning, dan scalability.
• Production readiness membutuhkan observability, restore plan, schema compatibility, dan ownership.