Part 078 — RabbitMQ and RabbitMQ Stream

Fokus part ini: memahami RabbitMQ sebagai broker queue/routing, RabbitMQ Stream sebagai log-oriented messaging, dan bagaimana membandingkannya dengan Kafka dalam service Java/JAX-RS enterprise.

Kafka bukan satu-satunya messaging system.

Banyak enterprise system memakai RabbitMQ karena kuat untuk:

• routing message
• work queue
• request/reply async
• command dispatch
• task distribution
• retry via DLX
• priority queue
• low-latency brokered messaging
• operational simplicity untuk queue workloads tertentu

RabbitMQ Stream menambahkan model stream/log yang lebih dekat ke Kafka, tetapi tetap berada dalam ekosistem RabbitMQ.

1. Core Mental Model

RabbitMQ classic mental model:

producer -> exchange -> binding -> queue -> consumer

Kafka mental model:

producer -> topic partition log -> consumer group

RabbitMQ Stream mental model:

producer -> stream log -> offset-based consumer

Perbedaan fundamental:

RabbitMQ classic routes messages into queues.
Kafka stores ordered logs in partitions.
RabbitMQ Stream stores append-only streams with offset consumption.

2. RabbitMQ Is Usually Routing-First

RabbitMQ classic unggul ketika pertanyaannya:

Where should this message go?
Which queue should receive it?
Which workers should compete for it?
How should routing key decide delivery?

Kafka unggul ketika pertanyaannya:

Can many independent consumers read the same event log at their own pace?
Can state be rebuilt by replaying historical events?
Can retention preserve events after consumption?

Queue dan log menyelesaikan masalah berbeda.

3. Exchange

Exchange menerima message dari producer.

Exchange tidak menyimpan message untuk consumer secara langsung.

Ia merutekan message ke queue berdasarkan type dan binding.

Type umum:

• direct
• topic
• fanout
• headers

Mental model:

exchange decides routing
queue stores pending delivery
consumer consumes from queue

4. Direct Exchange

Direct exchange route berdasarkan exact routing key.

routingKey = order.created

Cocok untuk routing command/event sederhana.

Contoh:

order.command.exchange
  routing key: order.submit
  -> order-submit-queue

5. Topic Exchange

Topic exchange route berdasarkan pattern.

order.*
order.created
order.cancelled
quote.#

Cocok untuk event category routing.

Risiko:

• wildcard terlalu luas
• consumer menerima message tidak diharapkan
• routing governance memburuk
• naming convention tidak konsisten

6. Fanout Exchange

Fanout mengirim message ke semua queue yang terikat.

Cocok untuk broadcast sederhana.

Tetapi untuk event enterprise dengan banyak consumer independen dan replay requirement, Kafka-style log sering lebih natural.

Fanout queue bukan pengganti event log dengan retention/replay semantics.

7. Queue

Queue menyimpan message sampai dikonsumsi, expired, dead-lettered, atau dihapus.

Queue memiliki concern:

• durable vs transient
• exclusive queue
• auto-delete
• quorum queue/classic queue
• message TTL
• queue length limit
• dead-letter exchange
• priority
• lazy queue behavior

Queue adalah operational object, bukan hanya nama string.

8. Binding

Binding menghubungkan exchange ke queue.

exchange + routing key pattern -> queue

Bug binding bisa menyebabkan:

• message tidak masuk queue
• message masuk queue salah
• duplicate routing
• silent data loss jika mandatory/publisher confirm tidak digunakan

Binding harus versioned dan dikelola seperti infrastructure/configuration.

9. Producer Publish Lifecycle

Producer flow:

create connection/channel
publish to exchange with routing key
broker routes to queues
publisher confirm indicates broker accepted

Tanpa publisher confirm, producer bisa mengira publish sukses padahal broker belum menyimpan message secara durable.

Untuk production, cek:

• publisher confirms
• mandatory flag jika perlu deteksi unroutable
• retry policy
• connection recovery
• message persistence
• timeout
• idempotency key

10. Consumer Acknowledgement

RabbitMQ consumer harus acknowledge message.

Mode umum:

• auto ack
• manual ack
• nack/reject
• requeue true/false

Manual ack lebih aman untuk processing yang bisa gagal.

Flow aman:

receive message
validate
process idempotently
commit side effect
ack message

Jika consumer crash sebelum ack, message bisa redelivered.

Karena itu idempotency tetap wajib.

11. Prefetch and Backpressure

Prefetch mengontrol berapa message bisa dikirim broker ke consumer tanpa ack.

Jika prefetch terlalu tinggi:

• consumer memory naik
• message tertahan di consumer lambat
• fair dispatch buruk
• shutdown lebih lama

Jika terlalu rendah:

• throughput rendah
• broker/consumer roundtrip overhead tinggi

Prefetch adalah backpressure control utama di RabbitMQ classic.

12. Work Queue Pattern

Work queue dipakai untuk membagi job ke beberapa worker.

one queue
many competing consumers
one message processed by one consumer

Cocok untuk:

• async task
• email send job
• document generation
• cleanup task
• external integration command

Tidak cocok untuk:

• banyak independent consumer yang semua perlu menerima event yang sama
• replay history jangka panjang
• event sourcing

13. Pub/Sub with RabbitMQ

RabbitMQ bisa melakukan pub/sub menggunakan exchange dan multiple queues.

exchange -> queue A -> consumer A
         -> queue B -> consumer B

Setiap consumer group biasanya punya queue sendiri.

Ini bekerja baik untuk fanout aktif.

Tetapi jika consumer baru ingin membaca event lama, RabbitMQ classic tidak otomatis menyediakan replay seperti Kafka retention.

14. Dead Letter Exchange

Dead-lettering memindahkan message ke exchange/queue lain saat:

• rejected/nacked without requeue
• TTL expired
• queue length limit exceeded
• delivery limit exceeded pada quorum queue

DLX pattern:

main exchange -> main queue -> consumer
                         failed -> DLX -> DLQ

DLQ bukan tempat sampah.

DLQ adalah recovery queue yang butuh owner, alert, triage, replay procedure, dan data retention.

15. Retry Patterns in RabbitMQ

Retry bisa dibuat dengan:

• immediate requeue
• delayed retry queue dengan TTL
• dead-letter routing kembali ke main queue
• delayed message exchange plugin jika tersedia
• application-managed retry count

Immediate requeue berbahaya karena bisa membuat hot loop.

Pattern lebih aman:

main queue
  fail -> retry queue with TTL
  TTL expires -> route back to main queue
  max retries exceeded -> DLQ

Butuh retry count header atau metadata.

16. Poison Message

Poison message adalah message yang selalu gagal diproses.

Gejala:

• retry loop
• queue stuck
• DLQ bertambah
• consumer CPU/log tinggi
• downstream dipukul terus

Handling:

• validate schema
• classify error retryable/non-retryable
• cap retry count
• DLQ with reason
• alert on DLQ growth
• provide replay/fix workflow

17. Ordering in RabbitMQ Classic

Queue menjaga order delivery dalam kondisi sederhana.

Namun ordering bisa berubah karena:

• multiple consumers
• nack/requeue
• retry queue
• priority queue
• consumer crash
• concurrent processing

Jika ordering per entity penting, desain harus eksplisit.

Opsi:

• one queue per shard/entity group
• single active consumer pattern
• message group routing
• external sequence/version check

Jangan mengandalkan FIFO global untuk workflow enterprise kompleks.

18. Idempotency

RabbitMQ classic delivery umumnya at-least-once saat manual ack.

Consumer harus idempotent.

Common idempotency keys:

• messageId
• business event id
• command id
• order id + version
• correlation id + operation id

Idempotency store bisa di:

• PostgreSQL inbox table
• Redis key with TTL
• domain table unique constraint
• processed message table

19. Request/Reply Pattern

RabbitMQ mendukung request/reply menggunakan:

• reply-to property
• correlation id
• temporary callback queue
• direct reply-to feature

Pattern ini berguna untuk async RPC.

Namun harus hati-hati.

Risiko:

• timeout ambiguity
• duplicate response
• caller crash
• callback queue lifecycle
• backpressure tidak jelas
• tight coupling seperti synchronous RPC

Untuk enterprise service, request/reply harus punya timeout dan fallback jelas.

20. RabbitMQ vs Kafka Decision Model

Gunakan RabbitMQ classic ketika:

work distribution matters
routing matters
message should be consumed and removed
per-task acknowledgement matters
broker-mediated command queue is natural

Gunakan Kafka ketika:

event log matters
multiple independent consumers need same history
replay matters
stream processing matters
partitioned ordering matters
retention after consumption matters

Gunakan RabbitMQ Stream ketika:

RabbitMQ ecosystem is already standard
stream/log semantics are needed
high throughput append-only stream is useful
Kafka is not available or not desired

21. RabbitMQ Stream Mental Model

RabbitMQ Stream berbeda dari classic queue.

Ia menyediakan:

• append-only stream
• offset-based consumption
• retention
• high throughput
• multiple consumers
• replay-like consumption

Mental model:

stream keeps messages based on retention
consumer tracks offset
many consumers can read independently

Ini lebih dekat ke Kafka daripada classic RabbitMQ queue.

22. RabbitMQ Stream vs Kafka

RabbitMQ Stream dan Kafka sama-sama log-oriented, tetapi ekosistem dan operations berbeda.

Pertanyaan review:

Is the organization already operating Kafka reliably?
Is RabbitMQ already the platform standard?
Is stream processing required?
Is ecosystem integration needed: Schema Registry, Kafka Streams, ksqlDB?
What is retention and replay requirement?
What is throughput target?
Who owns broker operations?

Kafka biasanya lebih matang untuk distributed event log ecosystem.

RabbitMQ Stream bisa menarik jika RabbitMQ sudah menjadi messaging platform utama dan kebutuhan stream tidak sebesar Kafka ecosystem.

23. Message Contract

RabbitMQ message harus punya contract eksplisit.

Minimal:

message type
schema version
message id
correlation id
causation id
tenant id if applicable
created at
producer service
retry count
content type

Tanpa metadata, debugging DLQ dan trace antar service menjadi sulit.

24. Header Strategy

Headers umum:

• message-id
• correlation-id
• causation-id
• traceparent
• baggage
• tenant-id
• schema-version
• producer-service
• retry-count
• original-exchange
• original-routing-key

Context propagation harus konsisten dengan HTTP dan Kafka strategy.

25. Schema Evolution

RabbitMQ tidak otomatis menyediakan schema governance.

Jika payload JSON/XML/Avro/Protobuf dipakai, compatibility harus dikelola.

Review:

• schema source of truth
• versioning
• backward compatibility
• unknown field tolerance
• required field policy
• old consumer compatibility
• DLQ for deserialization failure

RabbitMQ message tetap contract, meski tidak berada di Kafka topic.

26. Java Client Lifecycle

RabbitMQ Java client lifecycle:

ConnectionFactory
Connection
Channel
Consumer
Delivery callback
Ack/Nack
Shutdown listener

Concern:

• connection reuse
• channel per thread model
• automatic recovery
• topology recovery
• publisher confirm
• consumer cancellation
• graceful shutdown
• blocked connection notification

Channel biasanya tidak boleh dipakai sembarang secara concurrent tanpa memahami client semantics.

27. JAX-RS Integration Boundary

Pola integrasi umum:

Pattern A — HTTP command publishes message

POST /orders/{id}/submit
  validate request
  persist command/state
  publish message
  return accepted/result

Butuh outbox jika database commit dan publish harus recoverable.

Pattern B — Consumer updates database

queue message -> consumer -> DB update -> ack

Butuh idempotency dan transaction boundary jelas.

Pattern C — HTTP reads projection built by consumer

consumer builds read model
JAX-RS endpoint reads view

Butuh staleness semantics.

28. Publisher Confirms and Outbox

Jika API endpoint melakukan:

update database
publish RabbitMQ message

maka ada partial failure:

DB commit succeeds, publish fails
publish succeeds, response fails
publish uncertainty during network failure

Outbox pattern tetap relevan.

transaction writes domain state + outbox row
publisher job reads outbox and publishes
confirm received
mark outbox sent

Publisher confirms membantu, tetapi tidak menggantikan outbox untuk DB-message atomicity.

29. Consumer Transaction Boundary

Consumer safe pattern:

receive message
start DB transaction
check inbox/idempotency
apply state change
record processed message
commit transaction
ack message

Jika ack sebelum commit, message bisa hilang.

Jika commit sebelum ack, message bisa redeliver.

Karena itu idempotency/inbox wajib.

30. Operational Metrics

RabbitMQ metrics penting:

• queue depth
• ready messages
• unacked messages
• publish rate
• deliver rate
• ack rate
• redelivery rate
• consumer count
• consumer utilization
• DLQ depth
• connection/channel count
• memory alarm
• disk alarm
• node health

Application metrics:

• processing latency
• ack latency
• retry count
• failure reason
• DLQ reason
• duplicate skip count
• downstream dependency latency

31. Alerting Strategy

Alert bukan hanya queue depth tinggi.

Alert yang lebih berguna:

queue depth growing while consumer count stable
unacked messages growing
DLQ non-empty for critical queue
redelivery rate spike
consumer utilization low
publish rate drops unexpectedly
memory/disk alarm active
oldest message age exceeds SLO

Oldest message age sering lebih bermakna daripada queue depth.

32. Failure Modes

32.1 Unroutable Message

Message publish ke exchange tetapi tidak ada queue binding cocok.

Mitigasi:

• mandatory publish
• alternate exchange
• publisher confirms
• topology validation

32.2 Consumer Crash Before Ack

Message redelivered.

Mitigasi:

• idempotent consumer
• inbox table
• manual ack after commit

32.3 Retry Storm

Poison message terus direqueue.

Mitigasi:

• delayed retry
• max retry
• DLQ
• error classification

32.4 Queue Backlog

Producer lebih cepat dari consumer.

Mitigasi:

• scale consumers
• tune prefetch
• load shedding upstream
• rate limit producers
• inspect downstream bottleneck

32.5 Broker Memory/Disk Alarm

Broker menahan publish atau menurun performanya.

Mitigasi:

• queue length limit
• TTL
• capacity planning
• DLQ retention
• monitor disk/memory

32.6 Duplicate Processing

Redelivery, retry, or uncertain publish/ack.

Mitigasi:

• message id
• idempotency store
• unique constraints
• inbox pattern

33. Security Considerations

Review:

• TLS/mTLS to broker
• username/password or certificate auth
• vhost isolation
• exchange/queue permissions
• least privilege producer/consumer
• secret rotation
• payload encryption if required
• PII in payload/header
• audit access to management UI
• network policy from Kubernetes pods

RabbitMQ management UI access is operationally sensitive.

34. Multi-Tenancy Considerations

Tenant strategy bisa berupa:

• tenant id in message header
• tenant-specific routing key
• tenant-specific queue
• tenant-specific vhost
• tenant-specific exchange

Trade-off:

header tenant id -> simpler topology, weaker isolation
queue per tenant -> stronger isolation, more operational objects
vhost per tenant -> strong isolation, higher ops overhead

Untuk enterprise CPQ/order systems, tenant boundary harus jelas karena message bisa membawa order, pricing, catalog, dan customer data.

35. RabbitMQ Topology as Code

Exchange, queue, binding, policy, DLX, TTL, and permission harus dikelola sebagai code.

Anti-pattern:

someone manually created queue in management UI

Risiko:

• drift antar environment
• missing DLX
• wrong durability
• no retry policy
• no owner
• inconsistent permissions

Topology as code bisa melalui IaC, Helm values, operator, atau deployment initializer yang governed.

36. Internal Verification Checklist

Untuk konteks CSG Quote & Order atau service enterprise sejenis, jangan mengasumsikan RabbitMQ dipakai.

Verifikasi:

• apakah ada dependency RabbitMQ Java client atau Spring AMQP
• apakah ada RabbitMQ connection config
• apakah ada exchange/queue/binding definition
• apakah ada RabbitMQ Stream client usage
• apakah ada DLX/DLQ convention
• apakah publisher confirms aktif
• apakah manual ack dipakai
• apakah prefetch dikonfigurasi
• apakah retry queue/delayed exchange dipakai
• apakah idempotency/inbox pattern tersedia
• apakah message contract terdokumentasi
• apakah correlation/trace headers dipropagasi
• apakah broker TLS/mTLS dipakai
• apakah topology dikelola sebagai code
• apakah ada dashboard queue depth/oldest age/DLQ
• apakah ada runbook replay/requeue DLQ

37. PR Review Checklist

Saat mereview RabbitMQ change:

• exchange type sesuai use case
• routing key naming konsisten
• queue durable jika required
• DLX/DLQ tersedia
• retry policy tidak membuat hot loop
• publisher confirms dipakai untuk critical message
• manual ack setelah side effect commit
• idempotency/inbox tersedia
• prefetch masuk akal
• message schema/version jelas
• trace/correlation/tenant headers ada
• security permission least privilege
• topology dikelola sebagai code
• observability dan alert tersedia
• DLQ recovery procedure jelas

38. Senior Engineer Heuristics

RabbitMQ classic cocok untuk:

command queue
work distribution
routing-heavy integration
short-lived task processing
broker-mediated async RPC

Kafka cocok untuk:

event log
many independent consumers
history and replay
stream processing
materialized view rebuild

RabbitMQ Stream cocok untuk:

stream semantics in RabbitMQ ecosystem
append-only high-throughput messaging
offset-based consumption without adopting Kafka platform

Jangan memilih broker hanya karena familiar.

Pilih berdasarkan delivery model, replay need, ordering need, routing need, operations capability, and governance maturity.

39. Key Takeaways

• RabbitMQ classic adalah routing/queue broker, bukan event log seperti Kafka.
• Exchange, binding, queue, ack, prefetch, DLX, dan retry adalah konsep inti.
• Manual ack + idempotency adalah baseline untuk reliable consumer.
• DLQ perlu owner, alert, triage, dan replay procedure.
• RabbitMQ Stream memberi model stream/log dalam ekosistem RabbitMQ.
• RabbitMQ, RabbitMQ Stream, dan Kafka harus dibandingkan berdasarkan semantics, bukan popularitas.
• Semua message contract tetap perlu versioning, tracing, tenant context, dan governance.