RabbitMQ Observability
RabbitMQ observability for enterprise Java/JAX-RS systems: Management UI, Prometheus metrics, queue depth, ready/unacked messages, publish/deliver/ack/redelivery rates, consumer utilisation, connection/channel count, node health, alarms, DLQ/retry dashboards, alerting, debugging, and production readiness.
RabbitMQ Observability
1. Core idea
RabbitMQ observability is not just about checking whether the broker is up.
It is about answering production questions quickly:
Was the message published?
Was it routed?
Did it land in the expected queue?
Is a consumer attached?
Is the consumer keeping up?
Are messages stuck as ready or unacked?
Are messages being redelivered?
Are messages going to DLQ?
Is retry helping or amplifying failure?
Is the broker throttling publishers?
Is memory, disk, or network becoming the bottleneck?
Is the downstream database or service actually the slow component?
For senior backend engineers, RabbitMQ observability has three jobs:
1. Detect user-impacting degradation early.
2. Explain where the messaging lifecycle is blocked.
3. Provide evidence for capacity, correctness, and incident decisions.
A RabbitMQ dashboard that only shows "messages in queue" is not enough.
A production-ready observability model must cover:
producer
exchange/routing
queue
consumer
ack/nack/redelivery
retry/DLQ
broker node
cluster
network
storage
application side effects
In Java/JAX-RS systems, RabbitMQ metrics must also be joined with:
HTTP request rate
publisher success/failure rate
publisher confirm latency
consumer processing latency
PostgreSQL latency
DB connection pool saturation
Redis/Kafka/downstream dependency health
Kubernetes pod lifecycle
business state transition metrics
Otherwise, RabbitMQ will be blamed for failures caused somewhere else.
2. Monitoring vs observability
Monitoring asks:
Is something outside expected limits?
Observability asks:
Given this symptom, can we infer what is happening internally?
For RabbitMQ, monitoring might tell you:
queue_depth > 100000
consumer_count = 0
redelivery_rate increased
node memory alarm active
Observability should help explain:
queue_depth increased because consumers are alive but slow
consumers are slow because DB writes are blocked
DB writes are blocked because a migration changed an index plan
messages are being retried because one downstream integration returns 429
redelivery is spiking because consumers crash after side effect but before ack
The first level detects the fire.
The second level tells you where to send the engineer.
3. Observability surfaces
RabbitMQ can be observed through several surfaces.
3.1 Management UI
Use Management UI for quick operational inspection:
queues
exchanges
bindings
connections
channels
consumers
node memory/disk status
message rates
ready/unacked counts
DLQ growth
manual inspect/requeue/purge operations if authorized
Management UI is useful for incident triage, but should not be your long-term metrics system.
Use it to answer targeted questions:
Is queue X growing?
Does queue X have consumers?
Are messages ready or unacked?
Which connection/channel owns the consumers?
Which vhost contains the topology?
Is the node under memory/disk alarm?
Do not rely on humans refreshing the UI as your monitoring strategy.
3.2 Prometheus metrics
Prometheus is the common production path for long-term RabbitMQ metrics.
RabbitMQ exposes metrics via the Prometheus plugin.
Typical use:
RabbitMQ node -> /metrics endpoint -> Prometheus scrape -> Grafana dashboard -> alert rules
Important distinction:
aggregated metrics useful for cluster health and overview dashboards
per-object metrics useful for queue/exchange/connection/channel debugging
application metrics useful for producer/consumer correctness and latency
Do not scrape every possible high-cardinality detail blindly.
Per-queue and per-connection metrics can become expensive if the broker has many queues, connections, or channels.
3.3 Application metrics
RabbitMQ broker metrics are necessary but incomplete.
Your Java services must emit application-side metrics:
publisher attempts
publisher success
publisher failure
publisher confirm latency
returned/unroutable messages
message serialization failures
outbox rows pending
outbox publish lag
consumer deliveries received
consumer processing success
consumer processing failure
consumer processing latency
ack count
nack count
reject count
idempotency duplicate count
inbox insert conflict count
DLQ publish count if consumer-side DLQ publisher exists
Broker metrics tell you what RabbitMQ sees.
Application metrics tell you what the business service believes happened.
Both are required.
3.4 Logs
Logs should support traceability, not replace metrics.
Good logs include:
message_id
correlation_id
causation_id
trace_id / traceparent
tenant_id if applicable
message_type
message_version
exchange
routing_key
queue
consumer_name
attempt/retry count
x-death summary when present
business aggregate id
result: ack/nack/reject/dlq
error category
Bad logs include:
raw full payload with PII
only "failed to process message"
no correlation id
no queue/exchange/routing key
no message type
stack traces with no lifecycle context
3.5 Distributed tracing
Tracing is valuable when message flows cross async boundaries:
HTTP request -> JAX-RS resource -> service transaction -> outbox -> RabbitMQ publish -> consumer -> DB update -> downstream HTTP call
The trace context must be propagated through headers.
At minimum:
traceparent
correlation_id
message_id
causation_id
source_service
Trace spans should identify:
publish span
broker/routing boundary if modeled
consume span
processing span
ack/nack result
external dependency spans
Do not expect broker metrics alone to reconstruct distributed causality.
4. Observability model by lifecycle stage
A RabbitMQ message lifecycle can be observed in stages.
For each stage, ask a different question.
| Stage | Key question | Primary evidence |
|---|---|---|
| JAX-RS request | Did business input arrive? | HTTP metrics, access logs, request trace |
| Service transaction | Was durable business state created? | PostgreSQL row, transaction logs, business metric |
| Publisher/outbox | Was publish attempted and confirmed? | outbox lag, publish attempts, confirm latency |
| Exchange/routing | Was message routable? | returned message count, alternate exchange, bindings |
| Queue | Did queue receive and buffer? | ready messages, enqueue rate, queue depth |
| Consumer delivery | Is consumer receiving? | deliver/get rate, consumer count, consumer utilisation |
| Processing | Is business work succeeding? | processing success/failure latency, DB/downstream metrics |
| Ack/nack | What did consumer decide? | ack/nack/reject counts, unacked count |
| Retry/DLQ | Is failure isolated? | retry queue size, DLQ size, x-death headers |
A strong dashboard follows this lifecycle.
A weak dashboard shows only queue depth.
5. Queue metrics that matter
5.1 Total queue depth
Total queue depth is usually:
ready messages + unacknowledged messages
It answers:
How much work is buffered or in-flight?
But it does not explain why.
A high queue depth can mean:
producer spike
consumer down
consumer too slow
prefetch too high
messages stuck unacked
retry loop
downstream dependency slow
intentional backlog from batch workload
Never alert only on absolute depth without context.
5.2 Ready messages
Ready messages are queued and available for delivery.
High ready count usually means:
no consumers
not enough consumers
consumer prefetch saturated
consumer cannot keep up
broker cannot deliver fast enough
queue is intentionally backlogged
Debug path:
ready high
-> check consumer count
-> check deliver rate
-> check consumer utilisation
-> check consumer logs
-> check downstream dependency latency
5.3 Unacknowledged messages
Unacked messages were delivered but not acknowledged yet.
High unacked count usually means:
consumer processing is slow
prefetch is too high
consumer thread pool is blocked
consumer crashed/hung without closing connection
DB transaction is stuck
external call is slow
manual ack is not reached
Unacked growth is often more dangerous than ready growth because the broker has already handed work to consumers.
Debug path:
unacked high
-> identify consumers/channels
-> check prefetch
-> check processing latency
-> check thread dumps if Java process is alive
-> check DB connection pool and locks
-> check downstream timeouts
5.4 Consumer count
Consumer count answers:
Is anyone subscribed to the queue?
Zero consumers on a critical queue usually means:
deployment issue
consumer pod crashed
credential/permission failure
queue name mismatch
vhost mismatch
network/TLS failure
consumer cancelled by broker
feature flag disabled consumer startup
However, zero consumers can be normal for:
temporary queues
inactive workflows
manual replay queues
parking lot queues
low-priority scheduled workloads
Alert rules must distinguish critical always-on queues from intentional idle queues.
5.5 Consumer utilisation
Consumer utilisation measures how often the queue can immediately deliver to consumers.
Low consumer utilisation with backlog may mean:
consumers are busy
prefetch is saturated
consumer processing is slow
network path to consumers is slow
consumer acknowledgements are slow
High consumer utilisation with backlog may mean:
consumer is ready but broker/routing/storage is bottlenecked
or publish rate exceeds consume rate despite available consumers
Interpret utilisation with:
ready count
unacked count
consumer count
ack rate
deliver rate
processing latency
5.6 Publish, deliver, and ack rates
These rates expose the flow balance.
Healthy steady state often looks like:
publish rate ~= deliver rate ~= ack rate
queue depth stable
redelivery low
DLQ low
consumer latency within SLO
Backlog building looks like:
publish rate > ack rate
queue depth increasing
ready and/or unacked rising
Consumer problem often looks like:
publish rate normal
deliver rate high or normal
ack rate low
unacked rising
processing errors increasing
Routing/consumer absence often looks like:
publish rate high
deliver rate low or zero
ready messages rising
consumer count zero or low
5.7 Redelivery rate
Redelivery means messages are being delivered again after not being successfully completed.
Possible causes:
consumer crash before ack
nack/reject with requeue=true
connection loss while messages are unacked
consumer timeout / channel closure
application exception before ack
broker failover
retry topology returning messages
Redelivery is not always a bug, but a sustained redelivery spike is a correctness alarm.
It can indicate:
poison message
non-idempotent consumer risk
retry storm
external dependency outage
bad deployment
schema incompatibility
5.8 DLQ and retry queue size
DLQ size answers:
How much failed work is isolated?
Retry queue size answers:
How much failed work is waiting to be attempted again?
But these metrics must be paired with rates:
DLQ inflow rate
DLQ drain/replay rate
retry inflow rate
retry outflow rate
retry age
oldest message age
A DLQ with 20 messages can be severe if all 20 are high-value orders.
A DLQ with 50,000 low-priority cache invalidation messages may be operationally noisy but not immediately customer-critical.
Always connect queue metrics to business impact.
6. Broker/node metrics that matter
6.1 Node availability
Basic node health:
node up/down
RabbitMQ process running
Erlang VM responsive
cluster membership stable
listener accepting connections
management/metrics endpoint responsive
Health checks are useful, but they are not full observability.
A node can be "up" while:
memory alarm blocks publishers
disk alarm blocks writes
consumers are stalled
queues are growing
cluster has minority partition
publish confirms are slow
6.2 Memory usage
RabbitMQ uses memory for:
connections
channels
queues
messages
metadata
plugins
Erlang runtime
OS page cache interaction
Memory pressure can trigger flow control and publishing blocks.
Important questions:
Is memory usage rising with queue depth?
Is memory rising with connection/channel count?
Is a specific queue responsible?
Are large messages involved?
Are quorum queues or streams consuming expected memory/disk?
Is the OS left enough memory to avoid swapping?
Memory alarm is not just an infrastructure concern.
It changes publisher behavior.
6.3 Disk usage
Disk matters when:
messages are persistent
queues are durable
quorum queues are used
streams are used
classic queues page messages to disk
DLQ/retry queues accumulate
Disk alarm is severe because RabbitMQ protects itself from unsafe writes.
Track:
disk free
disk alarm status
disk read/write rate
fsync latency if available at infra level
queue index/message store activity
oldest messages in DLQ/retry
stream retention disk use
6.4 File descriptors and sockets
High connection counts consume OS resources.
Track:
connection count
channel count
socket descriptors
file descriptors
connection churn
channels per connection
Symptoms of connection/channel misuse:
many short-lived connections
connection storm during deploy
channel leak
per-request connection creation
publisher creates channel per message
consumer reconnect loop
6.5 Cluster metrics
For clusters, observe:
node count
cluster partition status
inter-node traffic
queue leader distribution
quorum queue member health
stream replica health
node alarms
rolling upgrade state
A single-node dashboard is not enough for a cluster.
You need both:
per-node view
cluster aggregate view
per-queue leader/replica awareness for critical queues
7. Connection and channel observability
RabbitMQ clients use long-lived connections and channels.
For Java systems, monitor:
connections per service
channels per service
consumer channels
publisher channels
connection churn
connection recovery count
channel closure count
blocked/unblocked connection events
heartbeat timeout count
TLS/auth failure count
Expected pattern:
small number of long-lived connections per JVM
bounded channel count
stable during normal traffic
brief controlled churn during deploy
Suspicious pattern:
connection count grows forever
channel count grows forever
connection count spikes per request
many connections idle but open
many channels closed by broker
heartbeats fail under load
blocked connection events ignored by publisher
Instrumentation should tag connections by:
service
pod
environment
vhost
role: publisher / consumer / admin / stream
Avoid tagging by high-cardinality values such as message ID.
8. Producer observability
A producer is not observable just because the HTTP endpoint returned 200.
Producer metrics should answer:
Did the app attempt to publish?
Was the message accepted by broker?
Was it confirmed?
Was it returned as unroutable?
Was the publish retried?
How long did confirm take?
Is outbox lag growing?
Recommended producer metrics:
rabbitmq_publish_attempt_total{service,message_type,exchange}
rabbitmq_publish_success_total{service,message_type,exchange}
rabbitmq_publish_failure_total{service,message_type,error_category}
rabbitmq_publish_confirm_latency_seconds{service,message_type}
rabbitmq_publish_returned_total{service,exchange,routing_key,message_type}
rabbitmq_publish_confirm_timeout_total{service,message_type}
rabbitmq_connection_blocked_total{service}
rabbitmq_outbox_pending{service,message_type}
rabbitmq_outbox_oldest_age_seconds{service,message_type}
Key failure signals:
confirm latency increases
returned messages appear
outbox pending rows increase
publish retry count increases
connection blocked events increase
publish success drops while HTTP success stays high
The last case is dangerous: the API may claim success while async work is not actually progressing.
9. Consumer observability
Consumer metrics should answer:
Did the consumer receive deliveries?
How long does processing take?
Did processing succeed?
Was the message acked, nacked, rejected, or dead-lettered?
Are duplicates being detected?
Is idempotency working?
Which downstream dependency is slow?
Recommended consumer metrics:
rabbitmq_consumer_delivery_total{service,queue,message_type}
rabbitmq_consumer_success_total{service,queue,message_type}
rabbitmq_consumer_failure_total{service,queue,message_type,error_category}
rabbitmq_consumer_processing_latency_seconds{service,queue,message_type}
rabbitmq_consumer_ack_total{service,queue}
rabbitmq_consumer_nack_total{service,queue,requeue}
rabbitmq_consumer_reject_total{service,queue,requeue}
rabbitmq_consumer_duplicate_total{service,queue,message_type}
rabbitmq_inbox_insert_conflict_total{service,message_type}
rabbitmq_consumer_shutdown_inflight{service,queue}
Consumer logs should include:
message_id
correlation_id
queue
message_type
attempt
business_key
ack_decision
error_category
Do not only log stack traces.
A stack trace without message identity is not enough for production recovery.
10. Retry and DLQ observability
Retry/DLQ observability must distinguish controlled failure isolation from uncontrolled retry storm.
Track:
retry queue depth
retry inflow rate
retry outflow rate
retry oldest age
DLQ depth
DLQ inflow rate
DLQ oldest age
parking lot depth
manual replay count
replay success/failure count
x-death reason distribution
message type distribution in DLQ
producer/consumer version associated with DLQ messages
Important debug questions:
Are messages failing once or repeatedly?
Are all failures from one message type?
Are all failures from one tenant/customer?
Are all failures from one consumer version?
Did DLQ spike after a deployment?
Did retry spike after a downstream outage?
Are messages stuck in retry because TTL/DLX binding is wrong?
Are replays causing duplicate business effects?
DLQ dashboard should show business impact, not only message count.
Examples:
failed order submissions
failed fulfillment commands
failed approval notifications
failed downstream integration messages
failed cache invalidation events
11. Dashboard design
A production RabbitMQ dashboard set should not be one giant page.
Use layered dashboards.
11.1 Broker overview dashboard
Purpose:
Is the RabbitMQ cluster healthy?
Include:
node up/down
cluster partition status
memory usage and alarm
disk usage and alarm
connection count
channel count
queue count
message rates
node CPU/network/disk IO
inter-node communication
leader distribution for critical quorum queues
Audience:
platform
SRE
on-call backend leads
11.2 Service-owned queue dashboard
Purpose:
Is my service's messaging flow healthy?
Include:
queue depth
ready messages
unacked messages
consumer count
consumer utilisation
publish/deliver/ack rates
redelivery rate
oldest message age
DLQ depth
retry queue depth
consumer processing latency
consumer error rate
outbox lag
inbox duplicate count
Audience:
backend service owner
feature team
incident responder
11.3 Retry/DLQ dashboard
Purpose:
What failed work is accumulating?
Include:
DLQ size by queue/message type
DLQ inflow rate
oldest DLQ age
retry inflow/outflow
retry attempt distribution
x-death reason
manual replay activity
business aggregate IDs sampled safely
release version correlation
Audience:
backend
support
SRE
integration team
11.4 Capacity dashboard
Purpose:
Are we approaching broker or consumer limits?
Include:
publish throughput trend
consume throughput trend
confirm latency trend
queue depth trend
message size trend
connection/channel trend
memory/disk trend
consumer latency trend
DB latency correlation
network IO
storage IO
Audience:
architecture
platform
capacity planning
12. Alerting strategy
Alerting must be symptom-aware and ownership-aware.
Bad alert:
queue_depth > 1000
Better alert:
critical queue depth increasing for 15 minutes
AND ack rate < publish rate
AND oldest message age > SLO threshold
Bad alert:
DLQ depth > 0
Better alert:
DLQ inflow > 0 for critical order command queue
OR DLQ oldest age > operational recovery SLO
OR DLQ growth rate increased after deployment
Alert on:
consumer count = 0 for always-on critical queue
oldest message age exceeds business SLO
unacked messages increasing continuously
redelivery rate spike
DLQ inflow spike
retry queue not draining
memory alarm active
disk alarm active
connection blocked active
publisher confirm latency high
outbox oldest age high
management/metrics endpoint unreachable
cluster partition detected
quorum queue unavailable
Do not alert on every temporary burst.
RabbitMQ often buffers bursts by design.
Alert when backlog threatens:
business SLO
data freshness
customer-visible workflow
broker resource safety
manual recovery effort
13. Observability anti-patterns
13.1 Queue depth only
Queue depth alone cannot distinguish:
producer spike
consumer outage
consumer slowness
unacked stuck
retry storm
intentional backlog
Always pair depth with:
publish rate
ack rate
consumer count
ready/unacked split
oldest message age
processing latency
13.2 No message identity in logs
Without message identity, you cannot safely replay, deduplicate, or trace failures.
Minimum log identity:
message_id
correlation_id
message_type
business_key
queue
attempt
13.3 No outbox/inbox metrics
Broker metrics cannot tell whether your database transaction and idempotency logic are healthy.
Missing metrics:
outbox pending
outbox oldest age
inbox duplicate count
consumer processing status
business transition success
13.4 Alerting on non-owned queues
Every alert must have an owner.
If a queue has no owner, it should not exist in production.
13.5 High-cardinality metric labels
Avoid labels like:
message_id
correlation_id
tenant_id if tenant count is high and unbounded
order_id
quote_id
raw routing key if extremely high cardinality
exception message
Use logs/traces for high-cardinality investigation.
Use metrics for bounded dimensions.
14. Java/JAX-RS impact
For a JAX-RS service, RabbitMQ observability starts at the HTTP boundary.
A typical async command flow:
POST /orders
validate request
insert order row
insert outbox row
commit
outbox publisher publishes command/event
consumer processes message
order state changes later
Required observability:
HTTP accepted count
idempotency key duplicate count
outbox pending count
publish confirm latency
consumer processing latency
order state transition latency
DLQ count for order messages
status endpoint freshness
If API returns 202 Accepted, dashboard must show whether async processing is actually progressing.
Otherwise, support teams will see:
API success
but order stuck
no obvious error
That is an observability failure.
15. PostgreSQL/MyBatis/JDBC impact
RabbitMQ consumer slowness is often PostgreSQL slowness in disguise.
Correlate queue metrics with:
DB connection pool active/waiting
transaction duration
row lock wait
deadlock count
query latency
outbox pending rows
inbox insert conflicts
business state transition failures
migration/deployment timeline
Common patterns:
unacked rising + DB pool exhausted
-> consumer threads blocked waiting for DB connection
ready rising + consumer count stable + processing latency high
-> consumer cannot keep up, downstream DB/service slow
redelivery rising + duplicate conflicts rising
-> consumer crashes or nacks after partial processing
outbox oldest age rising + queue publish rate low
-> publisher/poller issue, not RabbitMQ queue issue
MyBatis/JDBC instrumentation should expose:
query latency by mapper/operation
transaction duration
connection acquisition time
error category
commit/rollback count
16. Kubernetes impact
In Kubernetes, RabbitMQ observability must include both broker and client workloads.
For RabbitMQ broker pods:
pod restart count
node placement
PVC usage
CPU throttling
memory limit pressure
readiness/liveness probe failures
network policy changes
operator reconciliation events
For Java client pods:
consumer replica count
pod restarts
rolling update timeline
SIGTERM handling
in-flight message drain
connection recovery count
DNS resolution errors
secret reload/rotation events
CPU throttling
JVM heap/GC metrics
Important Kubernetes-specific failure pattern:
rolling deployment kills consumers
messages return to ready
new pods start
prefetch multiplies across replicas
DB pool saturates
redelivery and duplicate count spike
Observability must connect deployment events to message behavior.
17. AWS, Azure, on-prem, and hybrid considerations
17.1 AWS
If using Amazon MQ for RabbitMQ or self-managed RabbitMQ on EC2/EKS, verify:
CloudWatch metrics/logs integration
VPC/subnet/security group changes
broker maintenance window
storage usage
Multi-AZ behavior
TLS certificate status
managed broker limits
Application dashboards should still expose producer/consumer metrics.
Managed broker metrics do not replace service-level correctness metrics.
17.2 Azure
For AKS/VM-based RabbitMQ, verify:
Azure Monitor integration
VNet/NSG changes
Key Vault secret rotation
storage/PVC pressure
private connectivity status
AKS node health
17.3 On-prem
For on-prem RabbitMQ, verify:
OS-level metrics
disk layout
filesystem free space
network/firewall changes
certificate expiry
patch window
backup/restore evidence
monitoring stack ownership
17.4 Hybrid
For hybrid cloud/on-prem flows, add:
cross-region latency
link availability
federation/shovel lag
duplicate risk
ordering risk
credential expiry
message movement audit
18. Production debugging playbook
18.1 Symptom: message not processed
Check:
1. Was message published?
2. Was publisher confirm received?
3. Was message returned as unroutable?
4. Is exchange correct?
5. Is routing key correct?
6. Is binding present?
7. Did queue depth increase?
8. Is consumer count > 0?
9. Is message ready or unacked?
10. Did consumer log receive it?
11. Did consumer ack/nack/reject?
12. Did it move to retry/DLQ?
13. Did business DB state change?
18.2 Symptom: queue depth increasing
Check:
publish rate vs ack rate
ready vs unacked split
consumer count
consumer utilisation
processing latency
DB latency
external dependency error rate
pod restarts
recent deployment
18.3 Symptom: unacked increasing
Check:
prefetch
consumer thread dump
DB connection pool
external call timeout
manual ack path
consumer error swallowing
pod CPU throttling
18.4 Symptom: redelivery spike
Check:
consumer crash/restart
connection loss
nack/requeue=true path
schema incompatibility
poison message
bad deployment
retry topology
idempotency failures
18.5 Symptom: DLQ spike
Check:
message type distribution
x-death reason
consumer version
deployment timeline
tenant/customer concentration
payload schema version
business validation failure
external dependency outage
manual replay safety
19. Security and privacy observability
Do not observe RabbitMQ by leaking sensitive data.
Rules:
never log full payload by default
redact PII in headers and payload summaries
do not expose DLQ payload to broad dashboards
restrict Management UI access
restrict manual get/requeue/purge permissions
log administrative actions
track replay operations
track permission failures
track TLS/auth failures
Sensitive message stores include:
live queues
DLQs
parking lot queues
retry queues
outbox tables
inbox tables
logs
traces
metrics labels
manual replay exports
Compliance review should include observability artifacts.
20. RabbitMQ observability checklist
For each critical messaging flow, verify:
[ ] producer publish attempts are measured
[ ] publisher confirm latency is measured
[ ] returned/unroutable messages are measured
[ ] outbox pending count and oldest age are measured
[ ] queue ready/unacked/total depth are visible
[ ] publish/deliver/ack/redelivery rates are visible
[ ] consumer count and utilisation are visible
[ ] consumer processing success/failure/latency are measured
[ ] duplicate/idempotency metrics are measured
[ ] retry queue size and age are visible
[ ] DLQ size, inflow, and oldest age are visible
[ ] broker memory/disk alarms are visible
[ ] connection/channel count is visible
[ ] Kubernetes pod restart/deployment events are correlated
[ ] DB/downstream latency is correlated
[ ] logs contain message identity and correlation IDs
[ ] traces propagate across HTTP -> RabbitMQ -> consumer
[ ] alert rules have owners and runbooks
[ ] dashboards distinguish critical queues from best-effort queues
21. Internal verification checklist
Check these in the CSG/team environment before assuming anything:
[ ] Which RabbitMQ clusters/vhosts are used by Quote & Order?
[ ] Which dashboards are official: Grafana, CloudWatch, Azure Monitor, Datadog, custom?
[ ] Is rabbitmq_prometheus enabled?
[ ] Are per-object metrics scraped, or only aggregate metrics?
[ ] Which queues are customer-critical?
[ ] Which queues are allowed to backlog intentionally?
[ ] Which queues have zero-consumer alerts?
[ ] Which queues have DLQ/retry alerts?
[ ] Is oldest message age monitored?
[ ] Is publisher confirm latency monitored per service?
[ ] Is outbox lag monitored?
[ ] Is inbox duplicate count monitored?
[ ] Are message IDs/correlation IDs standardized?
[ ] Is traceparent propagated through message headers?
[ ] Are retry/DLQ messages searchable safely?
[ ] Who owns manual replay?
[ ] Who can purge/requeue/get messages from Management UI?
[ ] Are broker alarms linked to application incident procedures?
[ ] Are deployment events overlaid on RabbitMQ dashboards?
[ ] Are DB metrics correlated with consumer lag?
[ ] Are incident notes reviewed for recurring RabbitMQ symptoms?
22. PR review checklist
When reviewing code or architecture touching RabbitMQ, ask:
[ ] What metrics will prove publish succeeded?
[ ] What metrics will prove consume succeeded?
[ ] What logs identify the message without exposing sensitive payload?
[ ] What trace context crosses the async boundary?
[ ] How is outbox lag observed?
[ ] How is inbox/idempotency observed?
[ ] What happens to failed messages?
[ ] Is DLQ growth visible and alerted?
[ ] Is retry growth visible and bounded?
[ ] Is oldest message age visible?
[ ] Is consumer count monitored?
[ ] Is unacked growth monitored?
[ ] Are broker alarms monitored?
[ ] Is there a runbook for each alert?
[ ] Does the dashboard distinguish infrastructure failure from business validation failure?
[ ] Are metrics labels bounded and privacy-safe?
23. Senior engineer mental model
Do not ask only:
Is RabbitMQ up?
Ask:
Is the messaging lifecycle making forward progress?
Where is the first non-progressing boundary?
Is the failure in routing, buffering, delivery, processing, acknowledgement, retry, or side effect?
Can we prove the business state is correct?
Can we replay safely?
Can we explain customer impact?
Can the on-call engineer answer this in five minutes?
RabbitMQ observability is successful when production engineers can move from symptom to cause without guessing.
24. References
- RabbitMQ Documentation: Monitoring
- RabbitMQ Documentation: Monitoring with Prometheus and Grafana
- RabbitMQ Documentation: Production Deployment Guidelines
- RabbitMQ Documentation: Consumer Acknowledgements and Publisher Confirms
- RabbitMQ Documentation: Memory Use
- RabbitMQ Documentation: Persistence Configuration
- OpenTelemetry Documentation: Context Propagation and Messaging Semantic Conventions
You just completed lesson 35 in deepen practice. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.