producer transaction boundary
exchange routing
queue durability
consumer concurrency
ack timing
retry behavior
DLQ growth
duplicate processing
ordering guarantees
security permissions
observability quality
incident recovery path

Does the Java code compile?
Does the consumer receive the message?
Does the happy path pass?

Can this message flow survive duplicate delivery, broker failover, producer retry,
consumer crash, slow downstream dependency, partial database commit, poison message,
redelivery storm, bad deployment, and production debugging under pressure?

exchange declaration
queue declaration
binding declaration
routing key
message type
message schema
message headers
publisher code
consumer code
ack/nack/reject behavior
prefetch
consumer concurrency
retry topology
DLX/DLQ configuration
TTL/expiration
queue length limit
outbox table
inbox table
idempotency key
serialization format
connection/channel lifecycle
security permission
vhost usage
deployment manifest
Helm value
RabbitMQ policy/operator policy
observability dashboard
alert threshold
manual replay tool

# ADR: <message flow / topology / reliability change>

## Context
What business workflow or system pressure requires this change?

## Decision
What exchange, queue, routing, producer, consumer, retry, DLQ, and storage pattern will be used?

## Delivery semantics
At-most-once, at-least-once, or effectively-once through idempotency?

## Transaction boundary
How does the flow interact with PostgreSQL/MyBatis/JDBC transactions?

## Failure modes
What happens on producer crash, broker failover, consumer crash, poison message, duplicate delivery, and retry exhaustion?

## Observability
Which metrics, logs, traces, dashboards, and alerts prove this flow is healthy?

## Security/privacy
What sensitive data can appear in payload, headers, DLQ, logs, traces, exports, and replay tools?

## Rollout and rollback
How can this be deployed, migrated, disabled, or rolled back safely?

## Internal verification checklist
What must be checked in CSG/team codebase, RabbitMQ UI, Helm, GitOps, dashboards, and runbooks?

x-message-ttl: 60000
x-dead-letter-exchange: retry.exchange

Is this exchange for commands, events, tasks, replies, retry, DLQ, or integration?
Is the exchange owned by one bounded context or shared by many teams?
Is the exchange stable API surface or internal implementation detail?

Is the exchange durable?
What happens after broker restart?
Is topology declared by application or GitOps/operator?
Can this exchange be deleted safely?
Is there an owner and documentation page?

one global topic exchange for everything
wildcard consumers using # without explicit reason
business-critical unroutable messages silently dropped
exchange declared differently by multiple services
exchange name contains environment but vhost already separates environment
retry and business messages mixed on same exchange without clear convention

Who owns this queue?
Who consumes it?
Is it command, event subscription, task, retry, DLQ, parking lot, reply, temporary, or stream?
What business process is blocked if this queue stops draining?
What is the expected backlog tolerance?

durable queue
persistent message
durable exchange/binding topology

owning service/team
consumer service
purpose
message type
SLO expectation
DLQ/retry path
alert owner
runbook link
replay policy
retention policy
security classification

Is the routing key naming convention explicit?
Does it represent business event type, command type, tenant, version, priority, or operation?
Are routing key segments stable?
Can consumers safely use topic wildcards?
Can routing key changes break existing consumers?
Is tenant or environment encoded unnecessarily?

<domain>.<entity>.<event>
quote.quote.created
quote.quote.approved
order.order.submitted
order.fulfillment.requested
billing.invoice.generated

<target-domain>.<command>
pricing.calculate-quote
approval.request-review
fulfillment.create-order
notification.send-email

routing key contains raw customer ID
routing key contains sensitive data
routing key overloaded with payload semantics
routing key versioning inconsistent with payload versioning
routing key wildcard required because producer does not know message type

Connection lifecycle is clear.
Channel ownership is clear.
Channel is not shared unsafely across threads.
Message properties are explicit.
Content type is set.
Message ID is set.
Correlation ID is propagated.
Trace context is propagated.
Delivery mode is intentional.
Mandatory flag behavior is intentional.
Return listener exists when mandatory is used.
Publisher confirm is used for important messages.
Publish failure is measured.
Publish latency is measured.

Is confirm mode enabled?
Does the publisher wait for confirms synchronously, asynchronously, or in batches?
What is the confirm timeout?
What happens when confirm times out?
Can timeout cause duplicate publish?
Is duplicate publish safe for downstream consumers?
Is confirm integrated with outbox status update?

Can the message be unroutable?
Is mandatory=true used where needed?
Is a return listener implemented?
Is alternate exchange configured?
Is there alerting for returned/unroutable messages?
Does the publisher distinguish broker accepted vs business delivered?

DB commit then publish without outbox for critical state change
publish then DB commit for irreversible downstream command
publish inside transaction but no confirm or outbox
mark published before broker confirm
retry publish without deterministic message ID
publisher catches exception and returns success to HTTP client

Manual ack is used for important work.
Ack happens after durable side effects.
Nack/reject behavior is explicit.
Requeue policy is explicit.
Prefetch is configured.
Consumer concurrency is bounded.
Shutdown drains in-flight messages or allows safe redelivery.
Consumer logs include message ID and correlation ID.
Duplicate handling exists.
Poison message handling exists.
Metrics exist for success, failure, processing time, redelivery, and DLQ.

What exactly must complete before ack?
DB commit?
External API call?
Outbox insert?
State transition?
Audit write?
Cache invalidation?

consume -> validate -> idempotency check -> transaction -> durable side effect -> commit -> ack

consume -> ack -> process -> DB write

When is basicNack used?
When is basicReject used?
When is requeue=true allowed?
When is requeue=false used to dead-letter?
How is retry count bounded?
How are poison messages isolated?

catch (Exception e) {
  channel.basicNack(tag, false, true);
}

What is prefetch per consumer?
How many consumer instances run per pod?
How many pods run during normal load?
How many pods run during autoscaling?
What is the total possible unacked message count?
Does this exceed DB connection pool capacity?
Does this exceed downstream API rate limit?
Does this break ordering assumptions?

total_in_flight ≈ pod_count × consumers_per_pod × prefetch_per_consumer

10 pods × 4 consumers × prefetch 50 = 2,000 in-flight messages

What is the idempotency key?
Is it stable across retries and duplicate publish?
Is it technical message ID, business command ID, event ID, or aggregate transition ID?
Where is it stored?
Is there a unique constraint?
Is duplicate treated as success, ignored, or rejected?
Can idempotency survive process restart?
Can idempotency survive Redis loss if Redis is used?

Bad: random UUID generated on each retry
Good: commandId generated at request boundary and reused through outbox/retry

UPDATE quote
SET status = 'APPROVED'
WHERE quote_id = :quoteId
  AND status = 'PENDING_APPROVAL';

Is retry count bounded?
Where is retry count stored?
Is x-death used safely?
Is retry delay fixed or exponential?
Can retry storm overload downstream systems?
Is DLQ per consumer/subscriber or shared?
Is parking lot queue available?
Is manual replay safe and audited?
Does replay preserve idempotency key?

unbounded requeue=true
single shared DLQ for unrelated domains
manual replay by copying messages without audit
retry queue with no alerting
retry topology not documented
DLQ contains PII with no retention/access policy

Is business row and outbox row inserted in the same PostgreSQL transaction?
Does outbox row contain stable event ID/message ID?
Does poller use safe locking such as FOR UPDATE SKIP LOCKED?
Does publisher wait for RabbitMQ confirm before marking published?
Can mark-published fail after broker confirm?
Is duplicate publish safe?
Is outbox cleanup retention defined?
Are stuck outbox rows alerted?
Can outbox be replayed safely?

If business state commits, publish intent must be durable.
If publish is retried, consumers must tolerate duplicates.

service method commits DB
then publishes message directly
then returns success

DB commit succeeds
process crashes before publish
no message exists
business state becomes invisible to downstream systems

Is message ID stored before/with processing?
Is there a unique constraint on dedup key?
Is processing status tracked?
Is the business side effect and inbox state committed atomically?
Is ack sent only after commit?
Can failed processing be retried safely?
Can poison message be linked to inbox record?
Is replay detected as duplicate or reprocessable by policy?
Is inbox retention defined?

A duplicate delivery must not create duplicate business side effects.

RECEIVED
PROCESSING
PROCESSED
FAILED_TRANSIENT
FAILED_PERMANENT
IGNORED_DUPLICATE
REPLAYED

Is message type explicit?
Is schema documented?
Is content type set?
Is version set?
Are required and optional fields clear?
Are default values defined?
Can old consumers read new messages?
Can new consumers read old messages?
Are breaking changes versioned?
Are deprecated fields still produced until all consumers migrate?
Are contract tests present?

consumer depends on undocumented field
producer renames field without version bump
nullable field becomes required without migration
routing key version and payload version diverge
payload includes internal entity model directly

messageId
messageType
messageVersion
correlationId
causationId
traceparent or traceId
sourceService
tenantId if applicable
actorId if applicable
createdAt
publishedAt if available
idempotencyKey
retryCount if application-managed

Can one customer incident be traced from HTTP request to DB row to outbox to RabbitMQ to consumer to downstream side effect?
Can DLQ messages be correlated to original request?
Can replay be audited?
Are trace/log labels low-cardinality where needed?
Are sensitive values kept out of metrics labels?

global ordering
per tenant ordering
per quote ordering
per order ordering
per aggregate ordering
per queue ordering
per stream partition ordering

How many consumers read the queue?
Is single active consumer required?
Can retry break order?
Can redelivery break order?
Can priority queue break order?
Can horizontal scaling break order?
Is per-aggregate routing required?
Is Kafka/RabbitMQ Stream a better fit?

"RabbitMQ is FIFO, so order is guaranteed".

Does the service use a dedicated RabbitMQ user?
Is permission scoped to required vhost?
Are configure/write/read patterns least privilege?
Is TLS enabled where required?
Are certificates validated?
Are credentials sourced from approved secret manager?
Is credential rotation documented?
Is Management UI access restricted?
Are topic permissions used if needed?

Does payload contain PII?
Do headers contain PII?
Can DLQ retain sensitive data longer than intended?
Can replay tool expose sensitive data?
Do logs print full payload?
Do traces include sensitive fields?
Are metrics labels free of high-cardinality/customer data?

publish_attempt_total
publish_success_total
publish_failure_total
publish_confirm_latency
publish_confirm_timeout_total
returned_unroutable_total
outbox_pending_count
outbox_oldest_age_seconds

consume_success_total
consume_failure_total
processing_duration
redelivery_total
ack_total
nack_total
dlq_publish_total
inbox_duplicate_total
inbox_processing_failure_total

messages_ready
messages_unacknowledged
publish_rate
deliver_rate
ack_rate
redelivery_rate
consumer_count
consumer_utilisation
dlq_depth
retry_depth
oldest_message_age

Is there a dashboard?
Are alerts actionable?
Is alert ownership clear?
Are dashboard dimensions low-cardinality?
Do logs include correlation ID and message ID?
Can tracing cross async boundary?

Expected publish rate?
Expected consume rate?
Peak rate?
Message size P50/P95/P99?
Processing latency target?
Allowed backlog duration?
DLQ acceptable threshold?
Retry volume expectation?
Consumer CPU/IO-bound?
Downstream DB/API capacity?

persistent messages
publisher confirms
quorum queue replication
large payloads
high binding count
topic wildcard complexity
large unacked window
consumer concurrency
slow downstream dependency
cross-region network
TLS overhead

Does pod shutdown drain in-flight messages?
Is terminationGracePeriodSeconds sufficient?
Does readiness go false before consumer stops?
Can rolling update duplicate processing safely?
Can autoscaling multiply prefetch dangerously?
Are secrets mounted/rotated safely?
Does DNS/LB behavior match Java client recovery behavior?

StatefulSet/operator/managed service?
Persistent storage class?
Anti-affinity?
Pod disruption budget?
Backup/restore?
Upgrade plan?
Broker version?
Plugin version?
Policy/operator policy?

Network path?
Private connectivity?
Firewall/security group/NetworkPolicy?
TLS cert chain?
Monitoring integration?
Responsibility boundary?
Maintenance window?

Is topology declared as code?
Are changes reviewed before production?
Is there drift detection?
Can definitions be exported/imported?
Are policies declarative?
Are operator policies controlled by platform?
Are destructive changes guarded?
Is deletion reversible?
Is rollback possible?

renaming exchange
renaming queue
changing queue type
changing DLX
changing routing key
removing binding
changing TTL
changing max length/overflow
changing durable flag
changing vhost/permission

Who can replay?
From where: DLQ, parking lot, stream, outbox, backup, file export?
What message fields are preserved?
Is message ID preserved or regenerated?
Is idempotency key preserved?
Is correlation ID preserved?
Is replay audited?
Can replay be scoped by tenant/customer/message type/time window?
Can replay be dry-run?
Can replay overwhelm downstream services?

manual copy-paste from Management UI
no replay audit
replay changes message ID accidentally
replay bypasses idempotency
replay directly into business exchange without throttling

classic queue -> quorum queue
queue rename
exchange rename
routing key taxonomy change
payload schema version change
new consumer introduction
old consumer removal
DLQ topology redesign
outbox adoption
inbox adoption
RabbitMQ Stream adoption

1. Add new topology.
2. Publish to both old and new if required.
3. Deploy consumers that can read both schemas.
4. Observe parity.
5. Drain old queues.
6. Disable old publishers.
7. Remove old topology after retention window.

Can publisher failure be detected?
Can unroutable message be detected?
Can duplicate delivery be handled?
Can consumer crash be recovered?
Can poison message be isolated?
Can DLQ growth alert someone?
Can retry storm be stopped?
Can backlog be drained safely?
Can sensitive data be protected in DLQ/logs/traces?
Can flow be disabled without redeploying everything?
Can messages be replayed safely?
Can downstream systems tolerate catch-up load?
Can support explain customer-visible state during async processing?

Can you add retry?

This consumer currently nacks with requeue=true for all exceptions. That can create an infinite redelivery loop for permanent payload errors. Please classify transient vs permanent failures, cap retry attempts, and route exhausted messages to a DLQ/parking lot with messageId, correlationId, and x-death preserved.

Use outbox.

This endpoint commits quote status before publishing QuoteApproved. If the process crashes after DB commit and before broker accept, downstream systems will never see the transition. Please either use transactional outbox or document why message loss is acceptable for this flow.

Add logs.

Please log messageId, correlationId, messageType, routingKey, queue, retryAttempt, and business aggregate ID at consume start, success, permanent failure, and DLQ handoff. Avoid logging full payload because it may include customer/order data.

publisher abstraction
consumer abstraction
connection/channel lifecycle
manual ack/nack policy
prefetch configuration
outbox implementation
inbox implementation
message schema definitions
message metadata standard
idempotency helper
retry/DLQ handler
replay tool
logging/tracing instrumentation

vhosts
users
permissions
topic permissions
exchanges
queues
bindings
queue types
policies
operator policies
runtime parameters
plugins
DLX/DLQ topology
TTL and queue length limits
quorum delivery limit
stream plugin/stream queues if used

Helm chart
Kubernetes manifests
RabbitMQ Cluster Operator resources
StatefulSet/PVC/storage class
service/load balancer
NetworkPolicy
secrets/certificates
AWS/Azure/on-prem network controls
backup/restore process
upgrade process

RabbitMQ Management UI
Prometheus/Grafana dashboards
producer metrics
consumer metrics
DLQ/retry alerts
resource alarm alerts
connection/channel alerts
outbox/inbox dashboards
log search patterns
trace correlation
incident notes
runbooks

messaging ADR template
PR checklist
schema compatibility rules
topology ownership registry
DLQ ownership model
replay approval process
security review process
SRE escalation path
post-incident review history

What is the business invariant protected by this message flow?
What is the exact delivery semantic promised to downstream systems?
Where is the idempotency boundary?
What happens if producer publishes twice?
What happens if consumer processes twice?
What happens if broker accepts but publisher does not receive confirm?
What happens if consumer commits DB but dies before ack?
What happens if retry delay causes business SLA breach?
What happens if DLQ contains customer-sensitive data?
What happens if a deployment rolls all consumers at peak load?
What happens if RabbitMQ is healthy but PostgreSQL is slow?
What happens if downstream API is down for two hours?
What happens if an old consumer reads a new message version?
What happens if queue depth grows faster than consumers can drain?
Who gets paged, and what do they do first?

1. Routing contract
   Which exchange, routing key, binding, and queue will carry the message?

2. Delivery contract
   How does the system handle confirm, ack, duplicate, redelivery, retry, and DLQ?

3. State contract
   How does the message align with PostgreSQL/MyBatis/JDBC transaction boundaries?

4. Operational contract
   How will humans observe, alert, debug, replay, and recover this flow?

5. Governance contract
   Who owns schema, topology, security, privacy, and production readiness?