Part 036 — Workflow and RabbitMQ Integration

Goal: integrate Camunda workflow with RabbitMQ without mixing broker retry, workflow retry, worker retry, and business compensation into one unobservable failure loop.

RabbitMQ is useful for command delivery, work queues, routing, and reply patterns. Camunda is useful for explicit long-running process state. The integration must define ownership of retries, acknowledgements, correlation, and manual repair.

1. Core mental model

RabbitMQ and Camunda should not be treated as interchangeable.

RabbitMQ:
  exchange routing
  queue delivery
  consumer acknowledgement
  prefetch/backpressure
  dead-letter exchange/queue
  routing key patterns
  request/reply messaging
  broker-level delivery mechanics

Camunda:
  process definition
  process instance
  user task
  service task/job
  timer
  message catch
  incident
  retry exhaustion
  compensation/manual repair
  business process visibility

RabbitMQ delivers messages.

Camunda tracks process progress.

A RabbitMQ message being acknowledged does not mean the business process is complete.

A Camunda task being complete does not mean the RabbitMQ consumer ecosystem has processed every downstream message.

2. RabbitMQ vs Kafka in workflow integration

Kafka is usually event-log oriented.

RabbitMQ is usually command/work-queue/routing oriented.

Simplified distinction:

Kafka:
  "This happened; many consumers may observe it; replay may happen."

RabbitMQ:
  "Deliver this message to the right queue/consumer; handle ack/reject/DLQ."

Real systems can blur this distinction, but the mental model helps.

For workflow:

• Kafka commonly carries domain events.
• RabbitMQ commonly carries commands, task messages, request/reply messages, and integration queues.

Do not use RabbitMQ as a hidden workflow engine where queue presence becomes the only state of business progress.

3. Common integration patterns

3.1 Command message starts a process

A RabbitMQ message can start a Camunda process.

Example:

Queue: quote.approval.requested
Message: { quoteId, requestedBy, tenantId, correlationId }
Consumer:
  -> validates message
  -> checks idempotency
  -> starts Quote Approval process
  -> acknowledges message only after durable handling

Use this when an upstream system issues a command that must become an explicit workflow.

3.2 Process sends command message

A Camunda service task or worker can send a command to RabbitMQ.

Example:

Process step: Request credit validation
  -> worker inserts outbox command
  -> publisher sends to exchange credit.validation
  -> process waits for CreditValidationCompleted reply/message

Do not model the command send as completion of the external work.

The command send only means the request was handed off.

3.3 Reply message correlates process

RabbitMQ is often used for request/reply integration.

Example:

Workflow sends ProvisionOrder command
External provisioning service processes command
External service publishes ProvisionOrderResult reply
Reply consumer correlates Camunda message by fulfillmentRequestId
Process continues

The reply must include enough metadata to correlate deterministically.

3.4 RabbitMQ-backed worker command

A worker may use RabbitMQ internally to delegate sub-work.

Example:

Zeebe job: calculate-complex-price
  -> worker sends command to pricing queue
  -> worker either waits synchronously with timeout or returns and process waits for message

For long-running work, prefer process wait + reply correlation over blocking a worker thread for a long time.

3.5 External task style with RabbitMQ

Some systems build an external-task-like pattern using RabbitMQ queues.

Example:

Camunda creates work intent
Adapter publishes task message to RabbitMQ
External worker consumes task message
External worker reports completion to API/message
Camunda process continues

This can work, but it introduces two coordination systems.

You must define:

• who owns retry,
• who owns timeout,
• who owns DLQ,
• who owns incident creation,
• how duplicate completion is handled,
• how stuck messages are reconciled with stuck process instances.

4. Exchange, queue, routing key, and process semantics

RabbitMQ routing design should reflect integration semantics, not random technical naming.

Example:

Exchange: order.commands
Routing key: fulfillment.requested
Queue: fulfillment-service.requested

Exchange: order.replies
Routing key: fulfillment.completed
Queue: workflow.fulfillment.replies

Do not use vague routing keys:

process
camunda
message
update
backend

A routing key should help answer:

• what business action is requested?
• what business fact is reported?
• which bounded context owns the message?
• is it a command or reply/event?

5. Command vs event vs reply

RabbitMQ integration becomes brittle when message semantics are unclear.

5.1 Command

A command asks a receiver to do something.

ValidateOrder
ReserveInventory
RequestProvisioning
CancelFulfillment

A command can fail because the receiver rejects it or cannot perform it.

5.2 Event

An event states that something happened.

OrderValidated
InventoryReserved
ProvisioningCompleted
FulfillmentFailed

An event should not be named like an imperative command.

5.3 Reply

A reply answers a specific prior request.

CreditCheckResult
ProvisioningResult
PartnerEligibilityResult

A reply must carry correlation metadata.

Minimum reply metadata:

messageId: unique message id
correlationId: request/reply correlation id
causationId: original command id
businessKey: quoteId/orderId/etc
requestId: fulfillmentRequestId/commandId
status: SUCCESS | FAILED | REJECTED | TIMEOUT
errorCode: business/technical code if applicable
tenantId: if multi-tenant
traceparent: if tracing is used

6. Acknowledgement semantics

RabbitMQ acknowledgement is a delivery mechanism.

It is not business completion.

A safe consumer should acknowledge only when it has durably handled the message.

Example for process-start command:

Receive message
  -> validate payload
  -> insert inbox/idempotency record
  -> start process or detect already started
  -> commit durable state
  -> ack message

If the consumer acks before durable handling and then crashes, the command can be lost.

If it never acks after durable handling, the message can be redelivered.

Therefore, idempotency is mandatory.

7. Retry layering problem

RabbitMQ + Camunda can accidentally create too many retry layers.

Possible retry layers:

RabbitMQ redelivery
RabbitMQ delayed retry queue
Dead-letter retry policy
Consumer library retry
Worker internal retry
Camunda job retry
BPMN timer retry
Manual incident retry
External system retry

If all are enabled independently, a single failure can become a retry storm.

Example anti-pattern:

Worker fails to call external API
  -> worker throws exception
  -> Camunda retries job 5 times
  -> worker also retries HTTP 3 times each
  -> RabbitMQ message redelivered 10 times
  -> DLQ replay retries again

That is not resilience.

That is multiplication of side effects.

8. Decide the owner of retry

Use this rule:

Transport retry handles temporary delivery failure.
Worker retry handles short-lived local/transient call failure.
Workflow retry handles process-visible step failure.
Business retry/manual task handles recoverable business failure.
Compensation handles committed business side effects that must be undone.

Make the owner explicit per message/task.

8.1 Retry ownership table

Failure: RabbitMQ broker temporarily unreachable
Owner: consumer/publisher transport retry
Workflow visibility: maybe no incident unless prolonged

Failure: external API returns 503
Owner: worker/Camunda job retry with backoff
Workflow visibility: incident if exhausted

Failure: command payload invalid
Owner: consumer validation + DLQ/parked message
Workflow visibility: do not start process or create business fallout

Failure: business rejection
Owner: BPMN business path / BPMN error / user task
Workflow visibility: explicit modeled path

Failure: side effect committed but next step failed
Owner: compensation/manual repair/reconciliation
Workflow visibility: explicit incident or compensation path

9. Dead-letter queue is not a runbook

A DLQ is a parking place, not a solution.

A production DLQ needs:

• ownership,
• alerting,
• triage procedure,
• replay criteria,
• payload inspection rules,
• privacy controls,
• maximum age policy,
• business impact mapping,
• link to process instance/business key,
• safe replay tooling,
• audit log of manual action.

If nobody reviews DLQ messages, the system is silently losing business work.

For CPQ/order management, DLQ messages may represent:

• approval request not started,
• fulfillment command not delivered,
• cancellation reply not correlated,
• manual fallout not created,
• customer-visible status stuck.

10. Duplicate messages

RabbitMQ consumers must be idempotent.

Duplicates can happen because:

• consumer crashes after DB commit but before ack,
• broker redelivers unacked message,
• producer retries publish,
• DLQ message replayed,
• publisher confirms ambiguous,
• network issue during ack,
• manual replay operation.

Idempotency keys should come from business semantics.

Examples:

ValidateOrder:O-123:attempt-1
ProvisionRequest:FR-888
CancelOrder:O-123:CANCEL-REQUEST-999
QuoteApprovalRequested:Q-777

Do not use delivery tag as business idempotency.

Delivery tag is channel-specific delivery state, not stable business identity.

11. Reply correlation

Reply correlation must be deterministic.

Bad:

correlate by customer name
correlate by latest active order
correlate by non-unique external reference
correlate by timestamp window

Good:

correlate by fulfillmentRequestId
correlate by commandId
correlate by quoteApprovalRequestId
correlate by orderId + operationId

For long-running processes, create an operation/request id before sending the command.

operationId = UUID or deterministic command id
store operationId in DB/process variable
send operationId in RabbitMQ command
expect operationId in reply
correlate reply by operationId

This avoids ambiguity when the same order has multiple concurrent operations.

12. Timeout and late reply

Every request/reply workflow needs a late-reply policy.

Example:

Process sends ProvisioningRequest
Timer boundary waits 2 hours
If no reply:
  -> create manual fallout task
  -> maybe send cancellation command
  -> mark operation timed out

Then a reply arrives after 3 hours.

What happens?

Options:

• ignore and audit as late,
• correlate to fallout process,
• reopen repair task,
• compensate previously taken action,
• reject because operation is closed,
• update external reconciliation state only.

This is a business decision.

Do not leave it to accidental consumer behavior.

13. Prefetch and backpressure

RabbitMQ prefetch controls how many unacknowledged messages a consumer can hold.

Too high:

• one pod can hoard messages,
• slow consumer increases latency,
• shutdown redelivers many messages,
• memory pressure rises,
• task distribution becomes uneven.

Too low:

• throughput may be poor,
• network round-trips dominate,
• workers underutilized.

For workflow-triggering consumers, tune prefetch with downstream capacity in mind:

RabbitMQ prefetch
  -> consumer processing concurrency
  -> Camunda process start/correlation rate
  -> PostgreSQL writes
  -> worker/job volume
  -> external API load

Backpressure must be end-to-end.

14. Ordering

RabbitMQ does not guarantee global business ordering across queues/consumers.

Even within a queue, multiple consumers can process messages concurrently.

If strict ordering matters, design for it explicitly.

Options:

• single active consumer for a queue,
• partition/shard queues by business key,
• domain state machine guards transitions,
• operation sequence numbers,
• optimistic locking,
• park impossible transitions,
• reconciliation process.

Do not assume that because messages were published in order, side effects completed in order.

15. Process start from RabbitMQ

A robust process-start consumer should follow this shape:

consume command message
  -> validate message contract
  -> derive idempotency key
  -> insert inbox row
  -> check current business state
  -> if already started, link and ack
  -> start process instance
  -> persist process mapping
  -> commit durable state
  -> ack message

If process start fails after inbox insert, the message can be retried or the inbox row can be marked failed.

If ack fails after commit, duplicate redelivery must detect already-started process.

16. Process publishes RabbitMQ command

A process should not directly publish command and assume success without durable tracking.

Preferred pattern:

Zeebe/Camunda job worker
  -> starts DB transaction
  -> validates entity state
  -> creates operation row
  -> inserts outbox command
  -> commits DB
  -> completes job

Outbox publisher
  -> publishes RabbitMQ message
  -> waits for publisher confirm if configured
  -> marks outbox published

The process may then wait for a reply message.

This makes command intent durable and observable.

17. Publisher confirms and mandatory routing

For RabbitMQ publishing, consider:

• publisher confirms,
• unroutable messages,
• mandatory flag if applicable,
• exchange/queue existence,
• connection retry,
• channel lifecycle,
• serialization failure,
• credentials/permission failure.

A worker that "publishes" without knowing whether the broker accepted the message creates uncertain side effects.

If outcome is unknown, the design must be idempotent.

18. Camunda 7 integration patterns

Camunda 7 + RabbitMQ commonly appears as:

JavaDelegate publishes command
External task worker publishes/consumes message
JAX-RS endpoint receives reply and correlates message
RabbitMQ consumer starts process
Custom adapter bridges RabbitMQ and Camunda RuntimeService/ExternalTaskService

18.1 JavaDelegate caution

A JavaDelegate should not casually publish to RabbitMQ inside engine transaction.

Failure cases:

publish succeeds
engine transaction rolls back
message exists for a workflow step that did not commit

or:

publish succeeds
JavaDelegate throws exception
job retries
message is published again

Prefer external task worker or outbox pattern for durable side effects.

18.2 External task worker

External task workers can integrate with RabbitMQ cleanly if they:

• use deterministic idempotency,
• complete external task only after durable DB/outbox action,
• use lock duration safely,
• report BPMN error for business rejection,
• report failure for technical retry,
• record messageId/correlationId in logs.

19. Camunda 8 integration patterns

Camunda 8 + RabbitMQ commonly appears as:

Zeebe job worker publishes command
RabbitMQ inbound connector starts/correlates process
RabbitMQ outbound connector sends message
Custom consumer publishes Zeebe message
Custom worker manages outbox/RabbitMQ publish

Connector usage can be good for simple paths.

Custom workers/consumers are usually better when you need:

• DB transaction boundary,
• outbox/inbox,
• complex idempotency,
• domain validation,
• detailed observability,
• custom retry/DLQ behavior,
• tenant-specific routing,
• advanced security handling.

20. Java/JAX-RS API impact

JAX-RS endpoints may interact with RabbitMQ and workflow in several ways.

20.1 API writes command

POST /orders/{orderId}/fulfillment-requests
202 Accepted

Backend:

validate command
create operation row
insert RabbitMQ outbox command
maybe start workflow
return operationId

20.2 API receives callback/reply

POST /integrations/provisioning/replies

Backend:

validate signature/auth
persist inbox
correlate process message
return accepted

20.3 API exposes workflow status

GET /orders/{orderId}/workflow-status

Backend should clearly state whether status comes from:

• business table,
• process engine query,
• read model/projection,
• operational dashboard,
• combination with freshness indicator.

21. PostgreSQL/MyBatis impact

RabbitMQ integration is usually unsafe without durable tables.

21.1 Inbox for consumed RabbitMQ messages

rabbitmq_message_inbox (
  message_id text primary key,
  source_queue text not null,
  routing_key text not null,
  business_key text not null,
  correlation_id text null,
  payload jsonb not null,
  status text not null,
  received_at timestamptz not null,
  processed_at timestamptz null,
  error_message text null
)

21.2 Outbox for commands/replies

rabbitmq_message_outbox (
  outbox_id uuid primary key,
  exchange text not null,
  routing_key text not null,
  message_id text not null unique,
  correlation_id text null,
  business_key text not null,
  payload jsonb not null,
  status text not null,
  created_at timestamptz not null,
  published_at timestamptz null
)

21.3 MyBatis review points

• idempotency insert uses unique constraint,
• business update and outbox insert share transaction,
• mapper update uses version/optimistic locking,
• failed message state is visible,
• no hidden autocommit breaks atomicity,
• retry path handles duplicate key safely,
• process mapping is durable.

22. Redis impact

Redis is often tempting for RabbitMQ integration.

Possible useful roles:

• short-lived dedupe cache,
• consumer rate limit,
• kill switch,
• circuit breaker state,
• temporary reply wait optimization,
• dashboard cache.

Dangerous roles:

• only store of consumed message ids,
• only store of correlation mapping,
• only store of process start idempotency,
• distributed lock without correctness proof,
• authoritative workflow status cache.

Durable workflow correctness should survive Redis loss.

23. Observability

RabbitMQ/Camunda integration needs observability at broker, consumer, process, worker, and business levels.

Track:

RabbitMQ:
  queue depth
  unacked messages
  publish rate
  consume rate
  redelivery count
  DLQ count
  consumer count
  connection/channel errors
  unroutable messages

Camunda:
  process starts
  message correlations
  waiting receive/message events
  job failures
  incidents
  retries exhausted
  user task aging

Worker/consumer:
  processing latency
  failure rate
  ack/reject count
  duplicate message count
  idempotency conflict count
  external API latency

Database:
  inbox pending/failed
  outbox pending/failed
  optimistic lock conflict
  operation timeout count

Business:
  fulfillment requests pending reply
  cancellations waiting confirmation
  approvals waiting routing
  fallout cases created from messaging failure

Every message should be traceable by:

messageId
correlationId
causationId
businessKey
operationId
processInstanceKey/processInstanceId
jobKey/externalTaskId
traceId

24. Security and privacy

Review RabbitMQ integration for:

• credentials in connector/worker config,
• TLS/mTLS,
• virtual host isolation,
• exchange/queue permissions,
• tenant routing separation,
• sensitive payloads,
• PII in DLQ,
• payloads in logs,
• operational access to queues,
• replay permission,
• audit of manual replay/delete.

A DLQ can contain sensitive business data.

Treat it as production data, not a harmless debugging bucket.

25. Kubernetes, cloud, and on-prem concerns

25.1 Kubernetes

Check:

• consumer deployment replicas,
• graceful shutdown and message ack behavior,
• readiness before consuming,
• preStop hook if needed,
• liveness probe not too aggressive,
• config/secret rotation,
• NetworkPolicy to broker and Camunda,
• resource limits for payload size,
• autoscaling based on queue depth carefully.

25.2 AWS

Check:

• Amazon MQ RabbitMQ or self-managed broker topology,
• security groups,
• private subnet access,
• Secrets Manager,
• CloudWatch alarms,
• broker storage limits,
• multi-AZ behavior,
• connection limit.

25.3 Azure

Check:

• broker hosting model,
• VNet/private endpoint,
• Key Vault,
• Azure Monitor,
• NSG rules,
• managed identity if used indirectly,
• backup/restore responsibility.

25.4 On-prem/hybrid

Check:

• firewall rules,
• internal CA/TLS,
• broker reachability from workers,
• queue mirroring/quorum configuration if used,
• operational ownership,
• patching window,
• backup and restore,
• network partition behavior.

26. Failure modes

26.1 Message stuck in queue

Possible causes:

• no consumer,
• consumer unhealthy,
• routing key mismatch,
• consumer prefetch too high,
• downstream Camunda unavailable,
• DB blocked,
• poison message repeatedly redelivered,
• auth/permission issue.

Debug:

• check queue depth and unacked count,
• check consumer count,
• check logs by routing key/messageId,
• check Camunda API/worker health,
• check DB locks,
• check DLQ.

26.2 Message dead-lettered

Possible causes:

• rejected message,
• max retry exceeded,
• TTL expired,
• queue length limit,
• consumer validation failure,
• routing/dependency failure.

Debug:

• inspect DLQ payload safely,
• identify business key,
• identify process instance if any,
• classify technical vs business failure,
• decide replay, repair, compensate, or discard with audit.

26.3 Process waiting but reply in RabbitMQ

Possible causes:

• reply consumer down,
• wrong routing key,
• wrong queue binding,
• wrong correlation id,
• process waiting for different message name,
• process already timed out,
• tenant mismatch,
• message in DLQ.

Debug:

• check reply queue depth,
• check binding/routing key,
• inspect correlation id and operation id,
• inspect Camunda waiting activity,
• check timeout/fallout path.

26.4 Duplicate command executed

Possible causes:

• redelivery after consumer crash,
• ack lost,
• publisher retry,
• manual replay,
• no idempotency key,
• operation id regenerated on retry.

Repair:

• inspect operation rows,
• determine authoritative business state,
• identify duplicate side effects,
• compensate if necessary,
• add unique constraint/idempotency guard.

26.5 Retry storm

Possible causes:

• broker redelivery + Camunda retry + worker retry combined,
• no backoff,
• many messages fail due to same dependency outage,
• autoscaling adds more failing consumers,
• DLQ replay during active outage.

Mitigation:

• pause consumers,
• enable kill switch,
• reduce concurrency,
• stop replay,
• let dependency recover,
• retry in controlled batches,
• monitor business impact.

27. Production-safe debugging sequence

Use this sequence:

1. Identify business key and customer impact.
2. Find messageId/correlationId/operationId.
3. Check RabbitMQ queue/DLQ/unacked status.
4. Check consumer logs and deployment health.
5. Check inbox/outbox rows.
6. Check Camunda process instance state.
7. Check waiting message event or service task incident.
8. Check business DB state.
9. Classify duplicate, late, missing, poison, dependency outage, or schema error.
10. Choose controlled replay/retry/correlation/manual repair/compensation.
11. Record manual action and root cause.

Do not purge queues as a first response.

Queue purge is a destructive production action.

28. Design checklist

Before approving RabbitMQ/Camunda integration, answer:

• Is the message a command, event, or reply?
• What exchange/routing key/queue is used?
• What is the message id?
• What is the operation id?
• What is the correlation id?
• What is the business key?
• When is the message acknowledged?
• What happens if ack is lost?
• What happens if publish outcome is unknown?
• Who owns retry?
• What is the backoff policy?
• What sends message to DLQ?
• Who owns DLQ triage?
• How is duplicate message handled?
• How is late reply handled?
• How is process timeout modeled?
• Is outbox/inbox needed?
• How are credentials stored?
• What metrics and alerts exist?

29. PR review checklist

Review a PR touching RabbitMQ + Camunda for:

• clear command/event/reply semantics,
• deterministic routing key,
• stable message id,
• stable correlation id/operation id,
• durable inbox/outbox where needed,
• idempotency unique constraint,
• safe ack timing,
• safe publisher confirmation handling,
• retry ownership clarity,
• DLQ ownership and alerting,
• late reply policy,
• timeout modelling in BPMN,
• process incident mapping,
• sensitive payload review,
• observability fields in logs/metrics,
• tests for duplicate/redelivery/DLQ/timeout.

Reject the PR if RabbitMQ redelivery is treated as impossible.

Redelivery is part of the model.

30. Internal verification checklist

Verify inside CSG/team context:

• Which RabbitMQ exchanges/queues interact with quote/order workflow?
• Which messages start process instances?
• Which replies correlate to existing process instances?
• Which process steps publish RabbitMQ commands?
• Are integrations implemented with Camunda connectors, custom workers, Java delegates, external task workers, or separate consumers?
• What is the routing key naming convention?
• What is the message id convention?
• What is the correlation id/operation id convention?
• Are inbox/outbox tables used?
• What is the DLQ policy?
• Who owns DLQ triage?
• How are duplicate messages handled?
• How are late replies handled?
• How are RabbitMQ failures surfaced in dashboards?
• Are sensitive payloads stored in DLQ/logs?
• Are queue purge/replay operations audited?
• Are there incident notes for lost, duplicate, or late RabbitMQ messages?

31. Senior engineer heuristics

Use these heuristics:

If a message can be redelivered, the consumer must be idempotent.
If a command expects a reply, create an operation id before sending it.
If a reply can arrive late, the process must define late-reply behavior.
If RabbitMQ retry and Camunda retry both exist, one must be clearly dominant.
If a message is in DLQ, there must be an owner and business impact path.
If a worker publishes a command, the command intent should be durable before job completion.
If queue state is the only process state, the design is not observable enough.

RabbitMQ is good at delivering messages.

It is not enough to explain business lifecycle.

Camunda can explain lifecycle, but only if the RabbitMQ boundary is explicit.

32. Key takeaways

• RabbitMQ is a broker, not a workflow engine.
• Acknowledgement is delivery state, not business completion.
• DLQ is not a runbook.
• Retry ownership must be explicit to avoid retry storms.
• Request/reply needs stable operation/correlation id.
• Late reply behavior is a business/process decision.
• Duplicate redelivery is normal and must be idempotent.
• Outbox/inbox patterns are often required for durable reliability.
• Process timeout, manual repair, and compensation must be visible in BPMN and operations.
• RabbitMQ/Camunda integration is production-safe only when broker state, process state, and business state can be reconciled.

33. References for further study

• Camunda 8 RabbitMQ Connector documentation.
• Camunda 8 connector types documentation.
• Camunda 8 messages and correlation documentation.
• Camunda 8 job worker retry/timeout documentation.
• Internal RabbitMQ topology docs, exchange/queue catalog, DLQ runbook, incident notes, and platform/SRE operational guidelines.