Part 055 — Troubleshooting Playbook

Fokus part ini: memberi playbook debugging untuk workflow production system. Targetnya bukan menghafal error message Camunda, tetapi membangun cara berpikir investigatif: dari symptom, ke runtime state, ke side effect, ke recovery action yang aman.

Workflow troubleshooting harus lebih hati-hati daripada troubleshooting REST endpoint biasa. Dalam REST endpoint sinkron, failure sering berakhir pada request gagal. Dalam workflow, failure bisa meninggalkan state parsial:

Process instance sudah start.
Satu service task sudah menulis database.
Event Kafka sudah terkirim.
External API sudah dipanggil.
Worker crash sebelum complete job.
Incident baru muncul beberapa menit kemudian.
User task tidak terlihat oleh operator.
Timer masih aktif dan akan menembak escalation.

Artinya, debugging workflow harus menjawab tiga pertanyaan secara berurutan:

1. Where is the process now?
2. What side effects already happened?
3. What is the safest next action?

Jangan langsung retry, migrate, delete, manually update database, atau complete task sebelum ketiga pertanyaan itu jelas.

1. Troubleshooting Mental Model

1.1 Start from observable symptom, not assumption

Symptom umum:

• process tidak start;
• process start dengan version salah;
• token/element instance stuck;
• job tidak dibuat;
• worker tidak menerima job;
• job repeatedly failing;
• incident created;
• message tidak correlate;
• timer tidak firing;
• user task tidak visible;
• task tidak bisa complete;
• variable missing;
• variable serialization failure;
• BPMN error tidak tertangkap;
• compensation tidak jalan;
• migration failed;
• worker membuat duplicate side effect.

Setiap symptom bisa berasal dari beberapa layer:

BPMN model
Process deployment
Process version
Runtime state
Variable payload
Worker code
Database state
Message/event layer
Identity/authorization
Camunda engine/broker
Kubernetes/network/cloud dependency
Human operation

Senior engineer tidak boleh berhenti pada layer pertama yang kelihatan.

1.2 Debugging order

Gunakan order berikut:

1. Identify business impact.
2. Identify process definition/version.
3. Identify process instance/business key/correlation key.
4. Inspect current activity/element/job/task/incident.
5. Inspect worker/API/message/database logs with correlation ID.
6. Inspect variable state and payload shape.
7. Determine side effects already committed.
8. Choose recovery: retry, fix variable, correlate message, complete task, migrate, cancel, compensate, or manual repair.
9. Record evidence and post-incident learning.

Jangan menukar urutan side effect check dengan retry.

2. Core Debugging Invariants

2.1 Every workflow instance needs a stable business handle

Minimal harus bisa menemukan instance dengan:

processInstanceKey / processInstanceId
businessKey / businessId
correlationKey
quoteId / orderId / customerId
traceId / correlationId
processDefinitionKey / BPMN process id
processDefinitionVersion

Jika tim tidak bisa menemukan process instance dari quote/order ID, operability belum cukup matang.

2.2 Every worker side effect must be traceable

Untuk setiap worker/service task, harus bisa menjawab:

Worker apa yang jalan?
Job key / external task ID apa?
Input variable apa?
Business entity apa yang diubah?
External API apa yang dipanggil?
Event apa yang dipublish?
Apakah complete/fail job berhasil?

Tanpa ini, retry tidak aman.

2.3 Process state and business state are different

Contoh:

Process state: waiting at Validate Order task.
Order state: VALIDATION_PENDING.
Worker state: last attempt failed due to timeout.
External state: inventory reservation already created.
User task state: no user task created yet.

Troubleshooting harus membaca semua state, bukan hanya diagram.

3. Process Not Starting

3.1 Symptom

API start workflow berhasil secara teknis tetapi instance tidak muncul.
Event diterima tetapi process tidak dibuat.
Client mendapat 404/permission denied/wrong process id.
Process deployment ada tetapi start by key gagal.

3.2 Likely causes

• BPMN belum deployed ke environment tersebut.
• Process ID/key salah.
• Version yang diharapkan belum aktif.
• Start event type tidak cocok: none start vs message start.
• Message name/correlation key salah.
• Tenant ID mismatch.
• Authorization/service account tidak punya permission.
• Camunda 8 gateway/API unreachable.
• Camunda 7 engine belum boot atau process application belum deployed.
• Deployment pipeline gagal tapi tidak terdeteksi.

3.3 Debugging steps

1. Cek process definition tersedia di environment target.
2. Cek process ID/key persis sama dengan model.
3. Cek deployment timestamp dan version.
4. Cek start mechanism: direct API, message start, event consumer, scheduler.
5. Cek tenant/security context.
6. Cek API response dan error body.
7. Cek engine/gateway logs.
8. Cek pipeline deployment logs.

3.4 Java/JAX-RS checks

• Endpoint start process memakai process key yang benar.
• Request memakai idempotency key.
• Business key/correlation key diset dari domain identifier stabil.
• API tidak mengembalikan sukses sebelum command accepted.
• Error mapping tidak menyembunyikan Camunda error menjadi generic 500 tanpa detail operasional.

3.5 Internal verification checklist

• Di mana process definition dideploy?
• Apakah start process via REST/JAX-RS, worker, Kafka, RabbitMQ, scheduler, atau message start?
• Apakah ada tenant ID?
• Bagaimana mapping quote/order ID ke process instance?
• Dashboard apa yang menunjukkan deployed process versions?

4. Process Started with Wrong Version

4.1 Symptom

Instance baru mengikuti flow lama.
Worker menerima job type lama.
User task field/form tidak sesuai release terbaru.
Message correlation menunggu event yang sudah tidak dipakai.

4.2 Likely causes

• Start by key memakai latest version, tetapi deployment order tidak sesuai expectation.
• Environment memiliki version drift.
• Rollback worker dilakukan tanpa rollback BPMN.
• Running instance memang tetap berada di version lama.
• Process ID sama tetapi model berbeda antar branch/environment.
• CI/CD tidak memvalidasi deployed version.

4.3 Debugging steps

1. Catat processDefinitionVersion dari instance.
2. Bandingkan dengan deployment history.
3. Cek apakah instance ini start sebelum atau sesudah deployment baru.
4. Cek pipeline logs.
5. Cek worker version yang sedang running.
6. Cek variable/message schema compatibility.
7. Tentukan apakah perlu forward fix, migrate, atau biarkan drain.

4.4 Correct recovery

• Untuk instance lama yang masih valid: biarkan drain dengan worker compatible.
• Untuk instance baru yang salah version: evaluasi cancel + restart jika belum ada side effect.
• Untuk instance yang sudah punya side effect: gunakan recovery plan, bukan restart buta.
• Untuk deployment drift: deploy ulang via pipeline resmi.

4.5 Internal verification checklist

• Apakah pipeline menampilkan process version yang terdeploy?
• Apakah worker backward compatible dengan beberapa process version?
• Apakah ada version tag?
• Apakah start by latest diperbolehkan untuk process mission-critical?
• Apakah ada policy drain old version?

5. Token or Element Instance Stuck

5.1 Symptom

Process instance aktif tetapi tidak maju.
Tidak ada job aktif.
Tidak ada user task terlihat.
Timer/message wait state tidak berubah.
Operate/Cockpit menunjukkan activity tertentu aktif terlalu lama.

5.2 Likely causes

• Process sedang menunggu user task.
• Process sedang menunggu message correlation.
• Timer belum due.
• Job belum diambil worker.
• Incident menghentikan progress.
• Gateway path tidak punya valid outgoing condition.
• Parallel join menunggu path yang tidak akan pernah datang.
• Boundary event salah dipasang.
• Variable yang dibutuhkan condition tidak ada.

5.3 Debugging steps

1. Identifikasi current activity/element.
2. Tentukan jenis wait state: job, user task, message, timer, gateway, subprocess, incident.
3. Cek variable yang menentukan flow.
4. Cek active jobs/tasks/timers/messages.
5. Cek incident.
6. Cek BPMN model untuk join/gateway/boundary semantics.
7. Cek apakah external event memang belum datang atau gagal correlate.

5.4 BPMN checks

• Exclusive gateway punya default flow.
• Parallel split punya matching join jika dibutuhkan.
• Inclusive gateway tidak menciptakan join yang tidak reachable.
• Boundary event tidak memutus path yang masih diharapkan.
• Message catch event memakai correlation key benar.
• Timer expression valid dan timezone jelas.

5.5 Internal verification checklist

• Tool apa untuk melihat current activity?
• Apakah team menyimpan dashboard stuck process?
• Apakah ada SLA untuk active duration per activity?
• Apakah ada automated detector untuk token stuck?
• Siapa yang boleh melakukan manual repair?

6. Job Not Created

6.1 Symptom

Service task aktif di diagram tetapi worker tidak pernah mendapat job.
Tidak ada job/external task di runtime query.
Tidak ada incident.

6.2 Likely causes

• Service task tidak async di Camunda 7 saat expected async.
• Service task implementation/type salah.
• BPMN path belum mencapai activity tersebut.
• Condition gateway tidak memilih path service task.
• Input mapping gagal sebelum job creation.
• Model deployed bukan version yang expected.
• Camunda 8 job type expression menghasilkan nilai berbeda.
• Connector/service task configuration invalid.

6.3 Debugging steps

1. Cek apakah process benar-benar reached activity.
2. Cek active element/current activity.
3. Cek BPMN implementation properties.
4. Cek job type/topic.
5. Cek input mapping expression.
6. Cek deployment version.
7. Cek incident/error sebelum service task.

6.4 Review question

Is the problem that no job exists, or that a job exists but no worker can activate it?

Itu dua masalah berbeda.

7. Worker Not Receiving Job

7.1 Symptom

Job ada, tetapi worker tidak memproses.
Queue backlog naik.
Worker pod running tetapi idle.
External task topic tidak pernah terfetch.
Zeebe job activation kosong.

7.2 Likely causes

• Job type/topic mismatch.
• Worker credential/auth salah.
• Worker connect ke environment/cluster salah.
• NetworkPolicy/firewall/DNS/TLS issue.
• Worker max jobs active terlalu kecil.
• Backoff terlalu agresif.
• Camunda 8 gateway unavailable/backpressure.
• Camunda 7 external task lock masih dimiliki worker lama.
• Worker crashed setelah activate/fetch but before complete.
• Tenant/authorization filter mismatch.

7.3 Debugging steps

1. Cek job type/topic di BPMN dan worker config.
2. Cek worker logs saat startup dan activation/fetch.
3. Cek cluster endpoint dan credentials.
4. Cek network connectivity dari pod worker.
5. Cek worker concurrency/maxJobsActive/fetchAndLock parameters.
6. Cek locks/timeouts.
7. Cek gateway/engine health.
8. Cek metrics job activated/completed/failed.

7.4 Kubernetes checks

• Pod ready tetapi app belum connect ke Camunda.
• Secret/config map salah namespace.
• Service DNS salah.
• Egress NetworkPolicy memblokir gateway/engine.
• Rolling deployment membunuh worker tanpa graceful shutdown.
• Resource limit membuat worker throttled.

7.5 Internal verification checklist

• Apa naming convention job type/topic?
• Di mana worker config disimpan?
• Apakah worker punya health endpoint yang memvalidasi Camunda connectivity?
• Apakah dashboard membedakan worker alive vs worker processing?
• Apakah ada alert job backlog per type/topic?

8. Job Repeatedly Failing

8.1 Symptom

Same task repeatedly fails.
Retry count decreases.
Incident eventually created.
Worker logs show same exception.
External API repeatedly called.

8.2 Likely causes

• Input variable invalid.
• External dependency down.
• Business validation gagal tapi dimodelkan sebagai technical failure.
• Worker bug.
• DB constraint violation.
• Serialization/deserialization mismatch.
• Timeout terlalu pendek.
• Retry policy terlalu agresif.
• Non-idempotent side effect sudah terjadi pada attempt pertama.

8.3 Debugging steps

1. Identify job key/external task ID.
2. Inspect failure message and stack trace.
3. Inspect input variables used by worker.
4. Check whether side effect happened on previous attempts.
5. Check DB writes/external calls/events.
6. Decide whether failure is technical, business, or data quality.
7. Fix root cause before retry.
8. Retry only if idempotency is proven.

8.4 Recovery options

8.5 Internal verification checklist

• Apakah failure message cukup aman dan informatif?
• Apakah retry action audited?
• Apakah worker side effect idempotent?
• Apakah incident owner jelas?
• Apakah ada policy max retry/manual repair?

9. Incident Created

9.1 Meaning

Incident bukan sekadar error log. Incident berarti engine/runtime tidak dapat melanjutkan execution normal sampai sesuatu diperbaiki.

Incident biasanya muncul karena:

• retries exhausted;
• job failure with zero retries;
• failed async continuation;
• failed timer/message job;
• failed variable serialization/expression;
• unresolved runtime condition;
• platform/runtime issue.

9.2 Debugging steps

1. Read incident type/message.
2. Identify affected process instance and activity.
3. Identify job/task/worker related to incident.
4. Inspect variables and business entity state.
5. Inspect worker logs around failure timestamp.
6. Check prior attempts and side effects.
7. Fix root cause.
8. Resolve/retry through official tooling/API.
9. Verify process advances correctly.

9.3 Common mistake

Resolve incident first, investigate later.

Ini berbahaya. Resolving incident tanpa root cause bisa membuat process lanjut dengan data salah atau mengulang side effect.

9.4 Internal verification checklist

• Apakah incident alert punya severity mapping?
• Apakah incident dashboard bisa difilter per process/activity/customer/order?
• Siapa yang boleh resolve incident?
• Apakah incident resolution masuk audit log?
• Apakah ada post-incident review untuk recurring incident?

10. Message Not Correlated

10.1 Symptom

Callback/event sudah diterima tetapi process tetap menunggu.
Message correlation API returns not found/no matching subscription.
Kafka/RabbitMQ consumer logs successful but workflow unchanged.

10.2 Likely causes

• Message name mismatch.
• Correlation key mismatch.
• Process belum mencapai message catch event.
• Process sudah melewati/cancel message wait state.
• Message expired before subscription existed.
• Duplicate/late event ignored incorrectly.
• Wrong tenant/environment.
• Business key and correlation key confused.
• Event replay tries to correlate old messages to completed instances.

10.3 Debugging steps

1. Capture message name, correlation key, business key, tenant, timestamp.
2. Find process instance by business ID.
3. Verify it is currently waiting for that message.
4. Verify message subscription exists.
5. Compare exact key values and casing.
6. Check TTL/expiration.
7. Check duplicate/replay handling.
8. Decide whether to re-correlate, ignore, or repair state.

10.4 Kafka/RabbitMQ checks

• Event key equals expected correlation key.
• Schema version compatible.
• Consumer maps field correctly.
• Out-of-order event not processed before process subscription exists.
• DLQ has failed correlation attempts.
• Replay strategy prevents creating invalid duplicate correlations.

10.5 Internal verification checklist

• Apa standard correlation key untuk quote/order workflow?
• Apakah message correlation failure dimonitor?
• Apakah late message policy terdokumentasi?
• Apakah event replay pernah diuji terhadap workflow?
• Apakah DLQ correlation failure punya owner?

11. Timer Not Firing

11.1 Symptom

SLA timer lewat tetapi escalation tidak terjadi.
Timeout tidak memicu boundary event.
Timer start process tidak membuat instance.
Timer backlog meningkat.

11.2 Likely causes

• Timer expression invalid.
• Timezone assumption salah.
• Process belum mencapai timer wait state.
• Timer job incident/failure.
• Job executor/broker overloaded.
• Timer due date jauh di masa depan karena data salah.
• Business calendar tidak didukung seperti yang diasumsikan.
• Migration mengubah timer mapping.
• Clock drift/container time issue.

11.3 Debugging steps

1. Cek timer definition: date/duration/cycle.
2. Cek evaluated due date.
3. Cek timezone dan input variable.
4. Cek current process activity.
5. Cek timer job existence.
6. Cek incident/failure timer job.
7. Cek job executor/broker load.
8. Cek backlog metrics.

11.4 Internal verification checklist

• Apakah SLA memakai UTC atau timezone customer?
• Apakah business calendar dipakai atau tidak?
• Apakah timer backlog punya alert?
• Apakah timer migration pernah dites?
• Apakah due date terlihat di dashboard?

12. User Task Not Visible

12.1 Symptom

Process reached user task but operator cannot see task.
Task exists but not in expected queue.
Candidate group/user wrong.
Task appears for wrong tenant/team.

12.2 Likely causes

• Assignment expression salah.
• Candidate group mapping salah.
• Identity/group sync issue.
• Tenant ID mismatch.
• Task filter/search query salah.
• Authorization denies visibility.
• Task already claimed by another user.
• User task implementation differs across Camunda version/deployment.
• Form/tasklist configuration missing.

12.3 Debugging steps

1. Confirm task exists in runtime.
2. Inspect assignee/candidate users/groups/tenant.
3. Inspect due date/follow-up date.
4. Inspect authorization and identity mapping.
5. Check whether task claimed/completed/cancelled.
6. Check UI search/filter conditions.
7. Check backend task API logs.

12.4 Backend contract checks

• API filters by tenant and role correctly.
• Task visibility is not only a UI filter.
• Claim/complete operations check authorization server-side.
• Stale UI handles already completed/claimed task.
• Audit log records who acted and when.

12.5 Internal verification checklist

• Source of truth group/role dari mana?
• Apakah task visibility dikontrol Camunda, backend, UI, atau kombinasi?
• Apakah ada task aging dashboard?
• Apakah task assignment rules dites?
• Apakah operator punya escalation path jika task hilang?

13. Task Cannot Complete

13.1 Symptom

User clicks complete but task remains open.
API returns conflict/authorization/error.
Process incident muncul after complete.
Form validation passes in UI but engine rejects variables.

13.2 Likely causes

• Task already completed/claimed by another user.
• Authorization missing.
• Required variables missing.
• Variable type mismatch.
• Completion triggers gateway condition error.
• Completion triggers next service task failure.
• Backend complete endpoint not idempotent.
• UI sends stale task version.

13.3 Debugging steps

1. Check task current state.
2. Check assignee and claim state.
3. Check submitted variables.
4. Validate variable schema/type.
5. Check process activity after task completion attempt.
6. Check incident/job failure immediately after complete.
7. Check audit log.

13.4 Internal verification checklist

• Apakah complete task endpoint idempotent?
• Apakah save draft dan complete memakai schema berbeda?
• Apakah concurrent completion diuji?
• Apakah form validation dilakukan server-side?
• Apakah task completion punya audit trail lengkap?

14. Variable Missing or Wrong

14.1 Symptom

Gateway expression fails.
Worker cannot deserialize input.
Message correlation key empty.
DMN decision returns unexpected result.
Task form missing field.

14.2 Likely causes

• Input/output mapping salah.
• Worker completed job without required output variable.
• Variable scope local vs global salah.
• Parallel path overwrote variable.
• Variable name changed across process version.
• JSON/object shape changed across service release.
• Large variable stored as serialized object and not readable by new code.
• Sensitive variable intentionally filtered from API response.

14.3 Debugging steps

1. Inspect variables at process and local scope.
2. Check when variable was last written.
3. Identify writer: API, worker, user task, connector, message, DMN.
4. Compare expected schema vs actual payload.
5. Check version compatibility.
6. Check whether parallel path overwrote data.
7. Repair only through approved variable update path.

14.4 Correctness rule

Process variable should not become the only source of business truth. If variable and database disagree, recovery must decide which state is authoritative based on domain ownership.

14.5 Internal verification checklist

• Apakah variable naming convention ada?
• Apakah variable schema version disimpan?
• Apakah sensitive variables difilter?
• Apakah worker hanya membaca/menulis variable yang diperlukan?
• Apakah variable repair audited?

15. Variable Serialization Failure

15.1 Symptom

Worker fails reading variable.
Camunda 7 delegate cannot deserialize Java object.
Operate/Cockpit cannot display variable.
Process incident occurs before/after service task.

15.2 Likely causes

• Java serialized object class changed.
• Jackson object shape changed.
• Class not available in embedded engine classpath.
• Payload too large.
• Variable stored as complex object instead of stable JSON contract.
• Worker expects field that older process version does not have.

15.3 Debugging steps

1. Identify variable name and type.
2. Inspect raw payload if safe and allowed.
3. Check process version and worker version.
4. Check classpath/dependency changes.
5. Check schema evolution.
6. Decide whether to transform variable, deploy compatibility fix, or migrate instance.

15.4 Prevention

Prefer stable JSON-like contracts with explicit schema evolution over opaque Java object serialization for long-running process data.

16. BPMN Error Not Caught

16.1 Symptom

Worker throws BPMN error but process incidents instead.
Boundary error event does not trigger.
Expected business rejection path not reached.

16.2 Likely causes

• Error code mismatch.
• Boundary event attached to wrong activity/scope.
• Worker failed technically instead of throwing BPMN error.
• BPMN error thrown outside scope that catches it.
• Error event supported differently between Camunda 7 and Camunda 8 model implementation.
• Java exception confused with BPMN business error.

16.3 Debugging steps

1. Confirm worker/delegate actually throws BPMN error, not technical failure.
2. Compare error code exactly.
3. Check boundary event placement.
4. Check subprocess/call activity error propagation.
5. Check current activity scope.
6. Check incident/failure type.

16.4 Review rule

BPMN error should represent expected business deviation, not arbitrary exception handling.

17. Compensation Not Running

17.1 Symptom

Cancel/failure path reached but previous side effect not compensated.
Compensation handler never invoked.
Manual recovery required unexpectedly.

17.2 Likely causes

• Compensation event not thrown.
• Activity not completed, so compensation subscription not registered.
• Compensation handler attached incorrectly.
• Transaction subprocess misunderstood.
• Side effect was not actually compensatable.
• Compensation itself failed and created new incident.
• Business cancellation path bypasses compensation.

17.3 Debugging steps

1. Identify which completed activities need compensation.
2. Check compensation subscriptions/handlers.
3. Check whether compensation was triggered.
4. Inspect handler failure logs.
5. Inspect external state before and after compensation.
6. Decide manual repair if automatic compensation is impossible.

17.4 Internal verification checklist

• Apakah compensation path pernah diuji?
• Apakah side effect reversible?
• Apakah compensation idempotent?
• Apakah compensation failure punya incident owner?
• Apakah business menyetujui manual compensation procedure?

18. Process Migration Failed

18.1 Symptom

Migration command rejected.
Instance moved but stuck afterward.
User task/job/timer disappears or behaves unexpectedly.
Gateway condition invalid after migration.

18.2 Likely causes

• Activity mapping invalid.
• Target model tidak semantically compatible.
• Variable schema mismatch.
• Active element removed/renamed.
• Boundary/timer/message semantics changed.
• Worker incompatible with target job type.
• Migration performed without dry run.

18.3 Debugging steps

1. Preserve pre-migration evidence.
2. Identify source and target process version.
3. Inspect migration mapping.
4. Inspect active elements before migration.
5. Check variables and pending jobs/timers/tasks.
6. Determine whether forward repair is possible.
7. Escalate if rollback is not supported or unsafe.

18.4 Internal verification checklist

• Apakah ada migration runbook?
• Apakah dry run dilakukan?
• Apakah affected instance list disimpan?
• Apakah rollback/forward recovery plan ada?
• Siapa approver migration production?

19. Worker Duplicate Side Effect

19.1 Symptom

External API called twice.
DB row inserted twice.
Kafka event published twice.
Order action duplicated.
Payment/reservation/provisioning command duplicated.

19.2 Likely causes

• Worker completed side effect but crashed before complete job.
• Job timeout too short, then another worker picked it.
• Retry repeated non-idempotent operation.
• No processed job table.
• No external idempotency key.
• Kafka/RabbitMQ duplicate delivery not handled.
• Manual retry performed before checking side effect.

19.3 Debugging steps

1. Stop further retries if duplicate risk continues.
2. Identify all attempts by job key/external task ID/business key.
3. Inspect external system state.
4. Inspect DB unique constraints and audit table.
5. Inspect event outbox/inbox.
6. Determine whether duplicate can be ignored, merged, reversed, or compensated.
7. Patch idempotency before reopening retries.

19.4 Prevention

Use deterministic idempotency key:

businessOperation + businessId + processInstanceId + activityId

or a stronger domain-specific command ID.

19.5 Internal verification checklist

• Apakah setiap worker side effect punya idempotency key?
• Apakah ada processed job table?
• Apakah external API mendukung idempotency?
• Apakah event publish lewat outbox?
• Apakah duplicate side effect punya business recovery process?

20. Production-Safe Debugging Checklist

Sebelum melakukan aksi recovery:

[ ] Business impact jelas.
[ ] Process instance identified.
[ ] Process version identified.
[ ] Current activity/job/task/incident identified.
[ ] Variables inspected.
[ ] Worker logs inspected.
[ ] DB state inspected.
[ ] External side effects inspected.
[ ] Message/event state inspected.
[ ] Idempotency verified.
[ ] Recovery option selected.
[ ] Approval obtained if manual repair.
[ ] Evidence recorded.
[ ] Post-action verification planned.

21. Troubleshooting Decision Matrix

22. Senior Engineer Review Questions

Ask these during incident review:

Why did this failure become a production incident?
Was the failure mode expected or unexpected?
Did the BPMN model express the failure path?
Did the worker report failure correctly?
Was retry safe?
Was idempotency implemented at the right boundary?
Could observability detect this earlier?
Was the user/operator experience safe?
Was manual repair audited?
Do we need a BPMN change, worker change, data contract change, or runbook change?

23. Internal Verification Checklist

Use this part to inspect real CSG/team implementation:

[ ] How do we find process instance by quote/order/customer ID?
[ ] What tool is used: Cockpit, Operate, internal dashboard, logs, DB query?
[ ] Where are worker logs and metrics?
[ ] Are job key/external task ID/process instance ID logged?
[ ] Are correlation IDs propagated from JAX-RS to worker to DB to Kafka/RabbitMQ?
[ ] Are retry actions audited?
[ ] Who can resolve incidents?
[ ] Who can modify variables?
[ ] Who can migrate/cancel process instances?
[ ] Do we have stuck process detector?
[ ] Do we have message correlation failure dashboard?
[ ] Do we have task aging/SLA dashboard?
[ ] Do we have duplicate side effect runbook?
[ ] Do we have post-incident review notes for workflow failures?

24. Key Takeaways

• Workflow debugging is state debugging, not just log debugging.
• Retry is a mutation and must be treated as risky.
• Process state, business state, worker state, and external system state can diverge.
• Message correlation, timer, user task, variable, and worker failures require different playbooks.
• The safest engineer is the one who can prove what already happened before deciding what to do next.