Apa yang sedang terjadi?
Apa yang sudah terjadi?
Step mana yang gagal?
Kenapa gagal?
Apakah aman di-retry?
Apa dampaknya ke domain state?
Berapa biaya eksekusinya?
Apakah ada anomali yang butuh reconciliation?

ExecutionStarted
TaskStateEntered
TaskScheduled
TaskStarted
TaskSucceeded
ChoiceStateEntered
ExecutionSucceeded

create table case_audit_log (
  audit_id uuid primary key,
  case_id text not null,
  transition text not null,
  from_status text,
  to_status text,
  actor_type text not null,
  actor_id text,
  workflow_execution_id text,
  workflow_state_name text,
  reason text,
  occurred_at timestamptz not null
);

Workflow melewati state apa?

Mengapa status case berubah, oleh siapa, dan berdasarkan aturan apa?

Wait 30 seconds
CheckJobStatus
Choice not completed
Wait 30 seconds
CheckJobStatus
...

If a workflow waits for something external, prefer callback/event completion over tight polling.

{
  "timestamp": "2026-07-06T10:00:00Z",
  "level": "INFO",
  "message": "workflow task completed",
  "service": "case-service",
  "environment": "prod",
  "tenantId": "agency-a",
  "correlationId": "corr-abc",
  "workflowName": "CaseActivation",
  "workflowVersion": "v3",
  "executionArn": "arn:aws:states:...",
  "executionName": "CaseActivation-agency-a-case-123-cmd-456",
  "stateName": "ReserveCaseNumber",
  "taskAttempt": 1,
  "aggregateType": "Case",
  "aggregateId": "CASE-123",
  "commandId": "cmd-456",
  "idempotencyKey": "exec:state:case",
  "fromStatus": "DRAFT",
  "toStatus": "NUMBER_RESERVED",
  "durationMs": 82,
  "outcome": "SUCCESS"
}

{
  "level": "ERROR",
  "message": "workflow task failed",
  "errorType": "TransientDatabaseError",
  "retryable": true,
  "safeToRetry": true,
  "workflowName": "CaseActivation",
  "stateName": "ReserveCaseNumber",
  "caseId": "CASE-123",
  "commandId": "cmd-456",
  "correlationId": "corr-abc",
  "dbErrorCode": "deadlock_detected"
}

Any failed execution > 0 => wake everyone

Alert without runbook = anxiety, not operability.

Service mana yang lambat?

Kenapa case berubah status?

Step mana yang dijalankan workflow?

1. Identify executionArn and failed state.
2. Read execution input, failed state input, error, cause.
3. Inspect task logs by correlationId/executionArn/stateName.
4. Inspect domain aggregate current status/version.
5. Inspect command/idempotency record for failed state.
6. Inspect outbox/inbox records.
7. Inspect external side-effect ledger.
8. Determine if failure happened before or after commit.
9. Choose retry, redrive, compensation, manual repair, or no-op.
10. Record operational decision.

[ ] failed state identified
[ ] domain aggregate inspected
[ ] idempotency record inspected
[ ] external ledger inspected
[ ] outbox/inbox inspected
[ ] current workflow version understood
[ ] safe-to-redrive decision recorded
[ ] rollback/compensation path known

{
  "workflowName": "CaseActivation",
  "workflowAlias": "PROD",
  "workflowVersion": "17",
  "executionArn": "arn:aws:states:...",
  "stateName": "ValidateJurisdiction"
}

workflowName
alias
version
stateName
tenantId if safe/cardinality controlled

monthly_standard_cost ~= executions_per_month
                         * average_state_transitions_per_execution
                         * price_per_transition

100,000 executions/month
20 transitions happy path
5% executions retry average +4 transitions

base transitions = 100,000 * 20 = 2,000,000
retry transitions = 5,000 * 4 = 20,000
total = 2,020,000 transitions/month

monthly_express_cost ~= request_count_cost
                       + duration_memory_cost
                       + log_ingestion_cost

Wait 10 seconds
Check status
Repeat for 6 hours

6 hours * 60 minutes * 6 iterations/minute = 2160 iterations

Wait -> Task -> Choice = roughly 3 transitions

2160 * 3 = 6480 transitions for one waiting workflow

64,800,000 transitions just to wait

NormalizeString
ValidateEmailFormat
TrimWhitespace
ParseDate
CheckRequiredFields
MapDto
CallDatabase

ValidateCommand
PersistCommand

A state deserves to exist when it has different retry policy, timeout, owner, observability need, or business meaning.

{
  "caseId": "CASE-123",
  "commandId": "cmd-456",
  "stateName": "ValidateJurisdiction",
  "outcome": "REJECTED",
  "reasonCode": "OUT_OF_JURISDICTION"
}

EntityUpdated event emitted for every field update
EventBridge rule starts workflow for every update
Workflow does several DB reads and transitions

ValidateCommand
ReserveInventory
AuthorizePayment
WaitForApproval
GenerateDocument
SendNotice
MarkActive

ValidateFieldA
ValidateFieldB
ValidateFieldC
ConvertDate
BuildSqlParams
ExecuteSql
MapSqlResult

Panel 1: Executions started/succeeded/failed/timed out by workflow/version
Panel 2: Failure rate and top failed states
Panel 3: Execution duration p50/p95/p99
Panel 4: State retry count by error type
Panel 5: Callback wait age p95/max
Panel 6: Outbox pending age and count
Panel 7: DLQ depth for workflow-related queues
Panel 8: Invalid domain transition count
Panel 9: Compensation count by workflow
Panel 10: State transition volume/cost proxy
Panel 11: CloudWatch log ingestion volume
Panel 12: Reconciliation anomalies and repairs

fields @timestamp, workflowName, workflowVersion, executionName, stateName, errorType, message
| filter workflowName = "CaseActivation"
| filter outcome = "FAILED"
| sort @timestamp desc
| limit 50

fields @timestamp, aggregateId, fromStatus, attemptedTransition, executionName, stateName
| filter errorType = "InvalidTransition"
| sort @timestamp desc
| limit 100

fields @timestamp, workflowName, stateName, idempotencyKey, duplicateOutcome
| filter eventType = "IdempotencyDuplicate"
| stats count() by workflowName, stateName, duplicateOutcome

fields @timestamp, workflowName, executionName, stateName
| filter stateName like /Check.*Status|Wait.*/
| stats count() as events by workflowName, executionName
| sort events desc
| limit 20

select command_id,
       execution_id,
       state_name,
       aggregate_id,
       status,
       created_at,
       completed_at,
       failed_at,
       result
from workflow_commands
where execution_id = :execution_id
order by created_at;

select count(*) as pending_count,
       min(created_at) as oldest_pending_at,
       now() - min(created_at) as oldest_age
from outbox_messages
where publish_status = 'PENDING';

select job_id,
       case_id,
       status,
       lease_owner,
       lease_until,
       created_at,
       now() - created_at as age
from duplicate_detection_jobs
where status in ('PENDING', 'RUNNING')
order by created_at asc
limit 100;

select c.case_id,
       c.status,
       ws.execution_id,
       ws.current_state,
       ws.updated_at
from cases c
join workflow_shadow_status ws on ws.case_id = c.case_id
where ws.current_state = 'WaitForApproval'
  and c.status not in ('NEEDS_APPROVAL', 'APPROVED', 'REJECTED');

create table workflow_instances (
  execution_id text primary key,
  execution_name text not null unique,
  workflow_name text not null,
  workflow_version text not null,
  aggregate_type text not null,
  aggregate_id text not null,
  status text not null,
  current_state text,
  started_at timestamptz not null,
  updated_at timestamptz not null,
  completed_at timestamptz,
  failed_at timestamptz
);

Shadow status is a projection, not the Step Functions source of control.

Do not pass what you do not need.
Do not log what you cannot safely retain.
Do not expose what you cannot audit.

[ ] execution input contains only identifiers and minimum context
[ ] sensitive payload stored in database/S3 with proper access control
[ ] CloudWatch Logs retention configured
[ ] logging level excludes full payload in production
[ ] error cause does not leak secrets/PII
[ ] task token is not logged
[ ] callback URL/token never appears in user-visible logs
[ ] audit log has proper immutable retention policy

POST /cases returns accepted within 300ms p95

95% of CaseActivation workflows reach ACTIVE within 5 minutes excluding human approval.
99% of approval callbacks resume workflow within 30 seconds.
No accepted command remains without workflow execution for more than 2 minutes.
Outbox oldest pending age under 60 seconds.

CaseActivation p95 < 5 minutes

Machine processing time p95 < 5 minutes.
Human approval wait tracked separately.
Post-approval resume p95 < 30 seconds.

Expected CaseActivation volume: 100k/month
Expected avg transitions: 18
Expected transition budget: 1.8M/month
Alert at: 2.5M/month or +30% week-over-week

[ ] task fails before DB commit -> logs identify pre-commit failure
[ ] task fails after DB commit -> duplicate retry returns cached result
[ ] external provider timeout -> ledger shows UNKNOWN
[ ] callback job completes but callback fails -> reconciliation detects stuck workflow
[ ] workflow times out -> alarm fires and runbook explains expiry
[ ] outbox publisher stopped -> outbox age alarm fires
[ ] event source duplicate -> idempotency duplicate metric increments
[ ] workflow redrive -> version and operator action recorded
[ ] PII appears in fake payload -> log redaction test catches it
[ ] polling loop accidental change -> transition volume alarm catches it

# Workflow Incident Runbook: <WorkflowName>

## Severity Criteria
- SEV1:
- SEV2:
- SEV3:

## First 5 Minutes
1. Identify affected workflow/version.
2. Check execution failure rate.
3. Check top failed states.
4. Check domain invariant alarms.
5. Check DLQ/outbox/callback age.

## Debug Queries
- CloudWatch Logs Insights:
- Database command query:
- Outbox query:
- External ledger query:

## Safe Actions
- Retry task:
- Redrive execution:
- Requeue message:
- Release reservation:
- Pause event rule:

## Unsafe Actions
- Do not redrive SendPayment without ledger inspection.
- Do not delete command rows.
- Do not manually set case ACTIVE without audit entry.

## Escalation
- Workflow owner:
- Domain owner:
- Database owner:
- Security/compliance:

## Post-Incident
- Reconciliation required:
- Customer/regulatory notification required:
- Cost impact review:

[ ] Workflow type chosen intentionally: Standard vs Express.
[ ] Execution input/output contract documented.
[ ] Each state has owner, timeout, retry, and error policy.
[ ] Every mutating state is idempotent.
[ ] Domain audit is stored outside Step Functions history.
[ ] CloudWatch Logs configured with safe payload policy.
[ ] X-Ray/tracing decision documented.
[ ] Dashboard exists.
[ ] Alarms exist with runbooks.
[ ] Cost estimate exists for expected and worst-case volume.
[ ] Polling loops reviewed.
[ ] Callback token handling is secure.
[ ] Redrive policy documented.
[ ] Version/alias deployment strategy defined.
[ ] Reconciliation jobs cover ambiguous states.
[ ] Operators can answer: safe to retry or not?

Which execution failed?
Which state failed?
Did the database commit happen?
Was an external side effect sent?
Is retry safe?
What domain entities are affected?
What is the current cost/volume anomaly?
What runbook action is allowed?

Step Functions controls process.
Database owns truth.
Messages/events move facts.
Logs/metrics/traces explain behavior.
Runbooks make recovery repeatable.
Cost model keeps design honest.