Can the platform preserve commercial truth, order truth, tenant isolation, and operational recoverability when dependencies fail, users race, workflows stall, events duplicate, data evolves, and humans intervene?

[ ] Architecture diagrams
[ ] ADR index
[ ] Service ownership map
[ ] OpenAPI specs
[ ] JSON/event schemas
[ ] Database migration plan
[ ] Camunda BPMN/DMN deployment package
[ ] Kafka topic and consumer ownership list
[ ] Redis keyspace and TTL policy
[ ] Security threat model
[ ] Test evidence summary
[ ] Load/performance report
[ ] Observability dashboard links
[ ] Alert definitions
[ ] Runbooks
[ ] Backup/restore drill results
[ ] Failure drill results
[ ] Release and rollback/roll-forward plan
[ ] Known risks and accepted exceptions

owner, SLO, dashboard, alert, runbook, failure scenario, test evidence

Journey: Quote acceptance
SLO: 99.95% successful acceptance command for valid accepted quote requests over 30 days
Hard invariant: duplicate primary order count must be zero

If SLO burn exceeds threshold:
- freeze non-critical releases to quote/order services;
- prioritize acceptance failure root cause;
- increase monitoring and support coverage;
- run incident review if customer-impacting.

[ ] Quote states and transitions are explicit.
[ ] Order states and transitions are explicit.
[ ] Invalid transitions are tested.
[ ] Quote revision immutability is enforced.
[ ] Price result is snapshotted and reproducible.
[ ] Approval decision references exact quote revision.
[ ] Material quote change invalidates approval/pricing/document freshness.
[ ] Accepted quote creates at most one primary order.
[ ] Order cancellation is explicit lifecycle, not deletion.
[ ] Amendment/change order is modeled as new intent over baseline.
[ ] Fallout case has lifecycle and ownership.
[ ] Manual recovery actions are audited and authorization-checked.

- state machine tests
- concurrency tests
- database constraints
- command handler tests
- audit samples
- scenario catalog

[ ] All public/cross-service HTTP APIs have OpenAPI specs.
[ ] Specs are versioned and reviewed.
[ ] Generated DTOs do not leak into domain model.
[ ] Lifecycle mutations use command-shaped endpoints.
[ ] Error responses follow shared problem detail schema.
[ ] Idempotent commands define idempotency key behavior.
[ ] Optimistic concurrency behavior is documented.
[ ] Pagination/search semantics are stable.
[ ] Tenant and authorization behavior is described.
[ ] Breaking change detection runs in CI.

- OpenAPI files
- contract test results
- backward compatibility reports
- sample request/response payloads

[ ] Every Kafka topic has owner.
[ ] Every event has schema and version.
[ ] Event envelope includes eventId, eventType, eventVersion, occurredAt, tenantId, producer, correlationId.
[ ] Partition key is documented.
[ ] Retention policy is documented.
[ ] Replay policy is documented.
[ ] Consumer idempotency is implemented.
[ ] DLQ lifecycle is defined.
[ ] Event compatibility checks run in CI.
[ ] Outbox lag is monitored.

- event schema repository
- topic ownership matrix
- outbox table metrics
- consumer idempotency tests
- DLQ runbook
- replay drill result

If this event is published twice, what happens?
If this event arrives late, what happens?
If this event cannot be processed, who owns it?
If we replay a month of events, what breaks?

[ ] Schema migrations are versioned.
[ ] Migration lock impact is understood.
[ ] Expand-migrate-contract is used for risky changes.
[ ] Hot tables have appropriate indexes.
[ ] Foreign key constraints protect lifecycle references.
[ ] Partial unique indexes protect business uniqueness where needed.
[ ] Long-running transactions are monitored.
[ ] Connection pool size is budgeted across services.
[ ] Backup/restore drill has been performed.
[ ] Point-in-time recovery objective is defined if required.
[ ] Audit and outbox retention are planned.
[ ] Read replica lag impact is understood.
[ ] Camunda database boundary is separated or explicitly governed.

- migration dry-run logs
- explain plans for critical queries
- index list and ownership
- backup restore report
- database dashboard
- lock/bloat monitoring

[ ] Aggregate loading boundaries are explicit.
[ ] Lazy loading does not cross API serialization boundary.
[ ] Optimistic locking is used for mutable aggregates.
[ ] Transaction boundaries are command-scoped.
[ ] Bulk updates do not bypass invariants.
[ ] Entity mappings are covered by integration tests.
[ ] N+1 query risks are tested for critical paths.
[ ] EclipseLink cache behavior is understood and safe.
[ ] Entity graph/fetch strategy is defined for critical reads.
[ ] Persistence exceptions map to stable API errors.

- integration test logs
- SQL query count tests for critical endpoints
- mapping tests
- optimistic lock conflict tests

[ ] Camunda topology is documented.
[ ] Workflow Service is the only service using engine APIs directly.
[ ] Process variables are minimal and non-authoritative.
[ ] Business key format is standardized.
[ ] Process definitions are versioned and deployment-controlled.
[ ] Running instance behavior during deployment is understood.
[ ] External task workers are idempotent.
[ ] Retry policies are defined per task type.
[ ] BPMN errors vs technical failures are modeled explicitly.
[ ] Incident handling runbook exists.
[ ] History retention and cleanup policy exists.
[ ] Process migration policy exists.
[ ] Cockpit/Admin access is restricted.
[ ] Workflow metrics and alerts exist.

- BPMN/DMN validation result
- workflow scenario tests
- external task duplicate test
- incident drill
- process deployment runbook
- history cleanup configuration

If a process instance is stuck, can operations identify why, decide whether it is safe to retry, and recover without corrupting order state?

[ ] Topic naming convention is applied.
[ ] Topic owner is known.
[ ] Producer and consumer owners are documented.
[ ] Partition count and key are justified.
[ ] Retention policy fits replay/diagnostic needs.
[ ] Consumer lag is monitored.
[ ] DLQ is monitored and has owner.
[ ] Reprocessing procedure is tested.
[ ] Schema evolution is controlled.
[ ] Producer outbox is monitored.
[ ] Consumer inbox/deduplication is implemented where required.
[ ] Event replay does not duplicate side effects.

- topic registry
- consumer group dashboard
- DLQ replay drill
- schema compatibility report
- idempotent consumer tests

[ ] Redis use cases are explicitly non-authoritative.
[ ] Key naming includes tenant and version where required.
[ ] TTL policy exists for every key class.
[ ] Eviction policy is understood.
[ ] Memory usage and hot keys are monitored.
[ ] Cache stampede controls exist for hot paths.
[ ] Redis outage fallback is tested.
[ ] Redis stale data behavior is safe.
[ ] Redis locks are not used as sole correctness mechanism.
[ ] Persistence/failover expectations are documented.

- keyspace inventory
- TTL audit
- Redis failure test
- latency dashboard
- memory pressure runbook

If Redis is empty, slow, stale, or unavailable, what business truth can be corrupted?

None.

[ ] Threat model exists.
[ ] Tenant isolation is enforced at API, service, DB, Kafka, Redis, Camunda, and document boundaries.
[ ] Object-level authorization exists for quote/order/task/artifact access.
[ ] Approval authority is policy-driven and audited.
[ ] Control plane operations require privileged scoped roles.
[ ] Service-to-service authentication is enforced.
[ ] Secrets are not stored in repo/config maps/logs.
[ ] Sensitive fields are redacted from logs and events where needed.
[ ] Audit tampering risk is mitigated.
[ ] Security tests include cross-tenant and privilege escalation attempts.
[ ] Dependency and container scans run in CI.

- threat model document
- authorization test suite
- security scan report
- audit examples
- access control matrix

[ ] Structured logs include correlationId, traceId, tenantId, aggregateId where safe.
[ ] Traces propagate across JAX-RS, Kafka, workers, and workflow boundaries where possible.
[ ] Metrics exist for critical journeys, not only infrastructure.
[ ] Dashboards exist for quote, pricing, approval, order, workflow, outbox, Kafka, Redis, PostgreSQL.
[ ] Alerts map to user/business impact.
[ ] Alert runbooks exist.
[ ] Noise budget is controlled.
[ ] Audit logs are separate from diagnostic logs.
[ ] SLO burn is visible for critical journeys.

- dashboard screenshots/links
- alert definitions
- trace samples
- log samples
- SLO dashboard

CPU > 80%

Quote acceptance valid-command failure rate exceeds threshold for 10 minutes.

[ ] Quote acceptance duplicate detection
[ ] Pricing latency degradation
[ ] Catalog publication rollback/disable
[ ] Approval task stuck or missing
[ ] Camunda failed job / incident recovery
[ ] External task worker outage
[ ] Order fulfillment unknown outcome
[ ] Inventory reservation reconciliation
[ ] Billing handoff reconciliation
[ ] Kafka consumer lag
[ ] DLQ triage and replay
[ ] Outbox publisher stuck
[ ] Redis memory pressure
[ ] PostgreSQL lock/migration incident
[ ] Projection rebuild
[ ] Tenant suspension
[ ] Audit evidence export
[ ] Security incident / unauthorized access attempt

- symptom
- impact
- dashboards
- immediate mitigation
- diagnosis steps
- safe recovery actions
- unsafe actions
- escalation owner
- rollback/roll-forward decision
- customer/support communication notes if needed
- post-incident evidence to preserve

Unsafe actions

Do not manually update order status to COMPLETED if billing handoff is unknown.
Do not re-run ReserveInventory without checking external reservation status.
Do not delete Camunda process instances to clear incidents.
Do not flush all Redis keys in production without tenant/keyspace impact analysis.

- date
- environment
- scenario
- commands executed
- observed metrics/logs
- outcome
- gaps found
- follow-up actions

[ ] Release scope is clear.
[ ] Services included are listed.
[ ] OpenAPI/schema changes are listed.
[ ] Database migrations are listed.
[ ] BPMN/DMN deployments are listed.
[ ] Kafka topic/schema changes are listed.
[ ] Redis key changes are listed.
[ ] Feature flags are listed.
[ ] Deployment order is defined.
[ ] Smoke tests are defined.
[ ] Roll-forward plan is defined.
[ ] Rollback limitations are explicit.
[ ] Customer/support impact is known.
[ ] On-call coverage is scheduled.

safe rollout + feature flags + compatibility + roll-forward

[ ] RPO and RTO are defined.
[ ] Backup schedule is documented.
[ ] Restore drill has been completed.
[ ] Restore target environment exists or can be created.
[ ] Audit evidence recovery is tested.
[ ] Object storage/document artifact recovery is tested.
[ ] Kafka replay limitations are understood.
[ ] Projection rebuild is tested.
[ ] Camunda state recovery expectations are documented.
[ ] Redis recovery expectations are documented.

[ ] Accepted quote can be reconstructed.
[ ] Price result trace is stored.
[ ] Approval decision evidence is stored.
[ ] Document artifact hash/version is stored.
[ ] Customer acceptance evidence is stored.
[ ] Order creation references accepted quote revision.
[ ] Manual recovery is audited.
[ ] Admin/control-plane changes are audited.
[ ] Audit records include actor, authority, time, target, reason, correlation.
[ ] Sensitive data retention policy exists.
[ ] Audit export path exists.

For quote Q accepted on date D, can we prove what was configured, priced, approved, shown, accepted, and converted to order?

[ ] Support team can search quote/order by business identifiers.
[ ] Support can see lifecycle timeline.
[ ] Support can see workflow status without raw engine access.
[ ] Support can see fallout cases and owner.
[ ] Support can see external correlation IDs.
[ ] Support can distinguish customer error, validation error, system error, and pending async state.
[ ] Support has escalation path.
[ ] Support has safe manual recovery procedure.
[ ] Support cannot bypass domain invariants casually.

[ ] Expected traffic profile is documented.
[ ] Peak quote configuration sessions are estimated.
[ ] Pricing request volume is estimated.
[ ] Approval task volume is estimated.
[ ] Order fulfillment throughput is estimated.
[ ] Camunda job executor capacity is tested.
[ ] Kafka consumer throughput is tested.
[ ] PostgreSQL connection pool budget is tested.
[ ] Redis memory growth is estimated.
[ ] Object storage growth for quote artifacts is estimated.
[ ] Load test report exists.

Peak hour:
- 2,000 active sales users
- 12,000 quote preview pricing calls/hour
- 1,500 quote submits/hour
- 300 approval decisions/hour
- 900 quote acceptances/hour
- 900 order starts/hour
- 5,000 fulfillment step events/hour

Can pricing p95 remain within target?
Can outbox publish keep up?
Can Camunda jobs keep up?
Can DB pool avoid saturation?
Can Redis memory stay below threshold?

[ ] CRM availability assumptions are documented.
[ ] Catalog publication dependency is documented.
[ ] Inventory API timeout/retry/reconciliation is documented.
[ ] Billing API/event dependency is documented.
[ ] Document renderer dependency is documented.
[ ] Notification provider dependency is documented.
[ ] Identity provider dependency is documented.
[ ] Each dependency has timeout, retry, fallback, and owner.
[ ] Unknown outcome policy exists for side-effecting dependencies.

Any Red in security, data integrity, domain invariant, or recovery blocks launch.
Yellow can launch only with explicit condition, owner, due date, and risk acceptance.

Launch condition:
- launch only for tenant group: internal-sales-pilot
- maximum quote line count: 20
- no amendment/change order flow
- billing handoff in observe-only mode
- manual approval fallback enabled
- on-call coverage during business hours + 4 hours
- daily readiness review for first 2 weeks

# Production Readiness Packet: CPQ/OMS Release <version>

Date:
Release owner:
Technical owner:
Domain owner:
SRE owner:
Security owner:

## Scope

Services:
APIs:
Events:
Database migrations:
BPMN/DMN deployments:
Feature flags:
Tenants/customers affected:

## Critical Journeys

| Journey | Owner | SLO | Dashboard | Runbook | Test Evidence |
|---|---|---|---|---|---|

## Architecture Decisions

Linked ADRs:
Known deviations:
Accepted risks:

## Domain Invariants

| Invariant | Enforcement | Test | Monitoring |
|---|---|---|---|

## Failure Modes

| Failure mode | Behavior | Alert | Runbook | Drill |
|---|---|---|---|---|

## Security

Threat model:
Access matrix:
Security tests:
Known risks:

## Data and Recovery

Migration plan:
Backup restore evidence:
Projection rebuild evidence:
Audit export evidence:

## Operations

Dashboards:
Alerts:
Runbooks:
On-call:
Escalation:

## Release Plan

Deployment order:
Smoke tests:
Feature flag plan:
Roll-forward plan:
Rollback limitations:

## Decision

Ready / Conditional / Blocked
Conditions:
Approvers:

Billing handoff callback can be lost. OMS marks order line BILLING_REQUESTED but no reconciliation job exists.

High

Order may remain pending indefinitely and support cannot distinguish billing delay from lost callback.

- Add billing_handoff table with status and next_reconcile_at.
- Add reconciliation job querying billing by externalRequestId.
- Create fallout after SLA breach.
- Add dashboard for billing handoff pending age.
- Add runbook.
- Add failure drill: drop callback event and verify reconciliation.

Blocked for tenants using automated billing handoff.
Conditional launch allowed only if billing handoff feature flag is disabled.

Monitoring: yes
Runbook: yes
Security: yes

What is the first customer-visible symptom if this fails?
What invariant can never be violated, even under partial outage?
What data can be rebuilt, and from what authority?
What data cannot be rebuilt?
What operation is unsafe even if it seems to fix the issue?
What is the oldest running workflow instance after deployment, and what version is it on?
Can we reproduce a disputed quote exactly?
Can we explain why a discount was approved?
Can a duplicate event duplicate a side effect?
Can a timeout create duplicate external reservation?
What happens if outbox publisher is down for six hours?
What happens if a tenant is suspended while orders are in progress?
Who can manually recover fallout, and what prevents abuse?
What will wake up on-call before customers complain?
What will support see before engineering logs in?

Production Readiness Decision

Release:
Date:
Decision: Ready / Conditional / Blocked

Summary:

Critical blockers:

Conditional risks:

Accepted risks:

Required follow-up:

Approvers:
- Technical owner:
- Domain owner:
- SRE/Operations:
- Security:
- QA:
- Release manager:

commercial truth,
approval truth,
order truth,
workflow truth,
audit truth,
tenant boundary truth.

launch fast, corrupt business truth, recover manually, lose trust, then spend months rebuilding confidence.

the system can sell, approve, accept, fulfill, fail, recover, explain, and prove what happened.