Part 096 — Production Readiness Final Review

Ini part terakhir seri.

Tujuannya bukan menambah service baru.

Tujuannya adalah memastikan semua yang sudah dipelajari bisa dipakai untuk menjawab satu pertanyaan production:

Apakah sistem ini siap menerima traffic nyata, gagal secara terkendali, dipulihkan dengan bukti, dan dievolusi tanpa kehilangan correctness?

Production readiness bukan checklist kosmetik.

Production readiness adalah argumentasi teknis yang bisa diaudit.

1. The Final System Map

Sistem application + database di AWS biasanya terlihat seperti ini:

The production question is not “does every box exist?”

The production question is:

• what invariant does each box protect,
• what happens when it fails,
• who owns recovery,
• what is the blast radius,
• how do we prove data correctness after recovery,
• how do we safely change it later?

2. Production Readiness as Evidence

A production-ready system has evidence for these categories:

If you cannot show evidence, it is not ready. It is only hoped ready.

3. Invariants Register

Start with invariants, not services.

Example invariant register:

A serious readiness review asks:

• Where is this invariant enforced?
• Is enforcement atomic?
• What if enforcement fails after commit?
• What if the event is duplicated?
• What if the projection is stale?
• What if the workflow is replayed?
• How do we prove and repair violation?

4. Boundary Review

Every boundary must have an owner and contract.

Anti-pattern:

"The service owns it".

Better:

Case Service owns command validation and source-of-truth writes.
Projection Worker owns OpenSearch read model freshness and rebuild.
Workflow Orchestrator owns long-running process progress and compensation.
Operations owns incident coordination, but service team owns repair logic.

5. API Readiness

API readiness is not “endpoint returns 200”.

Review:

Minimum API production gates:

• Idempotency-Key or equivalent for non-trivial commands,
• correlation ID propagated,
• request validation exists,
• structured error shape documented,
• per-tenant abuse controls exist,
• load test covers p95/p99 and failure responses,
• dashboards show latency, error, throttling, dependency failure.

6. Queue Readiness

SQS readiness means the system can absorb work, slow down safely, and recover from bad messages.

Review:

Production gate:

• message age alarm,
• visible backlog alarm,
• DLQ growth alarm,
• worker error-rate alarm,
• replay runbook,
• idempotency/inbox table,
• partial batch failure support where applicable.

7. Event Readiness

Event-driven systems fail silently when no one owns the contract.

Review:

Production gate:

• event envelope standard,
• schema compatibility CI,
• consumer idempotency,
• DLQ and replay path,
• rule inventory,
• archive/replay governance,
• event storm and replay storm runbook.

8. Workflow Readiness

Step Functions readiness means long-running process correctness is explicit.

Review:

Production gate:

• state machine reviewed as code,
• Retry/Catch policies explicit,
• terminal states meaningful,
• timeout/heartbeat configured,
• compensation tested,
• stuck execution alarm,
• execution history/logging strategy documented.

9. Database Readiness

Database readiness is not “the table exists”.

Review:

9.1 RDS/Aurora gate

• engine selection ADR,
• parameter groups reviewed,
• connection pool sized,
• RDS Proxy considered/tested if needed,
• migration lock strategy defined,
• slow query visibility enabled,
• wait-event dashboard,
• backup/PITR restore test,
• failover test,
• read replica lag strategy.

9.2 DynamoDB gate

• access pattern matrix,
• PK/SK grammar documented,
• GSI/LSI decision documented,
• hot key load test,
• capacity mode justified,
• throttling alarms,
• conditional write and transaction failure handling,
• stream consumer idempotency,
• backup/restore/export strategy,
• global table conflict strategy if multi-Region.

9.3 Cache gate

• cache is not hidden source of truth unless using MemoryDB intentionally,
• staleness budget defined,
• invalidation strategy tested,
• stampede protection,
• fail-open/fail-closed policy,
• eviction/memory alarms,
• reconnect storm mitigation,
• hot key detection.

9.4 Specialized database gate

• DocumentDB compatibility tested,
• Neptune traversal performance tested,
• Keyspaces partition design validated,
• Timestream retention/rollup policy defined,
• OpenSearch treated as projection, not source of truth,
• rebuild/reconciliation path exists.

10. Migration and Change Readiness

Every production system must change safely.

Review:

Production gate:

• change has owner,
• rollback/forward-fix chosen,
• blast radius known,
• deploy sequence written,
• validation queries ready,
• observability ready before deploy,
• destructive step separated and approved.

11. Disaster Recovery and Recovery Correctness

Backups are not recovery.

Recovery is proven only when restore and validation have been executed.

Review:

Important recovery edge cases:

• database restored to T-15 minutes but SQS still contains messages after T,
• workflow executions reference state that no longer exists,
• outbox events replay and duplicate external notification,
• cache/projection contains data newer than restored source DB,
• OpenSearch index must be rebuilt,
• DMS/CDC target lags behind source,
• global tables have region conflict after failover.

Recovery plan must include dependent state cleanup.

12. Observability Readiness

Observability must answer:

• is the system healthy,
• why is it unhealthy,
• what user/business invariant is at risk,
• what should an operator do next?

Minimum dashboard sections:

Alarm design:

• alert on user impact or invariant risk,
• avoid unactionable noise,
• include runbook link,
• include owner,
• include dashboard link,
• include recent deploy/change context.

13. Load, Capacity, and Cost Readiness

Capacity test must reflect real shape, not average traffic.

Test dimensions:

Cost review must identify drivers:

• API requests,
• Lambda duration/concurrency,
• Step Functions state transitions or Express duration,
• SQS requests,
• EventBridge events/rules/pipes/scheduler,
• DynamoDB RCU/WCU/GSI/streams/global tables,
• RDS/Aurora instance/ACU/I/O/storage/backup,
• ElastiCache/MemoryDB node/serverless usage,
• OpenSearch storage/compute/ingest,
• data transfer,
• logging volume,
• snapshot retention.

Guardrails:

• tenant quotas,
• max throughput settings where supported,
• alarms on cost anomaly,
• per-feature cost attribution,
• load-shed policy,
• expensive query review.

14. Security and Compliance Readiness

Application + database security is not only IAM.

Review:

For regulatory systems, defensibility requires:

• immutable or append-only audit trail where needed,
• actor identity,
• command reason,
• previous state,
• new state,
• timestamp,
• correlation ID,
• decision basis,
• evidence link,
• repair/override trail.

15. Incident Runbook Readiness

Minimum runbooks:

Runbook format:

Symptom:
Impact:
Likely causes:
Immediate mitigation:
Diagnostic queries:
Rollback/repair:
Validation after repair:
Escalation:
Customer/compliance note:

16. Launch Readiness Review Template

Use this before production launch or major release.

16.1 System context

Service:
Owner:
On-call:
Business capability:
Critical users:
Dependencies:
Source-of-truth databases:
Derived stores:
Async systems:
External side effects:

16.2 Invariants

Invariant:
Enforced by:
Detection:
Repair:
Evidence:
Residual risk:

16.3 Failure model

Failure:
Expected behavior:
Blast radius:
Alarm:
Runbook:
Test evidence:

16.4 Data readiness

Schema reviewed:
Migration plan:
Backfill plan:
Validation plan:
Restore test date:
RTO/RPO:
Reconciliation owner:

16.5 Go/no-go criteria

Go if:
- critical invariants have enforcement and detection,
- high-severity runbooks tested,
- dashboards/alarms active,
- rollback/forward-fix clear,
- load/capacity evidence accepted,
- security/compliance sign-off complete.

No-go if:
- source of truth unclear,
- destructive migration lacks rollback/restore plan,
- DLQ/replay path missing for critical async work,
- backup restore untested,
- no owner for production incident,
- observability cannot identify user impact.

17. Final Architecture Review Questions

Ask these in the final review.

Correctness

• What can never be allowed to happen?
• Where is that enforced?
• Is enforcement atomic?
• How do we detect violation?
• How do we repair violation?

Consistency

• Which read paths are stale by design?
• What is the staleness budget?
• Which state is source of truth?
• Which state is derived?
• Can derived state be rebuilt?

Failure

• What happens if DB write succeeds but response fails?
• What happens if event publish fails?
• What happens if message is delivered twice?
• What happens if workflow times out?
• What happens if cache is empty or stale?
• What happens if Region fails?

Operations

• Which dashboard shows user impact?
• Which alarm wakes someone up?
• Which runbook do they follow?
• Can they safely pause consumers/backfill/replay?
• Can they prove recovery?

Evolution

• Can we add a field without breaking old app?
• Can we change event contract safely?
• Can we rebuild projection?
• Can we migrate database with low downtime?
• Can we roll back or forward-fix?

18. The Top 1% Mental Model

A strong engineer does not ask:

Should we use SQS, EventBridge, Step Functions, Aurora, or DynamoDB?

A strong engineer asks:

What invariant must hold?
What is the access pattern?
Where is the transaction boundary?
What happens under retry?
What happens under duplicate delivery?
What happens under stale read?
What happens under partial failure?
How do we observe, recover, and evolve it?

AWS services are implementation choices.

The real architecture is:

• boundary,
• ownership,
• state,
• consistency,
• failure model,
• recovery,
• operability,
• evolution.

If those are clear, service selection becomes disciplined.

If those are unclear, even the best managed service becomes a distributed failure amplifier.

19. Series Completion

This is the final part of the series:

learn-aws-application-database-part-096-production-readiness-final-review.mdx

The series covered:

• application/database mental model,
• integration style decision framework,
• API Gateway and AppSync,
• SQS,
• SNS and EventBridge,
• Step Functions,
• database selection,
• RDS and Aurora,
• Aurora DSQL,
• DynamoDB,
• cache and MemoryDB/ElastiCache,
• specialized databases,
• database migration,
• zero-downtime schema change,
• final production readiness review.

You now have the map.

The next level is not more service memorization.

The next level is building production-grade systems, injecting failures, measuring invariants, and forcing the design to survive reality.

References

• AWS Well-Architected Framework: https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html
• AWS Well-Architected Operational Excellence — Implement observability: https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/implement-observability.html
• AWS Well-Architected Reliability — Plan for disaster recovery: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/plan-for-disaster-recovery-dr.html
• AWS Well-Architected Reliability — Back up data: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/back-up-data.html
• AWS Operational Readiness Reviews: https://docs.aws.amazon.com/wellarchitected/latest/operational-readiness-reviews/wa-operational-readiness-reviews.html
• AWS Prescriptive Guidance — Transactional outbox pattern: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• AWS Prescriptive Guidance — Saga pattern: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga.html
• AWS Prescriptive Guidance — Database migration strategy: https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-migration/welcome.html
• Amazon SQS Developer Guide: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
• Amazon EventBridge User Guide: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html
• AWS Step Functions Developer Guide: https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
• Amazon RDS User Guide: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html
• Amazon DynamoDB Developer Guide: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html