Part 079 — Production Readiness Review for Compute and Storage Workloads

A production readiness review is not a meeting where people say "looks good."

It is a structured attempt to find failure before customers do.

It asks:

• what can fail?
• what happens when it fails?
• who knows?
• how fast do we know?
• how do we recover?
• what data can be lost?
• what data cannot be lost?
• what is the rollback plan?
• what is the cost/capacity risk?
• which permissions can destroy data?
• which backup was restored?
• which game day proved this?
• what will wake someone up at 03:00?
• what will the on-call person do?

This part is a rigorous production readiness review checklist for AWS compute and storage workloads.

Use it before launch, before major migration, before large-scale traffic event, before regulated workload onboarding, and after major architecture changes.

1. What Production Ready Means

A workload is production ready when it has proven answers for:

correctness
security
reliability
recoverability
operability
scalability
cost control
capacity readiness
change safety
ownership

Production ready does not mean perfect.

It means known risks are either:

• eliminated
• mitigated
• monitored
• tested
• documented
• accepted by the right owner

Unknown risks are reduced through testing, review, and operational rehearsal.

2. Review Inputs

Before the review, gather these artifacts.

2.1 Architecture artifacts

• architecture diagram
• data flow diagram
• sequence diagrams for critical workflows
• dependency map
• account/VPC/network diagram
• IAM role map
• storage classification table
• RPO/RTO table
• deployment pipeline diagram
• DR/failover diagram
• cost model
• capacity forecast

2.2 Operational artifacts

• dashboards
• alarm list
• runbooks
• incident response plan
• escalation policy
• backup plan
• restore test result
• game day result
• rollback plan
• change calendar
• ownership matrix

2.3 Security artifacts

• threat model
• IAM policies
• KMS key policy
• S3 bucket policies
• backup vault policies
• access review
• secrets inventory
• logging/audit plan
• public exposure scan
• compliance requirements

2.4 Evidence artifacts

• load test result
• failover test result
• restore test result
• chaos/game day result
• cost forecast
• quota check
• performance baseline
• migration validation if applicable

If these artifacts do not exist, the review should create action items before go-live.

3. Review Structure

3.1 Roles

3.2 Review outputs

A readiness review should produce:

decision: approved | approved-with-risks | blocked
risks:
  - description:
    severity:
    owner:
    mitigation:
    dueDate:
goLiveConditions:
  - condition:
    owner:
acceptedRisks:
  - risk:
    acceptedBy:
    expiry:
followUpReviewDate:

3.3 Severity of findings

3.4 Launch criteria

Do not launch critical production workload if:

• no owner
• no rollback
• no backup for source-of-truth data
• no restore test for critical state
• no on-call/runbook
• IAM can destroy critical data without guardrails
• no monitoring for user impact
• RPO/RTO undocumented
• quotas/capacity insufficient
• security baseline incomplete
• data classification unknown

4. Workload Intent Review

4.1 Business objective

Questions:

• What business capability does this workload provide?
• Who depends on it?
• What happens if it is unavailable?
• What happens if data is corrupted?
• What happens if data is leaked?
• What is the expected growth?
• What compliance requirements apply?

4.2 Service-level objectives

Define:

availability:
latency:
throughput:
dataDurability:
rpo:
rto:
security:
cost:

Example:

availability: 99.9%
apiP95Latency: 300ms
uploadCommitP95: 2m
rpo:
  evidence: near-zero after accepted
  catalog: 15m
rto:
  api: 4h
  evidenceRead: 4h

4.3 Data classes

Every durable data store must be classified:

4.4 Non-goals

Document what the workload does not guarantee.

Examples:

No active-active multi-region writes.
No restore of scratch directories.
No zero-RPO for generated reports.
No support for direct file mutation outside catalog.

Non-goals reduce false assumptions.

5. Architecture Decision Review

5.1 Compute selection

Questions:

• Why EC2/ECS/EKS/Lambda/Batch?
• Is compute stateless where possible?
• Are stateful servers unavoidable?
• Is AMI/container image versioned?
• How does compute scale?
• What happens on instance termination?
• How is graceful drain implemented?
• Can workload tolerate AZ loss?
• Is startup time measured?
• Is bootstrap idempotent?

5.2 Storage selection

Questions:

• Why S3/EBS/EFS/FSx/instance store?
• What is source of truth?
• What is cache/scratch/derived?
• Are storage semantics matched to app needs?
• Is S3 used with catalog/version IDs when needed?
• Is EBS root separated from data?
• Is shared file storage truly required?
• Is instance store only disposable?
• Is storage lifecycle recovery-aware?

5.3 Managed service evaluation

For self-managed state:

• Was managed service evaluated?
• Why rejected?
• Is exit strategy documented?
• Does self-managed system have failover/backup/runbooks?

5.4 Failure domain design

Questions:

• Which resources are AZ-scoped?
• Which are regional?
• Which are account-scoped?
• Which are global/control-plane dependencies?
• What happens if one AZ fails?
• What happens if one Region fails?
• What happens if account is compromised?
• What happens if KMS key is disabled?

5.5 Architecture Decision Records

For each major decision:

decision:
context:
optionsConsidered:
selectedOption:
tradeoffs:
failureModes:
reversalPlan:
reviewDate:

Top-tier engineering is explicit about trade-offs.

6. Reliability Review

6.1 Multi-AZ readiness

• Compute spans multiple AZs.
• Load balancer spans multiple AZs.
• ASG can replace failed instances.
• Remaining AZs can absorb traffic.
• Subnet IP capacity supports failover.
• EFS mount targets exist per AZ where required.
• EBS state has restore/failover strategy.
• FSx deployment type matches requirement.
• AZ evacuation game day completed.

6.2 Scaling readiness

• Scaling metric represents workload pressure.
• Min/desired/max capacity justified.
• Startup time measured.
• Warmup/cooldown tuned.
• Scheduled scaling for predictable peaks.
• Quotas checked.
• Mixed instance policy/Spot strategy tested if used.
• Scale-down drains safely.
• Backpressure exists.

6.3 Dependency resilience

• Critical dependencies identified.
• Timeouts configured.
• Retries use backoff/jitter.
• Circuit breakers where appropriate.
• Idempotency implemented.
• Dependency failure modes tested.
• Queue/DLQ configured where async.
• No infinite retry loops.

6.4 Data consistency

• Catalog and storage consistency model defined.
• Multi-object commit protocol defined.
• Version IDs/checksums stored where needed.
• Duplicate event/job handling tested.
• Partial output hidden.
• Split-brain prevention for stateful systems.
• Write authority clearly defined.

6.5 Reliability blockers

Block if:

• singleton stateful instance with no restore test
• ASG max lower than expected peak
• no backpressure for overload
• no idempotency for at-least-once processing
• shared file storage used as implicit database without review
• untested failover for critical RTO

7. Backup and Recovery Review

7.1 Backup coverage

• Every source-of-truth data store has backup/protection.
• Backup frequency matches RPO.
• Retention matches policy.
• Backup selection/tagging verified.
• Cross-account/cross-region copy evaluated.
• Immutable protection evaluated.
• KMS/key access tested.
• Backup failure alarm exists.
• Recovery point age dashboard exists.

7.2 Restore readiness

• Restore runbook exists.
• Restore target environment defined.
• Restore role permissions tested.
• KMS decrypt tested.
• Application validation script exists.
• Full restore tested for critical data.
• Item-level restore tested where promised.
• RTO/RPO actual measured.
• Restored resources cleaned up after test.

7.3 Application consistency

• Crash vs application consistency decided.
• Multi-volume snapshots coordinated.
• Database-native backup/PITR considered.
• DLM/SSM/VSS/pre-post scripts tested if used.
• Restore validates app state.

7.4 Ransomware resilience

• Backups protected from production credential compromise.
• Vault Lock/Object Lock/Snapshot Lock evaluated.
• Recovery account considered.
• KMS isolation considered.
• Clean-room restore process defined.
• Backup deletion attempts monitored.
• Retention weakening monitored.

7.5 Recovery blockers

Block if:

• critical state has never been restored
• backups depend on KMS key with untested recovery permissions
• app role can delete critical recovery points
• no clean recovery point selection for corruption/ransomware
• DB/catalog restore window exceeds object version retention

8. Disaster Recovery Review

8.1 Strategy

• DR strategy chosen: backup/restore, pilot light, warm standby, active-active.
• Strategy matches RTO/RPO.
• Cost and complexity accepted.
• DR Region/account selected.
• Data residency considered.
• Failover decision owner defined.

8.2 Data readiness

• Replication/backups exist in DR site.
• Replication lag monitored.
• Recovery point selection process.
• KMS in DR site.
• Secrets/config in DR site.
• Object/file/database consistency defined.

8.3 Compute readiness

• IaC can deploy DR stack.
• AMIs/images copied.
• Quotas sufficient.
• Subnets/IPs sufficient.
• Capacity Reservations evaluated if RTO needs.
• Standby health monitored if used.

8.4 Traffic readiness

• Route 53/ARC/DNS strategy.
• Health checks represent readiness.
• TTL/client caching understood.
• Split-brain guardrails.
• Failback plan.

8.5 DR blockers

Block if:

• DR plan cannot be executed by current on-call
• backup copied but no compute/KMS/network in DR
• failover can create two writers
• failback strategy unknown for stateful workload
• DR test older than acceptable threshold

9. Security Review

9.1 IAM

• Runtime roles least privilege.
• Admin/restore/backup roles separated.
• No broad wildcard destructive permissions.
• PassRole constrained.
• Cross-account trust reviewed.
• Break-glass role defined and monitored.
• No long-lived static keys unless exception.
• IAM Access Analyzer or equivalent review used.

9.2 KMS

• Customer managed keys where required.
• Key policies least privilege.
• Runtime role cannot administer key.
• Key deletion/disable monitored.
• Cross-account/DR decrypt tested.
• Backup/recovery key strategy documented.
• Encryption context considered where useful.

9.3 Network

• Public exposure minimized.
• No public SSH/RDP.
• SSM Session Manager or controlled access.
• Security groups scoped.
• VPC endpoints where appropriate.
• Private subnets for compute.
• WAF/edge protection if internet-facing.
• Egress risk considered.

9.4 Storage security

S3:

• Block Public Access.
• Versioning where needed.
• Object Lock where needed.
• Bucket policy denies non-TLS.
• Encryption required.
• Delete/version delete restricted.
• Lifecycle changes monitored.

EBS:

• Encryption enabled.
• Public snapshot blocked/monitored.
• Snapshot deletion restricted.
• Volume attachment permissions restricted.

EFS/FSx:

• Security groups scoped.
• Access points/export/share policies.
• POSIX/ACL model reviewed.
• TLS/encryption in transit where supported/required.
• AD/identity dependency secured.

9.5 Audit

• CloudTrail organization trail.
• S3 data events for critical buckets.
• Config rules.
• GuardDuty/Security Hub where used.
• Logs centralized/protected.
• Destructive changes alert.

9.6 Security blockers

Block if:

• public access to private data possible
• runtime role can delete immutable/protected data
• KMS key can be disabled/deleted without alert/approval
• critical control-plane changes not logged
• no incident runbook for credential compromise

10. Observability Review

10.1 User impact

• Availability metric.
• Latency p95/p99.
• Error rate.
• Request volume.
• Critical workflow success.
• Synthetic/canary checks.

10.2 Compute signals

• CPU/memory/disk/network.
• EC2 status checks.
• ASG desired/in-service/max.
• Launch failures.
• Spot interruptions.
• Bootstrap failures.
• Instance lifecycle drain.

10.3 Storage signals

• S3 4xx/5xx/latency/replication.
• EBS latency/queue/throughput/snapshot age.
• EFS PercentIOLimit/backup/replication/client errors.
• FSx capacity/throughput/backup/snapshot.
• KMS errors.
• Lifecycle/version growth.

10.4 Protection signals

• Latest backup.
• Latest restore test.
• Latest cross-account/region copy.
• Vault/Object/Snapshot lock status.
• Backup job failure.
• RPO actual.
• Replication lag.

10.5 Alert quality

• Every page has runbook.
• Severity defined.
• Owner defined.
• Dashboard linked.
• Alert tested.
• Noise reviewed.
• Composite alarms used where appropriate.
• Maintenance suppression defined.

10.6 Observability blockers

Block if:

• no alert for user-facing failure
• no storage/full capacity alert for stateful workload
• no backup failure/RPO breach alert for critical state
• no destructive API event monitoring on protected data
• on-call cannot diagnose with available dashboards/logs

11. Operations Review

11.1 Ownership

• Service owner.
• Technical owner.
• On-call owner.
• Data owner.
• Security owner.
• Cost owner.
• Escalation path.

11.2 Runbooks

Required runbooks:

• deployment rollback
• instance failure
• storage full
• backup failure
• restore data
• KMS denied
• high latency
• high error rate
• queue backlog
• DR failover/failback
• credential compromise
• cost anomaly

11.3 Incident response

• Severity model.
• Incident Manager/response plan or equivalent.
• Escalation.
• Communication template.
• Post-incident process.
• Game day cadence.

11.4 Change management

• CI/CD pipeline.
• IaC-managed infrastructure.
• Peer review.
• Automated tests.
• Canary/blue-green/rolling strategy.
• Rollback.
• Change observability.
• Maintenance windows for risky changes.

11.5 Operations blockers

Block if:

• no on-call
• no rollback
• no runbook for known high-risk failure
• console-only manual production setup
• no post-incident process
• no owner for alarms

12. Cost Review

12.1 Unit economics

• Unit of work defined.
• Cost per unit estimated.
• Major cost drivers identified.
• Retry/failure cost included.
• Backup/DR cost included.
• Observability cost included.
• Cost owner accepts model.

12.2 Cost controls

• Cost allocation tags.
• Budgets.
• Forecast alerts.
• Cost anomaly detection.
• Compute Optimizer review.
• Savings Plans/RI decision.
• Spot design where safe.
• Idle cleanup process.

12.3 Storage cost

• S3 lifecycle reviewed.
• Noncurrent version retention justified.
• Incomplete multipart cleanup.
• EBS snapshot lifecycle.
• Unattached EBS cleanup.
• EFS/FSx idle review.
• Backup retention by data class.
• Restore test cleanup.

12.4 Cost blockers

Usually cost is not a launch blocker unless:

• runaway cost risk without guardrail
• no owner for high-cost resources
• architecture cost exceeds approved budget
• cost optimization weakens required protection
• non-prod can consume unbounded expensive resources

13. Capacity Review

13.1 Quotas

• EC2 On-Demand vCPU quota.
• Spot quota if used.
• EBS/EFS/FSx quotas.
• S3/KMS request capacity considered.
• Backup/restore quotas.
• DR Region quotas.
• Service Quotas dashboard.
• Increase requests completed.

13.2 Compute capacity

• Peak forecast.
• Failure scenario forecast.
• ASG max enough.
• Mixed instance diversification.
• Capacity Reservations evaluated.
• Warm pool/startup time evaluated.
• Subnet IP capacity.

13.3 Storage capacity

• EBS size/performance forecast.
• EFS/FSx throughput/capacity.
• S3 layout/request pattern.
• KMS quotas.
• Backup copy throughput/lag.
• Restore target capacity.

13.4 Capacity blockers

Block if:

• quota insufficient for launch/DR
• ASG cannot scale to tested peak
• subnet IPs insufficient
• storage capacity/performance not tested
• Spot-only critical path without fallback
• DR restore cannot allocate capacity

14. Performance Review

14.1 Baseline

• Load test completed.
• p95/p99 latency measured.
• Throughput measured.
• CPU/memory/network observed.
• Storage latency observed.
• Cold start/warm start measured.
• Failure scenario tested.

14.2 Storage performance

• S3 request rate pattern validated.
• Multipart/range strategies where needed.
• EBS IOPS/throughput/latency validated.
• EFS/FSx throughput and metadata workload tested.
• Small file/pathological directory risk tested.
• Archive/cold access behavior understood.

14.3 Regression protection

• Performance test in CI/CD or release process.
• Canary metrics.
• Rollback threshold.
• Capacity model updated.

14.4 Performance blockers

Block if:

• no load test for critical launch
• workload exceeds storage performance assumptions
• p99 latency violates SLO under expected load
• restore performance violates RTO
• batch backlog cannot drain within SLO

15. Deployment Readiness

15.1 Build artifacts

• AMI/container image versioned.
• Artifact provenance known.
• Vulnerability scan.
• Rollback artifact retained.
• Config/secrets externalized.
• No manual server drift.

15.2 Infrastructure

• IaC applied.
• Drift checked.
• Resource tags applied.
• Security baseline.
• Backup plan attached.
• Monitoring attached.

15.3 Release strategy

• Canary/rolling/blue-green.
• Health checks.
• Automated rollback criteria.
• Manual rollback runbook.
• Database/storage migration plan.
• Pre-change backup/snapshot for risky changes.

15.4 Deployment blockers

Block if:

• no rollback artifact
• manual-only deployment
• storage migration lacks backup/rollback
• health check cannot detect broken dependency
• secrets only available from developer machine

16. Game Day Review

16.1 Required game days by workload class

Stateless web/API:

• instance termination
• AZ loss
• high latency
• rollback

Stateful EC2:

• EBS restore
• failover/fencing
• application-consistent backup
• KMS denied

S3/object:

• delete marker restore
• bad overwrite
• replication lag/KMS failure
• Object Lock delete attempt

EFS/FSx:

• item-level restore
• permission restore
• mount failure
• backup restore

Batch/worker:

• Spot interruption
• worker death
• duplicate job attempt
• poison message

DR:

• backup restore
• pilot light/warm standby scale-up
• traffic cutover
• failback

16.2 Evidence

A game day should produce:

scenario:
date:
participants:
result:
rtoActual:
rpoActual:
gaps:
actionItems:
nextTest:

16.3 Game day blockers

For critical workload, block if:

• no restore game day
• no AZ failure/instance failure test
• no rollback test
• no on-call participation
• no action items tracked

17. Go-Live Decision

17.1 Decision template

workload:
date:
reviewers:
decision: approved | approved-with-risks | blocked
rpo:
rto:
knownRisks:
acceptedRisks:
blockers:
mustFixBeforeLaunch:
mustFixAfterLaunch:
hypercareWindow:
rollbackOwner:
finalApprover:

17.2 Approved

Use when:

• critical controls pass
• risks are minor/accepted
• runbooks and monitoring ready
• rollback tested
• owners ready

17.3 Approved with risks

Use when:

• non-blocking high/medium risks exist
• explicit owner and due date
• risk accepted by correct authority
• mitigation exists

17.4 Blocked

Use when:

• data loss risk unacceptable
• no restore proof
• security exposure
• no rollback
• no owner/on-call
• capacity insufficient
• unknown critical dependency

17.5 Hypercare plan

After launch:

• elevated monitoring
• migration/launch team available
• daily health review
• cost review
• backup verification
• restore test if not already full
• action item triage
• final handoff

18. Production Readiness Scorecard

Use a simple scoring model.

Suggested interpretation:

0-10: not production ready
11-15: high risk
16-18: launch only with explicit risk acceptance
19-20: strong readiness

The score is not the goal. The conversations and action items are the goal.

19. Common Review Findings

19.1 "We have backups" but no restore

Finding:

Backup configured, restore never tested.

Action:

• restore to isolated environment
• app validation
• document RTO/RPO actual

19.2 "Stateless" app writes local files

Finding:

EC2 stores uploaded files under /var/lib/app.

Action:

• move to S3/EFS/FSx based on semantics
• add startup check forbidding local durable path
• backup interim state

19.3 S3 catalog lacks version ID

Finding:

DB stores s3://bucket/key only.

Action:

• store versionId/checksum
• align noncurrent retention with DB PITR
• add restore test

19.4 ASG max too low

Finding:

ASG max equals normal desired capacity.

Action:

• calculate peak + AZ failure + deployment surge
• update quota/subnet plan
• test scale-out

19.5 App role can delete protected data

Finding:

runtime role has s3:* on bucket.

Action:

• least privilege
• deny DeleteObjectVersion
• Object Lock/Vault Lock
• CloudTrail alert

19.6 EFS used as hidden queue

Finding:

workers poll directory for work.

Action:

• use SQS/EventBridge/catalog
• move file payload to S3/EFS only as storage
• idempotent processing

19.7 DR copies exist but no recovery capacity

Finding:

Backups copied to DR Region, but quota/VPC/KMS missing.

Action:

• DR capacity readiness
• full restore test
• KMS test
• route failover plan

20. Mini Case Study — Readiness Review Blocks Launch

20.1 System

A new document upload API is ready to launch.

20.2 Findings

• S3 bucket versioning enabled, but catalog does not store versionId
• app role can delete object versions
• no restore test
• ASG max equals desired capacity
• no CloudWatch Agent disk metrics
• no backup alarm for EFS templates

20.3 Decision

Blocked

20.4 Fixes

• store versionId/checksum
• restrict delete/version delete
• restore test object + EFS directory
• increase ASG max and quota
• install CloudWatch Agent
• add backup RPO alarm

20.5 Result

Launch delayed two days.

Two months later, a delete bug occurs; versioned restore succeeds in minutes.

20.6 Lesson

Readiness review did not slow delivery.
It prevented an outage from becoming data loss.

21. Summary

Production readiness is a structured proof that the system can survive normal production reality.

Key principles:

1. Review intent before implementation.
2. Classify data before choosing storage policy.
3. Review compute and storage semantics together.
4. Require restore evidence, not backup claims.
5. Treat KMS/IAM/backup as recovery dependencies.
6. Alert on user impact and data protection risk.
7. Map every page to a runbook.
8. Validate cost and capacity before launch.
9. Test failure with game days.
10. Document accepted risk with owner and expiry.
11. Block launch when data, security, or recovery risk is unacceptable.

The core rule:

A production readiness review is successful when it discovers painful truths while they are still cheap to fix.

Next, Part 080 closes the series with a capstone synthesis: the final mental models, operating principles, deliberate practice plan, and advanced projects that turn AWS compute/storage knowledge into top-tier engineering capability.

References

• AWS Well-Architected Framework — The pillars of the framework: https://docs.aws.amazon.com/wellarchitected/latest/framework/the-pillars-of-the-framework.html
• AWS Well-Architected Tool — Using lenses: https://docs.aws.amazon.com/wellarchitected/latest/userguide/lenses.html
• AWS Well-Architected Tool — Custom lenses for workloads: https://docs.aws.amazon.com/wellarchitected/latest/userguide/lenses-custom.html
• AWS Well-Architected Tool — Document a workload tutorial: https://docs.aws.amazon.com/wellarchitected/latest/userguide/tutorial.html
• AWS Resilience Hub — What is AWS Resilience Hub?: https://docs.aws.amazon.com/resilience-hub/latest/userguide/what-is.html
• AWS Resilience Hub — Concepts and terms: https://docs.aws.amazon.com/resilience-hub/latest/userguide/concepts-terms.html
• AWS Trusted Advisor — AWS Trusted Advisor check reference: https://docs.aws.amazon.com/awssupport/latest/user/trusted-advisor-check-reference.html
• AWS Well-Architected Operational Excellence Pillar — Use runbooks to perform procedures: https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_ready_to_support_use_runbooks.html