Part 063 — Backup, Disaster Recovery, and Data Protection Mental Model

A backup that has never been restored is not a recovery plan.

A replica that faithfully copies corruption is not a backup.

A snapshot that cannot be decrypted by the recovery role is not protection.

A disaster recovery document that nobody has executed is not operational readiness.

Data protection is not a checkbox. It is a system.

It includes:

• what data must survive
• what failure it must survive
• how much data loss is acceptable
• how fast service must return
• how restore is executed
• who can delete backups
• who can decrypt backups
• how application consistency is achieved
• how recovery is tested
• how cross-account/cross-region isolation works
• how operators know which recovery point is safe

This section starts the final major arc of AWS Compute and Storage: how to protect and recover everything we have built so far.

1. Problem yang Diselesaikan

Part ini membangun mental model untuk:

• backup vs snapshot vs replica vs archive vs cache
• RPO, RTO, RPA, MTD, and recovery confidence
• failure taxonomy
• application consistency vs crash consistency
• restore-first architecture
• immutable and isolated recovery
• cross-account/cross-region design
• catalog + bytes consistency
• backup policy by data class
• game days and recovery drills
• runbooks for delete, corruption, region failure, ransomware, KMS failure, and backup gap

Tujuannya bukan hanya "membuat backup."

Tujuannya adalah membuat sistem yang bisa kembali hidup dengan benar.

2. Core Vocabulary

2.1 RPO — Recovery Point Objective

RPO answers:

How much data can we afford to lose?

Examples:

RPO is about data loss window.

If backup runs daily at 02:00 and incident happens at 23:00, RPO could be up to 21 hours unless other protection exists.

2.2 RTO — Recovery Time Objective

RTO answers:

How long can the service be unavailable/degraded?

RTO includes:

• detection
• decision
• restore job start
• restore duration
• validation
• application configuration
• DNS/routing
• cache warmup
• user communication
• rollback decision

Do not confuse:

backup restore job completed

with:

service recovered

2.3 RPA — Recovery Point Actual

RPA is what you actually have at incident time.

latest usable backup: 2026-07-06 01:00
latest replica sync: 2026-07-06 03:17
latest application-consistent checkpoint: 2026-07-06 02:55
latest tested restore: 2026-06-30

Top-tier engineers distinguish objective from actual.

2.4 RTA — Recovery Time Actual

RTA is how long recovery actually took during a test or incident.

RTO target: 4 hours
RTA in last game day: 7h 20m

That is a design gap.

2.5 MTD — Maximum Tolerable Downtime

MTD is the business maximum downtime before consequences become unacceptable.

RTO should be less than MTD.

2.6 Recovery confidence

Recovery confidence is not a formal AWS metric. It is an engineering concept:

confidence = tested recovery path + current recovery point + permissions/KMS + app validation + operator readiness

A recovery plan with high confidence has been executed recently with representative data.

3. Backup vs Snapshot vs Replica vs Archive vs Cache

3.1 Backup

A backup is a recovery point retained for restore.

Characteristics:

• point-in-time or continuous
• retained by policy
• often stored separately from live data
• can be copied cross-account/region
• may be immutable/locked
• restore workflow required

Use for:

• accidental delete
• corruption rollback
• ransomware recovery
• compliance recovery
• point-in-time restore

3.2 Snapshot

A snapshot is a point-in-time image of a storage resource.

Examples:

• EBS snapshot
• FSx snapshot
• OpenZFS snapshot
• ONTAP snapshot

Snapshots are often the mechanism behind backups, but not always the complete recovery strategy.

Limitations:

• may be same account/region unless copied
• may depend on KMS/key permissions
• may not be application-consistent
• may not include catalog/application state
• may not be immutable

3.3 Replica

A replica is a copy updated from source.

Examples:

• S3 replication
• EFS replication
• database read replica
• FSx/ONTAP replication
• cross-region file copy

Replicas help with:

• lower RTO
• region standby
• read locality
• migration
• DR

Replicas can hurt when:

• bad delete replicates
• corrupt data replicates
• ransomware encrypts source and replica follows
• replica is used without catalog consistency
• split-brain occurs during failover

3.4 Archive

Archive is long-term storage optimized for cost/retention.

Examples:

• S3 Glacier classes
• backup cold storage
• long-term file tier

Archive helps with:

• compliance retention
• cost optimization
• long retention

Archive hurts when:

• restore latency is incompatible with RTO
• retrieval cost surprises users
• application expects interactive access

3.5 Cache

Cache improves performance and locality.

Examples:

• instance store cache
• local disk cache
• Amazon File Cache
• EFS cache-like directory
• CDN/cache

Cache is not recovery unless explicitly persisted and protected.

Rule:

Cache can be deleted without losing truth.

If not, it is not cache; it is state.

4. Failure Taxonomy

4.1 Human mistake

Examples:

• accidental delete
• wrong lifecycle rule
• bad cleanup script
• wrong bucket/prefix
• wrong IAM policy
• wrong KMS key deletion
• wrong migration cutover

Protection:

• versioning
• backups
• object lock/vault lock
• change review
• dry-run
• least privilege
• restore test

4.2 Application bug

Examples:

• corrupt output
• bad serializer
• duplicate overwrite
• schema migration bug
• checkpoint corruption
• wrong partition path
• catalog points to wrong object

Protection:

• point-in-time backup
• application-consistent checkpoint
• immutable output
• manifest commit
• canary migration
• rollback runbook

4.3 Infrastructure failure

Examples:

• instance loss
• volume failure
• AZ disruption
• file system failover
• region impairment
• network partition

Protection:

• multi-AZ design
• auto scaling
• EBS snapshots
• replicas
• cross-region copies
• recovery environment
• failover runbook

4.4 Security incident

Examples:

• compromised credentials delete data
• ransomware encrypts files
• malicious version delete
• backup vault deletion attempt
• KMS key disabled
• exfiltration

Protection:

• cross-account backups
• Vault Lock/Object Lock
• least privilege
• immutable retention
• MFA/change approval
• GuardDuty/malware scanning where applicable
• recovery account isolation
• KMS separation
• CloudTrail monitoring

4.5 Dependency failure

Examples:

• KMS unavailable/denied
• AD unavailable for SMB
• DNS/route issue
• IAM/SCP accidental deny
• backup service role broken
• VPC endpoint policy blocks restore

Protection:

• dependency inventory
• recovery role test
• break-glass access
• cross-account role
• KMS policy game day
• identity DR

4.6 Data lifecycle failure

Examples:

• lifecycle expired needed data
• archive transition breaks hot path
• noncurrent versions deleted too early
• backup retention shorter than legal requirement
• clone/snapshot sprawl fills storage

Protection:

• lifecycle review
• policy simulation/testing
• inventory reports
• retention ownership
• cost dashboard
• restore drills

5. Consistency Models

5.1 Crash-consistent backup

Crash-consistent means the backup captures storage as if the machine crashed at that instant.

It is usually enough for:

• many file workloads
• stateless app volumes
• logs
• systems with journaled filesystems
• workloads that can recover after crash

It may be insufficient for:

• databases without proper recovery
• multi-volume application state
• in-memory buffered writes
• distributed apps
• application-level transactions spanning services

AWS Backup creates crash-consistent backups of EBS volumes attached to the same EC2 instance by default, coordinating snapshots at the same point in time.

5.2 Application-consistent backup

Application-consistent means the application is quiesced/flushed so backup captures a valid application state.

Examples:

• VSS for Windows applications
• database pre/post scripts
• filesystem freeze
• application checkpoint
• transaction log flush
• coordinated multi-service snapshot

Use when:

• database or transaction system
• file system alone cannot ensure correctness
• business state spans multiple files/volumes
• restore must not require long repair

5.3 Multi-volume consistency

If application uses several EBS volumes:

/data
/wal
/index

Snapshots should be coordinated.

Do not snapshot each volume at random times and call it a database backup.

5.4 Catalog + object consistency

If metadata lives in database and bytes live in S3/EFS/FSx, recovery must restore a consistent pair.

Example:

catalog row -> s3://bucket/key?versionId=abc

If catalog restored to 03:00 but S3 object state restored to 02:00, references may break.

Patterns:

• versioned object references
• immutable object keys
• database transaction stores version ID
• point-in-time database restore + object version retention
• reconciliation job
• manifest commit

5.5 Event-driven consistency

If events trigger processing:

object uploaded -> event -> output -> catalog update

Backup/restore must handle:

• duplicate events
• missing events after restore
• output already written
• catalog rolled back
• idempotency table state

Recovery may need replay/reconciliation.

6. Restore-First Design

6.1 Start with restore

For every system, ask:

How do we restore it?

before:

How do we back it up?

Backup plan without restore plan is incomplete.

6.2 Restore target

Can you restore to:

• same account?
• recovery account?
• isolated VPC?
• another Region?
• new resource name?
• same KMS key?
• different KMS key?
• staging environment?
• production replacement?

6.3 Restore permissions

Test:

• backup service role
• restore operator role
• KMS decrypt
• IAM PassRole
• EC2/EBS permissions
• EFS/FSx creation permissions
• S3 bucket restore target permissions
• network/security group creation
• tagging and SCP constraints

6.4 Restore metadata

Some restores require metadata:

• subnet
• security group
• instance type
• IAM role
• KMS key
• VPC
• file system config
• database parameter group
• bucket target
• restore path
• mount target
• AD/domain

Store restore metadata or use AWS Backup inferred metadata where available, but test it.

6.5 Restore validation

A restore is not complete until validation passes.

Validation levels:

1. Storage validation: resource exists, bytes present.
2. Permission validation: app role can access/decrypt.
3. Application validation: app can start and read data.
4. Business validation: expected records/files visible.
5. Integration validation: dependencies and workflows work.
6. User validation: critical user journey works.

6.6 Restore environment

Use isolated restore environment for tests:

recovery VPC
separate subnets
no production writes
restricted outbound
temporary DNS
test-only users
cleanup after test

Avoid accidental conflict with production.

7. Protection Patterns by Storage Type

7.1 EC2 + EBS

Protection tools:

• EBS snapshots
• AMIs
• Data Lifecycle Manager
• AWS Backup
• application-consistent snapshots with SSM/VSS/pre-post scripts
• cross-region/cross-account snapshot copy
• image pipeline for rebuild
• user data/IaC for instance recreation

Pattern:

immutable AMI + EBS data snapshots + IaC + restore test

7.2 S3

Protection tools:

• versioning
• Object Lock
• MFA Delete where used
• lifecycle
• replication
• AWS Backup for S3
• cross-account protected bucket
• S3 Inventory
• CloudTrail data events

Pattern:

versioned object + catalog versionId + retention policy + protected replica/backup

7.3 EFS

Protection tools:

• AWS Backup
• EFS replication
• lifecycle
• access points/permissions
• DataSync migration/copy
• restore testing

Pattern:

access-point isolated file system + AWS Backup + replication for DR + restore game day

7.4 FSx

Protection tools vary by family:

• automatic/manual backups
• snapshots
• clones
• replication/migration features
• AWS Backup support depending service/resource type
• service-native restore

Pattern:

volume/share policy + snapshots for operational restore + backups for recovery + migration/DR test

7.5 Instance store and local scratch

Protection tool:

• none for local state
• rebuild from durable source
• output commit to durable store
• cache warmup
• graceful drain

Pattern:

local data is disposable; durable commit goes elsewhere

7.6 Backups themselves

Backups need protection:

• vault policies
• Vault Lock
• cross-account copy
• cross-region copy
• logically air-gapped vault
• deletion monitoring
• KMS protection
• least privilege
• restore testing

8. Immutable and Isolated Recovery

8.1 Why immutability matters

If attacker or bad admin can delete or modify backups, backup is weak.

Immutable/locked recovery helps protect against:

• ransomware
• malicious delete
• compromised application role
• bad lifecycle change
• insider threat
• accidental backup deletion

8.2 S3 Object Lock

Use for object versions that must be WORM-protected.

Good for:

• evidence
• audit logs
• backup objects
• compliance records

8.3 AWS Backup Vault Lock

Vault Lock can enforce retention on recovery points in a backup vault. In compliance mode after grace time, vault lock configuration cannot be changed/deleted while it contains recovery points.

Use for:

• immutable backup retention
• ransomware-resilient recovery points
• compliance backup policy
• protection from backup deletion

8.4 Cross-account backup

Put protected backups in a recovery/security account.

Benefits:

• blast-radius reduction
• source account compromise less likely to delete backups
• independent KMS/key policy
• separate operators
• restore through controlled workflow

8.5 Cross-region backup

Protects against regional impairment or regional operational failure.

Questions:

• which Region?
• data residency?
• KMS key in destination?
• restore environment in destination?
• network/DNS?
• dependency availability?
• cost?

8.6 Logically air-gapped vault

AWS Backup logically air-gapped vaults provide specialized backup vaults with additional security and sharing capabilities intended to improve recovery options and RTO.

Use when:

• recovery account isolation matters
• backup sharing for recovery needs controlled workflow
• ransomware-resilient posture is required
• organization wants centralized protected recovery

Design with security team.

9. Recovery Architecture Patterns

9.1 Pilot light

Minimal core infrastructure running in DR Region/account.

Data replicated/backed up.

On incident, scale up app.

Good for:

• moderate RTO
• cost control
• app can be rebuilt quickly

9.2 Warm standby

Reduced-size but functional stack in DR Region.

Data continuously replicated or restored periodically.

Good for:

• lower RTO
• periodic traffic testing
• higher cost acceptable

9.3 Active-active

Both Regions serve traffic.

Hardest mode.

Needs:

• conflict resolution
• multi-region data design
• idempotency
• traffic routing
• global control plane
• split-brain prevention

Do not choose active-active just because it sounds resilient.

9.4 Backup-and-restore

Cheapest but slowest.

Good for:

• non-critical systems
• dev/test
• rebuildable workloads
• long RTO

Still must be tested.

9.5 Recovery account pattern

Use separate account for:

• backup vault
• recovery infrastructure
• restore testing
• incident isolation
• security-owned access

10. RPO/RTO by Data Class

10.1 Data class examples

10.2 Class drives policy

Do not define backup policy by service name alone.

Define by data class:

dataClass: regulated-evidence
rpo: immediate after acceptance
rto: 4h
immutability: required
crossAccount: required
restoreTest: quarterly

10.3 Mixed data classes need separation

If one bucket/file system contains:

• temp
• cache
• source of truth
• audit
• exports

then one lifecycle/backup policy will be wrong for some data.

Separate:

• prefixes
• volumes
• file systems
• buckets
• access points
• backup selections
• lifecycle rules

11. Recovery Runbooks

11.1 Accidental delete

1. Stop delete source.
2. Identify affected data scope.
3. Determine storage type.
4. Check version/snapshot/backup.
5. Restore to staging when possible.
6. Validate content and permissions.
7. Copy back or update catalog.
8. Reconcile application state.
9. Patch delete guardrail.

11.2 Logical corruption

1. Stop writers.
2. Identify corruption start time.
3. Find last known good recovery point.
4. Restore to isolated environment.
5. Validate with application-specific checks.
6. Choose partial repair vs full rollback.
7. Replay safe changes if possible.
8. Resume service.
9. Add validation to prevent recurrence.

11.3 Region failure

1. Declare disaster.
2. Freeze source writes if possible.
3. Assess replica/backup state.
4. Choose recovery point.
5. Activate DR infrastructure.
6. Restore/promote data.
7. Reconfigure app.
8. Validate dependency: KMS/IAM/DNS/network.
9. Shift traffic.
10. Record data-loss window.
11. Plan failback.

11.4 KMS recovery failure

1. Identify encrypted resource and key.
2. Check key status.
3. Check recovery role permissions.
4. Check key policy/SCP.
5. Check cross-account grant/permissions.
6. Use alternate copy encrypted with recovery key if available.
7. Escalate key deletion/disable incident.
8. Add KMS recovery test.

11.5 Ransomware-style incident

1. Isolate compromised credentials/compute.
2. Stop replication if it is propagating corruption.
3. Protect backups from deletion.
4. Identify clean recovery point.
5. Restore into isolated account/VPC.
6. Scan/validate backups if malware risk.
7. Rebuild application from clean images/IaC.
8. Restore data.
9. Rotate credentials/keys.
10. Reconnect users after validation.

11.6 Backup policy gap

1. Identify unprotected resource.
2. Determine data class and required RPO/RTO.
3. Add backup plan/selection.
4. Create on-demand backup immediately if needed.
5. Test restore.
6. Add compliance/audit control.
7. Review similar resources.

12. Game Day Program

12.1 Why game days matter

Recovery is a skill. It decays.

Game days reveal:

• missing IAM permissions
• missing KMS permissions
• broken restore metadata
• outdated runbooks
• backups not created
• replication lag too high
• catalog not consistent
• DNS cutover unclear
• app cannot start from restored data
• operators do not know owner

12.2 Game day types

12.3 Minimum cadence

For critical systems:

• restore test quarterly
• DR failover exercise semiannually or annually
• backup policy audit monthly
• recovery role/KMS test quarterly
• incident runbook review after every major architecture change

12.4 Success criteria

A game day succeeds when:

• recovery point found
• restore completed
• application validated
• RTO/RPO measured
• gaps documented
• fixes owned
• next test scheduled

Not when:

we clicked restore and it started

12.5 Game day record

gameDay:
  date:
  system:
  scenario:
  recoveryPoint:
  restoreTarget:
  operators:
  rpoActual:
  rtoActual:
  validation:
    storage: pass
    permissions: pass
    appSmoke: pass
    businessChecks: pass
  gaps:
  actionItems:
  nextTest:

13. Observability and Audit

13.1 Backup coverage

Track:

• resources without backup plan
• resources with wrong retention
• resources without cross-account/cross-region copy
• critical resources without restore test
• backup jobs failed
• backup age
• backup vault lock status
• recovery points deleted
• backup plan changed

13.2 Recovery readiness

Track:

• last successful restore test
• last app validation
• restore role health
• KMS recovery test status
• DR environment drift
• IaC deployability
• replica lag
• backup vault policy changes
• recovery account access health

13.3 Data protection events

Alert on:

• DeleteRecoveryPoint
• PutBackupVaultAccessPolicy
• DeleteBackupVaultLockConfiguration
• KMS DisableKey/ScheduleKeyDeletion
• S3 DeleteObjectVersion
• S3 PutLifecycleConfiguration
• EFS/FSx backup disabled
• snapshot delete
• replication disabled
• Object Lock retention changes
• root/admin destructive actions

13.4 Audit evidence

Store:

• backup plans
• restore test reports
• recovery point inventory
• vault lock configuration
• cross-account copy status
• KMS key policy
• game day results
• RPO/RTO evidence
• incident recovery timelines

14. Design Checklist

14.1 For every workload

• Data classes identified.
• Source of truth defined.
• RPO documented.
• RTO documented.
• Backup mechanism chosen.
• Replication need evaluated.
• Immutability/isolation need evaluated.
• KMS recovery tested.
• Restore target designed.
• Restore validation defined.
• Game day scheduled.
• Owners assigned.

14.2 For every backup plan

• Resource selection correct.
• Schedule matches RPO.
• Retention matches policy.
• Vault selected intentionally.
• Cross-account/cross-region copy evaluated.
• Vault Lock/logically air-gapped vault evaluated for critical data.
• Backup job failure alarm exists.
• Restore test plan exists.
• KMS/decrypt path tested.
• Recovery point deletion monitored.

14.3 For every restore plan

• Restore metadata documented.
• Restore role has permissions.
• KMS key accessible.
• Network/VPC target exists.
• App can start from restored data.
• Catalog consistency checked.
• Users/clients can access restored system.
• Rollback plan exists.
• RTO measured.

15. Mini Case Study — Evidence Platform

15.1 System

An evidence platform stores:

• uploaded evidence blobs in S3
• evidence catalog in database
• temporary extraction files on instance store
• thumbnails/transcripts in S3
• audit logs in S3
• admin reports in EFS

15.2 Data classification

15.3 Recovery challenge

If database rolls back to 03:00 but S3 evidence remains current, catalog may reference older/newer version IDs.

Solution:

• catalog stores S3 version ID
• evidence keys immutable/content-addressed
• S3 noncurrent versions retained beyond DB PITR window
• reconciliation checks catalog refs
• restore test validates evidence open for restored catalog

15.4 Invariant

Evidence recovery restores catalog state and exact object versions together.

16. Mini Case Study — Ransomware-Resilient Shared File Storage

16.1 System

A finance team uses FSx Windows.

Risk:

• compromised AD admin or service account encrypts/deletes files
• replication mirrors encrypted files
• shadow copies deleted or insufficient

16.2 Protection

• FSx backups retained
• cross-account backup copy
• protected backup vault with lock
• restore test to isolated account/VPC
• AD recovery plan
• SMB access least privilege
• audit logs
• user shares separated by criticality
• incident runbook stops replication/clients

16.3 Recovery

1. Isolate compromised credentials.
2. Stop file access.
3. Identify encryption start time.
4. Choose clean backup before incident.
5. Restore to isolated FSx.
6. Validate files and ACLs.
7. Rebuild clean AD/app clients if needed.
8. Cut over via DFS namespace.
9. Keep old system isolated for forensics.

16.4 Invariant

Replication is for availability.
Locked backup is for malicious/corruption recovery.

17. Summary

Backup and DR design starts with recovery, not tooling.

Key principles:

1. Define source of truth.
2. Classify data role.
3. Set RPO and RTO.
4. Separate backup from replication.
5. Use application-consistent backups where needed.
6. Protect backups from deletion.
7. Test KMS and IAM for restore.
8. Restore to isolated environments.
9. Validate application and business state.
10. Measure actual RPO/RTO.
11. Run game days.
12. Treat recovery as production engineering.

The core rule:

Backups are only useful when they can be restored, decrypted, validated, and used by the application within the business recovery objective.

Next, we turn this mental model into implementation with AWS Backup: backup plans, vaults, selections, recovery points, copy jobs, Vault Lock, logically air-gapped vaults, restore testing, Audit Manager, and production operating patterns.

References

• AWS Backup Developer Guide — What is AWS Backup?: https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html
• AWS Backup Developer Guide — Restore testing: https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-testing.html
• AWS Backup Developer Guide — Restore testing validation: https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-testing-validation.html
• AWS Backup Developer Guide — Backup Vault Lock: https://docs.aws.amazon.com/aws-backup/latest/devguide/vault-lock.html
• AWS Backup Developer Guide — Logically air-gapped vault: https://docs.aws.amazon.com/aws-backup/latest/devguide/logicallyairgappedvault.html
• AWS Backup Developer Guide — Cross-Region backup copy: https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html
• AWS Backup Developer Guide — Amazon EBS multi-volume crash-consistent backups: https://docs.aws.amazon.com/aws-backup/latest/devguide/multi-volume-crash-consistent.html
• Amazon EBS User Guide — Automate application-consistent snapshots: https://docs.aws.amazon.com/ebs/latest/userguide/automate-app-consistent-backups.html
• AWS Prescriptive Guidance — Amazon EC2 backup and recovery with snapshots and AMIs: https://docs.aws.amazon.com/prescriptive-guidance/latest/backup-recovery/ec2-backup.html