Part 066 — EBS Snapshot Lifecycle, DLM, and Application Consistency

EBS snapshots are simple to create.

They are harder to operate correctly.

A real EBS snapshot strategy must answer:

• which volumes are protected?
• how often?
• how long?
• which snapshots are copied?
• which snapshots are immutable/locked?
• are snapshots crash-consistent or application-consistent?
• are multiple volumes captured at the same point in time?
• can encrypted snapshots be restored by the recovery role?
• how long does first read from restored volume take?
• do we need Fast Snapshot Restore?
• who can delete or share snapshots?
• which snapshots are orphaned?
• how much does all of this cost?

This part goes deep into EBS snapshot lifecycle and automation.

1. Problem yang Diselesaikan

Part ini membahas:

• how EBS snapshots work
• incremental block-level backup mental model
• snapshot creation lifecycle
• restoring volumes from snapshots
• first-read initialization and Fast Snapshot Restore
• multi-volume crash consistency
• application-consistent snapshots
• AWS Backup vs Amazon Data Lifecycle Manager
• DLM snapshot and AMI policies
• pre/post scripts with SSM
• Windows VSS snapshots
• Linux/database freeze/flush workflows
• snapshot copy, share, encryption, KMS
• Snapshot Lock and block public access
• cost and cleanup
• runbooks and game days

2. EBS Snapshot Mental Model

2.1 Snapshot is point-in-time volume state

An EBS snapshot captures a point-in-time state of an EBS volume.

It is not:

• a filesystem-level backup by itself
• an application-level transaction by itself
• a guarantee of database consistency by itself
• a full EC2 recovery plan by itself

It is a storage-level recovery point.

2.2 Snapshots are incremental

EBS snapshots are incremental backups. After the first snapshot, only changed blocks are saved.

Mental model:

volume blocks at t1 -> snapshot A
changed blocks between t1 and t2 -> snapshot B references unchanged blocks + stores changed blocks
changed blocks between t2 and t3 -> snapshot C

Deleting an old snapshot does not necessarily delete data needed by later snapshots. EBS manages referenced blocks so remaining snapshots stay restorable.

2.3 Snapshot data is stored in S3-managed infrastructure

You do not see snapshot objects in your S3 buckets. EBS stores snapshot data in AWS-managed storage.

Operational implication:

snapshot storage is managed by EBS, billed as snapshot storage, not as objects in your bucket

2.4 Snapshot completion vs restore usability

A snapshot can be in pending while data is being copied. For many operations, snapshot can be used before complete depending API/state, but production runbooks should track status and validate restore.

2.5 Snapshot is regional

EBS snapshots are regional resources. Volumes are created from snapshots in an Availability Zone in that Region.

For DR Region, copy snapshot to that Region.

3. Snapshot Lifecycle

3.1 Create

Input:

• source volume
• description
• tags
• optional multi-volume context
• optional encryption state inherited/selected
• optional application coordination

Command:

aws ec2 create-snapshot \
  --volume-id vol-1234567890abcdef0 \
  --description "orders-data pre-migration 2026-07-06"

3.2 Tag

Tags are operationally critical.

Recommended tags:

Name:
Owner:
Service:
Environment:
DataClass:
BackupTier:
CreatedBy:
SourceVolumeId:
Consistency: crash|application|unknown
Retention:
RestoreTested:

3.3 Retain

Retention should map to data class.

Examples:

hourly snapshots keep 24
daily snapshots keep 35
monthly snapshots keep 12
pre-migration snapshot keep 14 unless approved

3.4 Copy

Copy to:

• another Region
• another account
• another KMS key
• recovery account
• compliance account

3.5 Restore

Create volume from snapshot:

aws ec2 create-volume \
  --availability-zone ap-southeast-1a \
  --snapshot-id snap-1234567890abcdef0 \
  --volume-type gp3 \
  --size 500

3.6 Delete

Delete only when:

• retention expired
• not locked
• not required by policy
• no legal/compliance hold
• no incident preservation
• copied/alternate recovery point exists if required

Never delete manually during incident without inventory.

4. Restoring from Snapshot

4.1 Create volume in correct AZ

EBS volumes are AZ-scoped.

If replacement instance is in AZ ap-southeast-1a, create restored volume in same AZ.

4.2 Volume initialization

When you create a volume from snapshot, blocks are loaded lazily from snapshot storage as they are accessed. This can cause first-read latency on blocks not yet initialized.

Mitigations:

• pre-warm by reading all blocks
• use Fast Snapshot Restore for critical snapshots/AZs
• use application warmup
• accept latency if workload can tolerate
• restore before cutover

4.3 Fast Snapshot Restore

Fast Snapshot Restore lets you create volumes from enabled snapshots that are fully initialized at creation and immediately deliver provisioned performance.

Use when:

• low RTO
• latency-sensitive restore
• database/search volume
• many volumes must be created quickly from same snapshot
• game day shows lazy init violates RTO

Caution:

• FSR has cost
• enabled per snapshot per AZ
• plan credit/availability
• enable only for critical recovery points

4.4 Filesystem recovery

After attach:

lsblk
sudo file -s /dev/nvme1n1
sudo mkdir /mnt/restore
sudo mount -o ro /dev/nvme1n1p1 /mnt/restore

For production replacement, mount read/write only after validation.

4.5 UUID/fstab issue

If restored volume has same filesystem UUID as an existing volume, mounting can be confusing.

Use:

blkid
lsblk -f

Mount by explicit device or update UUID intentionally.

4.6 Restore validation

Check:

• filesystem mounts
• file count/sample checksum
• app can read data
• permissions
• database recovery logs
• expected timestamp
• volume type/performance
• KMS access
• monitoring/logging

5. Crash Consistency

5.1 What crash-consistent means

Crash-consistent snapshot is like disk state after sudden power loss.

Applications with journaling/recovery may recover.

But in-memory buffers may be lost.

5.2 AWS Backup multi-volume crash consistency

AWS Backup creates crash-consistent backups of EBS volumes attached to the same EC2 instance by default. The snapshots of every EBS volume attached to the instance are taken at the same moment.

This is crucial when data spans multiple volumes.

5.3 EBS multi-volume snapshots

EBS supports multi-volume snapshots for volumes attached to an EC2 instance. This provides point-in-time crash-consistent snapshots across volumes.

Use for:

• RAID/LVM
• data + WAL volumes
• multi-volume app state
• consistent volume group backup

5.4 Crash-consistent is not always enough

Not enough when:

• database needs transaction-aware backup
• app writes multiple resources outside volumes
• distributed cluster state
• memory-resident data not flushed
• app cannot recover cleanly from crash

Then use application-consistent backup.

6. Application Consistency

6.1 Definition

Application-consistent snapshot coordinates with the application so data on disk represents a valid application state.

Actions may include:

• flush buffers
• freeze filesystem
• quiesce application
• pause writes
• flush transaction logs
• create database checkpoint
• run pre/post scripts
• use Windows VSS
• validate app state after snapshot starts

6.2 DLM pre/post scripts with SSM

Amazon Data Lifecycle Manager integrates with AWS Systems Manager to run pre and post scripts for application-consistent snapshots.

DLM can:

• run pre script before snapshot creation
• freeze/flush I/O
• create snapshot
• run post script to resume application
• support Windows VSS, SAP HANA, and self-managed databases such as MySQL/PostgreSQL via templates/custom scripts

6.3 Windows VSS

For Windows workloads, AWS supports application-consistent EBS snapshots using Systems Manager Run Command and Windows Volume Shadow Copy Service. VSS-aware applications can flush pending transactions and coordinate consistent volume-level backups.

Use for:

• Microsoft SQL Server on EC2
• Windows file/application servers
• VSS-aware apps

6.4 Linux/database pre/post pattern

Example conceptual workflow:

pre:
  app health check
  database flush/checkpoint
  filesystem freeze /data
snapshot:
  create multi-volume snapshot
post:
  unfreeze filesystem
  resume writes
  verify app health

For PostgreSQL/MySQL, prefer database-native backup/PITR strategy if required. Snapshot scripts must be designed by database experts.

6.5 Filesystem freeze caution

Freezing too long can stall application writes.

Design:

• short freeze window
• timeout
• automatic unfreeze on failure
• monitoring
• post-script always runs
• test under load
• alert if freeze exceeds threshold

6.6 Application consistency shared responsibility

AWS can orchestrate SSM/DLM commands, but you own application correctness.

You must define:

• what to flush
• what to pause
• what to validate
• how long freeze is safe
• how to recover if post script fails
• restore validation

7. AWS Backup vs Data Lifecycle Manager

7.1 AWS Backup strengths

Use AWS Backup when you need:

• centralized multi-service backup
• vaults/recovery points
• cross-account/cross-region copy
• Vault Lock/logically air-gapped vault
• restore testing
• Audit Manager
• organizational governance
• broad resource support

7.2 DLM strengths

Use DLM when you need:

• EBS snapshot lifecycle automation
• EBS-backed AMI lifecycle automation
• tag-based EBS/EC2 policies
• retention/deletion of snapshots/AMIs
• application-consistent snapshots using SSM pre/post scripts
• focused EC2/EBS lifecycle control

7.3 Both can coexist

Example:

• AWS Backup for enterprise backup/compliance
• DLM for pre-change snapshots/AMI lifecycle in application account
• DB-native backup for database PITR
• S3 versioning/Object Lock for object data

Avoid duplicate uncontrolled backup policies that create cost and confusion.

7.4 Decision table

8. Amazon Data Lifecycle Manager

8.1 Policy types

DLM supports policy types including:

• EBS snapshot policy
• EBS-backed AMI policy
• cross-account copy event policy

8.2 Snapshot policy

Targets:

• volumes
• instances

Can automate:

• create snapshots
• retain snapshots
• delete old snapshots
• copy across Regions
• tag snapshots
• schedule policies

8.3 AMI policy

Targets instances and automates:

• EBS-backed AMI creation
• AMI retention
• deregistration of old AMIs
• underlying snapshot cleanup

8.4 Cross-account copy event policy

Automates cross-region copy actions for snapshots shared with your account.

Use for recovery account workflows.

8.5 Tag-based targeting

Example:

DLM targets volumes with:
BackupTier = Gold

Guardrail:

• ensure tag applied
• audit untagged volumes
• deployment blocks missing backup tags
• exception tags explicit

8.6 Retention

DLM retention can be:

• count-based
• age-based

Example:

create every 6 hours
retain last 28 snapshots

or:

retain for 35 days

Choose based on RPO/RTO and storage cost.

9. Snapshot Copy, Sharing, and Encryption

9.1 Copy snapshot

Use snapshot copy to:

• move to another Region
• change encryption key
• encrypt unencrypted snapshot copy
• create independent recovery point
• copy into recovery account

9.2 Full vs incremental copy

When copying to a new Region, the first copy is full. Subsequent copies can be incremental depending source/destination chain and conditions. Changing encryption state/key can cause full copy and higher cost.

9.3 Share snapshot

Snapshots can be shared privately with specific accounts. Public sharing is dangerous and should generally be blocked.

9.4 Block public access for EBS snapshots

Amazon EBS supports block public access for snapshots to prevent public sharing. Enable this as an account-level guardrail where appropriate.

9.5 Encrypted snapshot sharing

For encrypted snapshots:

• share snapshot
• key policy must allow destination account/principal
• destination may need to copy snapshot with its own KMS key
• test restore in destination account

9.6 Snapshot Lock

Amazon EBS Snapshot Lock can protect snapshots from deletion for a specified duration in governance or compliance mode.

Use for:

• critical recovery points
• ransomware resilience
• compliance retention
• pre-migration safety

Caution:

• compliance mode has strong immutability implications
• test governance mode first
• legal/security approval

10. Cost Engineering

10.1 Snapshot storage cost

Snapshots are incremental but still accumulate changed blocks.

High churn volumes can make snapshot storage grow fast.

10.2 Frequent snapshots

Higher frequency reduces RPO but increases:

• snapshot count
• changed-block retention
• copy job volume
• management overhead
• restore inventory complexity

10.3 Old AMIs

AMIs have backing snapshots. Deregistering AMI does not always mean all associated snapshots are gone unless lifecycle policy handles them.

Track:

• old AMIs
• orphaned snapshots
• AMI creation date
• last launch time if available
• owner

10.4 Fast Snapshot Restore cost

FSR can be expensive if enabled broadly.

Use only for:

• critical snapshots
• selected AZs
• limited duration
• tested RTO improvement

10.5 Restore test cleanup

Game days can create:

• restored volumes
• launched instances
• copied snapshots
• temporary AMIs
• FSR-enabled snapshots

Automate cleanup.

10.6 Cross-region copy cost

Copying snapshots across Regions incurs storage and potential data transfer costs. First copies can be full.

Plan cost for:

• Region DR
• monthly long-term copy
• recovery account copies
• test restores

11. Security Guardrails

11.1 Prevent public snapshot exposure

Enable block public access for EBS snapshots.

Monitor:

• ModifySnapshotAttribute
• public restore permissions
• snapshot sharing events

11.2 Restrict snapshot deletion

Use:

• IAM policy
• SCP
• Snapshot Lock for critical snapshots
• AWS Backup Vault Lock if using AWS Backup
• CloudTrail alerts

11.3 Restrict AMI sharing

Monitor:

• ModifyImageAttribute
• public AMI launch permission
• cross-account share to unknown accounts

11.4 KMS key protection

Alert on:

• DisableKey
• ScheduleKeyDeletion
• key policy changes
• grant changes
• denied decrypt during restore

11.5 Least privilege

Separate roles:

• backup creation
• restore operation
• snapshot copy
• security audit
• key administration

Application runtime roles should rarely delete snapshots.

12. Observability

12.1 Snapshot dashboard

Show:

• snapshots by owner/service
• snapshots by age
• snapshots by volume
• snapshots by encryption state
• snapshots by lock state
• snapshot storage cost
• failed snapshots
• pending snapshots age
• snapshots without tags
• snapshots shared/public
• snapshots not tested

12.2 DLM dashboard

Show:

• policy status
• last execution
• next execution
• target count
• created snapshots
• deleted snapshots
• failed actions
• disabled policies
• policies with no targets
• volume tags missing

12.3 Recovery dashboard

Show:

• latest snapshot per critical volume
• latest multi-volume recovery point
• latest app-consistent snapshot
• latest cross-region copy
• latest cross-account copy
• latest restore test
• KMS restore status
• RPO breach

12.4 Cost dashboard

Show:

• snapshot GB-month
• snapshot growth rate
• top changed volumes
• old AMI snapshot cost
• FSR enabled snapshots
• cross-region copy cost
• restored test resources not cleaned

13. Runbooks

13.1 Snapshot creation failed

1. Identify volume/instance.
2. Check volume state.
3. Check permissions.
4. Check KMS key.
5. Check DLM/AWS Backup job logs.
6. Check pre-script if app-consistent.
7. Run manual snapshot if RPO at risk.
8. Fix policy/role/script.
9. Validate next scheduled run.

13.2 Application-consistent snapshot failed

1. Check SSM agent status.
2. Check instance IAM role.
3. Check command document result.
4. Check pre-script timeout.
5. Verify app freeze/unfreeze.
6. Ensure post-script ran.
7. If filesystem left frozen, unfreeze immediately.
8. Re-run in maintenance window.
9. Patch script with failure-safe cleanup.

13.3 Restore volume is slow

1. Check if volume restored from snapshot and uninitialized.
2. Determine if FSR enabled.
3. Pre-warm by reading blocks if needed.
4. Check volume type/performance.
5. Check instance bandwidth.
6. Monitor latency.
7. Enable FSR for future critical snapshots if RTO requires.

13.4 Snapshot cannot be deleted

1. Check Snapshot Lock.
2. Check AWS Backup recovery point ownership.
3. Check IAM/SCP deny.
4. Check snapshot used by AMI.
5. Deregister AMI if appropriate.
6. Wait retention if locked.
7. Update lifecycle policy.

13.5 KMS restore denied

1. Identify snapshot key.
2. Check key state.
3. Check restore role.
4. Check key policy/grants.
5. Check destination account/Region.
6. Copy/re-encrypt if possible.
7. Use alternate recovery point.
8. Add KMS test to game day.

13.6 Public snapshot detected

1. Remove public permission immediately.
2. Identify exposure window.
3. Determine data sensitivity.
4. Rotate secrets if present.
5. Notify security/compliance.
6. Enable block public access.
7. Add preventive SCP/Config rule.

13.7 DLM policy disabled

1. Identify policy and resources.
2. Determine RPO breach.
3. Re-enable or create replacement.
4. Run on-demand snapshot for critical volumes.
5. Investigate actor/change.
6. Add alert on policy state change.

14. Game Days

Scenario 1 — Multi-volume restore

Expected:

• choose crash-consistent snapshot set
• create all volumes
• attach to recovery instance
• assemble filesystem/RAID/LVM
• app validates

Scenario 2 — Application-consistent backup under load

Expected:

• pre-script freezes safely
• snapshot starts
• post-script resumes
• app latency impact acceptable
• restore validates cleanly

Scenario 3 — FSR vs lazy restore

Expected:

• restore with and without FSR
• measure first-read latency
• decide critical snapshots/AZs for FSR

Scenario 4 — Encrypted cross-account restore

Expected:

• recovery account can copy/decrypt
• KMS policies correct
• volume restored
• app reads data

Scenario 5 — Snapshot deletion prevention

Expected:

• locked/protected snapshot cannot be deleted
• alert fires
• incident workflow triggered

15. Terraform/IaC Concepts

15.1 DLM snapshot policy concept

resource "aws_dlm_lifecycle_policy" "gold_ebs" {
  description        = "Gold EBS snapshot policy"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"

  policy_details {
    resource_types = ["VOLUME"]

    target_tags = {
      BackupTier = "Gold"
    }

    schedule {
      name = "every-6-hours"

      create_rule {
        interval      = 6
        interval_unit = "HOURS"
      }

      retain_rule {
        count = 28
      }

      tags_to_add = {
        CreatedBy   = "DLM"
        BackupTier  = "Gold"
        Consistency = "crash"
      }

      copy_tags = true
    }
  }
}

Validate provider version and supported fields.

15.2 DLM AMI policy concept

resource "aws_dlm_lifecycle_policy" "app_ami" {
  description        = "Application AMI lifecycle"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"

  policy_details {
    policy_type    = "IMAGE_MANAGEMENT"
    resource_types = ["INSTANCE"]

    target_tags = {
      CreateAmi = "true"
    }

    schedule {
      name = "daily-ami"

      create_rule {
        interval      = 24
        interval_unit = "HOURS"
      }

      retain_rule {
        count = 7
      }

      copy_tags = true
    }
  }
}

15.3 Snapshot lock concept

aws ec2 lock-snapshot \
  --snapshot-id snap-1234567890abcdef0 \
  --lock-mode governance \
  --lock-duration 30

Validate current API/CLI syntax before use.

15.4 Block public access concept

aws ec2 enable-snapshot-block-public-access \
  --state block-all-sharing

Validate state option according to account policy.

16. Design Checklist

16.1 Snapshot policy

• Volumes are tagged by data class.
• Snapshot frequency meets RPO.
• Retention meets policy.
• Cross-account/region copy evaluated.
• Snapshot encryption/KMS tested.
• Snapshot deletion restricted.
• Snapshot restore tested.
• Cost monitored.
• Orphan snapshots cleaned.

16.2 Consistency

• Crash consistency sufficient?
• Multi-volume consistency required?
• Application consistency required?
• Pre/post scripts tested?
• VSS required for Windows apps?
• Database-native backup considered?
• Restore validation proves app correctness?
• Freeze timeout safe?

16.3 DLM/AWS Backup

• Correct tool chosen.
• Policies are IaC-managed.
• Tags are governed.
• Policy failures alert.
• Disabled policy alerts.
• Restore testing exists.
• Duplicate policies reviewed.
• Owner/cost tags applied.

16.4 Security

• Block public snapshot access enabled.
• Snapshot sharing monitored.
• KMS key deletion guarded.
• Restore role least privilege tested.
• Critical snapshots locked or stored in locked backup vault.
• AMI sharing restricted.
• CloudTrail alerts configured.

17. Mini Case Study — PostgreSQL on EC2 with Two Volumes

17.1 Layout

/data -> data files
/wal  -> write-ahead logs

17.2 Bad backup

Daily independent snapshots:

snapshot /data at 02:00:03
snapshot /wal at 02:07:31

Risk:

• inconsistent recovery point
• WAL mismatch
• restore fails or loses data

17.3 Better backup

Options:

1. Use database-native backup/PITR.
2. Use application-consistent DLM with pre/post scripts.
3. Use AWS Backup multi-volume crash-consistent recovery point if crash recovery is acceptable.

17.4 Validation

Restore test must:

• create both volumes from same recovery point
• attach/mount
• start PostgreSQL in recovery
• verify database consistency
• run application smoke query
• measure RTO/RPO

17.5 Invariant

Database recovery is proven only when restored database starts and passes consistency checks.

18. Mini Case Study — Fast Restore for Critical Search Index

18.1 Workload

A search index volume is 4 TiB.

It is rebuildable from primary data, but rebuild takes 12 hours.

Restore from snapshot without pre-warm causes high latency for first hours.

18.2 Design

• daily snapshot
• index is derived but expensive to rebuild
• critical recovery snapshot has FSR enabled in primary AZ during release windows
• restore game day measures:
  • lazy restore latency
  • pre-warm time
  • FSR cost/benefit
• fallback rebuild path documented

18.3 Invariant

Derived does not mean unprotected if rebuild time exceeds RTO.

19. Summary

EBS snapshots are a core AWS recovery primitive, but production correctness depends on lifecycle and consistency design.

Key principles:

1. Snapshots are incremental point-in-time volume backups.
2. Restore performance can be affected by lazy block initialization.
3. Fast Snapshot Restore helps when immediate full performance is required.
4. Multi-volume applications need coordinated snapshots.
5. Crash consistency is not the same as application consistency.
6. DLM automates EBS snapshot/AMI lifecycle and supports app-consistent scripts.
7. AWS Backup centralizes backup governance and restore testing.
8. Encrypted snapshots need KMS recovery design.
9. Snapshot sharing/deletion must be controlled.
10. Cost requires lifecycle, owner tags, and cleanup.
11. Restore tests must validate the application, not just create a volume.

The core rule:

A snapshot is a recovery ingredient. A tested, consistent, encrypted, restorable snapshot policy is a recovery system.

Next, we continue data protection with S3/EFS/FSx backup and replication patterns: versioning, Object Lock, replication, AWS Backup for S3/EFS/FSx, and restore workflows across object and file storage.

References

• Amazon EBS User Guide — Amazon EBS snapshots: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html
• Amazon EBS User Guide — How Amazon EBS snapshots work: https://docs.aws.amazon.com/ebs/latest/userguide/how_snapshots_work.html
• Amazon EBS User Guide — Create Amazon EBS snapshots: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-creating-snapshot.html
• Amazon EBS User Guide — Amazon EBS fast snapshot restore: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html
• Amazon EBS User Guide — Automate data backups with AWS Backup and Amazon Data Lifecycle Manager: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-lifecycle.html
• Amazon EBS User Guide — How Amazon Data Lifecycle Manager works: https://docs.aws.amazon.com/ebs/latest/userguide/dlm-elements.html
• Amazon EBS User Guide — Create DLM custom policy for EBS snapshots: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-ami-policy.html
• Amazon EBS User Guide — Create DLM custom policy for EBS-backed AMIs: https://docs.aws.amazon.com/ebs/latest/userguide/ami-policy.html
• Amazon EBS User Guide — Automate application-consistent snapshots: https://docs.aws.amazon.com/ebs/latest/userguide/automate-app-consistent-backups.html
• Amazon EC2 User Guide — Application-consistent Windows VSS EBS snapshots: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/application-consistent-snapshots.html
• AWS Backup Developer Guide — Amazon EBS multi-volume crash-consistent backups: https://docs.aws.amazon.com/aws-backup/latest/devguide/multi-volume-crash-consistent.html
• Amazon EBS User Guide — Copy an Amazon EBS snapshot: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-copy-snapshot.html
• Amazon EBS User Guide — EBS encryption: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-encryption.html