local logs
temporary files
manual patches
attached EBS volumes
cached credentials
runtime-generated config
warm caches
worker checkpoints
leader locks
in-flight requests

detect failure quickly
remove bad node from service
preserve only state that is intentionally durable
replace compute from known artifact
verify readiness before traffic
quarantine evidence when needed
make the fleet converge automatically

A single EC2 instance can disappear without requiring human creativity.

repair instance < replace instance

slow
manual
non-repeatable
creates drift
keeps unknown state alive
hard to audit

AMI is known
bootstrap is deterministic
state is externalized
health checks are correct
capacity exists
rollback is versioned

A stateless API node fails.

terminate and replace.

spend 90 minutes repairing one node while capacity is degraded.

A stateful node owns a data volume.

fence old owner, preserve volume, attach to recovery node, verify application-level consistency.

ASG terminates node and deletes the only copy of state.

AWS physical host
Nitro/hypervisor path
EC2 instance lifecycle
OS/kernel
systemd
application process
runtime dependency
network attachment
storage attachment
application readiness

System status failed -> host/network/power/hardware-like issue; recovery/replacement often appropriate.
Instance status failed -> OS/kernel/network config/boot/app-level issue; reboot may help, but replacement may still be better for fleets.

process alive
listening socket open
critical config loaded
dependencies reachable or intentionally degraded
storage mounted
thread pools not exhausted
readiness endpoint responds within threshold

GET /health -> 200 if process is alive

/livez    -> process can answer simple request
/readyz   -> safe to receive production traffic
/startupz -> bootstrap completed

health should fail if volume is missing or mounted read-only unexpectedly

health may pass after scratch is recreated

ASG maintains desired capacity.
If an InService instance is unhealthy, ASG can terminate and replace it.
New instances launch from current ASG/Launch Template settings.

subnets have IP capacity
quotas allow new instances
launch template works
AMI boots
user data succeeds
target group health is correct
capacity is available for selected instance types

which EBS data volume belongs to which logical shard
whether old owner is fenced
whether the application data is consistent
which node should become leader

healthy-but-slow-booting nodes killed repeatedly

bad nodes stay in service too long

use lifecycle hooks for initialization
mark instance InService only when bootstrap/readiness is complete
keep health grace period tight and intentional

config fetch
volume attach/mount
warm cache
register with service discovery
run validation script
only then continue lifecycle

drain load balancer
stop accepting new work
checkpoint worker state
flush logs/metrics
detach or snapshot data volume
notify cluster membership

bad AMI
broken kernel/initramfs
bad fstab entry
missing EBS volume
cloud-init/user data hang
network dependency during boot
package repo unavailable
IAM role permission failure
wrong architecture image
bad systemd unit dependency
full root volume

what must happen before service starts
what may happen after service starts
what external systems are allowed during boot
what timeout fails the node
what logs prove bootstrap progress

install packages from internet during production boot
run database migrations during every node boot
fetch secret without timeout
silently continue after mount failure

AMI contains heavy dependencies
user data only binds environment-specific config
mounts are explicit
bootstrap is idempotent
timeouts are bounded
failure terminates/replaces node

EC2 system log
console screenshot
cloud-init logs
serial console if enabled
/var/log/messages or journal
systemd failed units

sudo cloud-init status --long
sudo journalctl -b -p warning
sudo systemctl --failed
sudo journalctl -u cloud-final --no-pager

stop instance
detach root volume
attach to rescue instance
inspect logs and fstab
fix or snapshot for forensic analysis
replace production node from known good AMI

security investigation required
unknown data corruption
kernel panic evidence needed
rare bug reproducibility
incident root-cause requires local logs
stateful recovery requires volume preservation

remove from traffic
prevent new work
preserve instance/volume/snapshot/logs
block accidental termination
capture evidence
replace capacity separately

Quarantine=true
IncidentId=INC-2026-07-06-001
DoNotTerminateUntil=2026-07-08
Owner=platform
Reason=kernel-panic-investigation

put instance in Standby
remove from load balancer
detach from ASG with or without decrementing desired capacity
increase desired capacity to launch replacement first

[ ] AMI is versioned and reproducible.
[ ] Launch Template version is tracked.
[ ] User data is idempotent.
[ ] Secrets are fetched from managed source, not stored manually.
[ ] Config is environment-scoped and versioned.
[ ] App artifact is immutable.
[ ] Local data is disposable or checkpointed.
[ ] Logs are shipped off-node.
[ ] Metrics/traces are shipped off-node.
[ ] Health checks prove readiness.
[ ] Replacement capacity can launch under quota/capacity constraints.
[ ] Rollback path exists.

Instance launch to ready: <= 5 minutes
ASG replacement after unhealthy: <= 8 minutes
Full AZ capacity restoration: <= 20 minutes

launch time
cloud-init time
artifact download time
dependency initialization time
target group healthy time
warmup time

self-managed database
legacy application with local data
single-writer volume processing
licensed software tied to host/volume
low-level storage appliance

logical owner
current attached instance
AZ
filesystem/application consistency model
snapshot policy
restore policy
failover policy
fencing mechanism
delete-on-termination setting

ASG node has EBS data volume.
No one knows if delete-on-termination is false.
No fencing.
No snapshot restore test.

logical node identity is separate from EC2 instance identity.
Data volume has tags and backup policy.
Old owner is fenced before attach.
New node validates data before serving.

terminate old instance
stop old instance
detach volume only after old instance is unreachable and ownership is clear
application-level lease with timeout
cluster manager fencing mechanism

snapshot first
copy evidence
then attempt repair

new Launch Template version points to bad AMI
ASG instance refresh starts
new instances fail health
old instances are terminated too aggressively
capacity drops

bake AMI with tests
launch canary ASG
validate systemd, mounts, app readiness
use instance refresh with minimum healthy percentage
keep previous Launch Template version
alarm on replacement failures

1. Pause instance refresh if active.
2. Set ASG Launch Template version back to last known good.
3. Ensure desired capacity and min healthy capacity are safe.
4. Terminate bad new instances gradually or start rollback refresh.
5. Preserve one bad node for evidence if needed.
6. Fix AMI pipeline or bootstrap config.
7. Re-run canary before retrying rollout.

wrong S3 path
missing IAM permission
region-specific package mirror
secret name mismatch
mount target DNS failure

fix Launch Template user data/config
create new version
replace failed nodes
avoid manually editing user data on instances

singleton-like instances
stateful instances where preserving instance identity matters
legacy workloads not yet rebuildable

Prefer replacement for fleets.
Use automatic recovery deliberately for instances whose identity/state must survive.

Goal: restore capacity, not save the node.

1. Confirm fleet has enough healthy capacity.
2. Remove instance from traffic or let target group mark unhealthy.
3. Check if issue is isolated or fleet-wide.
4. If isolated, terminate/replace instance.
5. If fleet-wide, stop rollout and inspect recent AMI/config/deploy.
6. Preserve one node only if root cause evidence is needed.
7. Verify ASG launches replacement and target becomes healthy.
8. Document cause and add detection/prevention.

1. Check whether this is isolated or many nodes.
2. Retrieve system log/console evidence if needed.
3. For stateless node: replace.
4. For singleton: reboot or recover based on policy.
5. For stateful node: snapshot/preserve volumes before repair.
6. Verify replacement or recovery through readiness check.

1. Treat as infrastructure/host-level impairment.
2. For fleet node: replace.
3. For singleton: use recover/stop-start if supported and safe.
4. Preserve EBS-backed data as required.
5. Check whether AZ or instance family pattern exists.

1. Stop rollout/instance refresh.
2. Roll ASG back to previous Launch Template version.
3. Launch one test instance with bad version in isolated subnet if needed.
4. Inspect cloud-init, systemd, fstab, app logs.
5. Fix image/config pipeline.
6. Re-run canary.

1. Confirm volume and target instance are in same AZ.
2. Confirm volume state is available/in-use as expected.
3. Confirm old owner is stopped/terminated/fenced.
4. Snapshot before repair if data integrity is uncertain.
5. Attach using known device mapping.
6. Mount by UUID/label.
7. Run application-level consistency check.
8. Only then serve traffic.

1. Remove from traffic if service is degraded.
2. Identify top disk consumers.
3. Check deleted-but-open files.
4. Free minimal space for safe operation.
5. Replace node if stateless.
6. Fix logging/temp/cache policy in AMI/config.
7. Add disk/inode alarms if missing.

any instance can be terminated at any time

external sessions/state
idempotent startup
readiness health
logs/metrics off-node
capacity headroom

job can be retried safely

visibility timeout > max processing window or heartbeat extension
idempotent writes
partial progress checkpoints
termination drain
DLQ/poison handling

only one writer owns the volume

fencing
snapshot
AZ-aware recovery
application consistency check
manual or automated runbook

subnet IP exhaustion
EC2 quota reached
Spot capacity unavailable
single instance type unavailable
Launch Template invalid
IAM instance profile broken
security group rule missing
AMI deleted/deprecated
KMS key denied for encrypted EBS

[ ] ASG spans multiple subnets/AZs where appropriate.
[ ] Mixed instance policy exists for flexible fleets.
[ ] Quotas cover failure replacement and deployments.
[ ] Subnet free IP addresses are monitored.
[ ] AMI/KMS/IAM dependencies are validated.
[ ] On-Demand fallback exists for critical workloads.
[ ] Warm pool or pre-baked AMI exists if startup is slow.

terminate one random instance
terminate 30% of fleet
break one AZ subnet path
roll out bad AMI to canary
fill root disk
detach non-critical data volume
simulate slow boot dependency
revoke temporary IAM permission in test
interrupt Spot instances

service remains within SLO or degrades predictably
replacement launches automatically
bad nodes leave traffic
alerts fire once with useful context
runbook steps are clear
no manual SSH required for stateless fleet
stateful recovery does not corrupt data

MTTD: time to detect
MTTA: time to acknowledge
MTTR: time to restore
launch-to-ready time
unhealthy-to-removed time
removed-to-replaced time
capacity deficit duration
customer-visible error budget impact

resource "aws_launch_template" "api" {
  name_prefix   = "api-"
  image_id      = var.ami_id
  instance_type = var.instance_type

  iam_instance_profile {
    name = aws_iam_instance_profile.api.name
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }

  user_data = base64encode(templatefile("${path.module}/user-data.sh", {
    environment = var.environment
    app_version = var.app_version
  }))

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size           = 30
      volume_type           = "gp3"
      encrypted             = true
      delete_on_termination = true
    }
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Service = "api"
      Managed = "terraform"
    }
  }
}

resource "aws_autoscaling_group" "api" {
  name                = "api-${var.environment}"
  min_size            = 3
  max_size            = 12
  desired_capacity    = 3
  vpc_zone_identifier = var.private_subnet_ids
  health_check_type   = "ELB"
  health_check_grace_period = 120

  launch_template {
    id      = aws_launch_template.api.id
    version = aws_launch_template.api.latest_version
  }

  target_group_arns = [aws_lb_target_group.api.arn]

  tag {
    key                 = "Service"
    value               = "api"
    propagate_at_launch = true
  }
}

mixed instance policy
instance refresh policy
lifecycle hooks
CloudWatch alarms
capacity rebalance for Spot
SSM access
IAM least privilege
KMS key policy
subnet IP monitoring

[ ] Instance can be terminated safely.
[ ] AMI is reproducible.
[ ] Launch Template is versioned.
[ ] User data is idempotent.
[ ] Health checks represent readiness.
[ ] Logs and metrics are off-node.
[ ] Secrets are not stored manually on disk.
[ ] Root volume contains no required state.
[ ] Data volume lifecycle is explicit.
[ ] ASG replacement has been tested.

[ ] Status check alarms exist.
[ ] Target group health alarms exist.
[ ] ASG failed launch alarms exist.
[ ] Instance refresh rollback process exists.
[ ] Quarantine process exists.
[ ] Stateful fencing process exists if needed.
[ ] Snapshot-before-repair rule is documented.
[ ] Subnet IP and EC2 quota headroom are monitored.
[ ] On-call can distinguish repair/replace/quarantine paths.

[ ] cloud-init logs preserved.
[ ] systemd journal accessible.
[ ] application logs shipped.
[ ] kernel/OOM/panic logs captured.
[ ] EBS snapshots tagged with incident ID.
[ ] instance metadata tags include service/version/owner.

new instances launch
app never starts
target group health fails
ASG keeps replacing nodes
capacity drops from 12 healthy to 7 healthy
p99 latency rises

SSH into failed instances.
Try restarting services.
Patch systemd unit manually.
Let ASG continue replacing.

1. Pause instance refresh.
2. Roll Launch Template to previous known-good version.
3. Ensure desired capacity remains safe.
4. Replace failed nodes from known-good AMI.
5. Quarantine one failed node for evidence.
6. Inspect cloud-init/systemd logs offline.
7. Fix AMI pipeline test to catch missing dependency.
8. Re-run canary before broad rollout.

Force detach volume.
Attach to new instance.
Start worker immediately.

old node may still write
filesystem may be inconsistent
partial checkpoint may be corrupt
duplicate output may be produced

1. Stop or terminate old instance if possible.
2. Confirm old writer is fenced.
3. Snapshot the data volume.
4. Attach to recovery instance in same AZ.
5. Mount safely.
6. Run filesystem/application consistency checks.
7. Resume from checkpoint only if valid.
8. Write idempotently to final output store.

Prefer replace over repair for stateless fleets.
Separate repair, replacement, and quarantine paths.
Treat health checks as service contracts.
Use ASG to converge capacity, not to magically understand application state.
Use lifecycle hooks to make launch and termination safe.
Make boot deterministic and bounded.
Version AMI and Launch Template changes.
Fence stateful ownership before moving volumes.
Test rebuildability with game days.