Start HereOrdered learning track

Launch Templates and Instance Fleet Design

Learn AWS Compute and Storage In Action - Part 013

Deep production guide to EC2 Launch Templates, versioning, fleet contracts, Auto Scaling integration, metadata options, block device mappings, tagging, rollout safety, and failure-aware instance fleet design.

13 min read2558 words
PrevNext
Lesson 1380 lesson track01–15 Start Here
#aws#ec2#launch-template#auto-scaling+3 more

Part 013 — Launch Templates and Instance Fleet Design

A production EC2 fleet should not be created from memory, console clicks, or tribal knowledge.

It needs a versioned launch contract.

In AWS, that contract is usually an EC2 Launch Template.

Bad mental model:

A launch template is a convenient form that stores EC2 launch options.

Better mental model:

A launch template is a versioned machine creation contract used by fleets, Auto Scaling Groups, EC2 Fleet, Spot Fleet, and deployment automation.

A single EC2 instance is easy to launch.

A safe fleet is harder because every instance must be created with the same rules:

same AMI lineage
same bootstrap contract
same IAM boundary
same metadata posture
same storage mapping
same tagging contract
same security group shape
same observability baseline
same termination semantics
same rollback path

This part teaches Launch Templates as the boundary between image construction and fleet operation.

Part 012 covered:

How do we build the machine artifact?

This part covers:

How do we repeatedly launch that artifact as a safe production fleet?

1. Problem yang Diselesaikan

Without a Launch Template discipline, EC2 fleets drift in invisible ways.

Common failure patterns:

FailureProduction Symptom
Console-launched instancesno reproducible launch contract
Mutable $Latest usageunexpected rollout after template edit
Unpinned AMInew node boots unknown image
Missing IMDSv2 enforcementmetadata exposure risk
Inconsistent tagscost, ownership, backup, and incident routing fail
Inconsistent block device mappingsome nodes lose data, some leak volumes
Incorrect delete-on-terminationorphaned EBS volumes create cost and compliance issues
IAM instance profile mismatchbootstrap succeeds in staging, fails in production
Network settings embedded casuallysubnet/AZ placement becomes accidental
User data too large or mutableboot behavior cannot be reasoned about
No version promotion pathrollback requires manual console edits
No instance refresh disciplineASG contains old and new node shapes indefinitely

The production question is not:

Can we launch an EC2 instance?

The production question is:

Can we launch, replace, scale, roll back, audit, and debug every instance in this fleet using an explicit immutable contract?

2. Core Mental Model

A Launch Template is not the fleet itself.

It is the definition that a fleet uses to create instances.

There are three different objects you must not confuse:

ObjectMeaning
AMIbase machine artifact
Launch Template Versionmachine launch contract
Auto Scaling Group / EC2 Fleetfleet control loop that uses the contract

The AMI answers:

What is inside the machine?

The Launch Template answers:

How should the machine be launched?

The fleet controller answers:

How many machines should exist, where, and when should they be replaced?

3. Why Launch Templates Matter More Than They Look

A Launch Template can store parameters such as:

  • AMI ID
  • instance type
  • key pair
  • security groups
  • network interfaces
  • IAM instance profile
  • block device mapping
  • EBS optimization
  • monitoring
  • shutdown behavior
  • instance market options
  • placement configuration
  • user data
  • tag specifications
  • metadata options
  • capacity reservation preference
  • CPU options
  • enclave options
  • hibernation options

That looks like a configuration list.

In production it is an invariant list.

Example:

MetadataOptions.HttpTokens = required

This is not merely an option. It is a fleet security invariant.

Example:

BlockDeviceMappings.Ebs.DeleteOnTermination = true

This is not merely a storage option. It is a lifecycle cost invariant.

Example:

TagSpecifications includes owner, service, environment, data-classification

This is not merely metadata. It is the basis for cost allocation, automation, backup policy, incident routing, and compliance evidence.


4. Launch Template as a Contract

A good Launch Template defines a contract between platform and workload.

Given this template version,
AWS can create an instance that belongs to this workload,
in this environment,
with this image,
with this identity,
with this storage layout,
with this metadata posture,
and with tags that make it governable.

4.1 Contract Dimensions

DimensionContract Question
ImageWhich AMI lineage and version boots?
RuntimeWhat user data/bootstrap behavior is allowed?
IdentityWhich IAM instance profile does the node receive?
NetworkWhich subnet/security group/ENI pattern is used?
StorageWhich root/data volumes exist and how are they destroyed?
MetadataIs IMDSv2 required? Are instance tags exposed?
ObservabilityAre detailed monitoring and agents expected?
TagsCan automation identify service/env/owner/data class?
LifecycleWhat happens on shutdown and termination?
CapacityIs On-Demand, Spot, or mixed capacity expected?

5. Versioning Model

Launch Templates are versioned.

This is powerful and dangerous.

The trap:

Using $Latest or mutable default version casually.

The safe principle:

Production fleets should generally consume an explicitly selected template version, not an accidental moving pointer.

5.1 Version Types

ReferenceMeaningProduction Risk
explicit version numberexact launch contractsafest for rollout/rollback
$Defaultpointer to chosen defaultacceptable only with disciplined promotion
$Latestnewest created versiondangerous for production ASG unless controlled

5.2 Release Manifest Pattern

Treat the fleet release as a manifest:

service: payment-api
environment: production
ami_id: ami-0abc123...
launch_template_id: lt-012345...
launch_template_version: 42
asg_name: payment-api-prod-a
release_id: payment-api-2026-07-06-001

This makes rollback explicit:

rollback = set ASG to previous launch_template_version and run controlled instance refresh

Not:

open console, guess old AMI, edit random settings

6. What Belongs in a Launch Template

6.1 AMI ID

The AMI ID defines the base artifact.

Good:

AMI is produced by image pipeline.
AMI is tagged with source commit/build/kernel/package baseline.
AMI is promoted from dev -> staging -> prod.
Launch Template version references a concrete AMI ID.

Bad:

AMI copied from an old instance.
AMI name pattern is used loosely.
Launch Template uses whichever AMI looks newest.

6.2 Instance Type

For simple workloads, the template may specify one instance type.

For resilient fleets, the Launch Template can be combined with ASG mixed instance policy.

Decision:

Stable capacity shape? specify instance type directly.
Need Spot/capacity diversification? use Launch Template + mixed instance overrides.

6.3 IAM Instance Profile

The instance profile is the machine identity.

It should contain only the permissions needed for:

  • bootstrap
  • configuration retrieval
  • secrets retrieval
  • log/metric publishing
  • SSM access
  • application AWS calls
  • storage access if the app uses S3/EFS/FSx/EBS APIs

Anti-pattern:

One giant EC2 role for all instances.

Better:

One role per workload boundary.

Even better:

Separate platform bootstrap permissions from application permissions where possible.

6.4 Metadata Options

Metadata options should be treated as security and operational invariants.

Recommended baseline for most production fleets:

HttpTokens = required
HttpEndpoint = enabled
HttpPutResponseHopLimit = 1 or carefully justified higher value
InstanceMetadataTags = disabled unless explicitly needed

Why this matters:

  • user data may use metadata to discover instance identity
  • agents may use metadata for region/instance details
  • overly permissive metadata access can increase blast radius
  • containerized workloads may require careful hop-limit reasoning

6.5 User Data

User data should be:

small
idempotent
observable
bounded in time
safe to retry
free of long-lived secrets
used to start controlled bootstrap, not contain the whole platform

Recommended pattern:

#!/bin/bash
set -euo pipefail

exec > >(tee -a /var/log/bootstrap.log | logger -t user-data -s 2>/dev/console) 2>&1

echo "bootstrap started at $(date -Is)"
/opt/platform/bin/bootstrap-node --environment prod --service payment-api
echo "bootstrap finished at $(date -Is)"

Do not put a 900-line deployment script in the template.

Use user data as the entry point into a versioned bootstrap tool.

6.6 Block Device Mapping

Block device mapping defines root and additional volumes at launch.

It answers:

What disks exist when the instance starts?
What type are they?
Are they encrypted?
Are they deleted on termination?
What size and performance do they have?

Example decisions:

VolumeSetting
rootsmall, encrypted, delete-on-termination true
app log volumeoften avoid; ship logs instead
data volumeonly if workload intentionally uses local block state
cache volumedelete-on-termination true
database volumeexplicit backup/snapshot lifecycle required

6.7 Tag Specifications

Tags must be applied to all relevant resource types:

  • instance
  • volume
  • network interface
  • spot instance request if applicable

Minimum useful tags:

Service: payment-api
Environment: prod
Owner: payments-platform
CostCenter: fintech-core
DataClassification: internal
ManagedBy: terraform
Runbook: https://internal/runbooks/payment-api
BackupPolicy: none-or-policy-name

Bad pattern:

Only tagging the instance.

Because orphaned volumes and ENIs need tags too.

6.8 Security Groups and Network Interfaces

For simple ASG fleets, specify security groups and let ASG place instances into subnets.

For advanced cases, network interface configuration may be explicit.

Be careful:

If you over-specify network interfaces, you may accidentally constrain subnet/AZ placement.

Network belongs partly to template and partly to fleet placement.

The Launch Template should define:

security posture
metadata posture
ENI behavior if required

The ASG should define:

which subnets/AZs the fleet spans

7. What Should Not Belong in a Launch Template

Do not put these in a Launch Template unless there is a strong reason:

ItemWhy
long-lived secretstemplate is not a secret store
environment-specific huge scriptshard to review, diff, test, rollback
per-instance unique configtemplate should be fleet-level contract
manual SSH keys for productionprefer SSM/session workflows where possible
unbounded package installationboot now depends on external package repos
data migration commandsevery new instance might repeat migration
hard-coded subnet for ASG fleetmay collapse AZ diversity
hard-coded private IP for scalable fleetprevents elastic replacement

8. Fleet Design: Template Is Not Enough

A Launch Template becomes useful when attached to a fleet controller.

The Launch Template defines node shape.

The Auto Scaling Group defines fleet behavior:

  • desired capacity
  • min/max capacity
  • subnet/AZ placement
  • health check type
  • replacement behavior
  • mixed instance strategy
  • lifecycle hooks
  • instance refresh
  • termination policy
  • warm pool

Production design requires both.


9. Fleet Contract Layers

Think of fleet configuration as layered contracts.

Image Contract
  AMI, OS, packages, runtime

Launch Contract
  launch template version, metadata, identity, storage, tags

Placement Contract
  ASG subnets, AZ spread, placement group if needed

Capacity Contract
  min, max, desired, scaling policy, mixed instance policy

Traffic Contract
  target group registration, readiness, deregistration delay

Termination Contract
  lifecycle hook, drain, log flush, volume handling

A production incident often happens because one layer is correct and another is implicit.

Example:

AMI is good.
Launch Template is good.
ASG uses only one subnet.
AZ outage takes down the fleet.

Example:

ASG spans three AZs.
Launch Template uses EBS data volume pattern that assumes fixed AZ-local state.
Replacement cannot recover state in another AZ.

10. Launch Template Design Patterns

10.1 One Template per Workload per Environment

payment-api-prod-lt
payment-api-staging-lt
ledger-worker-prod-lt
ledger-worker-staging-lt

Good when:

  • different environments need different roles/tags/settings
  • promotion is controlled
  • compliance requires separation

Trade-off:

  • more templates
  • more lifecycle management

10.2 One Template per Workload, Environment via Version/Variables

Good when:

  • platform is simple
  • environments are very similar
  • automation is mature

Risk:

  • environment-specific drift hides in variables
  • wrong version may be promoted to wrong environment

10.3 Base Template + ASG Overrides

Good for mixed instance fleets.

Launch Template defines common contract.
ASG mixed instance policy overrides instance types.

Example:

Base: m7i.large
Overrides: m7i.large, m7a.large, m6i.large, m6a.large

Use this when capacity availability matters more than a single exact hardware family.

10.4 Immutable Template Version per Release

Recommended for serious production.

Every release creates a new Launch Template version.
ASG points to explicit version.
Instance refresh rolls it out.
Rollback points to previous version.

This makes infrastructure deploys auditable.

10.5 Shared Platform Template with Workload Bootstrap

Sometimes a platform team provides a common hardened template and workloads inject only a bootstrap argument.

This works if the boundary is clear:

platform owns OS, agents, metadata, base IAM, storage defaults
workload owns app artifact and runtime config

Risk:

platform template becomes too generic and over-permissive

11. Launch Template Version Promotion Flow

A safe promotion flow:

Minimum checks before production:

AMI boots
cloud-init/user-data succeeds
SSM available
metrics/logging available
app reaches readiness
EBS mounts are correct
metadata posture is correct
tags are present on instance and volumes
health check passes after warmup
termination behavior is safe

12. Implementation: Terraform Launch Template

Example skeleton:

resource "aws_launch_template" "payment_api" {
  name_prefix   = "payment-api-prod-"
  image_id      = var.ami_id
  instance_type = "m7i.large"

  iam_instance_profile {
    name = aws_iam_instance_profile.payment_api.name
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
    instance_metadata_tags      = "disabled"
  }

  monitoring {
    enabled = true
  }

  user_data = base64encode(templatefile("${path.module}/user-data.sh", {
    service     = "payment-api"
    environment = "prod"
  }))

  block_device_mappings {
    device_name = "/dev/xvda"

    ebs {
      volume_size           = 30
      volume_type           = "gp3"
      encrypted             = true
      delete_on_termination = true
    }
  }

  tag_specifications {
    resource_type = "instance"

    tags = {
      Service            = "payment-api"
      Environment        = "prod"
      Owner              = "payments-platform"
      ManagedBy          = "terraform"
      DataClassification = "internal"
    }
  }

  tag_specifications {
    resource_type = "volume"

    tags = {
      Service            = "payment-api"
      Environment        = "prod"
      Owner              = "payments-platform"
      ManagedBy          = "terraform"
      DataClassification = "internal"
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

Key points:

AMI is passed as an explicit variable from image pipeline.
IMDSv2 is required.
Root volume is encrypted and deleted on termination.
Tags are applied to both instance and volume.
User data is rendered from a small template.

13. Implementation: ASG Consuming Explicit Version

resource "aws_autoscaling_group" "payment_api" {
  name                = "payment-api-prod"
  min_size            = 6
  max_size            = 30
  desired_capacity    = 9
  vpc_zone_identifier = var.private_subnet_ids

  health_check_type         = "ELB"
  health_check_grace_period = 300

  launch_template {
    id      = aws_launch_template.payment_api.id
    version = aws_launch_template.payment_api.latest_version
  }

  target_group_arns = [aws_lb_target_group.payment_api.arn]

  instance_refresh {
    strategy = "Rolling"

    preferences {
      min_healthy_percentage = 90
      instance_warmup        = 300
    }
  }

  tag {
    key                 = "Service"
    value               = "payment-api"
    propagate_at_launch = true
  }

  tag {
    key                 = "Environment"
    value               = "prod"
    propagate_at_launch = true
  }
}

Be careful with this line:

version = aws_launch_template.payment_api.latest_version

In simple Terraform workflows this may be fine because Terraform controls the version.

In stricter production workflows, prefer passing a promoted explicit version:

version = var.launch_template_version

That prevents accidental rollout when a template version is created but not approved.


14. Implementation: Mixed Instance Policy

Mixed instance policy separates:

common launch contract
from
capacity diversification

Example:

resource "aws_autoscaling_group" "worker" {
  name                = "ledger-worker-prod"
  min_size            = 20
  max_size            = 200
  desired_capacity    = 40
  vpc_zone_identifier = var.private_subnet_ids

  mixed_instances_policy {
    instances_distribution {
      on_demand_base_capacity                  = 10
      on_demand_percentage_above_base_capacity = 30
      spot_allocation_strategy                 = "price-capacity-optimized"
    }

    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.worker.id
        version            = var.launch_template_version
      }

      override {
        instance_type = "m7i.large"
      }

      override {
        instance_type = "m7a.large"
      }

      override {
        instance_type = "m6i.large"
      }

      override {
        instance_type = "m6a.large"
      }
    }
  }
}

Use this for:

  • queue workers
  • stateless APIs with enough horizontal capacity
  • batch processors
  • event consumers
  • workloads tolerant of instance replacement

Be cautious for:

  • strict performance workloads
  • licensed software pinned to CPU type
  • workloads sensitive to CPU architecture
  • JVM workloads not tested across architectures/families

15. Metadata Options: Production Baseline

A safe default:

metadata_options {
  http_endpoint               = "enabled"
  http_tokens                 = "required"
  http_put_response_hop_limit = 1
  instance_metadata_tags      = "disabled"
}

15.1 HttpTokens

required = IMDSv2 token required
optional = IMDSv1 still allowed

Use required unless you have a specific legacy dependency.

15.2 HttpPutResponseHopLimit

This controls metadata response hop limit.

Common reasoning:

WorkloadTypical Setting
app directly on EC2 host1
containers needing host metadatamay require >1, justify carefully
hardened platform1 with explicit metadata proxy/agent design

Do not set it higher just because something failed.

Find which component needs metadata and why.

15.3 Instance Metadata Tags

Exposing tags through metadata can simplify bootstrap.

But it also means any process able to access metadata can read tags.

Use only if:

tags do not contain sensitive information
bootstrap truly needs them
access boundary is understood

16. Storage Contract in Launch Template

Root volume baseline:

block_device_mappings {
  device_name = "/dev/xvda"

  ebs {
    volume_size           = 30
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }
}

Questions to ask:

Does this instance need only root volume?
Does app write data locally?
Is local data recoverable?
Should the data volume survive termination?
Who snapshots it?
What is the filesystem mount path?
How is device naming handled on Nitro/NVMe?

16.1 Stateless API

Usually:

root volume only
logs shipped out
delete root volume on termination
no data volume

16.2 Worker with Scratch Disk

Possible:

root volume + scratch EBS or instance store
scratch is disposable
delete-on-termination true
job retry handles loss

16.3 Stateful EC2 Workload

Requires strict design:

root volume separate from data volume
data volume lifecycle explicit
snapshot policy explicit
restore path tested
AZ locality understood
termination protection considered

17. User Data Contract

A Launch Template should not hide complex behavior in user data.

Use this pattern:

Launch Template user data starts bootstrap.
Bootstrap tool is versioned and tested.
Bootstrap emits status.
Systemd owns long-running process.
Health check waits for readiness.

Example user data:

#!/bin/bash
set -euo pipefail

LOG=/var/log/platform-bootstrap.log
exec > >(tee -a "$LOG" | logger -t platform-bootstrap -s 2>/dev/console) 2>&1

echo "starting bootstrap"

/opt/platform/bin/bootstrap \
  --service payment-api \
  --environment prod \
  --region "$(curl -sS -H "X-aws-ec2-metadata-token: $(curl -sS -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600')" http://169.254.169.254/latest/meta-data/placement/region)"

systemctl enable payment-api
systemctl start payment-api

echo "bootstrap complete"

In real systems, avoid duplicating metadata token logic everywhere.

Prefer a small, well-tested helper.


18. Tagging Contract

Tags are not decoration.

They are input to production systems.

Minimum contract:

TagWhy
Serviceworkload ownership
Environmentblast-radius and policy separation
Ownerescalation path
ManagedByconsole/manual drift detection
DataClassificationpolicy and audit
CostCenterfinancial ownership
BackupPolicybackup automation
Runbookincident response

Important:

Tag the instance, volume, and network interface where possible.

19. Security Group Placement Pattern

Launch Template can reference security groups.

But subnet placement usually belongs to ASG.

Good separation:

Launch Template:
  security groups
  metadata options
  IAM profile
  storage mapping

ASG:
  subnet IDs
  AZ spread
  desired/min/max
  health checks

Why?

Because the same node shape can be placed across multiple private subnets/AZs.

Over-specifying a network interface in the Launch Template can make the fleet less elastic.


20. Shutdown and Termination Behavior

Instances can stop, terminate, reboot, or be replaced.

Launch Template and ASG behavior must align.

Important questions:

If the OS shuts down, should the instance stop or terminate?
Should root volume be deleted?
Should data volume survive?
Should lifecycle hook drain traffic first?
Should app flush buffers?
Should queue worker release leases?

For stateless fleets:

termination is normal
root volume delete-on-termination true
node can be recreated from template

For stateful fleets:

termination is data-sensitive
volume lifecycle must be explicit
snapshot/restore must be tested

21. Launch Template and Instance Refresh

Changing a Launch Template version does not magically replace every running instance unless your fleet controller performs replacement.

For ASG:

New launches use the configured template version.
Existing instances remain until replaced or refreshed.

Use Instance Refresh to roll the version across the fleet.

Safe rollout pattern:

create new LT version
point ASG to new version
start instance refresh
respect min healthy percentage
wait for warmup
watch alarms
rollback if unhealthy

21.1 Canary First

For critical fleets:

1. launch one instance from new template
2. validate bootstrap
3. route small traffic or run synthetic check
4. update ASG
5. instance refresh gradually

21.2 Rollback

Rollback means:

set ASG back to known-good launch template version
start new instance refresh

Not:

SSH into broken nodes and patch them manually

22. Fleet Homogeneity vs Heterogeneity

Production fleets need a controlled level of sameness.

Too homogeneous:

one instance type, one AZ, one capacity pool

Risk:

capacity unavailable, hardware-specific failure, AZ blast radius

Too heterogeneous:

many instance families, architectures, AMI variants, disk layouts

Risk:

unpredictable performance, hard debugging, hidden compatibility bugs

Good fleet design:

same launch contract
small set of compatible instance types
explicit architecture boundary
measured performance envelope
clear rollback path

23. Capacity-Aware Template Design

A template can accidentally make capacity harder.

Examples:

Template ChoiceCapacity Effect
one rare instance typescale-out fails during regional pressure
huge EBS throughput requirementinstance choices limited
GPU instance hardcodedexpensive and scarce capacity
single architecture without fallbackless diversification
placement group requirementstronger placement constraints
specific AZ-local dependencyless mobility

Ask:

If this instance type is unavailable, what compatible alternatives exist?
If this AZ has pressure, can the fleet launch elsewhere?
If Spot is interrupted, can On-Demand absorb baseline?
If AMI is architecture-specific, did we separate arm64 and x86 templates?

24. Multi-Architecture Fleet Strategy

Do not casually mix x86_64 and arm64 in the same Launch Template unless your artifact pipeline supports it.

Better:

payment-api-prod-x86-lt
payment-api-prod-arm64-lt

or explicit manifest:

architectures:
  x86_64:
    ami: ami-x86...
    instance_types:
      - m7i.large
      - m6i.large
  arm64:
    ami: ami-arm...
    instance_types:
      - m7g.large
      - m6g.large

For Java workloads, test:

  • JDK version
  • native libraries
  • container images if used
  • JNI dependencies
  • crypto provider behavior
  • memory profile
  • GC behavior
  • performance per core

25. Production Checklist for Launch Templates

Before using a Launch Template in production:

[ ] AMI ID is explicit and produced by image pipeline
[ ] template version is explicit and auditable
[ ] user data is small and idempotent
[ ] IAM instance profile is least-privilege per workload
[ ] IMDSv2 is required
[ ] metadata hop limit is justified
[ ] root volume is encrypted
[ ] delete-on-termination is intentionally set
[ ] data volumes have explicit lifecycle policy
[ ] tags apply to instances and volumes
[ ] detailed monitoring decision is explicit
[ ] security groups are correct for workload boundary
[ ] subnet/AZ placement is controlled by ASG/fleet
[ ] SSM or break-glass access is available if needed
[ ] bootstrap logs are available
[ ] health check grace period matches startup time
[ ] rollback launch template version is known
[ ] instance refresh strategy is defined
[ ] capacity fallback is defined
[ ] quotas are checked for expected scale

26. Common Failure Modes

26.1 ASG Uses Unexpected Template Version

Symptom:

new instances behave differently than expected

Cause:

ASG uses $Latest or default pointer changed without rollout discipline

Fix:

pin explicit version; use release manifest; require approval for promotion

26.2 Broken AMI in New Template Version

Symptom:

instance refresh replaces healthy nodes with unhealthy nodes

Cause:

Launch Template version points to AMI not validated in production-like environment

Fix:

preflight launch, canary, staged refresh, alarm rollback

26.3 User Data Fails Silently

Symptom:

instance is running but app never becomes ready

Cause:

user data not logged, script not idempotent, cloud-init failure ignored

Fix:

log to /var/log/bootstrap.log and console; fail explicitly; expose readiness

26.4 IAM Profile Too Weak

Symptom:

bootstrap cannot fetch config/secrets or app cannot access required storage

Cause:

wrong instance profile attached in Launch Template

Fix:

role per workload; integration test bootstrap with production-like role

26.5 IAM Profile Too Broad

Symptom:

incident blast radius larger than expected

Cause:

shared admin-like EC2 role

Fix:

least privilege; split roles; monitor credential usage

26.6 Orphaned Volumes

Symptom:

EBS cost rises after instance replacement

Cause:

delete_on_termination false accidentally or volume not tagged

Fix:

explicit delete-on-termination; tag volumes; orphan cleanup report

26.7 Metadata Access Breaks After Hardening

Symptom:

app/agent cannot discover region, credentials, or instance identity

Cause:

IMDSv2 enforced but legacy script still uses IMDSv1

Fix:

update bootstrap helpers and agents; test IMDSv2 path before enforcing

26.8 ASG Cannot Launch Capacity

Symptom:

scale-out requested but instances do not launch

Possible causes:

instance type unavailable
subnet lacks IPs
quota exceeded
invalid AMI
bad launch template parameter
capacity reservation mismatch
security group or IAM issue

Fix:

inspect ASG activity history; diversify instance types; monitor subnet IPs and quotas

27. Debugging Runbook

When a new EC2 fleet deployment fails:

Step 1 — Identify Template Version

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names payment-api-prod \
  --query 'AutoScalingGroups[0].LaunchTemplate'

Confirm:

LaunchTemplateId
Version

Step 2 — Inspect Launch Template Data

aws ec2 describe-launch-template-versions \
  --launch-template-id lt-0123456789abcdef0 \
  --versions 42

Check:

ImageId
InstanceType
IamInstanceProfile
MetadataOptions
BlockDeviceMappings
SecurityGroupIds
UserData

Step 3 — Inspect ASG Activity

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name payment-api-prod \
  --max-items 20

Look for:

failed launch
capacity errors
invalid AMI
security group issues
instance profile issues
subnet IP exhaustion

Step 4 — Inspect Instance Console Output

aws ec2 get-console-output \
  --instance-id i-0123456789abcdef0 \
  --latest

Useful for:

  • cloud-init failure
  • kernel panic
  • disk mount failure
  • early bootstrap failure

Step 5 — Inspect Bootstrap Logs

If SSM works:

sudo journalctl -u cloud-init -n 200 --no-pager
sudo cat /var/log/cloud-init-output.log
sudo cat /var/log/platform-bootstrap.log
sudo systemctl status payment-api

Step 6 — Roll Back if Fleet Health Is Degrading

point ASG to previous known-good launch template version
start instance refresh
monitor target group health and error rate

Do not debug forever while the rollout is actively damaging production.


28. Anti-Patterns

28.1 The Console Snowflake

Launch some instances by hand.
Change settings until it works.
Create AMI from it.
Use it in prod.

Result:

No reproducibility. No provenance. No safe rollback.

28.2 The Giant User Data Script

User data installs everything, fetches app, migrates DB, edits config, starts service.

Result:

Boot is slow, fragile, hard to observe, hard to test.

28.3 The Moving $Latest Fleet

ASG always uses the latest template version.

Result:

Creating a template version becomes an implicit production rollout.

28.4 Untagged Volumes

Only instances are tagged.

Result:

Detached volumes become invisible cost and compliance risk.

28.5 One Template for Everything

All EC2 workloads use one generic template and a huge role.

Result:

Poor least privilege, hard audit, hidden coupling, broad blast radius.

29. Mini Case Study: Payment API Fleet Rollout

A payment API runs on EC2 behind an ALB.

Old setup:

ASG uses launch configuration
AMI changed manually
user data installs app from S3
root volumes not consistently encrypted
tags only on instances
rollback means editing ASG manually

Incident:

new AMI has bad JVM flag
instance refresh replaces 50% fleet
latency spikes
rollback takes 45 minutes because old version is unclear

New design:

Image pipeline builds AMI.
Launch Template version created per release.
ASG pins explicit template version.
Instance Refresh uses min healthy 95%.
Synthetic canary validates one node first.
Root volume encrypted/delete-on-termination.
Tags applied to instance and volume.
IMDSv2 required.
Rollback version stored in release manifest.

New failure behavior:

bad JVM flag found during canary
ASG not refreshed
rollback is not needed because release never reached fleet

The improvement is not a feature.

It is a contract.


30. Summary

A Launch Template is the production launch contract for EC2 machines.

It must define:

image
identity
metadata posture
storage mapping
bootstrap entrypoint
tagging contract
network/security posture
lifecycle defaults

A safe fleet design separates:

AMI = what is inside the machine
Launch Template Version = how the machine is launched
ASG/Fleet = how machines are placed, scaled, refreshed, and terminated

The most important production rules:

Use explicit versioning.
Avoid accidental $Latest rollouts.
Keep user data small.
Require IMDSv2.
Tag instances and volumes.
Make storage lifecycle explicit.
Use canary and instance refresh.
Keep rollback version known.

When Launch Templates are treated as contracts, EC2 becomes manageable at fleet scale.

When they are treated as convenience forms, fleets slowly become snowflakes.


References

Lesson Recap

You just completed lesson 13 in start here. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.