Deepen PracticeOrdered learning track

CloudWatch Metrics and Alarms

Learn AWS Security, Monitoring and Management - Part 056

Deep dive into Amazon CloudWatch Metrics and Alarms for production systems: namespaces, dimensions, statistics, metric math, alarm evaluation, composite alarms, anomaly detection, and alert design.

25 min read4942 words
PrevNext
Lesson 5672 lesson track40–59 Deepen Practice
#aws#cloudwatch#metrics#alarms+3 more

Part 056 — CloudWatch Metrics and Alarms

CloudWatch Metrics and Alarms are often treated as simple features:

Put numbers in CloudWatch. Add alarm. Done.

That is how noisy systems are born.

In production, metrics and alarms are part of the operational decision engine.

Metrics define what the system says about itself.

Alarms define when that statement becomes action.

If metric identity is wrong, alarms lie.

If dimensions are wrong, dashboards hide impact.

If thresholds are wrong, on-call becomes noise.

If missing data is wrong, outages disappear.

If alarm ownership is wrong, nobody acts.

This part builds a deep, practical model for CloudWatch Metrics and Alarms.


1. The Core Question

The real question is not:

How do I create a CloudWatch alarm?

The real question is:

How do I encode production health as reliable time-series signals and convert those signals into the right operational action?

That requires understanding:

  • metric identity,
  • namespace strategy,
  • dimension strategy,
  • statistic semantics,
  • period and resolution,
  • missing data behavior,
  • alarm evaluation,
  • metric math,
  • anomaly detection,
  • composite alarms,
  • cross-account visibility,
  • and alert governance.

2. CloudWatch Metric Mental Model

A CloudWatch metric is not just a name.

It is a time series identified by:

Namespace + MetricName + DimensionSet

Example:

Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
  FunctionName = payment-authorizer-prod

Another dimension set is a different metric.

That detail matters.

These are not the same metric:

AWS/Lambda Errors FunctionName=payment-authorizer-prod
AWS/Lambda Errors FunctionName=receipt-generator-prod

Nor are these the same metric:

Custom/Checkout Latency Operation=AuthorizePayment Region=ap-southeast-1
Custom/Checkout Latency Operation=CreateReceipt Region=ap-southeast-1

Metric identity is the first design decision.


3. Namespaces

A namespace is the top-level container for metrics.

AWS service metrics use namespaces such as:

  • AWS/EC2,
  • AWS/Lambda,
  • AWS/ApplicationELB,
  • AWS/RDS,
  • AWS/DynamoDB,
  • AWS/SQS,
  • AWS/ApiGateway,
  • AWS/ECS,
  • AWS/EKS,
  • AWS/States.

Custom metrics should use namespaces that communicate ownership and domain.

Examples:

Company/Checkout
Company/Payments
Company/CaseManagement
Company/Platform
Company/SecurityAutomation
Company/DataPipeline

Bad custom namespace:

CustomMetrics
Application
Prod
Monitoring

Those names do not communicate ownership, domain, or purpose.

Namespace Design Guidelines

GuidelineReason
Use domain-oriented namesEasier ownership and discovery
Avoid environment-only namespacesEnvironment should usually be a dimension
Avoid team names that change oftenOrganizational changes should not break telemetry semantics
Do not mix unrelated services in one namespacePrevent accidental aggregation
Keep naming stableMetric identity changes break dashboards and alarms

4. Metric Names

Metric names should describe the measured quantity.

Good examples:

RequestCount
RequestLatency
AuthorizationFailureCount
PaymentProviderTimeoutCount
OrderPersistFailureCount
QueueProcessingDelay
BusinessTransactionSuccessCount
BusinessTransactionFailureCount
DependencyCallDuration

Bad examples:

Error
BadThing
Metric1
LatencyMsProd
CheckoutStats

A metric name should not encode every dimension.

Bad:

CheckoutAuthorizePaymentProviderXProdLatency

Better:

MetricName: DependencyCallDuration
Dimensions:
  Service = checkout
  Operation = AuthorizePayment
  Dependency = provider-x
  Environment = prod

5. Dimensions

Dimensions are name/value pairs that identify a metric.

They are powerful and dangerous.

Useful dimensions let operators slice the system by decision boundary.

Dangerous dimensions create uncontrolled cardinality.

5.1 Good Dimension Candidates

DimensionUse
Serviceowner and routing
Environmentprod/staging/dev separation
Regionregional diagnosis
Operationbusiness/API operation
Dependencydownstream diagnosis
ErrorClassfailure classification
TenantTierimpact prioritization without per-user cardinality
DeploymentVersionregression diagnosis; use carefully
QueueNameasync backlog diagnosis
ConsumerGroupconsumer ownership

5.2 Dangerous Dimension Candidates

DimensionProblem
RequestIdnearly infinite cardinality
UserIdhigh cardinality and privacy risk
EmailPII leakage
FullUrlhigh cardinality and secret leakage risk
ExceptionMessageuncontrolled cardinality
StackTracemassive cardinality and cost
RawTenantIdmay be too high-cardinality unless carefully governed

Metrics should represent aggregatable behavior.

Logs and traces should hold high-cardinality detail.

5.3 Dimension Cardinality Example

Assume these dimensions:

Service: 20 values
Operation: 30 values
Region: 4 values
ErrorClass: 10 values
TenantTier: 4 values

Potential time series:

20 * 30 * 4 * 10 * 4 = 96,000

That is before adding deployment version, dependency, or status code.

A single “simple” metric can become expensive and hard to query.

Design dimensions intentionally.


6. Datapoints, Periods, and Resolution

A metric is made of datapoints over time.

Each datapoint has:

  • timestamp,
  • value or statistic set,
  • unit,
  • metric identity.

The period is the aggregation interval used for graphing and alarm evaluation.

Examples:

  • 1 minute,
  • 5 minutes,
  • 15 minutes,
  • 1 hour.

Period Trade-Off

Short PeriodLong Period
faster detectionsmoother signal
more noiseslower detection
useful for incidentsuseful for trend/capacity
higher risk of false positiveslower risk of alert noise

For paging alarms, do not choose the shortest period blindly.

Choose the shortest period that is reliable enough to act on.


7. Standard Resolution vs High Resolution

CloudWatch supports standard-resolution and high-resolution custom metrics.

A normal operational pattern:

Use CaseResolution
business SLO over minutesstandard resolution usually enough
Lambda/API error rate1-minute alarms often enough
very fast autoscaling signalhigh resolution may help
low-latency trading or realtime systemshigh resolution may help
dashboard trendstandard resolution

High resolution is not automatically better.

It can increase cost and noise.

Use it when the response loop really needs it.


8. Statistics

CloudWatch can calculate statistics over metric datapoints.

Common statistics:

StatisticMeaningGood For
Averagemean valuestable utilization trends
Sumtotal over periodrequest count, error count
Minimumlowest datapointavailability of floor behavior
Maximumhighest datapointsaturation spikes
SampleCountnumber of samplestraffic volume/context
percentilesdistribution thresholdlatency/user experience

Average Can Lie

Suppose 100 requests:

  • 99 requests finish in 100 ms,
  • 1 request finishes in 10 seconds.

Average latency:

(99 * 100 + 1 * 10000) / 100 = 199 ms

Average looks fine.

One user had a terrible experience.

For latency, prefer percentiles:

  • p50 for typical behavior,
  • p90/p95 for common tail behavior,
  • p99 for severe tail behavior.

Percentiles Need Sample Awareness

Percentiles with low sample count can be misleading.

If a service receives five requests in a period, p99 is not very meaningful.

Good dashboards pair latency percentile with request volume.

Good alarms often require both:

  • p95 latency high,
  • and request count above minimum threshold.

Metric math helps express this.


9. Units

Units are not decoration.

Units prevent accidental misinterpretation.

Examples:

MetricUnit
latencyMilliseconds or Seconds, but be consistent
request countCount
bytes transferredBytes
percentagePercent
queue delaySeconds
CPUPercent
memoryBytes or Percent

A common bug:

  • application emits latency in milliseconds,
  • alarm threshold assumes seconds,
  • alarm never fires or always fires.

Unit discipline is operational safety.


10. AWS Service Metrics vs Custom Metrics

AWS service metrics tell you what AWS can observe.

Custom metrics tell you what your application understands.

You usually need both.

AWS Service Metrics

Examples:

  • Lambda Errors, Duration, Throttles, ConcurrentExecutions,
  • ALB HTTPCode_Target_5XX_Count, TargetResponseTime, UnHealthyHostCount,
  • API Gateway 4XXError, 5XXError, Latency, IntegrationLatency,
  • SQS ApproximateAgeOfOldestMessage, ApproximateNumberOfMessagesVisible,
  • DynamoDB ThrottledRequests, ConsumedReadCapacityUnits,
  • RDS DatabaseConnections, CPUUtilization, FreeableMemory,
  • ECS service CPU/memory and deployment health signals.

These are essential, but they do not know your business outcome.

Custom Application Metrics

Examples:

  • CheckoutSuccessCount,
  • CheckoutFailureCount,
  • PaymentAuthorizationLatency,
  • CaseEscalationCreatedCount,
  • DocumentClassificationFailureCount,
  • RegulatoryDeadlineBreachCount,
  • IdempotencyConflictCount,
  • BusinessRuleRejectionCount.

These expose domain health.

A production-grade system should not depend only on AWS service metrics.


11. Embedded Metric Format

Embedded Metric Format, or EMF, lets applications emit structured logs that CloudWatch extracts into metrics.

This is useful because one emitted event can serve both:

  • log investigation,
  • metric aggregation.

Example conceptual event:

{
  "_aws": {
    "Timestamp": 1783332000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Company/Checkout",
        "Dimensions": [["Service", "Environment", "Operation", "Region"]],
        "Metrics": [
          { "Name": "RequestCount", "Unit": "Count" },
          { "Name": "Latency", "Unit": "Milliseconds" }
        ]
      }
    ]
  },
  "Service": "checkout-api",
  "Environment": "prod",
  "Operation": "AuthorizePayment",
  "Region": "ap-southeast-1",
  "RequestCount": 1,
  "Latency": 432
}

Use EMF when:

  • you want app metrics without separate PutMetricData calls,
  • you already emit structured logs,
  • you want CloudWatch-native custom metrics,
  • you need correlation between log record and metric.

Be careful with dimensions.

EMF can create many time series if dimension values are uncontrolled.


12. OpenTelemetry Metrics in CloudWatch

AWS increasingly supports OpenTelemetry-based telemetry flows.

OpenTelemetry is useful when:

  • you want vendor-neutral instrumentation,
  • you run hybrid/multi-cloud systems,
  • you already use OTLP collectors,
  • you want consistent metrics/traces/logs semantics,
  • you want future portability.

CloudWatch can ingest metrics through the CloudWatch API and OpenTelemetry paths depending on architecture.

Do not choose OpenTelemetry because it sounds modern.

Choose it when it simplifies instrumentation governance.


13. Alarm Mental Model

A CloudWatch alarm watches a metric or expression and transitions between states.

Main states:

StateMeaning
OKmetric is within defined threshold behavior
ALARMmetric breached the alarm condition
INSUFFICIENT_DATAalarm does not have enough data to determine state

An alarm is a state machine.

Alarm design is state transition design.


14. Threshold Alarms

A threshold alarm compares a metric/statistic to a threshold.

Examples:

Errors >= 10 for 3 out of 5 minutes
p95 Latency >= 1000 ms for 3 out of 5 minutes
QueueAge >= 300 seconds for 2 out of 3 periods
CPUUtilization >= 85% for 4 out of 5 periods

Threshold alarms are simple and useful when:

  • the healthy range is well-known,
  • the metric has stable behavior,
  • the threshold maps to action,
  • the signal is not strongly seasonal.

They are weak when:

  • traffic is highly variable,
  • baseline changes by hour/day,
  • metric is sparse,
  • threshold is arbitrary,
  • business impact depends on rate not absolute count.

15. Evaluation Periods and Datapoints to Alarm

CloudWatch alarms can evaluate multiple periods and require only some datapoints to breach.

This is often called M out of N.

Example:

EvaluationPeriods = 5
DatapointsToAlarm = 3
Period = 60 seconds

Meaning:

Alarm if 3 out of the last 5 one-minute datapoints breach the threshold.

This helps reduce noise from single-period spikes.

Examples

ConfigurationBehavior
1 out of 1fastest, noisiest
2 out of 3balanced for short sustained issues
3 out of 5filters transient spikes
5 out of 5only sustained continuous breach

For paging alarms, 2/3 or 3/5 often works better than 1/1.

But for critical availability checks, 1/1 may be justified if the signal is highly reliable.


16. Missing Data Behavior

Missing data is one of the most dangerous alarm settings.

An alarm can treat missing data as:

  • breaching,
  • not breaching,
  • ignore,
  • missing/insufficient data.

Choosing Missing Data Behavior

Metric TypeSuggested Thinking
continuous metric like CPUmissing may indicate telemetry failure
error count metric emitted only on errorsmissing should often mean zero, depending on emission model
heartbeat metricmissing should usually breach
low-traffic request metricmissing may be normal
security log ingestion metricmissing may be dangerous
batch job completion metricmissing may indicate failure if expected on schedule

Example trap:

Application emits ErrorCount only when errors occur.

If you set missing data as breaching, the alarm fires continuously when there are no errors.

Another trap:

Application emits Heartbeat=1 every minute.

If missing data is treated as not breaching, the service can disappear and alarm stays OK.

The metric emission contract determines missing data behavior.


17. Alarm Actions

CloudWatch alarms can trigger actions.

Common actions:

  • SNS notification,
  • EventBridge routing,
  • Systems Manager OpsItem,
  • incident management workflow,
  • Auto Scaling action,
  • Lambda remediation,
  • ticket creation via integration,
  • chat notification.

The action must match severity.

SeverityAction
user-impacting incidentpage on-call
SLO slow burnticket and notify service channel
dependency degradationnotify owner, enrich incident context
policy driftroute to platform/security workflow
telemetry pipeline degradationnotify platform operations
capacity trendbacklog/capacity planning ticket

Not every alarm should page.


18. Metric Math

Metric math lets you create expressions from metrics.

This is where alarms become more meaningful.

Error Rate

Instead of alarming on raw error count:

ErrorRate = Errors / Requests * 100

Why?

10 errors with 10 requests is catastrophic.

10 errors with 10 million requests may be irrelevant.

Success Rate

SuccessRate = SuccessfulRequests / TotalRequests * 100

This maps better to SLO.

Dependency Timeout Percentage

TimeoutRate = DependencyTimeoutCount / DependencyCallCount * 100

This distinguishes high traffic from high failure.

Queue Processing Lag

For async systems, combine:

  • age of oldest message,
  • processing rate,
  • consumer error rate,
  • visible message count.

A queue with 10,000 messages may be fine if consumers process fast.

A queue with 100 messages may be bad if the oldest message is 6 hours old.

Guarding Against Low Traffic

Alarm only when:

ErrorRate > 5% AND RequestCount > 100

This avoids paging on 1 failed request out of 1 total request during low traffic.


19. Composite Alarms

Composite alarms combine the states of other alarms using Boolean logic.

They are useful for reducing noise and representing higher-level health.

Example:

ALARM(ServiceErrorRateHigh) AND ALARM(ServiceTrafficNormal)

Another example:

ALARM(CheckoutSyntheticFailed) OR
(ALARM(CheckoutErrorRateHigh) AND ALARM(CheckoutLatencyHigh))

Composite alarms help build service health indicators.

Use composite alarms to page on meaningful combinations, not every raw symptom.


20. Anomaly Detection

Static thresholds fail when metric behavior is seasonal.

Examples:

  • traffic spikes every morning,
  • batch jobs run every night,
  • e-commerce traffic changes by day of week,
  • business activity drops on weekends,
  • reporting jobs have predictable monthly peaks.

CloudWatch anomaly detection can model expected values based on historical patterns and alarm when values move outside the expected band.

Good use cases:

  • request volume anomaly,
  • latency baseline anomaly,
  • error rate anomaly,
  • cost-ish operational usage proxy,
  • queue age that has predictable cycles.

Weak use cases:

  • brand-new metric with no stable history,
  • metric with chaotic behavior,
  • sparse event count,
  • security signal where any event is suspicious,
  • metric where fixed threshold maps directly to business harm.

Anomaly detection helps when the question is:

Is this metric unusual for this time and pattern?

It is not a replacement for SLOs.


21. Metrics Insights

Metrics Insights lets you query CloudWatch metrics using SQL-like queries.

This is useful for:

  • finding top-N offenders,
  • grouping by dimensions,
  • comparing services,
  • dashboard widgets,
  • dynamic fleets,
  • operational exploration.

Conceptual examples:

SELECT AVG(CPUUtilization)
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY AVG() DESC
LIMIT 10
SELECT SUM(Errors)
FROM SCHEMA("AWS/Lambda", FunctionName)
GROUP BY FunctionName
ORDER BY SUM() DESC
LIMIT 20

Metrics Insights is valuable when resources are dynamic and static dashboards become stale.


22. Common AWS Metrics Worth Knowing

22.1 Lambda

MetricMeaningAlarm Use
Errorsinvocation failureserror rate with invocation count
Durationexecution timep95/p99 latency and timeout proximity
Throttlesconcurrency throttlinguser impact or capacity issue
ConcurrentExecutionsactive concurrencysaturation/capacity
IteratorAgestream processing delayasync lag
DeadLetterErrorsDLQ delivery failurescritical async failure

22.2 Application Load Balancer

MetricMeaningAlarm Use
RequestCounttraffic volumedenominator/context
TargetResponseTimebackend latencyuser path latency
HTTPCode_Target_5XX_Countbackend errorsservice failure
HTTPCode_ELB_5XX_Countload balancer errorsinfrastructure/path failure
UnHealthyHostCountunhealthy targetscapacity/availability
TargetConnectionErrorCountconnection failuresbackend/network issue

22.3 API Gateway

MetricMeaningAlarm Use
Countrequest countdenominator/context
Latencyfull request latencyuser experience
IntegrationLatencybackend integration timedependency/backend issue
4XXErrorclient-side errorsusually not page unless abnormal
5XXErrorserver-side errorspage if sustained/impacting

22.4 SQS

MetricMeaningAlarm Use
ApproximateNumberOfMessagesVisiblebacklogcapacity/context
ApproximateAgeOfOldestMessageprocessing delayprimary async SLO signal
NumberOfMessagesSentinput rateworkload demand
NumberOfMessagesDeletedprocessing completionthroughput
DLQ visible messagesfailed workpage/ticket based on criticality

22.5 DynamoDB

MetricMeaningAlarm Use
ConsumedReadCapacityUnitsread usagecapacity trend
ConsumedWriteCapacityUnitswrite usagecapacity trend
ThrottledRequeststhrottlinguser impact or scaling issue
SystemErrorsAWS-side errorsavailability issue
latency metricsrequest latencyuser/dependency latency

22.6 RDS/Aurora

MetricMeaningAlarm Use
CPUUtilizationCPU pressuresaturation
DatabaseConnectionsconnection pressurepool/config issue
FreeableMemorymemory pressuresaturation
FreeStorageSpacestorage riskurgent capacity
ReadLatency/WriteLatencyIO performancedependency latency
ReplicaLagreplication delayread consistency/recovery risk

23. Business Metrics Are First-Class Metrics

AWS service metrics do not know whether your regulatory case breached deadline.

They do not know whether payment succeeded.

They do not know whether customer onboarding completed.

Business metrics must be instrumented by the application.

Examples:

DomainMetric
paymentsPaymentAuthorizationSuccessCount
case managementCaseEscalationCreatedCount
regulatory workflowDeadlineBreachCount
file processingDocumentProcessingCompletedCount
searchSearchNoResultCount
fraud/riskHighRiskDecisionCount
notificationNotificationDeliveryFailureCount

Business metrics should be treated carefully.

They can reveal sensitive behavior.

They need dimension governance.

They must not leak PII.


24. SLO-Aligned Alarm Design

A naïve alarm:

Errors > 0

A better alarm:

ErrorRate > 2% for 3 out of 5 minutes AND RequestCount > 100

An SLO-aligned alarm:

FastBurnErrorBudgetBreach for checkout success SLO

A practical structure:

AlarmPurposeAction
Fast burnsevere rapid SLO consumptionpage
Medium burnsustained degradationpage or urgent ticket
Slow burnreliability trendticket/planning
Symptom alarmuser impactpage
Cause alarmdiagnosisnotify/context
Telemetry alarmmonitoring system brokenplatform notification

The best alarm strategy separates:

  • symptom alarms,
  • cause alarms,
  • capacity alarms,
  • telemetry health alarms,
  • security alarms.

25. Example: Checkout CloudWatch Alarm Model

Metrics

Namespace: Company/Checkout
MetricName: RequestCount
Dimensions: Service, Environment, Region, Operation

Namespace: Company/Checkout
MetricName: RequestFailureCount
Dimensions: Service, Environment, Region, Operation, ErrorClass

Namespace: Company/Checkout
MetricName: RequestLatency
Dimensions: Service, Environment, Region, Operation

Namespace: Company/Checkout
MetricName: DependencyTimeoutCount
Dimensions: Service, Environment, Region, Operation, Dependency

Metric Math

FailureRate = RequestFailureCount / RequestCount * 100

Alarm Rules

AlarmConditionAction
Checkout high failure ratefailure rate > 2% and request count > 100 for 3/5 minpage
Checkout high p95 latencyp95 > 1200 ms and request count > 100 for 3/5 minpage if paired with symptom
Payment provider timeouttimeout rate > 5% for 3/5 minnotify dependency/channel
Receipt queue delayoldest message age > 300 sec for 2/3 minticket or page depending criticality
Synthetic checkout failurecritical flow fails in two locationspage

Composite Alarm

ALARM(CheckoutSyntheticFailure)
OR
(ALARM(CheckoutFailureRateHigh) AND ALARM(CheckoutTrafficNormal))
OR
(ALARM(CheckoutLatencyHigh) AND ALARM(CheckoutFailureRateHigh))

This pages on user-impacting states, not every supporting metric.


26. Metric Design for Async Workloads

For async systems, request latency is not enough.

Use:

  • accepted count,
  • processed count,
  • failed count,
  • retry count,
  • DLQ count,
  • processing age,
  • end-to-end age,
  • idempotency conflict count,
  • poison message count.

Queue Age Is Usually More Important Than Queue Depth

A queue with 100,000 messages may be healthy if consumers process 100,000 messages per minute.

A queue with 20 messages may be unhealthy if the oldest message is 6 hours old.

Alarm on age when the user promise is time-based.

Example:

ApproximateAgeOfOldestMessage > 300 seconds for 2 of 3 periods

If the business says “document must be classified within 5 minutes,” then queue age maps directly to user outcome.


27. Metric Design for Security Operations

Security metrics are not only findings.

Examples:

MetricPurpose
UnauthorizedApiCallCountunusual access attempts
RootUserActivityCountroot usage detection
SecurityFindingCountBySeverityposture/threat volume
RemediationFailureCountautomation reliability
PublicExposureFindingCountexposure posture
KmsDecryptDeniedCountkey access anomaly
CloudTrailDeliveryFailureCountaudit pipeline health
ConfigRecorderStoppedCountcompliance telemetry integrity

Security alarms should distinguish:

  • suspicious activity,
  • control failure,
  • detection pipeline failure,
  • remediation failure,
  • evidence integrity risk.

A security telemetry pipeline that cannot detect its own failure is not production-grade.


28. Alarm Ownership and Metadata

Every alarm should have metadata.

At minimum:

MetadataPurpose
serviceowner routing
environmentseverity/context
owner teamescalation
runbook URLactionability
severitypage/ticket/report
SLO mappingimpact interpretation
dashboard URLdiagnosis
escalation policyresponse path
created bygovernance
expiry/review dateprevent stale alarms

Use tags and IaC conventions.

Alarms created manually during emergencies should be backfilled into code or removed.


29. Cross-Account and Cross-Region Metrics

Production AWS environments are multi-account.

Metric visibility must match operational responsibility.

Typical pattern:

Design questions:

  • Which account owns source metrics?
  • Which monitoring account can view them?
  • Which team can create alarms?
  • Are alarms local, central, or both?
  • How are dashboards shared?
  • How are IAM permissions constrained?
  • Are security logs and operational metrics separated appropriately?
  • Are Regions covered consistently?

Cross-account observability improves visibility, but it does not replace ownership.


30. Dashboard vs Alarm

Do not alarm on every graph.

Use Dashboard WhenUse Alarm When
human exploration is neededaction is required
signal is diagnosticsignal represents state transition
trend mattersthreshold breach matters
no immediate response neededtimely response needed
context for incidentsincident trigger

A metric can be useful without being alarm-worthy.


31. Common Alarm Anti-Patterns

Anti-Pattern 1 — CPU > 80% Pages Everyone

CPU may be high during normal load.

Fix:

  • combine with saturation/error/latency,
  • use as diagnostic or capacity alarm,
  • page only if user impact or imminent failure.

Anti-Pattern 2 — Raw Error Count Without Denominator

Errors > 10 means different things at different traffic levels.

Fix:

  • use error rate,
  • require minimum request count.

Anti-Pattern 3 — Missing Data Misconfigured

Wrong missing-data behavior can hide outages or create constant noise.

Fix:

  • define metric emission contract,
  • configure missing data per metric type.

Anti-Pattern 4 — Alarm Without Runbook

If the operator does not know what to do, the alarm is incomplete.

Fix:

  • attach runbook,
  • include first checks,
  • include rollback/mitigation.

Anti-Pattern 5 — Static Threshold on Seasonal Metric

Traffic varies by time/day.

Fix:

  • use anomaly detection,
  • or compare to historical baseline,
  • or use SLO/error rate instead of raw volume.

Anti-Pattern 6 — High-Cardinality Custom Metrics

Dimensions explode cost and query complexity.

Fix:

  • move high-cardinality fields to logs/traces,
  • use controlled dimension dictionaries.

Anti-Pattern 7 — Alarming on Causes Instead of Symptoms

Cause alarms are useful, but paging should usually start from user impact.

Fix:

  • page on symptom,
  • attach cause alarms to context dashboard.

Anti-Pattern 8 — No Alarm Review

Old alarms remain after architecture changes.

Fix:

  • quarterly review,
  • stale alarm detection,
  • alarm ownership registry.

32. CloudWatch Alarm Implementation Pattern

A practical IaC pattern:

alarm:
  name: checkout-prod-high-failure-rate
  severity: sev2
  owner: payments-platform
  service: checkout-api
  environment: prod
  region: ap-southeast-1
  slo: checkout-success-rate-99.9
  runbook: https://runbooks.example.com/checkout/high-failure-rate
  dashboard: https://dashboards.example.com/checkout/prod
  metric:
    namespace: Company/Checkout
    expression: RequestFailureCount / RequestCount * 100
    threshold: 2
    comparison: GreaterThanThreshold
    period: 60
    evaluationPeriods: 5
    datapointsToAlarm: 3
    missingData: notBreaching
  action:
    page: true
    notify: payments-prod-alerts

The exact syntax depends on Terraform, CloudFormation, CDK, Pulumi, or internal tooling.

The important part is the contract.


33. Alarm Testing

An untested alarm is a hypothesis.

Test alarms by verifying:

  • metric exists,
  • dimensions match,
  • threshold can actually be reached,
  • missing data behaves correctly,
  • alarm action triggers,
  • routing works,
  • runbook link works,
  • dashboard link works,
  • escalation works,
  • recovery notification works,
  • alarm is suppressed during maintenance if needed.

Test Scenarios

ScenarioExpected Outcome
emit synthetic error metricerror alarm enters ALARM
stop heartbeat metricheartbeat alarm fires if configured
reduce traffic to zerolow-traffic alarm does not falsely page
trigger dependency timeoutdependency alarm and dashboard update
fill DLQDLQ alarm triggers correct owner
break telemetry permissiontelemetry health alarm fires

Observability needs its own game days.


34. Metric and Alarm Governance

In a large AWS environment, teams will create many metrics and alarms.

Governance prevents collapse.

Required Standards

StandardRule
namingstable, domain-oriented
dimensionsapproved dimension dictionary
ownershipevery alarm has owner
runbookevery page alarm has runbook
severitydefined mapping
SLO mappingrequired for user-impact alarms
IaCalarms defined as code
reviewperiodic stale/noisy alarm review
costhigh-cardinality metrics reviewed
retentiondashboard and metric needs understood

Operational Metrics for the Alarm System

Track:

  • alerts per service,
  • pages per service,
  • false positive rate,
  • duplicate alert rate,
  • mean time to acknowledge,
  • mean time to mitigate,
  • alarms without owner,
  • alarms without runbook,
  • alarms never triggered,
  • alarms always in insufficient data,
  • most noisy alarms.

The monitoring system itself must be monitored.


35. Practical Build: Minimal Production Metric Set

For every user-facing service, define:

RequestCount
RequestSuccessCount
RequestFailureCount
RequestLatency
DependencyCallCount
DependencyFailureCount
DependencyLatency
BusinessOperationSuccessCount
BusinessOperationFailureCount

Recommended dimensions:

Service
Environment
Region
Operation
Dependency
ErrorClass

Avoid by default:

UserId
RequestId
Email
RawUrl
ExceptionMessage
StackTrace

Then define alarms:

AlarmType
high failure rate with minimum trafficsymptom
high p95 latency with minimum trafficsymptom
critical synthetic failureoutside-in symptom
dependency timeout ratecause/context
queue age/backlog SLO breachasync symptom
telemetry heartbeat missingtelemetry health
DLQ has messagesasync failure

36. Practical Build: Metric Review Questions

Before accepting a metric, ask:

  1. What question does this metric answer?
  2. Who owns it?
  3. What dimensions are required?
  4. What dimensions are forbidden?
  5. Is it a symptom metric or cause metric?
  6. Is it useful for alerting, dashboarding, SLO, capacity, or audit?
  7. What is its unit?
  8. What is its expected emission frequency?
  9. What does missing data mean?
  10. What action follows a threshold breach?
  11. How will we test it?
  12. How will we retire it?

Before accepting an alarm, ask:

  1. Is this actionable?
  2. Does it indicate user impact or imminent impact?
  3. Is there a runbook?
  4. Is the owner clear?
  5. Is the severity correct?
  6. Is missing data configured correctly?
  7. Does it avoid low-traffic false positives?
  8. Does it duplicate another alarm?
  9. Does it need composite logic?
  10. Has it been tested?

37. Failure Modes

Failure Mode: Alarm Does Not Fire During Incident

Possible causes:

  • wrong dimension set,
  • metric not emitted,
  • missing data treated as OK,
  • threshold too high,
  • statistic wrong,
  • period too long,
  • alarm created in wrong Region/account,
  • insufficient traffic and no synthetic check,
  • dashboard looked at aggregate hiding one tenant/operation.

Failure Mode: Alarm Fires Constantly

Possible causes:

  • threshold too low,
  • metric too noisy,
  • period too short,
  • missing data treated as breaching incorrectly,
  • expected client errors counted as server errors,
  • maintenance not suppressed,
  • cause alarm paging instead of symptom alarm.

Failure Mode: Alarm Fires but Nobody Acts

Possible causes:

  • no owner,
  • no runbook,
  • wrong severity,
  • notification routed to dead channel,
  • alert fatigue,
  • alarm not trusted.

Failure Mode: Metric Cost Explosion

Possible causes:

  • high-cardinality dimensions,
  • deployment version dimension with too many values,
  • tenant/user/request dimensions,
  • EMF emitting uncontrolled dimension combinations,
  • per-resource custom metrics when service-level metric is enough.

Failure Mode: Dashboard Hides Incident

Possible causes:

  • aggregation across Regions,
  • p50 latency only,
  • no traffic denominator,
  • no error class breakdown,
  • no dependency view,
  • no deployment/change markers,
  • no synthetic/RUM signal.

38. Top 1% Engineering Heuristics

  1. Alert on symptoms; diagnose with causes.
  2. Every page must be actionable.
  3. Metrics are contracts; breaking metric identity breaks operations.
  4. Cardinality is a design budget.
  5. Missing data is a semantic decision, not a default.
  6. Averages are dangerous for latency.
  7. Request count is the denominator for most service alarms.
  8. Queue age beats queue depth for async SLOs.
  9. Business metrics are not optional for business systems.
  10. If an alarm has no owner and no runbook, it is not production-ready.
  11. Composite alarms reduce noise when designed around user impact.
  12. Anomaly detection is useful for baseline deviation, not a replacement for SLOs.
  13. Test alarms before incidents.
  14. Review noisy and stale alarms as operational debt.

39. Summary

CloudWatch Metrics and Alarms are not just AWS features.

They are the language your system uses to declare operational state.

Production-grade metric design requires:

  • stable namespaces,
  • precise metric names,
  • controlled dimensions,
  • correct units,
  • thoughtful periods,
  • appropriate statistics,
  • low-cardinality design,
  • metric math for ratios and SLOs,
  • correct missing data behavior,
  • composite alarms for meaningful state,
  • anomaly detection for seasonal behavior,
  • ownership metadata,
  • runbooks,
  • and regular review.

The next part goes deeper into CloudWatch Logs Insights.

Metrics tell you that the system is unhealthy.

Logs help you understand why.


References

Lesson Recap

You just completed lesson 56 in deepen practice. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.