Cheatsheet Observability Part 052 — Synthetic Monitoring

Fokus part ini: memahami synthetic monitoring sebagai observability dari sudut pandang luar sistem. Jika logs, metrics, dan traces melihat sistem dari dalam, synthetic monitoring bertanya: "Bisakah client atau user-like probe benar-benar menjalankan skenario penting sekarang?"

1. Core Mental Model

Synthetic monitoring adalah eksekusi request atau journey buatan secara terjadwal untuk memvalidasi availability, latency, routing, authentication, authorization, dependency behavior, dan business capability.

It is active monitoring.

Bukan menunggu user gagal dulu.

Bukan hanya membaca metric pasif.

Synthetic monitoring membuat traffic terkontrol untuk menguji path tertentu.

Passive telemetry:
Observe real traffic.

Synthetic monitoring:
Generate controlled traffic to test important paths.

Dalam enterprise Java/JAX-RS backend, synthetic monitoring dapat menguji:

• API gateway route;
• ingress/load balancer path;
• auth token validation;
• JAX-RS resource availability;
• request/response contract;
• database-backed read;
• quote creation smoke path;
• pricing path;
• order submission path;
• downstream dependency path;
• private network endpoint;
• region/zone-specific reachability.

2. Why Synthetic Monitoring Exists

Real telemetry is reactive.

Synthetic monitoring is proactive.

Real user traffic may not hit every critical path often enough.

A broken rarely used endpoint can stay broken for hours if no customer hits it.

Synthetic probes can detect:

• route misconfiguration;
• expired certificate;
• DNS issue;
• WAF/API gateway rejection;
• auth provider failure;
• deployment that passes pod health but fails real API path;
• broken request contract;
• broken dependency chain;
• regional endpoint outage;
• private connectivity failure;
• business journey regression;
• stale data path;
• unexpected latency degradation.

Synthetic checks are especially useful when:

• traffic is low but critical;
• business path is high-value;
• system is regionally distributed;
• deployment risk is high;
• on-prem/hybrid networking is complex;
• health check is too shallow;
• SLO needs outside-in validation.

3. Synthetic Monitoring vs Health Check

Health check asks:

Is this component alive or ready?

Synthetic monitoring asks:

Can a realistic path succeed from a realistic location?

Health check may pass while synthetic fails.

Example:

Pod /ready = UP
Ingress route = broken
Synthetic API check = DOWN

Synthetic monitoring catches the external path.

4. Synthetic Monitoring vs Real User Monitoring

Real User Monitoring observes actual user/client experience.

Synthetic monitoring simulates selected journeys.

For backend enterprise systems, RUM may be less available if the product is API/backend-heavy or B2B-integrated.

Synthetic checks can still validate key backend journeys.

Synthetic is not a replacement for real telemetry.

It complements it.

5. Types of Synthetic Checks

5.1 API Smoke Test

A lightweight check that verifies an API can respond correctly.

Example:

GET /api/quotes/health-smoke
Expected: 200 + known response schema

Useful after deployment and during ongoing monitoring.

5.2 Read-Only Business Check

Validates a safe business read path.

Example:

GET /api/catalog/products?synthetic=true
Expected: 200, non-empty response, latency < threshold

Good for dependency-backed read paths.

5.3 Write Path Synthetic Check

Validates a controlled write path.

Example:

POST /api/quotes
Header: X-Synthetic-Test: true
Payload: dedicated synthetic customer/test tenant
Expected: quote created and marked synthetic/test

High value but risky.

Requires strict test data governance.

5.4 User Journey Check

Multi-step scenario.

Example:

Authenticate
Create quote
Price quote
Submit approval
Cancel synthetic quote
Verify terminal state

Useful for CPQ/order management journeys, but expensive and potentially noisy.

5.5 Dependency Synthetic Check

Validates connectivity to a dependency through application path.

Example:

• DB-backed lookup;
• Redis-backed cache lookup;
• Kafka outbox publish path;
• RabbitMQ task enqueue path;
• Camunda process start path;
• downstream HTTP call path.

5.6 Regional Probe

Runs from different regions or network locations.

Useful for:

• regional outage detection;
• DNS/routing issue;
• latency geography;
• cloud/on-prem/hybrid path validation.

5.7 Private Endpoint Probe

Runs from inside private network or VPC/VNet.

Useful for internal APIs not reachable publicly.

5.8 Authenticated Synthetic Check

Uses controlled credentials or token flow.

High realism, high security responsibility.

6. Synthetic Check Design Principles

A synthetic check should be:

• purposeful;
• bounded;
• deterministic;
• safe;
• low-noise;
• low-cost;
• privacy-aware;
• tagged as synthetic;
• excluded or separated from business analytics where appropriate;
• connected to dashboards and runbooks;
• owned by a team.

Bad synthetic check:

Runs random production writes every minute with real customer-like data and no cleanup.

Good synthetic check:

Runs controlled journey using synthetic tenant/account, marks every request/event/audit as synthetic, validates expected response, cleans up or uses expiring data, and links failures to a runbook.

7. Synthetic Traffic Identification

Synthetic traffic must be identifiable.

Common identifiers:

X-Synthetic-Test: true
User-Agent: synthetic-monitor/<name>
synthetic=true
synthetic_check_id=<id>
synthetic_journey=<name>
tenant_id=synthetic-tenant
actor_id=synthetic-monitor
correlation_id=synthetic-<uuid>

But be careful.

Do not trust arbitrary external headers unless the source is controlled.

Internal services may treat X-Synthetic-Test as metadata only, not authorization.

Synthetic markers should propagate into:

• logs;
• traces;
• metrics where low-cardinality;
• audit logs where appropriate;
• business events if synthetic writes exist;
• dashboards;
• alert annotations.

Avoid high-cardinality synthetic IDs as metric labels.

Use bounded labels:

synthetic="true"
synthetic_journey="quote_create_smoke"

Do not use:

synthetic_run_id="uuid-per-run"

as a metric label.

Put run ID in logs/traces instead.

8. Synthetic Data Governance

Synthetic checks that perform writes need strict data discipline.

Questions:

• Which tenant/account/customer is used?
• Is the data clearly synthetic?
• Can it affect real reporting?
• Can it trigger real fulfillment?
• Can it trigger billing?
• Can it send real notifications?
• Can it create real orders?
• Can it enter real approval queues?
• Can it pollute audit/compliance records?
• Is cleanup required?
• What happens when cleanup fails?

For CPQ/order management systems, write synthetic journeys are powerful but dangerous.

Safer options:

• read-only synthetic checks;
• dry-run endpoints if officially supported;
• dedicated synthetic tenant;
• dedicated test catalog item;
• feature flag to prevent fulfillment side effects;
• synthetic marker that blocks external notification;
• TTL/cleanup job;
• separate lower environment check for deep write flows;
• production check limited to non-mutating critical path.

Internal verification checklist:

• verify whether synthetic production writes are allowed;
• verify synthetic tenant/account policy;
• verify whether synthetic records are excluded from reporting;
• verify whether synthetic records affect audit/compliance;
• verify cleanup and retention policy;
• verify with security/compliance/product ops before deep write probes.

9. API Smoke Test for JAX-RS Service

A simple synthetic API check should validate more than HTTP 200.

It may validate:

• status code;
• response schema;
• required fields;
• latency threshold;
• correlation ID returned or logged;
• auth behavior;
• content type;
• version header;
• deployment version marker;
• expected domain state.

Example conceptual request:

GET /quotes/synthetic/readiness-smoke HTTP/1.1
Authorization: Bearer <synthetic-token>
X-Correlation-ID: synthetic-quote-read-smoke-20260711T101530Z
User-Agent: synthetic-monitor/quote-read-smoke

Expected:

HTTP 200
Content-Type: application/json
Response contains expected schema
Latency < 500ms
Trace created
Logs contain correlation ID
Synthetic traffic marked

10. Multi-Step Journey Monitoring

A journey check validates workflow continuity.

Example CPQ journey:

This can reveal issues that single endpoint probes miss.

But multi-step checks are more fragile.

They need:

• stable synthetic data;
• idempotent design;
• cleanup;
• clear timeout per step;
• good failure classification;
• careful alert threshold;
• trace/log correlation per run;
• owner and runbook.

11. Authenticated Synthetic Checks

Authenticated checks are valuable because many production failures happen before application code:

• expired client secret;
• identity provider outage;
• bad token audience;
• bad scope;
• API gateway policy mismatch;
• mTLS certificate issue;
• WAF rule regression;
• clock skew;
• key rotation failure.

Security rules:

• use least-privilege synthetic identity;
• store credentials in approved secret manager;
• rotate credentials;
• avoid logging token;
• avoid exposing synthetic token to app logs;
• limit permissions to synthetic tenant or read-only scope;
• monitor failed auth separately;
• treat probe credentials as production secrets.

Internal verification checklist:

• verify identity provider integration;
• verify synthetic account owner;
• verify secret storage and rotation;
• verify token scopes;
• verify audit behavior for synthetic user;
• verify whether auth failures page app team, platform team, or identity team.

12. Regional and Network-Aware Probes

Synthetic checks should run from relevant locations.

Possible locations:

• public internet;
• same cloud region;
• different cloud region;
• inside Kubernetes cluster;
• inside VPC/VNet;
• on-prem network;
• partner network;
• private endpoint path;
• customer-like network segment.

A service can be healthy from inside cluster but broken from customer path.

Example:

Inside cluster: 200 OK
From public region: TLS error
From on-prem network: timeout
From partner VPN: DNS failure

Regional probes help identify blast radius.

During incident, ask:

Is this global, regional, network-specific, tenant-specific, or path-specific?

13. Synthetic Monitoring and Alerting

Synthetic alerts should be carefully tuned.

Common alert dimensions:

• check name;
• environment;
• region;
• endpoint/journey;
• status;
• failure reason category;
• owner;
• severity.

Avoid alerting on one failed probe unless the path is extremely critical and false positives are rare.

Typical strategies:

Fail 3 of 5 consecutive checks
Fail from 2 regions
Fail for 5 minutes
Latency above threshold for 10 minutes
Journey step failure repeated N times

Failure classification matters:

• DNS failure;
• TLS failure;
• connection timeout;
• HTTP 401/403;
• HTTP 404 route issue;
• HTTP 5xx application/dependency issue;
• schema mismatch;
• latency threshold breach;
• business assertion failed;
• cleanup failed.

Each class may route differently.

14. False Positives

Synthetic monitoring can be noisy.

False positive causes:

• probe location network issue;
• DNS resolver issue at probe provider;
• token expired due to probe credential problem;
• synthetic account disabled;
• test data changed;
• cleanup failure affects next run;
• transient dependency blip;
• threshold too tight;
• check too deep;
• probe timeout shorter than realistic latency;
• deployment temporarily shifts route;
• WAF blocks synthetic user-agent;
• rate limiter blocks probe.

Reducing false positives:

• use multiple probe locations;
• use consecutive failure windows;
• classify failure reason;
• maintain synthetic credentials;
• design idempotent checks;
• isolate test data;
• exclude probe from rate limits or explicitly budget it;
• tune latency thresholds using historical data;
• keep deep checks lower frequency;
• have a runbook for synthetic failure.

15. False Negatives

A synthetic check can pass while users still fail.

Causes:

• synthetic path too shallow;
• synthetic tenant has special access;
• synthetic data is too simple;
• real tenant-specific config broken;
• real payloads are larger;
• real network path differs;
• real auth scopes differ;
• synthetic check avoids expensive downstream calls;
• check validates status code but not response correctness;
• check only runs in one region;
• check frequency too low.

Mitigation:

• design representative scenarios;
• compare synthetic with real traffic metrics;
• use tenant-segment dashboards;
• validate response semantics;
• add regional/private probes;
• avoid privileged synthetic shortcuts;
• supplement with logs, traces, metrics, and business KPIs.

16. Synthetic Frequency

Frequency is a trade-off.

High frequency:

• faster detection;
• more cost;
• more traffic;
• more noise;
• more synthetic data;
• more chance of rate-limit effects.

Low frequency:

• lower cost;
• lower noise;
• slower detection;
• may miss short outages.

Suggested thinking:

Critical shallow API check: higher frequency.
Deep multi-step business journey: lower frequency.
Read-only dependency path: medium frequency.
Write journey in production: very careful, lower frequency, strict governance.

Frequency should reflect:

• business criticality;
• expected detection time;
• check cost;
• side-effect risk;
• false positive risk;
• SLO/alerting needs;
• probe location count.

17. Synthetic Latency Thresholds

Latency thresholds should not be arbitrary.

Bad:

Alert if synthetic latency > 100ms because 100ms feels good.

Better:

• use historical percentiles;
• align with user-facing latency SLO;
• separate network time from server time if possible;
• track regional latency separately;
• avoid overreacting to one spike;
• alert on sustained degradation;
• compare synthetic latency with real request latency.

Useful metrics:

synthetic.check.duration
synthetic.check.success
synthetic.check.failure
synthetic.journey.step.duration
synthetic.journey.failure.step
synthetic.check.latency.p95

Use bounded labels:

check_name
journey_name
step_name
environment
region
status
failure_category

Avoid run ID as metric label.

18. Synthetic Traces and Logs

Synthetic probes should produce correlated traces/logs.

A probe run should have:

• correlation ID;
• trace ID;
• synthetic marker;
• check name;
• region/source;
• journey step;
• expected result;
• failure category;
• run ID in logs/traces, not metric labels.

Example log fields:

{
  "event": "synthetic_check_failed",
  "synthetic": true,
  "synthetic_check": "quote_pricing_smoke",
  "synthetic_region": "private-vpc-a",
  "failure_category": "http_5xx",
  "correlation_id": "synthetic-quote-pricing-20260711T101530Z",
  "trace_id": "...",
  "duration_ms": 1840
}

Important:

• synthetic probe logs should not flood application logs;
• application should mark inbound synthetic requests;
• traces should allow drilling into dependency bottlenecks;
• synthetic dashboards should link to trace/log search.

19. Synthetic Monitoring and Business Metrics

Synthetic traffic can pollute business metrics.

Example bad outcome:

• quote created count inflated;
• approval aging includes synthetic tasks;
• conversion funnel distorted;
• order fallout rate affected;
• audit reports contain synthetic actions;
• SLA dashboard includes synthetic journeys;
• billing/reporting sees synthetic records.

Mitigations:

• mark synthetic records clearly;
• exclude synthetic tenant from business analytics;
• filter synthetic traffic in dashboards;
• separate synthetic metrics from business metrics;
• define audit policy for synthetic actions;
• cleanup synthetic records;
• use read-only checks where possible.

Never assume synthetic traffic is harmless.

20. Synthetic Monitoring for CPQ/Order Management

Relevant journeys:

Quote read smoke

Purpose:

• validate API route;
• validate auth;
• validate DB read;
• validate response schema.

Risk: low.

Catalog/pricing smoke

Purpose:

• validate catalog/pricing dependency;
• detect pricing degradation.

Risk: medium, depending on downstream calls.

Quote create dry-run

Purpose:

• validate write path without business side effects.

Risk: medium.

Requires official dry-run or synthetic tenant.

Approval workflow smoke

Purpose:

• validate workflow/Camunda path.

Risk: high.

May create tasks or workflow instances.

Order submission synthetic journey

Purpose:

• validate critical quote-to-order path.

Risk: very high.

May trigger fulfillment, billing, notifications, or external integrations.

Usually requires strong safeguards or non-production environment.

21. Synthetic Monitoring for Dependencies

Synthetic checks can validate dependency paths from the service perspective.

Examples:

• PostgreSQL read-only query through API;
• Redis-backed cached endpoint;
• Kafka outbox enqueue and later verification;
• RabbitMQ task enqueue with synthetic routing key;
• Camunda process definition availability;
• downstream service /status through API gateway;
• cloud object storage read of synthetic object;
• secret/config service retrieval path.

Avoid probes that directly hammer dependencies outside application semantics unless the goal is platform dependency monitoring.

For service ownership, prefer application-path synthetic checks.

For platform ownership, use dependency-native checks.

22. Synthetic Monitoring in Kubernetes and Cloud

Synthetic can run as:

• external SaaS probe;
• Kubernetes CronJob;
• in-cluster probe service;
• cloud function / serverless scheduled task;
• CI/CD post-deploy smoke test;
• pipeline gate;
• private network agent;
• on-prem probe agent.

Each has trade-offs.

Internal verification checklist:

• verify where synthetic checks run;
• verify network path from probe to service;
• verify whether probe tests public or private route;
• verify IAM/secrets used by probe;
• verify dashboards include probe source.

23. CI/CD Smoke Tests vs Continuous Synthetic Monitoring

Post-deploy smoke tests answer:

Did the new deployment work immediately after rollout?

Continuous synthetic monitoring answers:

Is the path still working over time?

Both are useful.

Post-deploy smoke tests catch:

• broken config;
• incompatible deployment;
• missing route;
• schema mismatch;
• startup issue;
• immediate dependency issue.

Continuous synthetic catches:

• certificate expiry;
• route drift;
• secret expiry;
• network change;
• dependency degradation;
• data drift;
• operational regressions after deployment.

Do not rely on only one.

24. Synthetic Monitoring and SLOs

Synthetic checks may inform SLOs but should be used carefully.

They can be useful for:

• low-traffic critical APIs;
• external availability validation;
• regional path availability;
• deployment confidence;
• private endpoint availability.

But synthetic success is not the same as real user success.

A synthetic SLI may be valid when:

• the check path is representative;
• frequency is sufficient;
• probe locations match user/client paths;
• synthetic credentials are realistic;
• false positives are controlled;
• exclusions are documented.

For high-traffic endpoints, real request SLIs are usually stronger.

Synthetic can act as complementary signal.

25. Alert Routing

Synthetic alert routing depends on failure classification.

Examples:

A good alert message should include:

• check name;
• environment;
• region/source;
• endpoint/journey;
• failure category;
• observed status/latency;
• last successful time;
• dashboard link;
• trace/log link;
• runbook link;
• owner/escalation path.

26. Production Debugging Workflow

When synthetic check fails:

1. Confirm which check failed.
2. Identify source region/network.
3. Classify failure: DNS, TLS, auth, route, status code, schema, latency, business assertion, cleanup.
4. Compare with real traffic metrics.
5. Check service health dashboard.
6. Check edge/API gateway/load balancer logs.
7. Check JAX-RS application logs by synthetic correlation ID.
8. Open trace for synthetic request.
9. Check dependency dashboard.
10. Check recent deployment/config/cert/secret changes.
11. Check whether synthetic data or credential expired.
12. Decide whether this is customer-impacting, probe-only, or early warning.

Do not immediately assume application outage.

Synthetic failures often occur outside the service process.

27. Synthetic Monitoring Anti-Patterns

Anti-pattern 1: Only checking /health

That is not synthetic journey monitoring.

Anti-pattern 2: Checking status code only

A 200 response can still contain wrong data.

Anti-pattern 3: Using real customer data

This creates privacy, correctness, and support risk.

Anti-pattern 4: No synthetic marker

You cannot filter or debug probe traffic.

Anti-pattern 5: Synthetic writes without cleanup

This pollutes business state.

Anti-pattern 6: Too many deep checks too frequently

This creates load and false positives.

Anti-pattern 7: Alerting on every single probe failure

This causes alert fatigue.

Anti-pattern 8: Synthetic account has excessive privileges

This hides real auth problems and increases blast radius.

Anti-pattern 9: Synthetic path is too privileged or too fake

This creates false confidence.

Anti-pattern 10: No owner/runbook

Alert fires, nobody knows what it means.

28. Design Example: Quote Read Synthetic Check

Purpose:

Validate public/private API route, auth, JAX-RS resource, DB read, response schema, and baseline latency.

Request:

GET /api/quotes/synthetic/reference-quote
Headers:
  Authorization: Bearer synthetic-readonly-token
  X-Correlation-ID: synthetic-quote-read-${timestamp}
  User-Agent: synthetic-monitor/quote-read

Assertions:

• HTTP 200;
• response schema valid;
• quote ID belongs to synthetic tenant;
• status is expected;
• latency below threshold;
• no PII returned;
• trace exists;
• logs contain correlation ID;
• metric emits check result.

Alert rule:

Page if check fails 3 consecutive times from 2 probe sources.
Ticket if only one source fails for 15 minutes.

Runbook:

• check gateway route;
• check auth status;
• check service health dashboard;
• check DB read latency;
• check recent deployment;
• check synthetic reference data existence.

29. Design Example: Pricing Journey Synthetic Check

Purpose:

Validate quote pricing path without creating real order or fulfillment side effects.

Journey:

1. Authenticate as synthetic user.
2. Create or load synthetic quote draft.
3. Request pricing.
4. Validate pricing response schema and bounded business expectation.
5. Cancel or discard synthetic quote if write occurred.

Assertions:

• each step succeeds;
• pricing latency under threshold;
• expected price rule applied for test product;
• no fulfillment event emitted;
• synthetic marker propagated;
• trace contains pricing dependency span;
• cleanup successful.

Risks:

• price rules change legitimately;
• catalog test item changes;
• downstream pricing dependency slow;
• synthetic quote cleanup fails;
• synthetic data appears in reporting.

Controls:

• dedicated synthetic catalog item;
• bounded assertion, not exact fragile price unless stable;
• synthetic tenant filtering;
• cleanup monitor;
• low frequency;
• clear owner.

30. Internal Verification Checklist

For your service/team, verify:

• whether synthetic monitoring exists;
• which tools run synthetic checks;
• which environments are covered;
• which regions/networks are covered;
• whether checks run from public internet, private cloud, Kubernetes, on-prem, or CI/CD;
• synthetic check inventory;
• check owner per journey;
• alert routing and severity;
• failure classification;
• runbook link;
• synthetic credentials and secret rotation;
• synthetic tenant/account policy;
• whether synthetic traffic is marked in logs/traces/metrics/audit;
• whether synthetic traffic is excluded from business analytics;
• whether synthetic writes are allowed in production;
• cleanup process for synthetic data;
• false positive history;
• false negative history;
• frequency and threshold rationale;
• relationship to SLOs;
• relationship to deployment smoke tests;
• trace/log correlation for synthetic requests;
• dashboard panels for synthetic checks;
• security review for authenticated probes.

31. PR Review Checklist

When reviewing synthetic monitoring changes, ask:

• What production question does this check answer?
• Is it health check, smoke test, synthetic API check, or business journey check?
• Is the check read-only or mutating?
• If mutating, what data does it create?
• How is synthetic data marked?
• Can it trigger real fulfillment, billing, notification, workflow, or audit side effects?
• Does it use least-privilege credentials?
• Are credentials stored and rotated safely?
• Is the frequency justified?
• Are timeouts bounded?
• Are assertions meaningful beyond HTTP status?
• Are false positives controlled?
• Are failure categories clear?
• Does alert route to the correct owner?
• Does it link to dashboard, logs, traces, and runbook?
• Does it create high-cardinality metric labels?
• Does it pollute business metrics?
• Is cleanup reliable?
• Is the check representative of real client path?

32. Summary

Synthetic monitoring is outside-in evidence.

It catches failures that internal health checks and passive metrics may miss:

• DNS;
• TLS;
• gateway routing;
• authentication;
• WAF/API policy;
• regional connectivity;
• private endpoint issues;
• low-traffic critical endpoint breakage;
• business journey regression.

But synthetic monitoring can also create noise, cost, data pollution, and security risk if poorly designed.

The senior engineer mental model:

Use health checks for platform control.
Use metrics/logs/traces for internal evidence.
Use synthetic monitoring for outside-in path validation.
Use SLOs to decide whether reliability is acceptable.
Use runbooks to make alerts actionable.

Synthetic monitoring is strongest when it is:

• representative;
• safe;
• low-noise;
• correlated;
• owned;
• privacy-aware;
• cost-aware;
• connected to incident response.