Cheatsheet Observability Part 048 — Observability Cost Management

Fokus part ini: mengelola biaya observability tanpa membuat production menjadi gelap. Observability yang baik bukan mengirim semua data. Observability yang baik adalah mengirim signal yang cukup benar, cukup lengkap, cukup murah, dan cukup cepat untuk menjawab pertanyaan production.

1. Core Mental Model

Observability memiliki biaya.

Biaya itu muncul di beberapa titik:

• telemetry generation di aplikasi;
• CPU/memory/network overhead;
• agent/collector processing;
• ingestion ke backend observability;
• storage hot/cold/archive;
• indexing;
• query execution;
• dashboard refresh;
• alert evaluation;
• retention;
• audit/compliance preservation;
• operational time untuk merawat noise.

Cost management bukan berarti mengurangi signal sembarangan.

Cost management berarti:

Preserve high-value evidence.
Reduce low-value noise.
Control cardinality.
Set retention by use case.
Sample intelligently.
Prevent accidental telemetry explosions.

The goal is not lowest cost. The goal is best evidence-per-cost.

2. The Observability Cost Equation

A simple model:

Observability cost = volume × cardinality × retention × query/index complexity × backend pricing model

Different signal types have different cost drivers.

A cheap signal that no one can use is still waste.

An expensive signal that prevents long outages may be worth it.

3. Cost vs Visibility Trade-off

Do not optimize cost in isolation.

Use this decision model:

Good cost management is not:

Turn off tracing because it is expensive.

Good cost management is:

Sample normal successful traces, retain all error/slow/business-critical traces, reduce noisy span attributes, and use tail sampling at collector when available.

4. Log Volume Cost

Logs are often the largest cost source because they are easy to add and hard to control.

Major log volume drivers:

• INFO log inside loops;
• DEBUG enabled in production;
• repeated stack traces;
• retry logs per attempt at WARN/ERROR;
• logging full request/response body;
• logging large payloads/events;
• logging SQL with parameters;
• verbose framework logs;
• noisy health check/access logs;
• high-frequency background job logs;
• logging every message consumed from Kafka/RabbitMQ;
• logging every Redis command.

Bad logging cost pattern

for (OrderLine line : order.getLines()) {
    log.info("Processing line {} for order {} payload={}", line.getId(), order.getId(), line);
}

Problems:

• high cardinality business identifiers;
• potentially large payload;
• unbounded volume based on order size;
• may leak commercial data;
• often not needed for normal operations.

Better pattern

log.info("Order line processing completed orderId={} lineCount={} failedLineCount={} durationMs={}",
    orderId,
    lineCount,
    failedLineCount,
    durationMs
);

For failed lines, log bounded details only:

log.warn("Order line processing failed orderId={} lineId={} errorCode={} retryable={}",
    orderId,
    lineId,
    errorCode,
    retryable
);

5. Log Event Value Classification

Classify logs by value.

A log should earn its ingestion cost.

Ask:

What incident question does this log answer that cannot be answered cheaper?

6. Structured Log Field Cost

Structured logging improves queryability but can increase indexing cost.

Not every field needs to be indexed.

Field categories:

Cost mistake:

Index every JSON field by default.

Better:

Define indexed fields explicitly based on incident, audit, and operational search needs.

7. Metric Cardinality Cost

Metric cost often explodes due to labels.

A metric time series is created for each unique combination of labels.

Example:

http_server_requests_total{
  service="quote-api",
  method="POST",
  route="/quotes/{id}",
  status="500"
}

This is bounded.

Dangerous:

http_server_requests_total{
  service="quote-api",
  method="POST",
  path="/quotes/Q-982173912",
  user_id="u-123456",
  request_id="req-abc",
  error_message="timeout after 1023ms to pricing tenant A"
}

This creates unbounded series.

Cardinality multiplication

series_count = routes × methods × status_codes × tenants × versions × pods × regions × extra_labels

Even bounded labels can multiply.

Example:

50 routes × 5 methods × 8 status codes × 100 tenants × 20 pods = 4,000,000 series

This is why tenant label policy matters.

8. High-Cardinality Label Risk Table

Rule of thumb:

Metrics are for aggregation. Logs/traces/audit are for individual identity.

9. Trace Volume Cost

Traces can become expensive when every request creates many spans.

Trace volume drivers:

• high request throughput;
• auto-instrumentation creates many spans;
• DB calls inside loops;
• Redis command per key;
• Kafka/RabbitMQ batch messages with per-message spans;
• background jobs processing large batches;
• verbose span events;
• large attributes;
• always-on 100% sampling;
• long retention.

Trace value is high because it gives causal path and latency breakdown.

But not every successful fast request must be retained.

Useful retention priority:

This naturally leads to sampling strategy, which is covered deeper in Part 049.

10. Sampling as Cost Control

Sampling reduces volume. It also creates blind spots if done badly.

Common approaches:

Cost-safe sampling principle:

Sample normal success aggressively. Preserve abnormal, slow, error, canary, and business-critical paths.

Do not sample audit logs in a way that violates compliance requirements.

11. Retention Cost

Retention should match use case.

Not every telemetry type needs the same retention.

Retention strategy:

Hot storage for fast incident search.
Cold storage/archive for compliance or long-term investigation.
Aggregated metrics for long-term trend.
Raw high-volume details for short windows.

Bad strategy:

Keep all raw debug logs for 1 year.

Better strategy:

Keep operational logs hot for 14-30 days, audit logs according to compliance, high-volume debug logs for hours/days only, and long-term aggregates for trend.

Exact retention must follow internal policy.

12. Dashboard Query Cost

Dashboards can be expensive even if telemetry volume is reasonable.

Cost drivers:

• too many panels;
• long time range by default;
• high refresh frequency;
• unbounded label filters;
• regex over high-cardinality labels;
• expensive log queries on every dashboard refresh;
• joining many sources dynamically;
• panels no one uses;
• per-tenant dashboards generated without governance.

Dashboard cost rules:

An incident dashboard should be fast under pressure.

Slow dashboards are operational risk.

13. Alert Cost and Noise Cost

Alert cost is not only backend evaluation cost.

The largest alert cost is human attention.

Noisy alerts create:

• alert fatigue;
• slower incident response;
• ignored pages;
• unnecessary escalations;
• context switching;
• bad trust in observability;
• hidden real incidents.

Cost-aware alerting means:

• page on symptoms, not every cause;
• group/deduplicate related alerts;
• use burn-rate alerts for SLOs;
• avoid per-pod pages unless directly actionable;
• avoid alerts without runbooks;
• review noisy alerts regularly;
• route ticket-level alerts separately from paging alerts.

An alert that fires often and requires no action is expensive noise.

14. Java/JAX-RS Cost Hotspots

Common application-level cost sources:

For JAX-RS services, a common mistake is logging full request and response bodies “for debugging”.

That creates:

• high cost;
• PII leakage;
• latency overhead;
• storage retention risk;
• noisy incident search.

Use targeted safe debug capture only with approval and expiry.

15. PostgreSQL/MyBatis/JPA/JDBC Cost Concerns

Database observability can be expensive if raw SQL and parameters are logged carelessly.

Cost and risk drivers:

• SQL parameter logging;
• large query plans in logs;
• per-row logs;
• transaction debug logs;
• ORM bind value logs in production;
• high-cardinality SQL labels;
• query text as metric label.

Better patterns:

Do not put raw SQL with user input into metric labels.

16. Kafka/RabbitMQ/Redis/Camunda Cost Concerns

Kafka/RabbitMQ

Avoid logging every successful message at INFO.

Prefer:

• consume rate metric;
• processing latency histogram;
• error count by error code;
• DLQ count;
• lag/queue depth;
• sampled trace for successful flows;
• detailed log only for failures or bounded summaries.

Redis

Avoid labels/fields with raw keys.

Prefer:

• command type;
• key pattern/category;
• cache name;
• hit/miss metric;
• latency histogram;
• slowlog for expensive commands.

Camunda/workflow

Avoid logging every variable or full process payload.

Prefer:

• process definition key/version;
• process instance ID only if policy allows;
• task aging metric;
• failed job count;
• incident count;
• correlation failure logs;
• audit/event for state transitions.

17. OpenTelemetry Collector as Cost Control Point

The collector can help manage cost before telemetry reaches backend.

Possible controls:

Collector policy examples:

Drop health check traces.
Keep all error traces.
Keep all traces above latency threshold.
Drop sensitive headers from span attributes.
Limit noisy instrumentation attributes.

Internal collector capabilities and allowed processors must be verified with platform/SRE.

18. Cost Guardrails

Cost guardrails prevent accidental explosions.

Recommended guardrails:

• telemetry budget per service/team;
• top log volume dashboard;
• top metric cardinality dashboard;
• top expensive dashboard queries;
• alert on sudden ingestion spike;
• alert on sudden series count spike;
• PR checklist for new metrics/logs/traces;
• disallowed metric label list;
• sensitive log field scanner;
• debug logging expiry mechanism;
• retention policy by signal type;
• ownership for each dashboard/alert/metric namespace.

Example ingestion spike alert:

Log ingestion for quote-api increased >3x compared with same time window yesterday for 30 minutes.

This is an observability incident: the telemetry system itself may become expensive or degraded.

19. Cost-Aware Signal Design Patterns

Pattern 1 — Summary log + metric + trace sample

For high-volume success flows:

• use metrics for count/latency;
• use sampled traces for path shape;
• use summary logs for batch/request completion;
• use detailed logs for failure only.

Pattern 2 — Error code instead of raw message

Bad metric:

order_failures_total{error_message="Timeout after 1032ms waiting for pricing tenant A"}

Better:

order_failures_total{error_code="PRICING_TIMEOUT", retryable="true"}

Pattern 3 — Route template instead of raw path

Bad:

path="/quotes/Q-12345/items/I-999"

Better:

route="/quotes/{quoteId}/items/{itemId}"

Pattern 4 — Event summary instead of full payload

Bad:

payload={...large quote/order object...}

Better:

eventName=QuotePriced schemaVersion=3 itemCount=12 totalAmountBucket=large

Pattern 5 — Temporary debug window

Debug mode should include:

• scope: service/route/tenant/correlation ID;
• expiry time;
• actor/change ticket;
• sensitive field protection;
• volume limit;
• automatic revert.

20. Failure Modes

21. Debugging an Observability Cost Spike

Use this flow:

1. Identify which signal spiked: logs, metrics, traces, profiles, dashboards, or alerts.
2. Identify source service/team/environment.
3. Identify whether spike started after deployment/config/debug change.
4. For logs: find top logger, level, event type, route, exception class.
5. For metrics: find top metric namespace and label cardinality.
6. For traces: find top service/span name/route and sampling behavior.
7. For dashboards: find expensive queries and refresh frequency.
8. For alerts: find noisy rules and grouping keys.
9. Decide immediate mitigation: lower log level, disable debug, drop/filter field, reduce sampling, pause dashboard refresh, fix label.
10. Add preventive guardrail: PR checklist, alert, linter, collector rule, dashboard owner review.

Cost spike analysis is incident debugging for the observability platform.

22. Cost vs Correctness Concerns

Reducing telemetry can break correctness of debugging.

Examples:

Cost optimization must preserve required questions:

Can we still detect the problem?
Can we still scope impact?
Can we still correlate logs/traces/metrics?
Can we still prove audit/security events?
Can we still perform RCA after retention window?

23. Security and Privacy Cost Interaction

Security/privacy and cost often align.

Not logging sensitive data reduces:

• breach risk;
• retention burden;
• access-control complexity;
• storage volume;
• purge complexity;
• compliance exposure.

But security controls can also add cost:

• immutable audit retention;
• restricted search backend;
• encryption;
• longer security log retention;
• access audit logs;
• data loss prevention scanning.

Cost decision must not override compliance requirements.

For sensitive telemetry:

Minimize collection.
Mask/redact at source where possible.
Restrict access.
Set policy-driven retention.
Audit access.

24. Operational Ownership

Every high-cost telemetry source should have an owner.

Ownership questions:

• Who owns this metric namespace?
• Who owns this dashboard?
• Who owns this alert rule?
• Who approves new indexed log fields?
• Who reviews high-cardinality labels?
• Who can enable debug logging?
• Who pays or is accountable for telemetry budget?
• Who decides retention exceptions?
• Who handles PII leakage in telemetry?

Without ownership, observability cost becomes everyone’s problem and no one’s responsibility.

25. PR Review Checklist

When reviewing instrumentation/logging changes, ask:

• Is this signal needed for a known production question?
• Is the signal already available elsewhere?
• Is the metric label set bounded?
• Are route/path labels templated?
• Are request/user/order/quote IDs avoided as metric labels?
• Are sensitive fields excluded or masked?
• Could this log fire inside a loop or high-volume path?
• Could retry behavior create log spam?
• Could this span be emitted per item/message/Redis key?
• Is debug logging temporary and scoped?
• Is retention appropriate for the signal type?
• Is the dashboard query cost acceptable?
• Does this change require collector filtering or sampling update?
• Does this preserve enough evidence for incident and RCA?

26. Internal Verification Checklist

Verify internally with CSG/team before assuming details:

• observability backend pricing/cost model;
• log ingestion dashboard;
• metric cardinality dashboard;
• trace volume dashboard;
• retention policy by signal type;
• audit retention/compliance policy;
• indexed log fields policy;
• allowed/disallowed metric labels;
• tenant label policy;
• debug logging policy and expiry mechanism;
• OpenTelemetry Collector filtering/sampling capabilities;
• top noisy services/loggers/metrics;
• top expensive dashboards/queries;
• cost ownership per team/service;
• alert on telemetry ingestion spike;
• process for approving high-cost telemetry;
• PII/secret scanning for logs;
• incident history involving telemetry cost or backend overload.

27. Practical Mastery Exercise

Pick one production service and answer:

1. What is its average daily log volume?
2. What are its top 10 log event types by volume?
3. Which logger produces the most data?
4. Which metrics have the highest cardinality?
5. Are any labels unbounded?
6. What percentage of traces are retained?
7. Are error/slow traces preserved better than normal success traces?
8. Which dashboards query the most data?
9. What telemetry is retained hot vs cold?
10. Which signal could be reduced without hurting incident debugging?
11. Which signal is expensive but must stay because it protects reliability/audit/security?

If no one can answer these, observability cost is unmanaged.

28. Key Takeaways

• Observability cost is driven by volume, cardinality, retention, indexing, query patterns, and human attention.
• Cost optimization must preserve production evidence.
• Logs are expensive when verbose; metrics are expensive when cardinality explodes; traces are expensive when everything is retained blindly.
• High-value signals should be protected; low-value noise should be reduced.
• Sampling, retention, collector filtering, and label governance are cost controls.
• Senior engineers should treat observability cost as architecture and production readiness concern, not only platform billing concern.