Part 050 — Kubernetes Cost and Capacity Management

Kubernetes cost problems rarely start as finance problems.

They usually start as engineering problems:

• inaccurate CPU and memory requests,
• oversized limits,
• idle replicas,
• inefficient node packing,
• uncontrolled log volume,
• unnecessary load balancers,
• expensive NAT traffic,
• unused persistent volumes,
• multi-AZ traffic patterns,
• autoscaling that reacts too late or too aggressively.

For enterprise Java/JAX-RS systems, cost and capacity management is part of production engineering. A service that is reliable but wildly overprovisioned is not mature. A service that is cheap but constantly throttled or OOMKilled is also not mature.

The goal is controlled efficiency: enough capacity to meet SLOs, enough headroom to survive bursts, and enough visibility to know what is waste versus resilience.

CSG/internal note: this part does not assume CSG uses a specific cost platform, FinOps model, EKS/AKS pricing structure, node pool strategy, autoscaler, logging stack, metrics retention policy, or chargeback/showback model. All environment-specific cost, capacity, and ownership details must be verified through internal cloud accounts, cluster metrics, dashboards, IaC, platform/SRE/DevOps teams, finance/FinOps contacts, and production incident history.

1. Core Mental Model

Kubernetes capacity is allocated through requests, constrained through limits, and paid for through infrastructure.

Application behavior
   ↓
Pod resource requests/limits
   ↓
Scheduler placement
   ↓
Node utilization
   ↓
Autoscaling behavior
   ↓
Cloud/on-prem infrastructure cost

The scheduler does not place pods based on actual average usage. It places pods based on requests.

That means a wrong request can waste money even when the application is idle.

High request + low actual usage = poor bin packing / wasted capacity
Low request + high actual usage = noisy neighbors / throttling / eviction risk

Cost optimization is therefore not simply “reduce resources.” It is about matching declared capacity with actual workload behavior and reliability needs.

2. Kubernetes Cost Surface Area

Kubernetes cost comes from more than worker nodes.

Common cost drivers:

• worker nodes or VM instances,
• control plane fees where applicable,
• persistent volumes,
• snapshots and backups,
• load balancers,
• NAT gateways or egress gateways,
• inter-zone traffic,
• cross-region traffic,
• log ingestion,
• log retention,
• metrics ingestion,
• metrics retention,
• trace volume,
• container registry storage and transfer,
• managed database/messaging/cache services,
• private endpoints,
• public IPs,
• idle dev/test clusters,
• over-retained environments.

For a backend engineer, the most directly controllable levers are usually:

• pod requests and limits,
• replica count,
• autoscaling settings,
• logging volume,
• metric cardinality,
• persistent volume sizing,
• load balancer creation,
• topology and cross-zone behavior,
• environment lifecycle.

3. Requests Drive Scheduling and Cost

A CPU or memory request is a scheduling reservation.

Example:

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "1"
    memory: "2Gi"

This tells Kubernetes:

Only schedule this pod on a node that has at least 500m CPU and 1Gi memory allocatable.

Even if the pod uses only 50m CPU most of the day, the scheduler reserves 500m.

If every service overstates requests, cluster utilization looks bad:

Node capacity:       16 vCPU / 64Gi
Scheduled requests:  15 vCPU / 58Gi
Actual usage:         4 vCPU / 22Gi

The cluster appears full to the scheduler but underutilized in reality.

This causes unnecessary scale-out.

4. Limits Affect Runtime Stability

Limits affect runtime behavior.

CPU limit:

• can cause throttling,
• can increase latency,
• can hurt Java GC performance,
• can make readiness/liveness probes time out,
• can reduce Kafka/RabbitMQ consumer throughput.

Memory limit:

• creates a hard ceiling,
• can trigger OOMKilled,
• must include heap and non-heap memory,
• must account for direct memory, metaspace, thread stacks, native memory, JIT, buffers, and agents.

For Java/JAX-RS services:

container memory limit ≠ JVM heap size

A pod with 2Gi memory limit cannot safely use -Xmx2g.

There must be headroom for non-heap memory.

5. Overprovisioning vs Underprovisioning

Overprovisioning

Symptoms:

• low average CPU usage,
• low memory usage relative to request,
• many underutilized nodes,
• low bin packing efficiency,
• idle replicas,
• high cost per request,
• dev/test clusters running 24/7 without need.

Risks:

• unnecessary cloud spend,
• hidden capacity waste,
• autoscaler adds nodes too early,
• teams normalize inefficient defaults.

Underprovisioning

Symptoms:

• OOMKilled,
• CPU throttling,
• high p95/p99 latency,
• frequent GC pressure,
• readiness probe failures,
• consumer lag grows,
• pods evicted under node pressure,
• HPA constantly at max replicas.

Risks:

• production instability,
• failed rollout,
• cascading latency,
• customer-visible outage,
• emergency overcorrection.

The engineering target is not minimum cost. The target is efficient reliability.

6. Request Accuracy

Request accuracy compares declared request with observed usage.

Useful questions:

• What is average usage?
• What is p95 usage?
• What is p99 usage?
• What is peak during rollout?
• What is peak during traffic burst?
• What is usage during batch/consumer spikes?
• What is usage during GC pressure?
• What is usage during dependency degradation?

Do not size only from average usage.

For production Java services, request should account for:

• normal traffic,
• deployment startup spike,
• GC behavior,
• TLS overhead,
• serialization/deserialization,
• Kafka/RabbitMQ bursts,
• connection pool behavior,
• retry storms,
• observability agent overhead,
• safety headroom.

A useful policy:

Use observed p95/p99 + workload-specific headroom, then revisit periodically.

7. Java/JAX-RS Capacity Profile

Java services often have a different capacity profile than small native processes.

Important dimensions:

• heap,
• metaspace,
• direct memory,
• thread stacks,
• JIT compiled code,
• GC overhead,
• HTTP server thread pool,
• DB connection pool,
• Kafka/RabbitMQ consumer concurrency,
• Redis client pool,
• OpenTelemetry agent overhead,
• TLS/session overhead,
• JSON serialization/deserialization cost.

A REST API service may be CPU-bound due to:

• JSON mapping,
• validation,
• crypto/TLS,
• business rules,
• high logging volume,
• excessive object allocation.

A consumer service may be throughput-bound due to:

• batch size,
• downstream DB writes,
• retry/backoff,
• message acknowledgement pattern,
• concurrency level,
• partition count or queue behavior.

Do not copy resource settings from one Java workload type to another.

8. Node Utilization and Bin Packing

Bin packing is the scheduler’s ability to place pods efficiently onto nodes.

Poor bin packing happens when:

• requests are inflated,
• pods have awkward CPU/memory shapes,
• anti-affinity is too strict,
• topology constraints are too restrictive,
• daemonsets consume significant node resources,
• node instance sizes do not match workload shapes,
• too many dedicated node pools fragment capacity.

Example:

Node size: 4 vCPU / 16Gi
Pod request: 2200m CPU / 2Gi

Only one such pod fits by CPU, leaving nearly half the node CPU and most memory unused.

A better shape may be:

• reduce request if measured usage supports it,
• choose larger node type,
• split workload type,
• tune autoscaling,
• isolate only when justified.

Capacity is a packing problem, not only a resource limit problem.

9. Autoscaling and Cost

Autoscaling can reduce waste, but it can also amplify cost if configured poorly.

HPA decisions are influenced by:

• resource requests,
• target utilization,
• metric delay,
• stabilization window,
• scale-up/down policy,
• workload startup time,
• readiness delay,
• max replica limit.

Cluster autoscaling decisions are influenced by:

• unschedulable pods,
• node group size,
• instance type availability,
• node provisioning time,
• pod disruption constraints,
• topology requirements.

Common cost anti-patterns:

• HPA target too low, causing excessive replicas.
• Max replicas very high without traffic/load guardrails.
• Cluster autoscaler adds expensive nodes for poorly shaped requests.
• Scale-down blocked by strict PDB or local storage.
• VPA recommendation ignored for months.
• Queue-based autoscaling scales consumers beyond downstream DB capacity.

Autoscaling must be validated against end-to-end bottlenecks.

10. Scale-to-Zero and Idle Workloads

Scale-to-zero can reduce idle cost for certain workloads, but it is not free.

Good candidates:

• dev/test workloads,
• asynchronous workers with acceptable cold start,
• scheduled/batch workloads,
• low-priority internal tools,
• event-driven consumers where startup latency is acceptable.

Poor candidates:

• customer-facing APIs with strict latency SLO,
• services with slow JVM warmup,
• services requiring warm caches,
• consumers where lag must be near-zero,
• services with expensive connection initialization,
• workflows requiring immediate availability.

For Java/JAX-RS systems, cold start includes:

• JVM startup,
• class loading,
• JIT warmup,
• dependency initialization,
• DB pool creation,
• Kafka/RabbitMQ connection setup,
• cache warmup,
• readiness transition.

The cost saving must be balanced against latency and operational risk.

11. Spot and Preemptible Nodes

Spot/preemptible capacity can reduce compute cost, but it introduces interruption risk.

Suitable workloads:

• stateless replicas with enough redundancy,
• batch jobs that can retry,
• idempotent workers,
• non-critical dev/test workloads,
• horizontally scalable consumers with safe rebalance handling.

Risky workloads:

• single replica services,
• stateful workloads without safe failover,
• long-running non-idempotent jobs,
• latency-critical traffic without enough on-demand baseline,
• workloads with weak graceful shutdown.

For Kafka/RabbitMQ consumers, interruption handling must be correct:

• stop accepting new messages,
• finish or safely abandon in-flight work,
• commit/ack only after successful processing,
• rely on retry/DLQ patterns,
• avoid duplicate side effects.

Spot/preemptible savings require workload design discipline.

12. Reserved Capacity and Baseline Load

For stable baseline workloads, reserved capacity or committed usage may reduce cost.

But from an engineering view, the first question is:

What is our actual baseline load after right-sizing requests and replicas?

Do not reserve capacity based on inflated requests.

Good sequence:

1. Measure actual usage.
2. Right-size requests and limits.
3. Review replica baseline.
4. Review node family and size.
5. Identify stable baseline.
6. Apply reservation/commitment strategy if appropriate.

Reservation is a financial optimization after engineering sizing is reasonably accurate.

13. Storage Cost

Kubernetes storage cost can hide in:

• oversized PVCs,
• unused PVCs after workload deletion,
• retained PVs with Retain reclaim policy,
• snapshots,
• backup retention,
• high-performance disk classes,
• cross-zone replication,
• file shares,
• orphaned volumes.

Common failure:

StatefulSet deleted, PVCs remain, volumes keep billing.

That may be correct for data safety, but it must be intentional.

Questions to ask:

• Which workloads use PVCs?
• Are PVC sizes based on real data growth?
• Are unused PVCs detected?
• What is reclaim policy?
• Who owns backup retention?
• Are snapshots expired?
• Is high-performance storage justified?
• Is managed database cheaper/safer than self-managed storage?

14. Load Balancer Cost

Every Kubernetes Service of type LoadBalancer may provision cloud load balancer resources depending on the environment.

Cost anti-pattern:

kind: Service
spec:
  type: LoadBalancer

used casually for internal services.

Better pattern:

• use ClusterIP for internal service-to-service traffic,
• use Ingress/Gateway for shared HTTP edge routing,
• use dedicated LoadBalancer only when justified,
• consolidate routes where appropriate,
• review public vs internal load balancer need.

A load balancer is not just cost. It is also:

• public exposure risk,
• DNS ownership,
• certificate management,
• firewall/security group/NSG scope,
• operational dependency.

15. NAT Gateway, Egress, and Data Transfer Cost

Egress cost can become significant when pods frequently call external or cross-zone services.

Potential drivers:

• pulling images over public path,
• calling public cloud APIs through NAT,
• cross-AZ service traffic,
• log export,
• metrics export,
• object storage transfer,
• external SaaS calls,
• cross-region replication,
• database traffic across zones or networks.

Private endpoints can reduce exposure and sometimes change data path economics, but they also add cost and operational complexity.

Questions:

• Does pod-to-cloud-service traffic go through NAT?
• Are VPC endpoints/private endpoints used?
• Is DNS resolving to private or public endpoint?
• Is traffic crossing zones unnecessarily?
• Are logs/traces exported across region?
• Is image pull path optimized?

Network cost is often invisible to application teams until it becomes large.

16. Logging, Metrics, and Tracing Cost

Observability is necessary, but uncontrolled observability becomes expensive.

Common drivers:

• excessive INFO logs,
• debug logs left enabled,
• request/response body logging,
• high-cardinality metric labels,
• per-user/per-order/per-quote metric labels,
• unbounded trace sampling,
• long retention for low-value logs,
• duplicate shipping to multiple tools,
• noisy health check logs,
• stack traces repeated at high rate.

For Java/JAX-RS systems, be careful with labels such as:

customerId
quoteId
orderId
requestId
sessionId
email
phone
full URL with dynamic ID

These can create high cardinality, privacy risk, and cost explosion.

Observability should be designed with:

• signal value,
• retention policy,
• sampling strategy,
• privacy constraints,
• cardinality control,
• incident usefulness.

17. Multi-AZ Cost and Resilience Trade-off

Running across multiple availability zones improves resilience, but can increase cost.

Cost contributors:

• cross-zone load balancing,
• inter-zone traffic,
• replicated storage,
• duplicated node baseline,
• multi-AZ managed databases,
• zone-aware autoscaling headroom.

Do not optimize away zone redundancy casually.

Instead ask:

• What is the availability requirement?
• What is the blast radius of one zone loss?
• Are pods spread across zones?
• Are dependencies also multi-AZ?
• Does cross-zone traffic dominate cost?
• Can topology-aware routing reduce unnecessary cross-zone traffic?
• Is the cost justified by SLO/customer impact?

Cost and resilience must be reviewed together.

18. Environment Cost

Non-production environments can dominate waste if unmanaged.

Common waste sources:

• dev/test clusters running 24/7,
• preview environments not cleaned up,
• duplicated stateful dependencies,
• large default node pools,
• excessive log retention in lower environments,
• production-sized replicas in staging,
• old namespaces with orphaned resources,
• unused load balancers and PVCs.

Controls:

• TTL for preview environments,
• scheduled scale-down for dev/test,
• namespace owner labels,
• resource quota,
• cost dashboard by environment,
• automatic cleanup for abandoned branches,
• lower retention outside production,
• explicit exception process.

19. Cost Attribution

Cost attribution requires metadata discipline.

Useful Kubernetes labels:

metadata:
  labels:
    app.kubernetes.io/name: quote-order-service
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: quote-order
    app.kubernetes.io/managed-by: argocd
    cost-center: example
    team: backend
    environment: production

Without ownership labels, cost becomes someone else’s mystery.

Good attribution should answer:

• Which team owns this workload?
• Which product/domain does it support?
• Which environment is it in?
• Which customer/tenant if applicable and allowed?
• Which cost center pays for it?
• Is it production, staging, dev, or preview?

Do not put sensitive identifiers into labels.

20. Cost-Aware PR Review

When reviewing a Kubernetes PR, ask:

• Are requests based on measurement or copied defaults?
• Are limits safe for JVM behavior?
• Is replica count justified?
• Is HPA configured?
• Is max replica bounded?
• Does the workload need dedicated node pool?
• Does affinity reduce bin packing?
• Does this create a new LoadBalancer?
• Does this create persistent storage?
• Does this increase log/metric cardinality?
• Does this introduce cross-zone or public egress?
• Does this require private endpoint or NAT path?
• Are labels sufficient for cost attribution?
• Is the cost worth the reliability/security benefit?

Cost review is not about blocking changes. It is about making cost visible when architecture decisions are made.

21. EKS, AKS, On-Prem, and Hybrid Concerns

EKS

Verify cost drivers such as:

• EC2 node groups,
• Fargate profiles if used,
• EBS/EFS volumes,
• ALB/NLB resources,
• NAT Gateway traffic,
• VPC endpoint cost,
• CloudWatch logs/metrics,
• cross-AZ data transfer,
• ECR storage and scanning,
• Karpenter/Cluster Autoscaler behavior.

AKS

Verify cost drivers such as:

• VMSS/node pool size,
• Azure Disk/Azure Files,
• Azure Load Balancer/Application Gateway,
• NAT Gateway/firewall if used,
• Private Endpoint,
• Azure Monitor/Log Analytics ingestion,
• ACR storage/replication,
• cross-zone traffic,
• Cluster Autoscaler behavior.

On-prem

Cost may show up as:

• hardware capacity,
• virtualization quota,
• storage array capacity,
• network appliance capacity,
• operational labor,
• patching burden,
• over-reserved cluster resources.

On-prem cost is often less visible but not free.

Hybrid

Hybrid cost includes:

• private connectivity,
• cross-network data transfer,
• duplicated environments,
• proxy/firewall capacity,
• observability export,
• operational complexity.

22. Capacity Review Workflow

A practical review loop:

1. List workloads by namespace/team/environment.
2. Collect requests, limits, replicas, and HPA settings.
3. Compare against actual CPU/memory usage p50/p95/p99.
4. Identify over-requested and under-requested workloads.
5. Review OOMKilled and CPU throttling history.
6. Review node utilization and unschedulable events.
7. Review HPA/cluster autoscaler behavior.
8. Review persistent volumes and load balancers.
9. Review log/metric/trace volume.
10. Review cost attribution labels.
11. Tune one workload class at a time.
12. Observe production impact before broad rollout.

Do not mass-edit resource settings across many services without measurement and rollback plan.

23. Failure Modes

24. Debugging Cost and Capacity Incidents

For workload resource behavior:

kubectl top pod -n <namespace>
kubectl top node
kubectl describe pod <pod> -n <namespace>
kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>

For scheduled requests:

kubectl get pods -n <namespace> -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_LIM:.spec.containers[*].resources.limits.memory

For storage:

kubectl get pvc -A
kubectl get pv

For load balancers:

kubectl get svc -A | grep LoadBalancer
kubectl get ingress -A

For events:

kubectl get events -A --sort-by=.lastTimestamp

Then correlate with:

• cloud billing/cost explorer,
• node utilization dashboard,
• autoscaler logs,
• logging/metrics ingestion dashboard,
• network egress metrics,
• storage inventory,
• release/deployment history.

25. Cost Optimization Checklist

Resource sizing

• CPU requests reflect observed usage and headroom.
• Memory requests reflect JVM heap + non-heap behavior.
• CPU limits do not cause unacceptable throttling.
• Memory limits avoid OOMKilled under normal spikes.
• Resource settings differ by workload type.
• VPA recommendations are reviewed if available.

Replicas and autoscaling

• Replica count is justified.
• HPA target is intentional.
• Max replicas are bounded.
• Queue-based scaling respects downstream capacity.
• Scale-down behavior is not blocked unnecessarily.
• Cluster autoscaler/Karpenter behavior is understood.

Node and placement

• Node pool sizes match workload shapes.
• Affinity/anti-affinity does not over-fragment capacity.
• Topology spread is justified.
• Dedicated nodes are used only when needed.
• Spot/preemptible usage is safe for workload semantics.

Cloud resources

• LoadBalancer resources are justified.
• PVCs are actively used.
• Snapshots/backups have retention policy.
• NAT/private endpoint/egress paths are understood.
• Cross-zone/cross-region traffic is reviewed.
• Registry storage and retention are governed.

Observability

• Logs are structured but not excessive.
• Debug logging is disabled in production unless temporary.
• Metric labels avoid high cardinality.
• Trace sampling is intentional.
• Retention differs by environment and value.
• PII/sensitive data is not logged.

Attribution

• Workloads have owner/team labels.
• Environment labels are present.
• Cost center/product/domain labels are present if required.
• Orphaned namespaces/resources are detected.
• Cost dashboard maps spend to accountable owners.

26. Internal Verification Checklist

Verify these in the CSG/team environment:

• Existing resource request/limit standard.
• Whether requests are measured, guessed, or copied.
• JVM memory sizing policy for containers.
• CPU throttling dashboard.
• OOMKilled history.
• HPA/VPA usage.
• Cluster Autoscaler/Karpenter usage for EKS if relevant.
• Cluster Autoscaler/node pool autoscaling usage for AKS if relevant.
• Node pool strategy.
• Spot/preemptible policy.
• Production vs non-production replica standards.
• Dev/test scheduled scale-down policy.
• LoadBalancer ownership and review process.
• Persistent volume inventory and cleanup process.
• Snapshot/backup retention.
• NAT gateway / egress cost visibility.
• Cross-zone/cross-region traffic visibility.
• Logging/metrics/tracing ingestion cost dashboards.
• Metric cardinality guidelines.
• Label standard for cost attribution.
• Cost dashboard and owner mapping.
• FinOps review cadence if any.
• Incident history related to underprovisioning or cost spikes.

27. Key Takeaways

• Kubernetes cost is driven by engineering choices, not only finance settings.
• Requests drive scheduling and bin packing.
• Limits affect runtime stability, especially for JVM workloads.
• Cost optimization without SLO awareness creates outages.
• Reliability without efficiency creates waste.
• Logs, metrics, traces, load balancers, NAT, storage, and cross-zone traffic can cost as much as compute.
• Java/JAX-RS workloads need sizing based on heap, non-heap, GC, thread pools, connection pools, and traffic shape.
• Autoscaling must respect downstream bottlenecks, not only pod CPU.
• Cost attribution requires disciplined labels and ownership metadata.
• A senior backend engineer should treat cost, capacity, reliability, and operability as one system.