Part 040 — Debugging Kubernetes Workloads

Part sebelumnya membahas observability: logs, metrics, traces, events, dashboard, alerting, dan cara membuat production system bisa dipahami.

Part ini membahas debugging Kubernetes workload. Untuk senior backend engineer, debugging Kubernetes bukan menghafal command kubectl. Debugging adalah proses membangun hypothesis, memilih evidence, mengecilkan search space, dan melakukan mitigasi tanpa memperburuk production.

CSG note: jangan mengasumsikan akses kubectl, namespace, cluster role, production permission, ephemeral container permission, port-forward permission, log backend, atau debug policy. Semua harus diverifikasi melalui internal platform/SRE/DevOps/security process. Di production, debugging harus mengikuti access control, audit, dan change-management yang berlaku.

1. Core Concept

Kubernetes debugging menjawab:

Desired state apa yang kita minta?
Observed state apa yang terjadi?
Controller mana yang mencoba merekonsiliasi?
Layer mana yang gagal: image, scheduling, config, runtime, network, storage, identity, dependency, atau aplikasi?

Debugging efektif selalu membandingkan:

2. Production-Safe Debugging Discipline

Debugging production harus punya guardrail.

Prinsip:

1. Observe before changing.
2. Prefer read-only commands first.
3. Do not exec destructive commands in production pod.
4. Do not dump env vars if secrets may appear.
5. Do not restart blindly before capturing evidence.
6. Do not scale to zero unless mitigation plan requires it.
7. Do not patch live resources outside GitOps unless emergency process allows it.
8. Record what changed and why.
9. Confirm customer impact and mitigation success.

Dangerous debugging behavior:

kubectl delete pod <pod>
kubectl edit deployment <deployment>
kubectl exec <pod> -- env
kubectl exec <pod> -- cat /var/run/secrets/...
kubectl scale deployment <deployment> --replicas=0

These may be valid in controlled scenarios, but they are not safe default moves.

3. First Five Commands

For many incidents, start with these:

kubectl get deploy,rs,pod,svc,ep,endpointslice -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers=true
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl rollout status deployment/<deployment> -n <namespace>

Why this order?

1. get shows shape and high-level state.
2. describe shows events and pod-level details.
3. logs shows application/runtime evidence.
4. events shows scheduling/kubelet/controller evidence.
5. rollout status shows Deployment health.

Avoid starting with exec. If the app is not running, exec may not work. If it works, it can tempt you into poking rather than reasoning.

4. Namespace and Context Safety

Before debugging:

kubectl config current-context
kubectl config get-contexts
kubectl get ns

Production incidents often get worse because someone ran a command in the wrong context or namespace.

Use explicit namespace:

kubectl get pods -n quote-order-prod

Avoid relying on implicit namespace during incident.

If internal tooling supports read-only kube context, prefer it for first investigation.

5. Reading kubectl get

Command:

kubectl get pods -n <namespace> -o wide

Look at:

• READY,
• STATUS,
• RESTARTS,
• AGE,
• IP,
• NODE,
• NOMINATED NODE,
• READINESS GATES if present.

Example:

NAME                            READY   STATUS             RESTARTS   AGE
quote-service-7dd9d7d7c5-a1b2c  0/1     CrashLoopBackOff   8          12m
quote-service-7dd9d7d7c5-d3e4f  1/1     Running            0          34m

Interpretation:

Deployment still has some serving capacity, but one replica is crash-looping.
Investigate crashed pod logs and events before deleting it.

6. Reading kubectl describe pod

Command:

kubectl describe pod <pod> -n <namespace>

Important sections:

• node placement,
• labels and annotations,
• container image,
• container state,
• last state,
• exit code,
• reason,
• readiness/liveness/startup probe config,
• environment references,
• volume mounts,
• conditions,
• events.

Example signal:

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

This means the container exceeded its memory limit or was killed under memory pressure.

Example signal:

Readiness probe failed: HTTP probe failed with statuscode: 503

This means app process may run, but Kubernetes will not route Service traffic to that pod.

7. Logs: Current and Previous Container

Commands:

kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl logs <pod> -n <namespace> -c <container>
kubectl logs deployment/<deployment> -n <namespace> --tail=200

Use --previous for:

• CrashLoopBackOff,
• OOMKilled,
• app starts then exits,
• failed migration or startup validation.

Do not only inspect the newest container attempt. The useful exception may be in the previous attempt.

Java-specific log clues

8. Events: Control Plane Evidence

Command:

kubectl get events -n <namespace> --sort-by=.lastTimestamp

Common event patterns:

Events answer questions application logs cannot answer.

If container never started, logs may be empty. Events become primary evidence.

9. Rollout Debugging

Commands:

kubectl rollout status deployment/<deployment> -n <namespace>
kubectl rollout history deployment/<deployment> -n <namespace>
kubectl describe deployment <deployment> -n <namespace>
kubectl get rs -n <namespace> -l app=<app-label>

Look for:

• old ReplicaSet still serving,
• new ReplicaSet unavailable,
• revision number,
• image change,
• config checksum change,
• rollout stuck condition,
• maxUnavailable/maxSurge behavior.

Rollback command exists:

kubectl rollout undo deployment/<deployment> -n <namespace>

But in GitOps environment, direct rollback may be overwritten by controller unless Git desired state is also reverted. Verify internal process first.

10. Resource Debugging with kubectl top

Commands:

kubectl top pod -n <namespace>
kubectl top pod <pod> -n <namespace> --containers
kubectl top node

kubectl top shows current usage, not full history. Use metrics backend for trends.

Useful for:

• CPU saturation,
• memory near limit,
• node pressure hint,
• pod resource comparison.

Limitations:

• does not show CPU throttling directly,
• does not show historical peak,
• may be unavailable if metrics-server is broken,
• memory usage after restart can hide pre-OOM growth.

For Java services, compare:

container memory limit vs JVM heap + non-heap + native memory
CPU usage vs request/limit vs throttling metric

11. Debugging with kubectl exec

Command:

kubectl exec -it <pod> -n <namespace> -- sh

But many hardened images do not contain shell. Distroless images often have no shell and no package manager. That is a security feature, not a defect.

Safer uses of exec:

kubectl exec <pod> -n <namespace> -- cat /etc/resolv.conf
kubectl exec <pod> -n <namespace> -- wget -qO- http://localhost:8080/health

Be careful:

• do not print environment variables if secrets may appear,
• do not install packages into production container,
• do not mutate files unless runbook explicitly allows,
• do not run load-generating commands from production pod.

12. Ephemeral Container Debug

Ephemeral containers allow attaching a debug container to an existing pod.

Example pattern:

kubectl debug -it <pod> -n <namespace> --image=busybox --target=<container>

Useful when:

• app image has no shell,
• need DNS/network tools,
• need inspect network namespace,
• need minimal non-invasive debugging.

Limitations:

• may require special RBAC,
• may be disabled by policy,
• cannot fix the original container,
• production use may need approval/audit.

Use ephemeral containers to observe, not to turn production pod into a mutable pet.

13. Port-Forward Debugging

Command:

kubectl port-forward pod/<pod> 8080:8080 -n <namespace>
kubectl port-forward svc/<service> 8080:80 -n <namespace>

Useful for:

• testing internal endpoint,
• checking metrics path,
• comparing pod-direct vs service path,
• debugging readiness endpoint.

Limitations:

• bypasses ingress/load balancer,
• bypasses some network policy paths,
• may not represent production client flow,
• should not be used as workaround for real traffic.

If port-forward to pod works but ingress fails, suspect:

• Service selector/targetPort,
• ingress route,
• TLS/backend protocol,
• load balancer health check,
• NetworkPolicy,
• cloud security group/NSG.

14. CrashLoopBackOff

CrashLoopBackOff means container repeatedly starts and exits, and kubelet backs off restart attempts.

First commands:

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Common causes:

• app startup exception,
• missing config,
• missing secret,
• DB migration failure,
• dependency required at startup unavailable,
• wrong entrypoint/CMD,
• permission issue with non-root user,
• port binding failure,
• JVM memory error,
• liveness probe killing too early,
• app exits after running one-time task in Deployment.

Debug decision tree:

Did container start? 
  no  -> image/entrypoint/config/permission
  yes -> inspect previous logs
Exit code 137?
  yes -> OOMKilled
Exit code 1?
  yes -> app error/startup validation
Probe events?
  yes -> probe config or slow startup

Java-specific checks:

• main class exists,
• jar path correct,
• config file path correct,
• truststore/keystore mounted,
• DB URL resolves,
• secret name/key exists,
• JVM options compatible with container memory.

15. ImagePullBackOff and ErrImagePull

ErrImagePull is initial pull failure. ImagePullBackOff is repeated pull failure with backoff.

Commands:

kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Common causes:

• image tag does not exist,
• wrong registry path,
• registry auth failure,
• image pull secret missing,
• node cannot reach registry,
• registry outage,
• ECR/ACR permission issue,
• image policy/admission rejected,
• digest mismatch,
• private registry TLS issue.

EKS direction:

• ECR repository exists,
• node/IRSA permission model for pull,
• VPC endpoint/NAT path to ECR,
• security group/NACL path,
• image region/account mismatch.

AKS direction:

• ACR integration,
• managed identity permission,
• private endpoint/DNS for ACR,
• imagePullSecret if not integrated,
• registry firewall.

16. OOMKilled

Signal:

Reason: OOMKilled
Exit Code: 137

Common causes:

• memory limit too low,
• JVM heap too large relative to container limit,
• direct memory growth,
• metaspace growth,
• too many threads,
• large request payload,
• memory leak,
• large cache,
• native library allocation,
• log buffering,
• batch processing too large.

Debug checklist:

• compare memory limit with JVM MaxRAMPercentage,
• inspect memory usage before kill from metrics,
• inspect GC logs if available,
• inspect heap/non-heap/direct memory metrics,
• check recent traffic/payload size,
• check deployment changed memory limit or JVM option,
• check whether OOM happens during startup, steady traffic, or batch job.

Important:

OOMKilled is not always solved by increasing memory.

Increasing memory may be mitigation. Root cause may be leak, unbounded cache, batch size, payload handling, or direct memory.

17. Pending Pod

Pending means pod has not been scheduled or cannot start due to dependencies like volumes.

Commands:

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get nodes

Common causes:

• insufficient CPU/memory,
• nodeSelector mismatch,
• affinity impossible,
• taint without toleration,
• PVC not bound,
• quota exceeded,
• image pull not yet reached,
• topology spread constraint too strict,
• no node in required zone.

Event examples:

0/10 nodes are available: insufficient memory
0/10 nodes are available: node(s) didn't match Pod's node affinity/selector
0/10 nodes are available: pod has unbound immediate PersistentVolumeClaims

Do not debug Pending as an application bug. The app has not run yet.

18. Readiness Failure

Readiness failure means pod is running but should not receive traffic.

Commands:

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl get endpointslice -n <namespace>

Common causes:

• app still starting,
• wrong readiness path,
• wrong port,
• readiness endpoint depends on downstream dependency,
• startup probe missing,
• timeout too low,
• app returns 503 due to internal warmup,
• management port mismatch,
• TLS vs HTTP mismatch.

Debug distinction:

Pod Running + Not Ready = traffic excluded by Service
Pod Running + Ready but users fail = routing/app/runtime issue after readiness

Readiness should answer:

Can this pod safely receive new traffic now?

Not:

Are all dependencies globally perfect?

19. Liveness Failure

Liveness failure means Kubernetes believes container is unhealthy and should be restarted.

Common causes:

• liveness endpoint too strict,
• dependency check inside liveness,
• timeout too low under GC/CPU throttling,
• startup probe missing,
• long stop-the-world pause,
• thread pool starvation,
• deadlock,
• event loop blocked.

Danger:

Bad liveness probe can convert temporary slowness into restart storm.

Debug checks:

• event timestamps,
• liveness config,
• GC pause metrics,
• CPU throttling metrics,
• thread pool saturation,
• application logs around probe failure,
• whether startup probe should absorb startup time.

20. Service Routing Failure

Symptoms:

• service unreachable,
• ingress returns 503,
• pod works via port-forward but service fails,
• service has no endpoints,
• wrong backend receives traffic.

Commands:

kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>
kubectl get pods -n <namespace> --show-labels
kubectl describe svc <service> -n <namespace>

Common causes:

• Service selector does not match pod labels,
• targetPort wrong,
• named port mismatch,
• pod not ready,
• app listening on different port,
• NetworkPolicy blocks traffic,
• service points to old labels after chart refactor,
• headless service expectation misunderstood.

Key invariant:

Service routes to ready endpoints selected by labels.
No matching ready pod = no endpoint = no traffic.

21. Ingress Failure

Common symptoms:

Commands:

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc,endpointslice -n <namespace>
kubectl logs <ingress-controller-pod> -n <ingress-namespace>

Check:

• host rule,
• path rule,
• IngressClass,
• TLS secret,
• backend service name,
• backend service port,
• rewrite annotation,
• backend protocol,
• ingress controller logs,
• cloud load balancer target health.

22. DNS Failure

Java symptoms:

UnknownHostException
Name or service not known
Temporary failure in name resolution

Commands:

kubectl exec <pod> -n <namespace> -- cat /etc/resolv.conf
kubectl exec <pod> -n <namespace> -- nslookup <service>.<namespace>.svc.cluster.local
kubectl get svc -n <namespace>
kubectl get pods -n kube-system -l k8s-app=kube-dns

If image has no DNS tools, use ephemeral container if allowed.

Check:

• service exists,
• namespace correct,
• DNS search path,
• CoreDNS health,
• NetworkPolicy allows DNS egress,
• private DNS zone for cloud/private endpoints,
• ndots behavior,
• typo in hostname.

23. ConfigMap and Secret Failure

Symptoms:

• startup exception,
• missing file,
• missing env var,
• FailedMount event,
• wrong credential,
• app uses stale config.

Commands:

kubectl describe pod <pod> -n <namespace>
kubectl get configmap -n <namespace>
kubectl get secret -n <namespace>

Be careful with secrets. Do not casually decode or print secret values.

Check:

• object exists,
• key exists,
• namespace correct,
• volume mount path correct,
• env var reference correct,
• immutable config behavior,
• restart-on-config-change pattern,
• ExternalSecret sync status if used,
• sealed secret or secret manager sync status if used.

24. RBAC and Identity Failure

Kubernetes RBAC symptom:

forbidden: User "system:serviceaccount:..." cannot get resource ...

Cloud identity symptom:

AccessDenied
Unauthorized
Forbidden
NoCredentialProviders
ManagedIdentityCredential authentication unavailable

Debug direction:

kubectl auth can-i get pods --as=system:serviceaccount:<ns>:<sa> -n <ns>
kubectl get serviceaccount <sa> -n <ns> -o yaml
kubectl describe pod <pod> -n <namespace>

Check:

• ServiceAccount assigned to pod,
• automount token setting,
• Role/RoleBinding,
• ClusterRole/ClusterRoleBinding,
• IRSA annotation on EKS,
• Azure Workload Identity labels/annotations,
• cloud trust policy/federated credential,
• SDK credential chain,
• token audience.

25. NetworkPolicy Blocked

Symptoms:

• connection timeout,
• DNS timeout,
• service unreachable only from some pods,
• dependency reachable from debug namespace but not app namespace.

Commands:

kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get pods -n <namespace> --show-labels

Check:

• default deny policy,
• podSelector match,
• namespaceSelector labels,
• egress to DNS,
• egress to DB/Kafka/RabbitMQ/Redis,
• egress to private endpoint/cloud service,
• ingress from ingress controller namespace,
• CNI actually enforces NetworkPolicy.

NetworkPolicy failure often looks like app timeout.

26. PVC and Mount Failure

Symptoms:

• pod Pending,
• FailedMount,
• app file path missing,
• permission denied on mounted volume.

Commands:

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get storageclass

Check:

• PVC bound,
• StorageClass exists,
• access mode compatible,
• zone topology,
• CSI driver healthy,
• reclaim policy,
• mount path,
• filesystem permissions for non-root user.

27. HPA Not Scaling

Commands:

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>
kubectl top pod -n <namespace>

Common causes:

• metrics server unavailable,
• missing CPU request,
• metric target wrong,
• custom metric missing,
• external metric adapter issue,
• maxReplicas too low,
• scale behavior too conservative,
• stabilization window,
• pod pending due to cluster capacity,
• Cluster Autoscaler/Karpenter delay.

For Kafka/RabbitMQ consumers, CPU may be bad scaling signal. Queue lag or message age may be better if available.

28. Debugging Workflow by Symptom

Symptom: users get 503

Ingress status/logs
Service endpoints
Pod readiness
Deployment available replicas
Recent rollout
NetworkPolicy

Symptom: users get 504

Ingress timeout
App request logs/traces
Downstream latency
DB pool wait
Kafka/RabbitMQ publish or consume latency
CPU throttling/GC pause

Symptom: pod restarts

describe pod
previous logs
exit code
OOMKilled or probe failure
resource metrics
recent config/image change

Symptom: pod pending

describe pod events
resource requests
node selector/affinity/taints
PVC binding
quota
cluster autoscaler

Symptom: cloud SDK access fails

ServiceAccount
IRSA/Azure Workload Identity config
cloud permission
private endpoint DNS
egress policy
SDK credential chain

29. Internal Verification Checklist

For CSG/team verification, check:

• permitted kube contexts,
• production access model,
• read-only vs write permissions,
• namespace naming convention,
• standard debug commands,
• whether kubectl exec is allowed in production,
• whether ephemeral containers are allowed,
• whether port-forward is allowed,
• log backend and query convention,
• metrics dashboard location,
• tracing tool and trace ID propagation,
• runbook repository,
• incident escalation path,
• GitOps emergency patch process,
• rollback procedure,
• service ownership mapping,
• ingress controller namespace,
• platform/SRE contact path,
• security restrictions for secret inspection,
• audit requirements for production access,
• known failure modes from previous incidents.

30. Debugging Checklist

Use this checklist during workload incidents:

[ ] Confirm cluster context and namespace.
[ ] Identify impacted service, deployment, pod, and version.
[ ] Check deployment/replicas/pods/service/endpoints.
[ ] Describe failing pod.
[ ] Capture events sorted by time.
[ ] Capture current and previous logs.
[ ] Check rollout status and recent revision.
[ ] Check resource usage and restart count.
[ ] Check readiness/liveness/startup probe behavior.
[ ] Check config/secret mount/reference errors.
[ ] Check service selector and endpoint slice.
[ ] Check ingress/gateway/backend health if traffic-facing.
[ ] Check DNS if hostname resolution fails.
[ ] Check NetworkPolicy if timeout is selective.
[ ] Check RBAC/identity if access denied.
[ ] Check PVC/storage if pod pending or mount failing.
[ ] Check HPA/autoscaler if capacity/scaling is involved.
[ ] Check dashboard/traces for customer impact.
[ ] Identify recent deployment/config/platform change.
[ ] Apply mitigation through approved process.
[ ] Prove recovery using metrics/logs/traces/events.
[ ] Record finding for postmortem/runbook update.

31. PR Review Checklist

When reviewing changes that affect debuggability:

• Are labels consistent so pods/services/endpoints can be linked?
• Is app version exposed through label, metric, or endpoint?
• Does the service emit structured logs?
• Is correlation ID propagated?
• Are probes debuggable and not misleading?
• Are startup errors explicit?
• Are config validation errors actionable?
• Are dependency timeouts visible?
• Are metrics available for new failure modes?
• Is dashboard updated?
• Is alerting updated?
• Is runbook updated?
• Are GitOps rollback steps clear?
• Are security constraints compatible with debugging needs?
• Are secrets protected from accidental debug exposure?

32. Senior Engineer Mental Model

Weak debugging:

Restart it and see if it works.

Stronger debugging:

What changed?
What layer is failing?
What signal proves it?
What is the safest mitigation?
What evidence proves recovery?
What should be added so this is faster next time?

Kubernetes debugging is not command memorization.

It is layered systems reasoning under operational constraints.

33. Summary

Key takeaways:

• Start with desired state vs observed state.
• Use read-only evidence before mutation.
• describe, logs --previous, and events are essential for pod failure.
• CrashLoopBackOff, ImagePullBackOff, OOMKilled, Pending, readiness failure, and service routing failure have different debug paths.
• Java/JAX-RS failures often surface as Kubernetes symptoms but originate in JVM, config, dependency, or shutdown behavior.
• In GitOps environments, direct cluster patches can drift or be reverted.
• Production-safe debugging requires access discipline, audit awareness, and measurable recovery proof.

Next part: Common Kubernetes Failure Modes.