Part 061 — Kubernetes Events Operations

Tujuan

Kubernetes Events adalah catatan singkat tentang sesuatu yang terjadi pada object Kubernetes: pod gagal dijadwalkan, image gagal ditarik, volume gagal di-mount, probe gagal, container dibunuh, node menekan pod karena resource pressure, dan sebagainya.

Untuk backend engineer, Events adalah lapisan observability pertama sebelum masuk ke logs, metrics, traces, atau dashboard dependency.

Events membantu menjawab:

• apakah pod gagal karena aplikasi atau scheduler?
• apakah container crash atau tidak pernah sempat start?
• apakah image pull gagal karena registry/auth/network?
• apakah readiness failure berasal dari probe atau aplikasi lambat?
• apakah volume/secret/config gagal di-mount?
• apakah pod di-kill karena liveness probe, eviction, rollout, atau node drain?
• apakah issue muncul setelah deployment terbaru?
• apakah failure isolated ke satu pod/node/namespace atau widespread?

Part ini membahas cara membaca Kubernetes Events secara operasional, bukan sekadar menjalankan kubectl describe.

1. Events Mental Model

Events are not application logs. They are Kubernetes control-plane and node-side signals.

A useful interpretation rule:

Logs tell what the application said. Events tell what Kubernetes tried to do to the workload.

Examples:

Events should be read early in an incident because they often reduce the search space quickly.

2. What Events Are Good For

Events are excellent for detecting lifecycle and orchestration failures:

• scheduling failures
• image pull failures
• container start failures
• mount failures
• probe failures
• restart loops
• node pressure and eviction
• rollout-related pod replacement
• PDB/drain-related disruption
• HPA-related replica changes
• ingress/controller reconciliation messages
• service endpoint changes in some controllers

Events are weaker for:

• application business logic errors
• SQL query performance
• Kafka consumer lag root cause
• RabbitMQ message redelivery logic
• Redis hot-key behavior
• Camunda process incident details
• request-level latency diagnosis
• distributed trace propagation issues

Use Events as the first Kubernetes truth layer, then correlate with logs, metrics, traces, and dependency dashboards.

3. Event Object Fields That Matter

A Kubernetes Event usually has fields similar to:

Operationally, reason, message, involvedObject, lastTimestamp, and count are the most immediately useful.

4. Production-Safe Event Commands

Safe read-only commands:

kubectl get events -n <namespace> --sort-by=.lastTimestamp

kubectl get events -n <namespace> \
  --field-selector involvedObject.kind=Pod \
  --sort-by=.lastTimestamp

kubectl describe pod <pod-name> -n <namespace>

kubectl describe deployment <deployment-name> -n <namespace>

kubectl describe job <job-name> -n <namespace>

kubectl get events -A --sort-by=.lastTimestamp

Use cluster-wide -A carefully in large production clusters. Prefer namespace-specific investigation unless incident impact is unclear.

Useful watch during active incident:

kubectl get events -n <namespace> --watch

Use watch only when needed. For evidence capture, static snapshots are easier to attach to incident notes.

5. Event Triage Flow

The goal is not to memorize every reason. The goal is to map each reason to the next safest investigation step.

6. Common Event Reasons and Meaning

Important: event reason alone is not enough. The message usually contains the real clue.

7. FailedScheduling Events

FailedScheduling means the pod has not been assigned to a node.

Typical messages include:

0/12 nodes are available: 8 Insufficient cpu, 4 node(s) had untolerated taint.

0/8 nodes are available: 8 node(s) didn't match Pod's node affinity/selector.

0/5 nodes are available: persistentvolumeclaim "..." not found.

Operational interpretation:

Backend engineer should usually not modify node capacity directly. The safe action is evidence capture and escalation with precise event messages.

8. Image Pull Events

Image pull events include:

• Pulling
• Pulled
• ErrImagePull
• ImagePullBackOff

Common failure modes:

Production-safe investigation:

kubectl describe pod <pod> -n <namespace>

kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

kubectl get secret -n <namespace>

Do not print secret data. Validate secret existence/reference, not credential contents.

Internal verification checklist:

• is the image tag/digest expected?
• did pipeline push the image?
• does the workload use correct imagePullSecrets?
• does ECR/ACR permission exist?
• is registry reachable from node/pod network?
• is GitOps rendering the expected image?

9. FailedMount Events

FailedMount often means Kubernetes cannot mount a volume, Secret, ConfigMap, projected token, or PVC.

Examples:

MountVolume.SetUp failed for volume "app-config": configmap "quote-service-config" not found

MountVolume.SetUp failed for volume "db-secret": secret "quote-db-secret" not found

Unable to attach or mount volumes: unmounted volumes=[data]

Common causes:

• ConfigMap not created
• Secret not created
• wrong namespace
• typo in manifest/values
• GitOps sync ordering issue
• External Secrets sync failure
• CSI driver issue
• PVC pending
• cloud volume attach failure
• projected service account token issue

Backend relevance:

• missing DB credential can prevent Java app startup
• missing truststore/keystore secret can break TLS
• missing ConfigMap can cause CrashLoopBackOff
• stale or missing external secret can break cloud access
• PVC mount failure can block batch/file processing job

Production-safe commands:

kubectl describe pod <pod> -n <namespace>

kubectl get configmap <name> -n <namespace>

kubectl get secret <name> -n <namespace>

kubectl get pvc -n <namespace>

Do not decode secret values during normal investigation.

10. Unhealthy Probe Events

Unhealthy means kubelet probe failed.

Common messages:

Readiness probe failed: HTTP probe failed with statuscode: 503

Liveness probe failed: Get "http://10.x.x.x:8080/health": context deadline exceeded

Startup probe failed: dial tcp 10.x.x.x:8080: connect: connection refused

Operational interpretation:

Backend-specific causes:

• Java startup slower than configured initial delay
• wrong management port
• wrong health path
• dependency health check blocks readiness
• DB pool initialization slow
• Kafka/RabbitMQ connection attempted during startup
• Redis/Camunda dependency check too strict
• GC pause or CPU throttling causes timeout
• app server bound to localhost instead of container interface

Safe investigation:

kubectl describe pod <pod> -n <namespace>

kubectl logs <pod> -n <namespace> --previous

kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>

Escalate if probe failures correlate with node pressure, CNI issue, ingress controller issue, or platform-wide DNS/dependency failure.

11. BackOff Events

BackOff means Kubernetes is delaying restart attempts after repeated container failures.

Typical clue:

Back-off restarting failed container app in pod quote-service-...

Common backend causes:

• missing environment variable
• invalid configuration
• missing secret/config mount
• JVM exits during bootstrap
• incompatible database migration
• port already unavailable inside container
• application fails to bind expected port
• OOM before readiness
• fatal dependency initialization
• classpath/library mismatch

Investigation order:

1. Read pod events.
2. Read current logs.
3. Read previous logs.
4. Check exit code and reason.
5. Check recent deployment/config/secret changes.
6. Check resource and OOM status.
7. Decide rollback vs config fix vs escalation.

Commands:

kubectl describe pod <pod> -n <namespace>

kubectl logs <pod> -n <namespace> --previous

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

12. Killing Events

Killing can be normal or dangerous.

Normal cases:

• rolling update replaces old pod
• scale down reduces replicas
• node drain evicts pod
• manual deletion by authorized operator

Dangerous cases:

• liveness probe keeps killing app
• pod evicted due to node pressure
• application ignores SIGTERM and receives SIGKILL
• rollout kills too many pods due to bad maxUnavailable/PDB
• HPA scales down while queue consumer still processing

Questions to ask:

• Was there a deployment or scale event?
• Did liveness probe fail before killing?
• Was node drained?
• Did termination grace period expire?
• Did the app shut down gracefully?
• Were in-flight HTTP requests or Kafka/RabbitMQ messages interrupted?
• Did Camunda worker finish/extend/return active jobs safely?

13. Event Retention and Limitations

Kubernetes Events are not long-term audit logs.

Limitations:

• events expire quickly depending on cluster configuration
• repeated events are aggregated by count
• event messages can be truncated
• not every controller emits enough detail
• application-level causes may not appear in events
• event timestamps may need correlation with other systems
• event volume can be high during incidents

Implication:

Capture events early during an incident.

Evidence capture example:

kubectl get events -n <namespace> --sort-by=.lastTimestamp > events-<incident-id>.txt

Only do this according to internal incident evidence policy.

14. Events in Rollout Debugging

During rollout, Events help distinguish:

• new pod cannot schedule
• new image cannot pull
• new config/secret cannot mount
• new pod starts but fails readiness
• new pod crash loops
• old pod being killed normally
• deployment progress deadline exceeded

Sequence:

A rollout is not just kubectl rollout status. It is a sequence of event-backed state transitions.

15. Events and Java/JAX-RS Backend

Kubernetes Events do not understand JAX-RS, but they expose runtime symptoms around it:

Events show the Kubernetes-side symptom. Logs/metrics/traces explain the application-side reason.

16. Events and Dependencies

Events can expose dependency setup failure indirectly:

Be careful: dependency runtime degradation usually appears more clearly in application metrics/traces/logs than Kubernetes Events.

17. EKS/AKS/On-Prem Considerations

EKS

Event clues may relate to:

• VPC CNI IP exhaustion causing scheduling/networking symptoms
• EBS CSI volume attach/mount failures
• ECR auth/image pull failures
• AWS Load Balancer Controller reconciliation events
• node group capacity and spot interruption-related pod movement

AKS

Event clues may relate to:

• Azure CNI/subnet capacity
• Azure Disk CSI mount/attach failures
• ACR pull permission failures
• Application Gateway Ingress Controller events
• node pool capacity and VMSS issues

On-prem/hybrid

Event clues may relate to:

• private registry access
• proxy/NO_PROXY misconfiguration
• internal CA/trust issues
• storage provisioner failures
• corporate DNS problems
• firewall-blocked dependency routes

Internal verification is mandatory. Do not infer platform topology from generic Kubernetes behavior.

18. Safe Mitigation Based on Events

Do not mutate production blindly because an event looks obvious. Use events to narrow the hypothesis, then validate safely.

19. Internal Verification Checklist

For each critical backend workload, verify:

• namespace event visibility
• event retention window
• access to kubectl get events
• access to kubectl describe for Pod/Deployment/Job
• event export into observability stack if available
• dashboard panel for recent Kubernetes events
• alert correlation with events
• common event runbooks
• policy for evidence capture during incident
• policy for reading events across namespaces
• mapping from event reason to owner/escalation path
• known platform-specific event patterns for EKS/AKS/on-prem
• runbook for FailedScheduling
• runbook for FailedMount
• runbook for ImagePullBackOff
• runbook for Unhealthy probe
• runbook for BackOff/CrashLoopBackOff
• runbook for Evicted

CSG/team-specific items to verify internally:

• which observability stack stores Kubernetes events
• whether events are retained beyond Kubernetes default retention
• whether backend engineers have read access to events in production
• which team owns CSI/storage failures
• which team owns registry/image pull failures
• which team owns ingress controller events
• which team owns CNI/networking-related events
• whether incident templates require event snapshots

20. PR Review Checklist

When reviewing Kubernetes workload changes, check whether the PR can create event-visible failure:

• image tag/digest changed?
• imagePullSecrets changed?
• resource requests increased beyond namespace/node capacity?
• node selector/affinity/toleration changed?
• PVC/volume reference changed?
• ConfigMap/Secret reference changed?
• probe path/port/timeout changed?
• lifecycle hook changed?
• termination grace period changed?
• service account changed?
• NetworkPolicy changed?
• ingress/service port changed?
• rollout strategy changed?
• Helm/Kustomize overlay changed only in one environment?

Ask: if this fails, what event would we expect to see, and do we have a runbook for it?

21. Operational Anti-Patterns

Avoid these patterns:

• jumping to application logs before reading pod events
• treating Normal events as irrelevant
• ignoring event count and last timestamp
• assuming CrashLoopBackOff always means code bug
• assuming Pending always means cluster issue
• decoding secrets during routine event investigation
• deleting pods to “fix” unknown failures without root cause
• using kubectl get events -A as first step in known namespace incident
• failing to capture events before retention expires
• ignoring node-related events affecting only one subset of pods

22. Final Mental Model

Kubernetes Events are operational breadcrumbs.

They do not replace logs, metrics, or traces, but they tell you what Kubernetes attempted and where orchestration failed.

Use this sequence:

Symptom
→ Identify workload
→ Read Events
→ Map Event reason to lifecycle phase
→ Correlate with logs/metrics/traces
→ Check recent changes
→ Choose safe mitigation
→ Escalate with evidence when outside backend ownership

For senior backend engineers, Events are a fast way to avoid wrong debugging paths.

A production-ready service owner should know the common Events for their workload, what they mean, what evidence to capture, and which team owns the next action.