Part 041 — Common Kubernetes Failure Modes

Failure mode analysis is the difference between “I know kubectl commands” and “I can reason during a production incident”.

In enterprise Java/JAX-RS systems, Kubernetes failure rarely comes from one isolated object. A visible symptom often crosses several layers:

Git commit
  -> CI image build
  -> registry
  -> GitOps/Helm/Kustomize
  -> Kubernetes API
  -> scheduler
  -> node/kubelet/runtime
  -> pod lifecycle
  -> service/endpoints
  -> ingress/load balancer/DNS
  -> Java runtime
  -> PostgreSQL/Kafka/RabbitMQ/Redis/Camunda/cloud dependency

This part focuses on the failure modes you will repeatedly see in production-like Kubernetes environments. The goal is not memorizing every error string. The goal is building a triage map: what signal means what, what layer owns the issue, what to inspect first, what not to do blindly, and how to reduce blast radius.

CSG/internal note: do not assume a specific CSG cluster topology, namespace design, ingress controller, GitOps tool, CNI, registry, cloud provider, or runbook. Treat every “how it is configured in CSG” point as an internal verification item.

1. Failure Mode Mental Model

A Kubernetes symptom is usually an observed state mismatch.

Desired state says:

Run 4 replicas of service quote-api using image X, config Y, secret Z,
with 1 CPU request, 2Gi memory limit, exposed through Service S and Ingress I.

Observed state says:

Only 2 pods are ready.
One pod is CrashLoopBackOff.
Two pods cannot pull image.
Service has no endpoints.
Ingress returns 503.

The senior-engineer question is not “what command do I run?”. It is:

Which reconciliation boundary is failing?

1. Desired state invalid?
2. Image unavailable?
3. Scheduler cannot place pod?
4. Node/runtime cannot start container?
5. App starts but is unhealthy?
6. Pod is healthy but service does not route?
7. Service routes but ingress/load balancer fails?
8. Request reaches app but dependency fails?
9. Autoscaler/rollout/controller is stuck?

That framing prevents random command execution during incident pressure.

2. Fast Triage Sequence

A practical initial sequence:

kubectl get deploy,rs,pod,svc,ingress -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get endpointslice -n <namespace>
kubectl rollout status deploy/<deployment> -n <namespace>
kubectl describe deploy <deployment> -n <namespace>

Use this order to separate:

• controller-level failure,
• scheduler failure,
• image/runtime failure,
• application failure,
• readiness/routing failure,
• ingress/network failure.

Do not start by exec-ing into pods unless you already know the pod is running and the issue is runtime/network/config inspection.

3. CrashLoopBackOff

CrashLoopBackOff means the container process repeatedly exits and kubelet is backing off before restarting it again.

It does not mean Kubernetes itself is broken. Usually the app process fails.

Common causes:

• Java process exits during startup.
• Missing environment variable.
• Missing config file.
• Wrong secret value.
• Cannot connect to PostgreSQL/Kafka/RabbitMQ/Redis/Camunda/cloud service during startup.
• Database migration failure.
• JVM OOM before readiness.
• Invalid JVM option.
• Entrypoint script fails.
• Permission denied writing to filesystem.
• App binds to wrong port.
• Health probe kills process repeatedly.

Basic inspection:

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous

Important distinction:

Current logs  = logs from current container attempt
Previous logs = logs from prior failed container attempt

For Java/JAX-RS services, always check:

• exception stack trace,
• framework startup logs,
• failed config binding,
• port binding error,
• datasource initialization,
• Kafka/RabbitMQ consumer initialization,
• Flyway/Liquibase migration error,
• illegal JVM option,
• OutOfMemoryError,
• permission denied under non-root container.

Dangerous anti-pattern

Do not blindly increase restart delay, memory limit, or disable probes without understanding why the process exits.

CrashLoopBackOff is a symptom. The process exit reason is the root signal.

Java-specific examples

Missing DB password
  -> application fails startup
  -> container exits with code 1
  -> CrashLoopBackOff

Wrong JVM flag
  -> JVM refuses to start
  -> container exits immediately
  -> CrashLoopBackOff

Read-only root filesystem but app writes temp file to /tmp without writable mount
  -> startup fails
  -> CrashLoopBackOff

CSG internal verification checklist

• Which services fail fast on dependency unavailability?
• Are DB migrations executed inside app startup or separate Job?
• Are config validation errors explicit in logs?
• Are secrets mounted as env var or volume?
• Are startup failures visible in dashboards/alerts?
• Are previous logs retained long enough for incident analysis?

4. ImagePullBackOff and ErrImagePull

ErrImagePull means kubelet failed to pull the image. ImagePullBackOff means it keeps retrying with backoff.

Common causes:

• image tag does not exist,
• image registry unavailable,
• wrong registry URL,
• missing imagePullSecret,
• expired registry credential,
• permission denied to ECR/ACR/private registry,
• node cannot reach registry due to network/proxy/firewall,
• digest mismatch or deleted artifact,
• GitOps updated manifest before image was pushed,
• cross-region registry replication lag.

Inspect:

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get secret -n <namespace>

Look for event messages like:

Failed to pull image
pull access denied
repository does not exist
manifest unknown
no basic auth credentials
403 Forbidden
TLS handshake timeout

CI/CD race condition

A common enterprise failure:

1. Pipeline builds image.
2. Pipeline updates Helm values/GitOps manifest.
3. GitOps sync applies Deployment.
4. Registry push or replication has not completed yet.
5. Nodes attempt pull.
6. ImagePullBackOff.

Prevent with:

• push image before manifest update,
• verify image digest exists,
• use immutable digest promotion,
• gate GitOps sync on artifact availability.

EKS-specific checks

• ECR repository exists.
• Node IAM role or pull-through permission is valid.
• Private subnet has route/VPC endpoint/NAT to reach ECR.
• ECR API and ECR DKR endpoint connectivity exists for private clusters.

AKS-specific checks

• AKS is attached to ACR or has valid pull secret.
• Managed identity has AcrPull permission.
• Private ACR endpoint DNS resolves correctly.
• Firewall/NSG/UDR permits registry access.

CSG internal verification checklist

• What registry is authoritative for production images?
• Is deployment pinned by tag or digest?
• Are image tags immutable?
• Is image promotion separate from rebuild?
• Who owns registry credentials?
• Are registry outages part of incident runbooks?

5. OOMKilled

OOMKilled means the container exceeded its memory limit and the kernel killed the process.

Inspect:

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
kubectl top pod <pod> -n <namespace>

Common causes in Java services:

• heap too large relative to container limit,
• direct memory not budgeted,
• metaspace not budgeted,
• thread stack memory not budgeted,
• native libraries consuming memory,
• Netty/direct buffers,
• large request payloads,
• large JSON serialization/deserialization,
• connection pool too large,
• unbounded cache,
• Kafka/RabbitMQ batch size too large,
• memory leak,
• sidecar consuming memory inside same pod budget assumption.

A JVM memory budget is not only heap:

container memory limit
  = Java heap
  + metaspace
  + thread stacks
  + direct memory
  + code cache
  + GC/native overhead
  + JNI/native libraries
  + process overhead

A bad configuration:

resources:
  limits:
    memory: "512Mi"

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-Xmx512m"

This leaves no room for metaspace, thread stacks, direct memory, or native overhead.

Better pattern:

memory limit: 1024Mi
heap target: 50-70% depending on workload
leave headroom for non-heap/native memory
observe actual RSS, GC, direct memory, thread count

Failure signatures

• Kubernetes reports OOMKilled.
• App logs may stop abruptly without Java stack trace.
• JVM may not produce heap dump unless configured and enough disk exists.
• Pod restarts during high traffic or large batch processing.
• Latency rises before kill due to GC pressure.

Dangerous anti-pattern

Increasing memory limit may hide the problem but worsen node pressure and cost. It is valid as mitigation, not always root fix.

CSG internal verification checklist

• What are standard memory request/limit ratios for Java services?
• Are JVM heap settings derived from container limit?
• Are heap dumps enabled or disabled in production?
• Where would heap dumps be written?
• Are Kafka/RabbitMQ consumer batch sizes bounded?
• Are JSON payload sizes bounded?
• Are memory dashboards available per pod/container?

6. CPU Throttling

CPU throttling happens when a container with CPU limit tries to use more CPU than allowed.

Symptoms:

• p99 latency spikes,
• request timeout,
• GC takes longer,
• readiness probe timeout,
• thread pool backlog,
• consumer lag grows,
• CPU usage seems “below node capacity” but app is still slow.

Important distinction:

CPU request = scheduling reservation
CPU limit   = runtime ceiling

For Java services, CPU limits can cause surprising latency issues because JVM, GC, JIT, request handlers, TLS, JSON serialization, and compression all compete under the same quota.

Inspect metrics:

container_cpu_cfs_throttled_seconds_total
container_cpu_cfs_periods_total
container_cpu_usage_seconds_total

Operational clues:

• CPU throttling correlates with latency spike.
• GC pauses stretch under CPU pressure.
• Startup probe fails because CPU-starved startup is slow.
• Kafka/RabbitMQ consumers cannot keep up despite low replica count.

Trade-off

No CPU limit:

• better burst behavior,
• lower throttling risk,
• possible noisy-neighbor impact.

CPU limit:

• stronger isolation,
• better cost/control boundaries,
• higher throttling risk if badly sized.

CSG internal verification checklist

• Are CPU limits required by platform policy?
• Are latency-sensitive Java APIs exempt from tight CPU limits?
• Are throttling metrics available?
• Are probes tuned for CPU-starved startup?
• Are consumer autoscaling rules based on lag/backlog instead of CPU only?

7. Pod Pending

Pending means the pod has not been scheduled or is waiting for required resources before running.

Common causes:

• insufficient CPU/memory on nodes,
• node selector matches no nodes,
• affinity/anti-affinity impossible,
• topology spread constraint impossible,
• taints not tolerated,
• PVC not bound,
• image pull secret dependency not ready,
• quota exceeded,
• LimitRange violation,
• no nodes in required zone,
• autoscaler cannot provision matching node.

Inspect:

kubectl describe pod <pod> -n <namespace>
kubectl get nodes --show-labels
kubectl describe nodes
kubectl get quota -n <namespace>
kubectl get limitrange -n <namespace>
kubectl get pvc -n <namespace>

Look at scheduler events:

0/10 nodes are available: insufficient memory
node(s) didn't match Pod's node affinity/selector
node(s) had taint that the pod didn't tolerate
pod has unbound immediate PersistentVolumeClaims

Senior reasoning

A Pending pod is often not an application bug. It is a placement/capacity/policy mismatch.

Do not change application image or probes when the scheduler has not even placed the pod.

CSG internal verification checklist

• What node pools exist?
• Are node labels documented?
• Which workloads require dedicated nodes?
• Are quotas/LimitRanges enforced per namespace?
• Is cluster autoscaler/Karpenter enabled?
• Are PVC zones compatible with pod scheduling?

8. NodeNotReady

NodeNotReady means Kubernetes considers a node unhealthy or unreachable.

Impact:

• pods on that node may become unavailable,
• endpoints may be removed,
• workloads may be rescheduled after eviction timing,
• PDB may slow disruption,
• stateful workloads may require careful recovery,
• latency/error rate may spike if replicas were not spread.

Common causes:

• kubelet down,
• container runtime failure,
• node disk pressure,
• memory pressure,
• network plugin failure,
• cloud VM failure,
• node reboot,
• CNI issue,
• certificate issue,
• security patch/restart,
• spot/preemptible interruption.

Inspect:

kubectl get nodes
kubectl describe node <node>
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Workload design impact

A well-designed stateless Java service should tolerate one node failure when:

• replicas > 1,
• pods are spread across nodes/zones,
• PDB is configured sanely,
• readiness removes bad pods,
• clients retry safely,
• dependencies are not node-local.

CSG internal verification checklist

• Are critical services spread across zones/nodes?
• Are PDBs defined?
• Are node failures tested?
• Are spot/preemptible nodes used?
• Are stateful workloads protected from single-node dependency?
• Who owns node remediation?

9. DNS Failure

DNS failure in Kubernetes can appear as application failure:

java.net.UnknownHostException
Temporary failure in name resolution
connection timeout to service name
Kafka bootstrap DNS failure
Redis host unresolved
cloud private endpoint unresolved

Common causes:

• CoreDNS unhealthy,
• wrong service name,
• wrong namespace,
• wrong search domain assumptions,
• excessive ndots causing unexpected queries,
• DNS caching stale result,
• headless service expectation wrong,
• ExternalName issue,
• private DNS zone misconfigured,
• hybrid DNS forwarding issue,
• NetworkPolicy blocks DNS egress.

Inspect from a debug pod:

kubectl run dns-debug -n <namespace> --rm -it --image=busybox:1.36 -- sh
nslookup <service>
nslookup <service>.<namespace>.svc.cluster.local
cat /etc/resolv.conf

Inspect CoreDNS:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system deploy/coredns

Java-specific concern

The JVM and application libraries may cache DNS results. A Kubernetes DNS change may not be picked up as quickly as expected depending on JVM/security/network cache settings and client connection pooling.

CSG internal verification checklist

• Is DNS egress allowed by NetworkPolicy?
• Are service names fully qualified in config?
• Are private DNS zones used for cloud services?
• Are hybrid DNS forwarders documented?
• Are DNS failures visible in logs/metrics?
• Are JVM DNS cache settings understood?

10. Service Has No Endpoint

A Service with no endpoints means Kubernetes has no ready pod backing that Service.

Symptoms:

• Ingress returns 503.
• Internal callers get connection refused/timeout.
• Service DNS resolves but traffic goes nowhere.
• kubectl get endpointslice shows empty or missing endpoints.

Common causes:

• Service selector does not match pod labels,
• pods exist but are not Ready,
• readiness probe failing,
• targetPort mismatch,
• deployment has zero replicas,
• rollout stuck,
• pods crash before readiness,
• namespace mismatch.

Inspect:

kubectl get svc <service> -n <namespace> -o yaml
kubectl get pods -n <namespace> --show-labels
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>
kubectl describe pod <pod> -n <namespace>

Key distinction

Service DNS resolves
  does not mean
Service has healthy backends

CSG internal verification checklist

• Are labels standardized?
• Are Helm/Kustomize templates generating selectors correctly?
• Are readiness probes too strict or too weak?
• Are targetPort and containerPort aligned?
• Are endpoint slices monitored?

11. Ingress 404, 502, 503, 504

Ingress failures are often misread because the status code may come from different layers.

Approximate meaning:

404 = no matching ingress route or app returned not found
502 = ingress reached backend but upstream connection/protocol failed
503 = no healthy backend/endpoints or upstream unavailable
504 = upstream timeout

But exact meaning depends on ingress controller and gateway implementation.

Inspect:

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc -n <namespace>
kubectl get endpointslice -n <namespace>
kubectl logs -n <ingress-namespace> <ingress-controller-pod>

Check:

• host rule,
• path rule,
• path type,
• rewrite annotation,
• backend service name,
• backend service port,
• TLS secret,
• ingress class,
• controller ownership,
• timeout annotation,
• backend protocol HTTP/HTTPS,
• service endpoints,
• app context path.

Java/JAX-RS-specific traps

• App serves under /api but ingress rewrites incorrectly.
• App expects X-Forwarded-Proto for redirects/security.
• TLS terminates at ingress but app thinks request is HTTP.
• JAX-RS base path differs from ingress path.
• Management port is accidentally exposed instead of application port.
• Long-running endpoint exceeds ingress timeout.

CSG internal verification checklist

• Which ingress controller is used?
• Are NGINX annotations or Gateway API routes used?
• What are default timeouts/body limits?
• Where is TLS terminated?
• Are forwarded headers trusted by Java service?
• Are ingress logs accessible to backend engineers?

12. Secret Missing or Wrong Secret

Secret-related failures often look like application failures.

Common symptoms:

• pod fails to start,
• environment variable missing,
• mounted file missing,
• app authentication fails,
• DB connection rejected,
• cloud SDK credential fails,
• TLS keystore/truststore missing,
• secret rotation breaks running service.

Inspect:

kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
kubectl get deploy <deployment> -n <namespace> -o yaml

Avoid printing secret values during incident handling unless explicitly allowed by security policy.

Common causes:

• secret name mismatch,
• key mismatch inside secret,
• secret created in wrong namespace,
• ExternalSecret sync failure,
• wrong secret version,
• base64 encoding mistake,
• app expects file but manifest injects env var,
• app expects env var but manifest mounts file,
• rotation happened without reload/restart plan.

CSG internal verification checklist

• Is secret source Kubernetes Secret, External Secrets, AWS Secrets Manager, Azure Key Vault, or another vault?
• Who owns secret rotation?
• Is secret reload supported or restart required?
• Are secret sync failures alerted?
• Are secret values prevented from logs and crash dumps?

13. ConfigMap Missing or Wrong Config

Config failure can produce silent correctness issues, not only startup failure.

Examples:

• wrong feature flag,
• wrong endpoint URL,
• wrong Kafka topic,
• wrong RabbitMQ exchange/queue,
• wrong Redis key prefix,
• wrong timeout/retry,
• wrong database schema name,
• wrong Camunda worker topic,
• wrong cloud region,
• wrong base path.

Inspect:

kubectl get configmap -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get deploy <deployment> -n <namespace> -o yaml

Dangerous pattern

A pod may stay healthy while doing the wrong thing because config is semantically wrong but syntactically valid.

That is why startup validation and runtime metrics matter.

CSG internal verification checklist

• Are configs validated at startup?
• Are environment overlays reviewed?
• Are config changes audited through GitOps?
• Are wrong-topic/wrong-endpoint failures detectable?
• Are config changes tied to deployment history?

14. RBAC Denied

RBAC failure appears as Forbidden from the Kubernetes API.

Common cases:

• app tries to read Kubernetes resources,
• operator/controller lacks permission,
• GitOps controller cannot apply resource,
• CI/CD service account lacks namespace permission,
• workload service account token disabled,
• wrong namespace binding,
• ClusterRole vs Role mismatch.

Inspect:

kubectl auth can-i <verb> <resource> -n <namespace> --as=system:serviceaccount:<namespace>:<serviceaccount>
kubectl get role,rolebinding,clusterrole,clusterrolebinding -n <namespace>
kubectl describe sa <serviceaccount> -n <namespace>

Senior concern

Do not fix RBAC by granting cluster-admin. That turns an operational issue into a security incident waiting to happen.

CSG internal verification checklist

• Which workloads need Kubernetes API access?
• What permissions do GitOps controllers have?
• What permissions do CI runners have?
• Are namespace-scoped roles preferred?
• Are RBAC exceptions reviewed by security/platform?

15. PVC Pending

PVC Pending means storage could not be bound/provisioned.

Common causes:

• StorageClass missing,
• CSI driver unavailable,
• requested access mode unsupported,
• requested size invalid or quota exceeded,
• zone mismatch,
• dynamic provisioning failure,
• cloud storage permission failure,
• on-prem storage backend unavailable.

Inspect:

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass
kubectl get pv
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Impact:

• StatefulSet pod stuck Pending,
• Job cannot start,
• Redis/RabbitMQ/PostgreSQL/Kafka-like workload unavailable,
• migration job stuck,
• backup/restore workflow blocked.

CSG internal verification checklist

• Which StorageClasses are supported?
• What access modes are allowed?
• Are volume zones understood?
• Is CSI driver managed by platform/cloud provider?
• Are backup/restore procedures documented?
• Are stateful workloads self-managed or managed service?

16. NetworkPolicy Blocked

NetworkPolicy failures usually look like timeout.

Symptoms:

• service DNS resolves but connection times out,
• DB/Kafka/RabbitMQ/Redis unreachable,
• cloud endpoint unreachable,
• DNS itself fails because egress to CoreDNS is blocked,
• ingress to pod blocked,
• only some namespaces can connect.

Inspect:

kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get pods -n <namespace> --show-labels

Use a debug pod only if allowed by policy:

nc -vz <host> <port>
nslookup <host>
curl -v <url>

Common mistake

Applying default deny without explicitly allowing:

• DNS egress,
• ingress from ingress controller namespace,
• egress to PostgreSQL,
• egress to Kafka/RabbitMQ/Redis,
• egress to cloud private endpoints,
• egress to observability collectors.

CSG internal verification checklist

• Is default deny enforced?
• Are namespace labels stable?
• Are required egress paths documented?
• Are DB/message broker/cache ports explicitly allowed?
• Are network policies tested before production?

17. HPA Not Scaling

HPA failure may be silent until load rises.

Common causes:

• metrics-server unavailable,
• resource requests missing,
• CPU utilization target cannot compute,
• wrong scale target reference,
• custom metrics adapter failing,
• external metrics unavailable,
• minReplicas/maxReplicas too restrictive,
• stabilization window delays scale,
• queue-based scaling not configured,
• app bottleneck not CPU/memory.

Inspect:

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>
kubectl top pod -n <namespace>
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<namespace>/pods

For Kafka/RabbitMQ consumers, CPU may not represent backlog. Queue depth/consumer lag is often a better scaling signal.

CSG internal verification checklist

• Are HPAs using CPU, memory, custom metrics, or external metrics?
• Are resource requests set?
• Are consumer lag/backlog metrics available?
• Is KEDA used?
• Are maxReplicas and downstream capacity aligned?
• Are scale events visible in dashboards?

18. Rollout Stuck

A rollout is stuck when the Deployment cannot progress to desired updated replicas.

Common causes:

• new pods fail readiness,
• new pods crash,
• image pull failure,
• insufficient capacity,
• bad maxUnavailable/maxSurge,
• PDB or quota blocks progress,
• readiness probe too strict,
• dependency unavailable,
• migration incompatibility,
• long startup beyond progress deadline.

Inspect:

kubectl rollout status deploy/<deployment> -n <namespace>
kubectl describe deploy <deployment> -n <namespace>
kubectl get rs -n <namespace>
kubectl get pods -n <namespace>
kubectl describe pod <new-pod> -n <namespace>

Rollback:

kubectl rollout undo deploy/<deployment> -n <namespace>

But in GitOps environments, imperative rollback may be overwritten unless Git desired state is also reverted.

Senior concern

Rollback is not always safe if the new deployment ran database migrations, changed message schema, changed Redis key format, or changed workflow behavior.

CSG internal verification checklist

• Is rollback done through GitOps revert or kubectl?
• Are database migrations backward compatible?
• Are Kafka/RabbitMQ message schemas backward compatible?
• Are feature flags available?
• Are rollout metrics and progress deadline configured?

19. Failure Mode Decision Table

20. Production-Safe Debugging Principles

During incident response:

1. Preserve evidence before deleting pods.
2. Capture events, logs, rollout history, image digest, and config version.
3. Prefer read-only inspection first.
4. Avoid ad-hoc changes outside GitOps unless emergency process allows it.
5. Understand whether rollback is data/schema compatible.
6. Do not print secrets.
7. Do not scale stateful workloads blindly.
8. Do not disable NetworkPolicy/RBAC broadly as a shortcut.
9. Document exact command, timestamp, and observed result.
10. Convert repeated manual fixes into runbooks and guardrails.

21. Internal Verification Checklist

For CSG/team verification, collect answers to these:

Cluster and platform

• Which clusters host Quote & Order workloads?
• Are they EKS, AKS, on-prem, or hybrid?
• What CNI is used?
• What ingress/gateway stack is used?
• What GitOps/IaC stack is used?
• What observability stack is authoritative?

Workload

• What are standard labels/selectors?
• What are standard resource requests/limits?
• Are CPU limits required?
• What are standard probe endpoints?
• What is the standard graceful shutdown period?
• Are PDBs required?

Registry and image

• What registry is used?
• Are images deployed by tag or digest?
• Are tags immutable?
• How is image scanning enforced?
• What happens during registry outage?

Networking

• How does traffic flow from client to pod?
• Where is TLS terminated?
• Are forwarded headers trusted?
• Are NetworkPolicies default deny?
• How are private endpoints reached?
• Who owns DNS records?

Dependencies

• Are PostgreSQL/Kafka/RabbitMQ/Redis/Camunda managed services or in-cluster?
• What are dependency timeout/retry standards?
• What dashboards show dependency health?
• What is the rollback strategy for schema/message changes?

Operations

• What are common incidents from the past 6–12 months?
• Are runbooks current?
• Who can execute rollback?
• What changes are allowed outside GitOps during incident?
• How are postmortem actions tracked?

22. Senior Engineer Review Checklist

Before approving a Kubernetes change, ask:

• What failure mode does this change introduce?
• How will we detect it?
• How will we roll it back?
• Does rollback remain safe after database/message/schema changes?
• Will readiness protect traffic correctly?
• Are resource limits aligned with JVM behavior?
• Could this cause CPU throttling or OOMKilled?
• Could this break service endpoints?
• Could this break ingress routing or DNS?
• Could this break secret/config loading?
• Could this violate RBAC/NetworkPolicy/security policy?
• Could this increase cost unexpectedly?
• Are dashboards and alerts updated?
• Is there an internal verification checklist item unresolved?

23. Key Takeaways

Kubernetes failures become manageable when you map symptoms to layers.

Remember these invariants:

• CrashLoopBackOff is usually process failure.
• ImagePullBackOff is registry/auth/network/artifact failure.
• OOMKilled is memory budget failure.
• CPU throttling is runtime quota pressure.
• Pending is scheduling/capacity/policy failure.
• Service DNS resolution does not guarantee healthy endpoints.
• Ingress errors often reflect backend endpoint, timeout, protocol, or routing issues.
• Secret/config failures often present as application startup or correctness failures.
• RBAC and NetworkPolicy should not be bypassed casually.
• Rollback must consider data/schema/message compatibility.

A senior backend engineer does not need to own the whole platform. But they must understand enough to reason across application, container, Kubernetes, networking, cloud, and operational boundaries.