source code
build
image
registry
manifest
Kubernetes API
scheduler
node
container runtime
pod lifecycle
service discovery
network path
ingress path
identity
configuration
secret
storage
autoscaling
observability
application behavior
dependency behavior

delete pod
restart deployment
scale to zero
edit live manifest
disable policy
remove NetworkPolicy
increase timeout blindly

observe
classify symptom
identify failing layer
collect evidence
test hypothesis
apply smallest safe mitigation
confirm recovery
document root cause

kubectl get pod <pod> -n <namespace> -o yaml > pod-before.yaml
kubectl describe pod <pod> -n <namespace> > pod-describe-before.txt
kubectl logs <pod> -n <namespace> --previous > pod-previous.log
kubectl get events -n <namespace> --sort-by=.lastTimestamp > events-before.txt

- deleting all pods
- disabling admission policy globally
- removing NetworkPolicy broadly
- scaling every service
- editing GitOps-managed resources manually without coordination
- restarting stateful clusters blindly
- changing ingress timeout globally
- changing DNS/CoreDNS without platform owner

Is this customer-facing?
Is this one pod, one service, one namespace, one node, one cluster, one region?
Is stateful data involved?
Is traffic still flowing?
Is rollback possible?
Who owns the dependency?

kubectl get ns
kubectl get all -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

kubectl get deploy,statefulset,daemonset,job,cronjob -n <namespace>
kubectl describe deploy <deployment> -n <namespace>
kubectl rollout status deploy/<deployment> -n <namespace>
kubectl rollout history deploy/<deployment> -n <namespace>

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous

kubectl get svc,endpoints,endpointslices -n <namespace>
kubectl describe svc <service> -n <namespace>
kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>

kubectl exec -it <pod> -n <namespace> -- sh
kubectl port-forward pod/<pod> 8080:8080 -n <namespace>
kubectl top pod -n <namespace>
kubectl top node

container exits immediately
application cannot bind port
config missing
Java cannot find main class
permission denied
file not found
works on host but not in container

docker build -t app-debug .
docker run --rm app-debug
docker run --rm -it app-debug sh
docker inspect app-debug

- jar path wrong
- main class missing
- shaded jar not built
- WAR copied but runtime expects JAR
- Maven build artifact name changed
- non-root user cannot read file
- app writes to read-only path
- port config differs from Dockerfile
- missing trust store
- missing timezone/locale data

What process is PID 1?
Does it receive signals?
Does Java see the expected env vars?
Does the artifact exist at runtime?
Can the runtime user read it?
What port is actually listening?
Does the process write to a writable path?

docker build fails
Maven dependency download fails
BuildKit cache not used
permission denied
file not found in build context
COPY fails
tests fail in image build

docker build --no-cache -t app-test .
docker build --progress=plain -t app-test .

- required file excluded by .dockerignore
- copying from wrong build stage
- Maven repository unavailable
- private dependency credentials missing
- base image tag changed
- no network from CI runner
- corporate proxy not configured
- architecture mismatch amd64/arm64
- dependency cache invalidated by COPY order

denied: requested access to the resource is denied
unauthorized
repository does not exist
tag invalid
TLS certificate error

ImagePullBackOff
ErrImagePull

kubectl describe pod <pod> -n <namespace>

Failed to pull image
pull access denied
manifest unknown
not found
unauthorized
x509 certificate signed by unknown authority
toomanyrequests

- image tag does not exist
- image pushed to different registry/repo
- imagePullSecret missing
- node cannot reach registry
- registry auth expired
- private registry certificate issue
- wrong architecture image
- tag overwritten unexpectedly
- registry outage
- pull rate limit

kubectl get secret -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

pod remains Pending
0/10 nodes are available
insufficient cpu
insufficient memory
node affinity mismatch
untolerated taint
pod has unbound immediate PersistentVolumeClaims

kubectl describe pod <pod> -n <namespace>
kubectl get nodes --show-labels
kubectl describe node <node>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get quota,limitrange -n <namespace>
kubectl get pvc -n <namespace>

insufficient cpu
insufficient memory
insufficient ephemeral-storage

nodeSelector mismatch
node affinity too strict
pod anti-affinity impossible
topology spread impossible
untolerated taint

PVC pending
volume zone mismatch
StorageClass unavailable

pod starts then exits repeatedly
restart count increases
CrashLoopBackOff

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses}'

missing config
missing secret
invalid DB URL
migration failure
port conflict
Java main class error
unhandled exception
license/config validation failure

Liveness probe failed
Startup probe failed

startupProbe missing
liveness too aggressive
endpoint path wrong
management port wrong
timeout too short
Java startup too slow

permission denied
read-only file system
cannot write temp file
non-root user cannot access path

PostgreSQL unavailable
Kafka unavailable
Redis unavailable
secret manager unavailable
cloud identity unavailable

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

kubectl describe pod <pod> -n <namespace>
kubectl top pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous

heap
metaspace
thread stacks
direct memory
JIT/code cache
GC structures
native memory
TLS buffers
libraries

heap too large relative to container limit
MaxRAMPercentage too high
too many threads
large batch/message processing
large JSON payload
unbounded cache
direct buffer growth
connection pool leak
memory leak
large file loaded into memory

reduce heap percentage
increase memory limit after evidence
reduce batch size
reduce prefetch
limit payload size
fix leak
cap caches
reduce thread count
separate heavy workload
add memory dashboards

high latency
low average CPU
high throttling
GC pauses increase
startup slow
readiness delay
consumer lag grows

CPU limit too low
CPU request too low for scheduling
JVM warmup needs more CPU
GC threads throttled
JSON serialization CPU spike
TLS overhead
compression overhead
too many concurrent requests

kubectl top pod -n <namespace>
kubectl top node

READY 0/1
Readiness probe failed
service has no endpoints
rollout stuck

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl get endpointslices -n <namespace>
kubectl describe deploy <deployment> -n <namespace>

readiness path wrong
readiness port wrong
management port not exposed
app startup incomplete
DB dependency check failing
Kafka dependency check failing
secret missing
config invalid
slow JVM startup
CPU throttling delays startup
thread pool exhausted

Kubernetes may be protecting traffic from a bad pod.

curl service DNS fails
connection refused
connection timeout
service has no endpoints
one service cannot call another

kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpointslices -n <namespace>
kubectl get pods -n <namespace> --show-labels
kubectl describe svc <service> -n <namespace>

nslookup <service>.<namespace>.svc.cluster.local
curl -v http://<service>.<namespace>.svc.cluster.local:<port>

Service selector does not match pod labels
pods not ready, so no endpoints
targetPort wrong
named port mismatch
container not listening on expected port
NetworkPolicy blocks traffic
DNS failure
wrong namespace
service name typo

Service selector:
  app: quote-api

metadata:
  labels:
    app.kubernetes.io/name: quote-api

UnknownHostException
Temporary failure in name resolution
nslookup fails
service discovery intermittent
cloud private endpoint resolves wrong IP

cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local
nslookup <service>.<namespace>.svc.cluster.local
nslookup <private-endpoint-domain>

kubectl get pods -n kube-system | grep coredns
kubectl logs -n kube-system deploy/coredns
kubectl get svc -n kube-system kube-dns

CoreDNS unavailable
NetworkPolicy blocks DNS egress
wrong service namespace
ndots/search behavior surprises app
ExternalName target wrong
private DNS zone not linked
split-horizon DNS issue
node DNS issue
DNS cache stale

404
502
503
504
TLS error
host not found
wrong backend
timeout

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc,endpointslices -n <namespace>
kubectl logs -n <ingress-namespace> deploy/<ingress-controller>

curl -vk https://<host>/<path>

host/path rule mismatch
rewrite mismatch
request reaches ingress but no route matches
wrong ingress class

backend connection refused
backend protocol mismatch
pod not listening
service targetPort wrong
TLS to backend misconfigured

service has no endpoints
pods not ready
backend unavailable

backend timeout
app slow
DB slow
ingress timeout shorter than app processing
connection draining issue

client timeout
CDN/front door timeout
cloud load balancer timeout
ingress timeout
service mesh timeout
app server timeout
HTTP client timeout
database timeout

certificate expired
certificate name mismatch
unknown authority
TLS handshake failure
backend TLS error

kubectl describe ingress <ingress> -n <namespace>
kubectl get secret <tls-secret> -n <namespace> -o yaml
curl -vk https://<host>
openssl s_client -connect <host>:443 -servername <host>

certificate expired
wrong certificate secret
secret not synced
DNS points to wrong load balancer
SNI mismatch
private CA not trusted by Java
backend protocol mismatch
mTLS config mismatch
certificate chain incomplete

PKIX path building failed
unable to find valid certification path
handshake_failure

CreateContainerConfigError
secret not found
configmap not found
environment variable missing
volume mount failed

kubectl describe pod <pod> -n <namespace>
kubectl get configmap,secret -n <namespace>
kubectl get deploy <deployment> -n <namespace> -o yaml

wrong namespace
wrong name
GitOps sync order
External Secret not reconciled
Sealed Secret not decrypted
secret key missing
config key missing
Helm value typo
Kustomize overlay omitted resource
RBAC prevents controller from creating secret
cloud secret manager permission denied

kubectl get externalsecret -n <namespace>
kubectl describe externalsecret <name> -n <namespace>

kubectl describe pod <pod> -n <namespace>
kubectl logs <secret-provider-pod> -n <provider-namespace>

Forbidden
User cannot get resource
serviceaccount cannot list secrets
cannot patch deployments

kubectl auth can-i get pods -n <namespace> --as=system:serviceaccount:<namespace>:<serviceaccount>
kubectl get role,rolebinding -n <namespace>
kubectl get clusterrole,clusterrolebinding
kubectl describe rolebinding <name> -n <namespace>

wrong ServiceAccount
default ServiceAccount used accidentally
RoleBinding references wrong subject
namespace mismatch
ClusterRole required but Role used
GitOps controller lacks permission
CI/CD identity lacks permission
automount token disabled but app expects Kubernetes API

connection timeout
DNS timeout
service reachable from one pod but not another
works before policy, fails after policy
cloud private endpoint unreachable

kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get pod <pod> -n <namespace> --show-labels
kubectl get ns --show-labels

nslookup <target>
curl -v http://<target>:<port>
nc -vz <target> <port>

default deny without allow rule
DNS egress not allowed
namespaceSelector label mismatch
podSelector label mismatch
wrong port/protocol
egress to DB/Kafka/RabbitMQ/Redis not allowed
egress to cloud private endpoint not allowed
ingress from ingress controller not allowed
CNI does not enforce NetworkPolicy as expected

source pod labels
source namespace labels
target pod labels
target namespace labels
policy selectors
ports
direction

PVC Pending
pod stuck ContainerCreating
Multi-Attach error
MountVolume.SetUp failed
volume node affinity conflict

kubectl get pvc,pv -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get storageclass
kubectl get csidrivers
kubectl get pods -n kube-system | grep csi

StorageClass missing
dynamic provisioning failed
CSI driver unavailable
volume zone mismatch
access mode incompatible
volume already attached to another node
quota exceeded
cloud IAM/identity issue for CSI
reclaim policy surprise

HPA shows unknown
current metrics unavailable
replicas do not increase
consumer lag grows but pods do not scale

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>
kubectl top pod -n <namespace>
kubectl get apiservice | grep metrics

kubectl get scaledobject -n <namespace>
kubectl describe scaledobject <name> -n <namespace>

metrics server unavailable
custom metrics adapter unavailable
wrong metric name
missing resource requests
HPA target wrong
maxReplicas too low
scaleDown stabilization too slow
KEDA trigger auth failed
consumer lag metric missing
cluster autoscaler cannot add nodes
pods Pending after scaling

Is the scaling signal actually correlated with the bottleneck?

AccessDenied
Unable to locate credentials
InvalidIdentityToken
ExpiredToken
AssumeRoleWithWebIdentity failed

CredentialUnavailableException
ManagedIdentityCredential authentication unavailable
ClientAuthenticationException
AuthorizationFailed

kubectl get sa <serviceaccount> -n <namespace> -o yaml
kubectl describe pod <pod> -n <namespace>
kubectl exec <pod> -n <namespace> -- env | sort

wrong ServiceAccount used
annotation missing
OIDC/federated credential mismatch
namespace mismatch
audience mismatch
cloud role lacks permission
SDK uses env/static credential before workload identity
token file not mounted
network cannot reach cloud API
private DNS resolves incorrectly

authentication = who is the pod?
authorization = what can it do?
network = can it reach the API endpoint?

connection timeout
connection refused
DNS resolves public IP instead of private IP
TLS name mismatch
works from node but not pod
works from one namespace but not another

1. DNS resolution
2. Routing
3. Firewall / SG / NSG / NACL
4. NetworkPolicy
5. Proxy
6. TLS trust
7. Service authorization
8. Application timeout

nslookup <private-domain>
curl -vk https://<private-domain>
nc -vz <private-domain> <port>

consumer lag increasing
processing latency high
rebalance storm
DLQ increasing
consumer pods healthy but backlog grows

not enough partitions
not enough replicas
consumer stuck on poison message
downstream DB/API slow
retry storm
CPU throttling
memory pressure
large messages
rebalance due to rolling restarts
max poll interval exceeded

queue depth grows
message age grows
unacked messages high
redeliveries high
DLQ grows

consumers down
prefetch too high
consumer processing slow
downstream dependency slow
poison messages requeued forever
ack not sent
connection/channel instability
consumer concurrency too low

too many connections
connection timeout
pool exhausted
API latency spike
consumer processing slows
database CPU high

HPA scaled pods
pool size too high per pod
connection leak
slow queries
long transactions
database failover/reconnect storm
migration lock
background jobs competing with API traffic

Redis timeout
cache miss spike
distributed lock stuck
duplicate processing
rate limit failure
idempotency failure

Redis unavailable
network policy blocks Redis
connection pool exhausted
retry storm
slow Redis command
large keys/values
lock TTL too long
lock TTL too short
credential rotation failed
TLS trust issue

task backlog grows
incidents increase
task lock expires
duplicate task processing
worker completes fail
workflow stuck

worker pods down
worker version incompatible
lock duration too short
processing time increased
downstream dependency slow
retry storm
bad process data
task type mismatch
worker shutdown unsafe
database latency

deployment rollout stuck
new ReplicaSet not progressing
old pods not terminating
new pods not ready
progressDeadlineExceeded

kubectl rollout status deploy/<deployment> -n <namespace>
kubectl describe deploy <deployment> -n <namespace>
kubectl get rs -n <namespace>
kubectl get pods -n <namespace>
kubectl describe pod <new-pod> -n <namespace>

new image cannot pull
new pod cannot schedule
new pod crashloops
readiness fails
PDB blocks old pod eviction
maxUnavailable/maxSurge impossible
quota exceeded
NetworkPolicy blocks startup dependency
secret/config missing

kubectl rollout undo deploy/<deployment> -n <namespace>

Argo CD/Flux sync failed
drift detected
resource out of sync
admission webhook denied
no matches for kind
forbidden
helm render error
kustomize build error

invalid manifest
deprecated API
missing CRD
CRD sync order wrong
RBAC denied
policy violation
Helm values error
Kustomize patch error
manual drift
resource managed by another controller

1. Time detected
2. Alert or symptom
3. Customer impact
4. Affected service/namespace/cluster
5. Recent changes
6. Current state
7. Evidence collected
8. Hypotheses
9. Actions taken
10. Result of each action
11. Current mitigation
12. Root cause if known
13. Follow-up items

If pod is Pending, start with scheduler events.
If pod is CrashLooping, start with previous logs and exit reason.
If pod is OOMKilled, think JVM memory beyond heap.
If pod is Running but not Ready, inspect readiness semantics.
If Service fails, check selector and EndpointSlice before DNS.
If DNS fails, verify CoreDNS and NetworkPolicy DNS egress.
If Ingress returns 503, check backend endpoints.
If Ingress returns 504, check timeout chain and app/dependency latency.
If RBAC says forbidden, do not grant cluster-admin.
If NetworkPolicy changed, compare labels before changing ports.
If HPA does not scale, inspect metric availability first.
If cloud SDK fails, separate identity, permission, and network.
If private endpoint fails, debug DNS before firewall.
If consumer lag grows, check downstream bottleneck before scaling.
If rollback is needed, check GitOps source of truth.

- deleting pods before reading previous logs
- restarting everything without classifying failure
- bypassing readiness
- disabling liveness permanently
- removing NetworkPolicy globally
- granting cluster-admin during incident
- scaling consumers without checking downstream capacity
- increasing DB pool size during DB exhaustion
- increasing timeouts without checking latency source
- deleting PVCs to fix mount issues
- applying kubectl edits to GitOps-managed resources without follow-up
- ignoring customer impact while debugging technical details

user symptom
traffic path
Kubernetes routing
pod lifecycle
container runtime
application runtime
dependency path
cloud/on-prem network
identity/secret/config
observability evidence