Part 042 — Kubernetes Operations and Runbooks

A production Kubernetes system is not production-ready because it has manifests, Helm charts, dashboards, and alerts. It is production-ready when humans can operate it under pressure without guessing.

Runbooks are the bridge between platform design and incident response.

For enterprise Java/JAX-RS services in a Quote & Order / CPQ / order-management context, runbooks must account for more than pod health. They must account for:

• customer-facing API availability,
• quote/order workflow correctness,
• PostgreSQL transaction safety,
• Kafka/RabbitMQ message processing,
• Redis/cache behavior,
• Camunda/workflow worker behavior,
• NGINX/ingress/API gateway behavior,
• cloud dependencies,
• hybrid/private connectivity,
• rollback and data compatibility,
• security/privacy constraints,
• auditability.

This part defines how to think about Kubernetes operations and how to structure runbooks that are actually usable during incidents.

CSG/internal note: this part does not assume internal CSG runbooks, severity definitions, cluster topology, tooling, or escalation paths. Every operational detail must be verified with platform/SRE/DevOps/security/backend teams.

1. What a Runbook Is

A runbook is a controlled operational procedure for a known class of situation.

A good runbook answers:

What is happening?
How do we confirm it?
How severe is it?
Who owns response?
What is safe to inspect?
What is safe to change?
What is the mitigation?
What is the rollback path?
What evidence must be preserved?
When do we escalate?
How do we verify recovery?
What follow-up is required?

A bad runbook is just a page of random commands.

A production-grade runbook includes context, decision points, risks, and stop conditions.

2. Runbook Anatomy

Use a consistent structure.

# Runbook: <incident type>

## Purpose
What this runbook is for.

## Scope
Which services/clusters/namespaces/components this applies to.

## Symptoms
Observable signals that indicate this runbook may apply.

## Severity and Customer Impact
How to classify impact.

## First Checks
Read-only commands and dashboards.

## Diagnosis Flow
Decision tree from symptom to likely cause.

## Mitigation Options
Safe immediate actions.

## Rollback Procedure
How to revert safely.

## Validation
How to confirm recovery.

## Escalation
Who to contact and when.

## Evidence to Capture
Logs, events, metrics, config versions, image digest, timelines.

## Known Risks
Actions that may worsen impact.

## Post-Incident Follow-Up
Permanent fixes, tests, alerts, documentation updates.

This structure keeps incident handling repeatable.

3. Operations Mental Model

Kubernetes operations should be viewed as layered ownership.

Application layer
  Java/JAX-RS code, config, probes, shutdown, dependency behavior

Container layer
  image, entrypoint, runtime user, JVM options, filesystem, signals

Workload layer
  Deployment, StatefulSet, Job, HPA, PDB, rollout strategy

Service/network layer
  Service, EndpointSlice, DNS, Ingress, NetworkPolicy, load balancer

Platform layer
  nodes, CNI, CSI, CoreDNS, ingress controller, autoscaler, GitOps

Cloud/hybrid layer
  IAM, private endpoint, DNS, route, firewall, registry, cloud dependency

Operations layer
  alerts, dashboards, runbooks, escalation, rollback, audit, postmortem

During incidents, avoid collapsing all layers into “Kubernetes is broken”. Identify the failing boundary.

4. Workload Runbook

Use this for a service degradation where pods exist but workload health is questionable.

Symptoms

• increased 5xx,
• rising latency,
• readiness failures,
• pod restarts,
• consumer lag,
• thread pool saturation,
• memory pressure,
• partial endpoint failure.

First checks

kubectl get deploy,pod,svc,hpa,pdb -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe deploy <deployment> -n <namespace>
kubectl get pods -n <namespace> -o wide
kubectl top pod -n <namespace>

Application checks:

• API error rate,
• latency percentiles,
• JVM heap/non-heap,
• GC pause,
• thread pool usage,
• DB pool usage,
• Kafka/RabbitMQ lag,
• Redis latency,
• Camunda worker failure/retry rate.

Mitigation options

• rollback recent deployment,
• scale replicas if downstream can handle it,
• disable feature flag,
• reduce traffic via gateway/routing if available,
• pause consumers if duplicate/incorrect processing risk exists,
• increase resources as temporary mitigation,
• route to previous version if blue-green/canary exists.

Stop conditions

Do not scale blindly when:

• database is already saturated,
• Kafka/RabbitMQ downstream commit is failing,
• issue is data corruption or semantic bug,
• external dependency is rate limiting,
• memory leak causes every replica to die eventually.

Internal verification checklist

• Which dashboards show service health?
• Which metrics define customer impact?
• Who can scale/rollback?
• Are feature flags available?
• Are consumers safe to pause/resume?
• What is the downstream capacity limit?

5. Deployment Runbook

Use this when a rollout is failing, stuck, or causing degraded service.

Symptoms

• rollout does not complete,
• new pods not ready,
• old pods terminating too fast,
• ingress returns 503/504 after deployment,
• increased error rate after release,
• migration job failed,
• HPA behavior changed unexpectedly.

First checks

kubectl rollout status deploy/<deployment> -n <namespace>
kubectl describe deploy <deployment> -n <namespace>
kubectl get rs -n <namespace>
kubectl get pods -n <namespace> --show-labels
kubectl describe pod <new-pod> -n <namespace>
kubectl logs <new-pod> -n <namespace> --previous

GitOps checks:

• commit deployed,
• image tag/digest,
• Helm values/Kustomize overlay diff,
• Argo CD/Flux sync status,
• drift/manual changes,
• pipeline logs.

Decision points

Ask:

• Did the image change?
• Did config/secret change?
• Did resource/probe change?
• Did ingress/service selector change?
• Did database migration run?
• Did message schema/topic/queue change?
• Did cloud permission/network setting change?

Mitigation

• pause rollout if possible,
• rollback to previous known-good version,
• revert GitOps commit,
• disable problematic feature flag,
• restore previous config,
• scale stable ReplicaSet if safe,
• block traffic to bad version if canary/progressive delivery is used.

Rollback caution

Rollback is unsafe without verification when release changed:

• database schema,
• message schema,
• durable workflow state,
• Redis key format,
• API contract,
• external integration behavior.

Internal verification checklist

• Is rollback done by Git revert or kubectl?
• Are deployment artifacts traceable to commit?
• Are image digests recorded?
• Are DB migrations reversible or forward-compatible?
• Are canary metrics defined?
• Is rollout history retained?

6. Rollback Runbook

Rollback is an operational decision, not a reflex.

Preconditions

Before rollback, confirm:

• current bad version,
• previous known-good version,
• data compatibility,
• message compatibility,
• config compatibility,
• migration status,
• customer impact,
• whether rollback will be overwritten by GitOps.

Kubernetes rollback command

kubectl rollout history deploy/<deployment> -n <namespace>
kubectl rollout undo deploy/<deployment> -n <namespace>
kubectl rollout status deploy/<deployment> -n <namespace>

But in GitOps environments, preferred rollback is often:

revert Git commit
  -> GitOps controller syncs desired state
  -> monitor rollout
  -> validate service health

Validation

After rollback:

• replicas ready,
• endpoints present,
• ingress/gateway healthy,
• error rate normal,
• latency normal,
• dependency metrics normal,
• consumer lag recovers,
• no new crash loops,
• no schema/message incompatibility errors.

Evidence to preserve

• bad image digest,
• previous image digest,
• manifest diff,
• config diff,
• migration status,
• logs from failed pods,
• events,
• alert timeline,
• customer impact window.

Internal verification checklist

• Who is authorized to rollback?
• Is rollback documented per service?
• Are database and message schema compatibility policies defined?
• Are rollback drills performed?
• Does GitOps overwrite manual rollback?

7. Pod Crash Runbook

Use this for CrashLoopBackOff, repeated restarts, or unexpected exits.

First checks

kubectl get pod <pod> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Diagnose by exit pattern

Exit code 0 repeatedly
  -> app exits successfully but should be long-running

Exit code 1
  -> startup/application error

Exit code 137
  -> often OOMKilled/SIGKILL

Exit code 143
  -> SIGTERM, may be normal during shutdown

Java-specific checks

• startup exception,
• config binding failure,
• secret missing,
• datasource init failure,
• Kafka/RabbitMQ connection failure,
• Redis connection failure,
• JVM flag error,
• permission denied,
• write failure on read-only filesystem,
• port already used or wrong bind address.

Mitigation

• rollback if caused by new release,
• restore config/secret if changed,
• increase memory as temporary mitigation for OOM,
• disable startup dependency fail-fast only if safe,
• fix entrypoint or filesystem permission,
• separate migration from app startup if migration causes repeated crash.

Internal verification checklist

• Are app logs structured enough for startup failure?
• Are previous logs available?
• Are crash alerts tied to deployment events?
• Are startup dependency policies documented?
• Are OOM heap dumps configured safely?

8. Ingress Outage Runbook

Use this when external or internal HTTP traffic cannot reach the service.

Symptoms

• external 404/502/503/504,
• TLS errors,
• host not resolving,
• path not routing,
• only one route broken,
• internal service works but ingress fails,
• ingress works but app returns wrong redirects.

First checks

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc <service> -n <namespace>
kubectl get endpointslice -n <namespace>
kubectl get pods -n <namespace>

Ingress controller checks:

kubectl get pods -n <ingress-namespace>
kubectl logs -n <ingress-namespace> <ingress-controller-pod>

External checks:

curl -v https://<host>/<path>
nslookup <host>

Diagnose by status

404
  route/host/path mismatch or application path issue

502
  backend protocol/connection failure

503
  no healthy backend/endpoints or upstream unavailable

504
  backend timeout or dependency slowness

Java/JAX-RS checks

• JAX-RS application path,
• reverse proxy headers,
• X-Forwarded-Proto,
• context path,
• HTTP vs HTTPS backend protocol,
• request body limit,
• timeout for long-running endpoints.

Internal verification checklist

• Which ingress controller/API gateway is authoritative?
• What is default timeout/body size?
• Where is TLS terminated?
• Are route changes deployed through GitOps?
• Are ingress controller logs accessible?
• Who owns DNS and certificates?

9. DNS Issue Runbook

Use this when service names, external names, or private endpoints fail to resolve.

Symptoms

• UnknownHostException,
• intermittent dependency resolution failure,
• cloud private endpoint not resolving,
• service DNS resolves in one namespace but not another,
• DNS timeout,
• sudden increase in connection errors after DNS change.

First checks

kubectl get svc -A | grep <name>
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system deploy/coredns

From debug pod:

cat /etc/resolv.conf
nslookup <service>
nslookup <service>.<namespace>.svc.cluster.local
nslookup <private-endpoint-host>

Decision points

• Is it Kubernetes service DNS?
• Is it external DNS?
• Is it private cloud DNS?
• Is it hybrid/on-prem DNS forwarding?
• Is NetworkPolicy blocking DNS egress?
• Is the application caching stale DNS?

Internal verification checklist

• Are DNS policies documented?
• Is CoreDNS monitored?
• Are private DNS zones managed by platform/cloud team?
• Are hybrid forwarders monitored?
• Are JVM DNS cache settings understood?
• Are DNS incidents part of postmortem history?

10. Node Issue Runbook

Use this when nodes are NotReady, under pressure, or disrupting workloads.

Symptoms

• node NotReady,
• many pods evicted,
• disk pressure,
• memory pressure,
• kubelet not responding,
• CNI errors,
• pods stuck terminating,
• sudden workload concentration on fewer nodes.

First checks

kubectl get nodes
kubectl describe node <node>
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
kubectl top node

Mitigation

Depends on platform ownership:

• cordon node,
• drain node if safe,
• let autoscaler replace node,
• restart node-level components only if platform process allows,
• move workload away from bad node,
• scale replicas if capacity allows,
• escalate to platform/cloud provider.

Commands:

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Use drain carefully. PDB, stateful pods, local storage, and disruption budgets matter.

Internal verification checklist

• Who can cordon/drain nodes?
• Are PDBs configured for critical workloads?
• Are stateful pods safe to evict?
• Are spot/preemptible nodes used?
• Are node-level incidents visible to backend teams?
• What is the escalation path to platform/SRE?

11. Storage Issue Runbook

Use this for PVC pending, volume attach failure, mount failure, storage latency, or stateful workload disruption.

Symptoms

• PVC Pending,
• pod Pending due to unbound PVC,
• volume mount timeout,
• read/write errors,
• database/broker/cache pod stuck,
• storage latency spike,
• disk full,
• backup/restore failure.

First checks

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl get pv
kubectl get storageclass
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Risk classification

Storage incidents are high-risk when they affect:

• PostgreSQL data,
• Kafka/RabbitMQ durable queues,
• Redis persistence,
• Camunda workflow state,
• migration jobs,
• audit records,
• customer order/quote state.

Mitigation

• do not delete PVC without explicit approval,
• do not recreate StatefulSet blindly,
• check backup/restore status,
• verify volume zone and attachment,
• escalate to platform/storage owner,
• reduce write load if storage latency is causing cascading failure.

Internal verification checklist

• Which workloads use persistent storage?
• Are stateful dependencies self-managed or managed service?
• Are backups tested?
• Is restore time objective defined?
• Who owns CSI/storage backend?
• Are disk-full alerts configured?

12. Secret Rotation Runbook

Secret rotation can be routine or incident-triggered. Both need controlled execution.

Rotation risks

• pods keep old env var until restart,
• mounted secret updates but app does not reload,
• external secret sync delay,
• database password changed before app update,
• cloud credential permission mismatch,
• partial rollout causes mixed credentials,
• logs accidentally expose secret material.

Safe rotation flow

1. Identify secret consumers.
2. Confirm reload behavior: env var requires restart, mounted file may update but app may not reload.
3. Create new secret/version.
4. Update external system to accept old + new credentials during transition if possible.
5. Roll pods safely.
6. Validate authentication success.
7. Remove old credential after all consumers moved.
8. Confirm no failures or leaked values.

Kubernetes checks

kubectl get secret -n <namespace>
kubectl describe externalsecret -n <namespace>
kubectl rollout restart deploy/<deployment> -n <namespace>
kubectl rollout status deploy/<deployment> -n <namespace>

Internal verification checklist

• What secret manager is source of truth?
• Are External Secrets/Key Vault/Secrets Manager used?
• Which services consume each secret?
• Does app support live reload?
• Is dual credential rotation supported?
• Are secret access logs audited?

13. Certificate Expiry Runbook

Certificate expiry can break ingress, internal mTLS, database TLS, cloud private endpoints, or outbound calls.

Symptoms

• TLS handshake failure,
• browser/client certificate warning,
• ingress HTTPS unavailable,
• Java PKIX path building failed,
• expired truststore/keystore,
• cloud SDK TLS failure,
• private endpoint TLS validation failure.

First checks

kubectl get secret -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -dates

Java-specific concern

Java services may depend on:

• JVM default truststore,
• custom truststore,
• mounted CA bundle,
• application keystore,
• mTLS client certificate,
• sidecar/proxy certificate.

Mitigation

• renew certificate,
• update Kubernetes TLS Secret or certificate manager resource,
• restart/reload ingress if required,
• roll app pods if truststore/keystore is mounted and not reloaded,
• validate full chain from client and pod.

Internal verification checklist

• Who owns public certificates?
• Who owns internal CA?
• Is cert-manager used?
• Are expiry alerts configured?
• Are truststores built into image or mounted?
• Are mTLS certificates rotated safely?

14. Autoscaling Incident Runbook

Autoscaling incidents happen when scaling does not happen, happens too slowly, or happens too aggressively.

Symptoms

• HPA does not scale during load,
• replicas oscillate,
• nodes do not scale,
• pods pending after scale-out,
• queue lag grows,
• cost spikes,
• downstream dependency overloaded after scale-out.

First checks

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>
kubectl top pod -n <namespace>
kubectl get pods -n <namespace>
kubectl describe pod <pending-pod> -n <namespace>

Cluster autoscaler/Karpenter checks depend on platform tooling and access.

Decision points

• Is scaling signal correct?
• Are metrics available?
• Are resource requests set?
• Are maxReplicas too low?
• Is cluster capacity available?
• Are new pods Pending?
• Is downstream dependency the bottleneck?
• Is consumer scaling causing duplicate or out-of-order processing risk?

Mitigation

• temporarily adjust min/max replicas,
• fix metrics adapter issue,
• scale node pool if platform-owned process allows,
• throttle incoming traffic,
• pause lower-priority workloads,
• reduce consumer concurrency if downstream is overloaded,
• disable aggressive scaling behavior if thrashing.

Internal verification checklist

• Are HPA metrics reliable?
• Is KEDA used for queue/lag scaling?
• Are maxReplicas aligned with downstream capacity?
• Are autoscaling events alerted?
• Are node autoscaler limits documented?
• Are cost guardrails defined?

15. Cloud Dependency Incident Runbook

Use this when a pod cannot call AWS/Azure/cloud-managed services.

Symptoms

• AWS/Azure SDK authentication failure,
• 403/AccessDenied,
• timeout to cloud endpoint,
• private endpoint unreachable,
• DNS resolves public instead of private IP,
• rate limiting,
• cloud service regional incident,
• increased latency from cloud dependency.

First checks

From application logs:

• SDK exception type,
• HTTP status,
• endpoint host,
• region,
• request ID/correlation ID,
• retry count,
• timeout type.

Kubernetes checks:

kubectl describe pod <pod> -n <namespace>
kubectl describe sa <serviceaccount> -n <namespace>
kubectl get networkpolicy -n <namespace>

Network checks from debug pod if allowed:

nslookup <cloud-service-host>
curl -v https://<cloud-service-host>

AWS-specific decision points

• IRSA configured?
• OIDC trust policy valid?
• IAM role permission valid?
• STS reachable?
• VPC endpoint exists?
• private DNS enabled?
• security group permits traffic?
• regional AWS service issue?

Azure-specific decision points

• Workload Identity configured?
• federated credential valid?
• Managed Identity permission valid?
• Private Endpoint DNS correct?
• NSG/UDR/firewall permits traffic?
• Azure service regional issue?

Internal verification checklist

• Which cloud services are called by each workload?
• Are permissions least-privilege and documented?
• Are private endpoints used?
• Are SDK retries/timeouts standardized?
• Are cloud request IDs logged?
• Are cloud dependency dashboards available?

16. Incident Severity and Customer Impact

Severity should be based on customer/business impact, not only technical weirdness.

A practical classification model:

For Quote & Order-like systems, impact dimensions may include:

• quote creation unavailable,
• order submission unavailable,
• order state stuck,
• pricing/configuration incorrect,
• async workflow backlog,
• integration callbacks failing,
• customer-specific tenant impacted,
• data correctness risk,
• SLA breach risk,
• security/privacy exposure.

Impact checklist

Ask:

• Which API endpoints are affected?
• Which customers/tenants/regions are affected?
• Is the issue availability, latency, correctness, or data integrity?
• Are orders/quotes blocked or delayed?
• Are messages accumulating in Kafka/RabbitMQ?
• Are workflow tasks stuck?
• Is manual remediation needed?
• Is there security/privacy exposure?

17. Communication During Incidents

Technical accuracy matters, but so does coordination.

A good incident update includes:

Current impact:
What changed:
Current hypothesis:
Actions taken:
Next action:
ETA if known, otherwise say unknown:
Need from other teams:
Risk:

Avoid:

• blaming a component/team prematurely,
• claiming root cause before evidence,
• saying “fixed” before validation,
• hiding uncertainty,
• changing multiple variables at once without recording them.

For senior engineers, incident communication is part of system reliability.

18. Evidence Capture

Before deleting pods or rolling back, capture evidence where possible.

Kubernetes evidence

kubectl get deploy <deployment> -n <namespace> -o yaml
kubectl get pod <pod> -n <namespace> -o yaml
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl rollout history deploy/<deployment> -n <namespace>

Release evidence

• Git commit,
• image tag,
• image digest,
• Helm values diff,
• Kustomize overlay diff,
• Argo CD/Flux sync event,
• pipeline run,
• migration logs,
• feature flag changes.

Runtime evidence

• request/error/latency metrics,
• JVM metrics,
• DB pool metrics,
• Kafka/RabbitMQ lag,
• Redis metrics,
• workflow metrics,
• ingress logs,
• cloud request IDs.

Evidence is not bureaucracy. It prevents repeated incidents.

19. Post-Incident Review

A Kubernetes incident postmortem should not stop at “pod crashed”.

Ask:

• Why did the pod crash?
• Why did readiness/rollback/autoscaling not protect users?
• Why was the failure not caught in CI/staging/canary?
• Why was the alert late or noisy?
• Why was the runbook incomplete?
• Why was blast radius larger than expected?
• What guardrail would prevent recurrence?

Useful outputs:

• better probe design,
• safer rollout policy,
• improved dashboard,
• alert threshold correction,
• stronger validation in CI,
• resource tuning,
• NetworkPolicy/RBAC fix,
• secret rotation automation,
• DNS/private endpoint documentation,
• runbook update,
• architecture decision record.

20. Runbook Testing

A runbook that has never been tested is an assumption.

Test runbooks through:

• staging drills,
• game days,
• rollback rehearsals,
• node drain tests,
• DNS failure simulation,
• secret rotation rehearsal,
• certificate renewal rehearsal,
• load and autoscaling tests,
• dependency outage simulation,
• restore drills for stateful systems.

For Java/JAX-RS services, especially test:

• graceful shutdown,
• readiness removal before termination,
• long-running request handling,
• DB transaction behavior during shutdown,
• consumer shutdown and offset/ack behavior,
• retry/backoff during dependency failure.

21. Production Operations Checklist

Workload readiness

• Deployment/StatefulSet/Job type is appropriate.
• Probes are correct.
• Resources are sized and monitored.
• PDB is defined where needed.
• HPA/KEDA is configured where needed.
• Graceful shutdown is tested.

Networking readiness

• Service selectors match pods.
• Ingress/Gateway rules are correct.
• TLS termination is understood.
• DNS names are documented.
• NetworkPolicy allows required paths only.
• Private endpoints are tested.

Security readiness

• Workload runs as non-root.
• RBAC is least-privilege.
• Secrets do not leak into logs.
• Cloud identity is secretless where possible.
• Admission policies are satisfied.
• Audit logs are available.

Observability readiness

• Logs are structured.
• Metrics include business and technical signals.
• Traces propagate correlation IDs.
• Dashboards exist.
• Alerts map to action.
• Kubernetes events are accessible.

Operations readiness

• Rollback path is documented.
• Runbook exists and is tested.
• Escalation path is known.
• Ownership is clear.
• Incident evidence can be captured.
• Postmortem actions are tracked.

22. CSG Internal Verification Checklist

Use this checklist to align with actual team/platform practices.

Ownership

• Who owns application runtime issues?
• Who owns Kubernetes manifests?
• Who owns Helm/Kustomize overlays?
• Who owns GitOps sync?
• Who owns ingress/DNS/certificates?
• Who owns cloud IAM/private endpoints?
• Who owns node/CNI/CSI/CoreDNS issues?

Access

• What cluster access does a backend engineer have?
• Can backend engineers read events/logs across namespaces?
• Who can rollback production?
• Who can scale deployments?
• Who can read secrets metadata?
• Who can exec into pods?
• Is production exec restricted or audited?

Tooling

• Is Argo CD, Flux, or another GitOps tool used?
• Is Helm, Kustomize, or both used?
• Which dashboard is source of truth?
• Which alerting tool is used?
• Where are runbooks stored?
• Where are incident notes/postmortems stored?

Service-specific

• Which services are customer-facing APIs?
• Which services are async consumers/workers?
• Which services interact with PostgreSQL?
• Which services interact with Kafka/RabbitMQ?
• Which services interact with Redis?
• Which services interact with Camunda/workflow engine?
• Which services call AWS/Azure APIs?

Operational safety

• What changes are allowed during incident?
• Are manual kubectl changes allowed?
• Does GitOps revert manual changes?
• What is the emergency break-glass process?
• Are rollbacks tested?
• Are schema/message changes backward compatible?

23. Senior Engineer Operating Principles

A senior backend engineer operating on Kubernetes should follow these principles:

1. Diagnose by layer, not by guess.
2. Prefer evidence before action.
3. Preserve logs/events before deleting resources.
4. Understand GitOps before applying manual fixes.
5. Treat rollback as a compatibility decision.
6. Avoid broad security bypasses.
7. Consider downstream capacity before scaling.
8. Treat stateful systems with extra caution.
9. Keep customer impact visible.
10. Convert every repeated manual fix into automation, guardrail, or runbook.

24. Key Takeaways

Production Kubernetes operations are about controlled response under uncertainty.

The core mindset:

Observe first.
Identify layer.
Protect customers.
Mitigate safely.
Preserve evidence.
Validate recovery.
Fix permanently.

For enterprise Java/JAX-RS systems, the most important operational skill is cross-layer reasoning: from pod lifecycle to JVM behavior, from ingress to endpoint readiness, from Kubernetes identity to cloud API access, from rollout strategy to database/message compatibility.

A runbook is not a document you write after everything is understood. It is a reliability interface between engineering teams, platform teams, and incident responders.