This part does not assume CSG Quote & Order uses a specific Kubernetes version,
managed platform, workload controller, deployment strategy, Helm chart,
GitOps controller, service mesh, or runtime convention.

Treat Kubernetes version, workload type, rollout strategy, lifecycle hooks,
probe endpoints, resource policy, job model, and operational runbooks as
internal verification items.

startup behavior
readiness behavior
traffic eligibility
graceful shutdown
request draining
connection pool cleanup
consumer/worker stop behavior
job retry semantics
rollout blast radius

container running = service ready
pod deleted = request langsung berhenti
Deployment = always safe for every workload
Job retry = harmless
CronJob schedule = exactly once
StatefulSet = just Deployment with volume
liveness probe = health check for dependencies
readiness probe = same as liveness probe

desired state: apa yang dideklarasikan manifest
observed state: apa yang benar-benar terjadi di cluster
control loop: siapa yang mencoba memperbaiki selisihnya

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3

manual change akan cenderung dikembalikan oleh controller
Pod yang mati dapat diganti jika controller menginginkan replica tetap ada
rollout adalah perubahan desired state yang direkonsiliasi bertahap

Apakah container saya bisa jalan?

Apa yang terjadi saat Pod dibuat?
Apa yang terjadi saat Pod belum ready?
Apa yang terjadi saat Pod diganti saat ada request berjalan?
Apa yang terjadi saat node hilang?
Apa yang terjadi saat rollout gagal di tengah?
Apa yang terjadi saat Job retry menjalankan logic yang sama dua kali?

one main application container
init containers
sidecar containers
shared network namespace
shared volume mounts
shared lifecycle boundary

service mesh proxy
log shipper
secret agent
certificate agent
metrics exporter
runtime security agent

kubectl get pod <pod-name> -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl logs <pod-name> -n <namespace> -c <container-name>

Pending
Running
Succeeded
Failed
Unknown

PodScheduled
Initialized
ContainersReady
Ready

Running tetapi Ready=false

readiness probe gagal
application belum selesai warmup
dependency health check terlalu ketat
sidecar belum siap
config missing tetapi process belum crash

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.phase}'
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.conditions}'
kubectl describe pod <pod-name> -n <namespace>

A Java service must be safe during startup, safe during steady state, and safe during termination.

wait for dependency bootstrap
render config template
fetch certificate/config artifact
run lightweight preflight check
prepare shared volume

long migration in init container
business data repair in init container
external dependency wait with no timeout
complex shell script with hidden failure mode

kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <init-container-name>

Apakah init container punya timeout?
Apakah failure-nya jelas?
Apakah idempotent?
Apakah seharusnya ini menjadi Job terpisah?
Apakah init container membuat startup service bergantung pada dependency eksternal yang tidak perlu?

startupProbe: beri waktu aplikasi boot/warmup sebelum liveness menilai
readinessProbe: boleh menerima traffic atau tidak
livenessProbe: perlu restart karena stuck atau unrecoverable

startupProbe cocok untuk slow classloading, DI bootstrap, schema cache, warmup
readinessProbe cocok untuk traffic eligibility
livenessProbe harus shallow dan tidak tergantung dependency eksternal berat

liveness check query database
readiness selalu 200 walaupun app belum bisa melayani request
startupProbe terlalu pendek untuk cold start JVM
probe endpoint memakai thread pool yang sama dan bisa starvation

startupProbe too aggressive -> CrashLoopBackOff during normal warmup
liveness too deep -> restart storm during downstream outage
readiness too shallow -> broken pods receive traffic

request terputus di tengah
Kafka offset committed sebelum work selesai
DB transaction abort tidak jelas
file upload rusak
outbox publisher duplicate
connection pool leak sampai forced kill

stop accepting new traffic via readiness=false
stop scheduling new internal jobs
stop consuming new Kafka/RabbitMQ messages
finish or safely cancel in-flight request
flush telemetry/logs if needed
close DB pool, HTTP clients, producer/consumer
respect terminationGracePeriodSeconds

stateless HTTP API service
JAX-RS service with external database/cache/message broker
horizontally scalable service
interchangeable replicas

replica identity penting
stable network identity per replica diperlukan
persistent volume per replica harus dipertahankan
ordered startup/shutdown diperlukan

Pod lokal bisa diganti kapan saja tanpa kehilangan source-of-truth state.

service tidak pernah menyimpan data apa pun.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

image pull failure
bad config
startup probe failure
readiness never true
crash loop
insufficient resources
quota exceeded
node scheduling constraint

kubectl rollout status deployment/<name> -n <namespace>
kubectl rollout history deployment/<name> -n <namespace>
kubectl describe deployment <name> -n <namespace>
kubectl get rs -n <namespace>

kubectl rollout undo deployment/<name> -n <namespace>

Pod restart
node drain
rolling update
availability zone disruption
brief dependency warmup

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: quote-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: quote-api

Apakah service punya cukup replica untuk rollout?
Apakah PDB sesuai dengan replica count?
Apakah HPA dapat menurunkan replica di bawah safe minimum?
Apakah service dependency juga highly available?

stable network identity
stable ordinal identity
persistent volume per replica
ordered rollout/startup/shutdown semantics

database cluster
message broker cluster
stateful coordination component

Mengapa replica identity penting?
Apa state lokal yang harus dipertahankan?
Apakah state itu seharusnya pindah ke DB/cache/object storage?
Bagaimana backup/restore dilakukan?
Bagaimana rolling update memengaruhi state?

menggunakan StatefulSet hanya karena butuh disk temporary
menyimpan business source-of-truth di local volume Pod
menganggap volume otomatis membuat aplikasi reliable

one-off data repair
migration assistant
backfill
bulk export
report generation
batch reconciliation
manual operational task

Job may retry Pods
Job can run same logic more than once
success means desired completions reached
failure depends on backoffLimit and pod failures

Job logic must be idempotent
Job must have progress marker
Job must have observable output
Job must support safe resume or safe rerun

Job writes side effects without idempotency key
Job scans full production table without limit/batch
Job has no owner/runbook
Job success only means process exited 0, not business result reconciled

cleanup job
retry stuck records
archival job
reconciliation job
scheduled report
catalog/pricing sync

apiVersion: batch/v1
kind: CronJob
metadata:
  name: quote-reconciliation
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: job
              image: example/quote-reconciliation:1.0.0

overlapping runs process same data
missed schedule after controller downtime
long-running job piles up
retry duplicates side effects
job succeeds technically but reconciliation incomplete

Deployment for HTTP APIs
Job/CronJob for reconciliation/backfill/cleanup
StatefulSet for platform dependencies, not ordinary API service

1. readiness becomes false
2. load balancer stops sending new requests
3. service stops accepting long-running internal work
4. in-flight requests complete within grace period
5. background publishers/consumers stop safely
6. resources close
7. process exits cleanly

how HTTP server handles SIGTERM
whether runtime supports graceful shutdown
whether request thread pool drains
whether DB pool closes
whether Kafka consumers stop polling
whether telemetry flushes

public final class ApplicationShutdown {
    private final AtomicBoolean shuttingDown = new AtomicBoolean(false);

    public void beginShutdown() {
        if (!shuttingDown.compareAndSet(false, true)) {
            return;
        }

        // Stop accepting background work first.
        // Let in-flight request handling finish if platform supports graceful drain.
        // Close clients/pools after work is stopped.
    }

    public boolean isReady() {
        return !shuttingDown.get();
    }
}

Pod stays Pending
Deployment rollout stuck
no new replica becomes ready

insufficient CPU/memory
node selector cannot match
taint not tolerated
PVC cannot bind
image pull secret missing
quota exceeded

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe node <node-name>
kubectl get quota -n <namespace>

Pending is usually scheduling/resource/platform constraint, not application code failure.

Pod repeatedly starts and crashes
restart count increases
rollout does not complete

missing config
bad secret
startup exception
JVM exits due to invalid flags
port binding failure
startup probe too aggressive

kubectl logs <pod-name> -n <namespace> --previous
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses}'

Always inspect previous logs for crashing containers.
Current logs may only show the latest short failed attempt.

wrong image tag
image not pushed
registry unavailable
image pull secret missing
network path to registry broken
architecture mismatch

kubectl describe pod <pod-name> -n <namespace>
kubectl get secret -n <namespace>
kubectl get serviceaccount <sa-name> -n <namespace> -o yaml

Is image tag immutable?
Is registry accessible from cluster?
Is pull secret/platform identity configured?
Is promotion process copying images across environments?

kubectl rollout status deployment/<name> -n <namespace>
kubectl describe deployment <name> -n <namespace>
kubectl get rs -n <namespace>
kubectl get pod -n <namespace> -l app=<app>

readiness probe failing
new config incompatible
new DB migration not applied
new image crash loop
new version cannot talk to dependency
resource pressure

Never debug rollout only from Deployment object.
Follow Deployment -> ReplicaSet -> Pod -> Container -> Logs -> Events.

finalizer blocks deletion
volume detach issue
container ignores SIGTERM
preStop hangs
application never exits
node unreachable

kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl logs <pod-name> -n <namespace>

Does the Java process handle SIGTERM?
Does preStop have a bounded timeout?
Does terminationGracePeriodSeconds match worst-case request duration?
Are long-running requests cancelled or drained safely?

send invoice twice
publish duplicate event without idempotency key
recalculate pricing and overwrite newer value
archive records while active transaction still uses them
delete records without checkpoint

use job_run_id
use batch checkpoint
use idempotency key
use unique constraint
use reconciliation table
write progress before/after side effect deliberately
make rerun safe

order state
quote version
catalog effective date
pricing effective date
tenant boundary
currency rounding
outbox/inbox state

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: quote-api
  template:
    metadata:
      labels:
        app: quote-api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: app
          image: registry.example.com/quote-api:1.2.3
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "500m"
              memory: "768Mi"
            limits:
              memory: "1024Mi"
          startupProbe:
            httpGet:
              path: /health/startup
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            periodSeconds: 10

Traffic failing?
  -> Check endpoint/service/ingress later in networking part
Rollout failing?
  -> Deployment -> ReplicaSet -> Pod -> Container -> Logs/Events
Pod pending?
  -> Scheduler/resource/node/quota/PVC
Pod crashing?
  -> previous logs + env/config/secrets/JVM flags
Pod running but not receiving traffic?
  -> readiness condition + Service endpoints
Job duplicate?
  -> Job retry/concurrency/idempotency/checkpoint

kubectl get deploy,rs,pod -n <namespace>
kubectl describe deploy <name> -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers=true
kubectl logs <pod-name> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Is Deployment/StatefulSet/Job/CronJob the correct workload type?
Is the application stateless enough for Deployment?
Is the replica count compatible with rollout and availability needs?
Are startup/readiness/liveness probes semantically correct?
Is terminationGracePeriodSeconds aligned with shutdown behavior?
Does the app handle SIGTERM safely?
Are resource requests/limits declared and realistic?
Can rollout fail safely?
Can rollback work with current DB/API/event compatibility?
For Jobs: is logic idempotent and resumable?
For CronJobs: is concurrencyPolicy explicit?
For sidecars: is lifecycle dependency understood?
Are labels/selectors stable and non-ambiguous?
Are dashboards/alerts tied to rollout and Pod health?

no readiness probe
liveness probe depends on DB/downstream
single replica for critical service
CronJob with Allow concurrency for mutating reconciliation
Job with no idempotency story
termination grace too short for request duration
manual kubectl rollout in GitOps-managed environment

Kubernetes version and managed platform
standard workload controller for JAX-RS services
whether Helm, Kustomize, GitOps, or custom deployment tooling is used
standard labels/selectors/annotations
standard probe endpoints and semantics
standard terminationGracePeriodSeconds
whether preStop hooks are used
whether service runtime supports graceful shutdown
whether PodDisruptionBudget is required
replica count policy per environment
HPA/VPA usage
Job/CronJob standards
allowed concurrencyPolicy for CronJobs
runbook for failed rollout
runbook for failed Job/CronJob
monitoring dashboards for Deployment, Pod, Job, and CronJob
ownership boundary between application team and platform team

Every Pod can be created, made ready, removed from traffic, terminated,
replaced, retried, and rolled back without corrupting business state or
causing uncontrolled user impact.