This part does not assume CSG uses Argo CD, Flux CD, Jenkins, GitHub Actions,
GitLab CI, Azure DevOps, Helm, Kustomize, Terraform, Pulumi, Crossplane,
Atlantis, Backstage, Spinnaker, Harness, or any specific internal deployment platform.

Treat every tool, repository, pipeline, approval gate, promotion rule,
and runtime ownership model as an internal verification item.

In production engineering, deployment is not "run kubectl apply".
Deployment is a controlled state transition from source intent to runtime state.

GitOps and IaC make that transition reviewable, reproducible, auditable,
and recoverable.

Where is the desired state declared?
Who can change it?
What validates it before merge?
What reconciles it after merge?
How do we know the cluster actually matches Git?
How do we promote the same artifact across environments?
How do we rollback safely when API, DB, Kafka, config, and feature flags changed?
What is controlled by application team versus platform team?

Source state
  Code, tests, build files, migration scripts, API contracts, event contracts.

Desired runtime state
  Kubernetes manifests, Helm values, Kustomize overlays, config references,
  image tags/digests, resource limits, rollout strategy, ingress rules.

Observed runtime state
  What the cluster, cloud, database, gateway, and monitoring systems actually run.

CI builds and verifies artifacts.
CD/GitOps reconciles desired runtime state.
IaC provisions platform resources used by runtime state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-order-api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
  template:
    spec:
      serviceAccountName: quote-order-api
      containers:
        - name: app
          image: registry.example.com/quote-order-api@sha256:...
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: quote-order-api-config
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "2Gi"
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
          livenessProbe:
            httpGet:
              path: /live
              port: 8080

How much memory the JVM may consume.
How many replicas can serve traffic.
Which identity the Pod uses.
How readiness is detected.
Which image is trusted.
Which config source is mounted.
How rollout happens.

quote-order-service/
  src/
  pom.xml
  Dockerfile
  deploy/
    base/
    overlays/dev/
    overlays/staging/
    overlays/prod/

App and deployment changes are close together.
Developer can reason about code + runtime together.
Simple for small teams.

Harder to centralize platform governance.
App developers may accidentally own too much platform detail.
Promotion may require code repo commits.

quote-order-service/
  src/
  pom.xml
  Dockerfile

platform-deployments/
  services/
    quote-order-service/
      dev/
      staging/
      prod/

Clear separation between build artifact and runtime desired state.
Environment changes are auditable.
Platform team can own common deployment controls.

Developers must understand two repositories.
Traceability from code commit to deployment commit must be designed.

clusters/
  eks-prod-us-east-1/
    namespaces/
      quote-order/
        quote-order-api/
  aks-prod-eastus/
    namespaces/
      quote-order/
        quote-order-api/

Cluster state is explicit.
Good for platform governance.
Easy to detect drift against cluster ownership.

High blast radius if repo permissions are weak.
Needs strong CODEOWNERS and policy validation.

image: quote-order-api:latest

image: quote-order-api:1.42.7

image: registry.example.com/quote-order-api@sha256:abc123...

Tags can be overwritten.
Digest identifies exact artifact bytes.
Rollback becomes deterministic.
Incident RCA can prove what code actually ran.
Supply chain scanning/signing can bind to immutable artifact identity.

Rebuild separately for dev, staging, and prod.

You are not testing the same artifact you deploy.
Dependency resolution, build plugin behavior, timestamps, generated code,
or base image changes can create environment-specific artifacts.

Service is small.
Duplication is low.
Environment variation is minimal.

Copy-paste drift across environments.
Manual patching.
No shared abstraction for common labels, probes, policies, resource defaults.

deploy/
  base/
    deployment.yaml
    service.yaml
    ingress.yaml
  overlays/
    dev/
      kustomization.yaml
      patch-replicas.yaml
    prod/
      kustomization.yaml
      patch-resources.yaml

You want Kubernetes-native overlay/patch behavior.
You want base + environment overlay without templating language.

Patch stacking can become hard to reason about.
Generated output must be inspected in CI.

chart/
  templates/
    deployment.yaml
    service.yaml
    ingress.yaml
  values.yaml
  values-prod.yaml

You need parameterized reusable deployment package.
You have many services with similar structure.

Templating hides rendered Kubernetes state.
Values sprawl becomes a second programming language.
Invalid combinations can render dangerous manifests.

Review rendered manifests, not only templates or values.

helm template ./chart -f values-prod.yaml > rendered.yaml
kubeconform -strict rendered.yaml
conftest test rendered.yaml

VPC / VNet
subnets
route tables
security groups / NSG
EKS / AKS clusters
node groups / node pools
container registries
IAM roles / managed identities
DNS zones
load balancers / gateways
PostgreSQL instances
Kafka clusters
Redis clusters
S3 / Blob buckets
Key Vault / Secrets Manager
monitoring workspaces
private endpoints

Deployment
Service
Ingress
ConfigMap
ExternalSecret
ServiceAccount
HorizontalPodAutoscaler
PodDisruptionBudget
NetworkPolicy

Who owns the network?
Who owns the namespace?
Who owns the ServiceAccount/IAM binding?
Who owns the ExternalSecret definition?
Who owns DB schema migration?
Who owns API gateway route?
Who owns rollout gate?

Application team changes Terraform network rules through emergency PRs
without platform review.

Platform team changes app resource limits without service owner visibility.

Maven build reproducibility
unit tests
JAX-RS endpoint tests
OpenAPI lint/diff
AsyncAPI/schema compatibility
DB migration safety
container image security
SBOM generation
signature/provenance
Kubernetes manifest policy
resource requests/limits
probe existence
secret reference correctness
feature flag default state
observability instrumentation

replica count
resource size
external endpoint hostnames
feature flag targeting
rate limits
log level within allowed policy
cloud account/subscription/project
secret values

different code artifact
different database migration history
different API contract
different event schema compatibility mode
different authorization policy
different JSON serialization config
different retry timeout behavior
different timezone/currency behavior
different feature flag default semantics

Which artifact was tested?
Which digest is being promoted?
Which config changed between staging and prod?
Which migrations already ran?
Which feature flags are enabled?
What is the rollback path?

Someone manually changed replicas with kubectl scale.
A hotfix changed a ConfigMap directly in cluster.
A Secret was rotated outside the declared path.
A platform controller mutated ingress annotations.
An HPA changed replica count as designed.
A cloud load balancer changed IP or health state.

What should GitOps correct automatically?
What should GitOps alert but not auto-correct?
What fields are ignored because controllers own them?
What manual operations are allowed during incident?
How are emergency changes reconciled back to Git?

Production incident fixed manually.
Git not updated.
Next reconciliation or rollout removes the fix.

Emergency manual change is documented, time-limited, and backported to desired state.

backward-compatible application deployment
backward-compatible database migration
feature flag controlled behavior
roll-forward-first mindset for data changes
explicit rollback playbook for runtime-only changes

Every Pod starts.
Every Pod tries to run migration.
One Pod succeeds.
Others fail or block.
Application rollout becomes migration coordination mechanism.

Dedicated migration Job before deployment.
Pipeline-controlled migration stage.
DB migration service with lock and explicit approval.
Expand-contract migration across releases.

Release N:
  expand schema: add nullable column / new table / compatible index
  deploy app that can read old and new

Release N+1:
  backfill data
  enable new behavior via feature flag

Release N+2:
  stop writing old field
  verify no old readers

Release N+3:
  contract schema: remove old column only after compatibility window

Reducing timeout can break slow downstream calls.
Changing retry count can overload dependencies.
Changing feature flag default can expose unfinished behavior.
Changing tenant config can isolate or leak tenant behavior.
Changing Kafka topic name can silently stop event processing.
Changing DB pool size can saturate database.

owner review
environment diff
policy validation
rollout plan
rollback plan
observability expectation

rotation schedule
dual-read/dual-write where needed
old secret expiry window
credential usage dashboard
failed authentication alert

canary by replica percentage
canary by traffic weight
blue-green deployment
header-based routing
tenant-based rollout
region-based rollout
feature-flag-based behavior enablement

Deployment rollout controls which binary serves traffic.
Feature flags control which behavior path executes.
Gateway traffic splitting controls which route receives traffic.

No latest image tag.
All containers must have resource requests and limits.
Readiness probe required.
Liveness probe must not hit dependency-heavy endpoint.
No privileged container unless approved.
No hostPath unless approved.
No plaintext secret in ConfigMap.
Ingress must use approved TLS policy.
ExternalSecret must reference allowed secret store.
ServiceAccount must be least privilege.
Production namespace requires CODEOWNERS approval.

package kubernetes.admission

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("container %s uses mutable latest tag", [container.name])
}

unsafe desired state must fail before merge or before admission.

build duration
test duration
failed quality gates
artifact digest
SBOM generation
scan result
deployment commit
GitOps sync status
reconciliation latency
rollout progress
pod readiness time
restart count after deployment
error rate after deployment
latency after deployment
customer-impacting alerts after deployment
rollback frequency
change failure rate
mean time to restore

service.name
service.version
image.digest
git.commit
build.id
deployment.environment
cluster.name
namespace
release.id

Did Maven fail?
Did tests fail?
Did contract diff fail?
Did image build fail?
Did scanner block?
Did artifact signing fail?
Did manifest validation fail?

mvn -version
mvn -DskipTests=false verify
mvn dependency:tree

Did controller see the desired state commit?
Did rendered manifest validate?
Did admission controller reject it?
Did RBAC block apply?
Did namespace quota block it?
Did CRD exist?
Did sync wave/order fail?

kubectl get events -n quote-order --sort-by=.metadata.creationTimestamp
kubectl describe deployment quote-order-api -n quote-order
kubectl get pods -n quote-order
kubectl describe pod <pod> -n quote-order

Did image pull fail?
Did container start?
Did readiness pass?
Did app bind to correct port?
Did JVM OOM before ready?
Did config/secret mount fail?
Did DB migration block startup?
Did network policy block dependencies?

Did error rate increase?
Did latency increase?
Did downstream calls fail?
Did DB pool saturate?
Did Kafka lag grow?
Did feature flag enable risky path?
Did only one tenant/region fail?

Issue disappears after manual kubectl edit, then reappears after sync.

Runtime changed outside desired state.

Backport emergency change to Git or revert manual change explicitly.

Rollback to previous tag still runs wrong binary.

Tag was overwritten.

Pin immutable digest and preserve artifact history.

Works in staging, fails in prod.

Different timeout, auth issuer, DB pool, topic, schema compatibility, or feature flag.

Generate environment diff and validate allowed differences.

Image rollback breaks because schema is already contracted.

Database migration was not backward-compatible.

Use expand-contract and roll-forward strategy.

A bad manifest change affects many namespaces/clusters.

Controller scope or repo permissions are too broad.

Limit scope, use CODEOWNERS, policy gates, environment separation.

Artifact
  Is image pinned by digest or immutable version?
  Is SBOM/scanning/signing complete?
  Is the same artifact promoted across environments?

Kubernetes manifest
  Are resource requests/limits reasonable?
  Are readiness/liveness/startup probes correct?
  Does rollout strategy match service behavior?
  Is termination grace period compatible with request processing?
  Are ServiceAccount and RBAC least privilege?

Config
  Are config changes explicit?
  Are environment differences expected and documented?
  Are safe defaults defined?
  Can config be rolled back?

Secret
  Are secrets referenced, not embedded?
  Is rotation considered?
  Are logs redacted?

Network
  Are ingress/DNS/TLS changes understood?
  Are private endpoint and network policy effects reviewed?

Database
  Is migration backward-compatible?
  Is migration ordering clear?
  Is rollback/roll-forward strategy defined?

Event/API contracts
  Are OpenAPI/AsyncAPI/schema changes compatible?
  Are generated clients affected?
  Are consumers identified?

Operations
  Is there a dashboard/alert for rollout?
  Is rollback playbook updated?
  Is customer impact understood?

Repository model
  Where are Kubernetes manifests stored?
  Is there separate app repo and deployment repo?
  Who owns environment overlays?

GitOps controller
  Is Argo CD, Flux, or another tool used?
  Which clusters/namespaces does it reconcile?
  What is the sync policy?
  Is auto-sync enabled?
  Is prune enabled?
  Are manual changes allowed?

IaC
  Is Terraform, Pulumi, Crossplane, CloudFormation, Bicep, or ARM used?
  Which team owns cloud network, cluster, database, registry, gateway, and identity?

Artifact promotion
  Are images promoted by tag or digest?
  Are artifacts signed?
  Is SBOM generated?
  Is the same artifact used in dev/staging/prod?

Validation
  Are manifests rendered in CI?
  Is OpenAPI/AsyncAPI/schema checked?
  Are policy-as-code checks used?
  Are resource/probe/security policies enforced?

Migration
  How are Liquibase/Flyway migrations executed?
  Are migrations part of app startup, CI/CD, Job, or separate process?
  Is expand-contract required?

Release strategy
  Is canary/blue-green/progressive delivery supported?
  Are feature flags tied to releases?
  What is the rollback playbook?

Observability
  Are deployment metadata attached to logs/traces/metrics?
  Is GitOps sync status observable?
  Are rollout failures alerted?

Can we prove what changed?
Can we reproduce it?
Can we review it?
Can we promote it safely?
Can we detect drift?
Can we rollback or roll forward?
Can we explain an incident from artifact to runtime?