Part 033 — Progressive Delivery, Zero-Downtime Reload, and Blast Radius Control

1. Core Mental Model

Traffic-layer deployment is different from application deployment.

When a Java/JAX-RS service is deployed badly, one service may fail. When an NGINX or Ingress routing change is deployed badly, the failure may affect many services, many tenants, many customers, or an entire environment.

NGINX and Ingress changes control:

• where requests go,
• whether TLS works,
• whether headers survive,
• whether large requests are accepted,
• whether long-running calls time out,
• whether WebSocket/SSE connections stay alive,
• whether old and new backends receive traffic safely,
• whether rollback is possible without a second incident.

The safe deployment mental model:

change proposal
  -> config/render validation
  -> risk classification
  -> staged rollout
  -> small traffic exposure
  -> observe correctness and health
  -> expand traffic
  -> keep rollback path warm
  -> close with post-change evidence

Key principle:

Progressive delivery at the NGINX layer is not about being fancy. It is about reducing the number of requests harmed by an incorrect traffic decision.

2. What "Zero Downtime" Really Means

"Zero downtime" is often used too casually.

At traffic layer, it can mean different things:

These are not equivalent.

A reload may preserve listeners but still break:

• WebSocket connections,
• SSE streams,
• in-flight uploads,
• long downloads,
• sticky session routing,
• traffic to an old path,
• canary weight expectation,
• header assumptions in Java/JAX-RS backend.

Senior-engineer framing:

Treat zero downtime as an objective to verify, not a property to assume.

3. NGINX Reload vs Restart

NGINX reload and restart are operationally different.

Reload is preferred for config changes, but reload does not make a logically bad config safe.

Bad config examples that can reload successfully:

• route points to wrong service,
• timeout is syntactically valid but too low,
• rewrite strips wrong path prefix,
• CORS allows wrong origin,
• auth snippet bypasses expected check,
• rate limit key collapses many users into one bucket,
• TLS secret points to valid but wrong certificate,
• backend protocol is HTTP while backend expects HTTPS.

4. Safe Reload Lifecycle

A safe reload should be treated as a controlled lifecycle.

1. Render final config
2. Validate syntax
3. Validate policy
4. Validate route intent
5. Deploy to lower environment
6. Smoke test externally
7. Deploy to small production scope
8. Observe health
9. Expand rollout
10. Preserve rollback path

For standalone NGINX:

nginx -t
nginx -s reload

For containerized NGINX:

kubectl exec deploy/nginx -- nginx -t
kubectl rollout restart deploy/nginx

For Ingress Controller:

kubectl apply -f ingress.yaml
kubectl describe ingress <name> -n <namespace>
kubectl logs deploy/<ingress-controller> -n <controller-namespace>

For GitOps:

PR merged
  -> GitOps controller syncs
  -> Ingress/ConfigMap changes
  -> controller renders new NGINX config
  -> controller reloads NGINX
  -> traffic behavior changes

Risk:

kubectl apply success does not mean traffic behavior is correct.

5. Kubernetes Rolling Update Mental Model

Kubernetes rolling update is controlled replacement of Pods.

For NGINX/Ingress-related systems, rolling update affects:

• controller pod availability,
• NGINX worker lifecycle,
• load balancer target health,
• readiness state,
• in-flight connection survival,
• external LB target deregistration delay,
• long-lived connections,
• metrics continuity.

Important deployment fields:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
terminationGracePeriodSeconds: 60
readinessProbe:
  httpGet:
    path: /healthz
    port: 10254
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

This example is not a universal recommendation. It shows the types of controls that matter.

Senior review questions:

• Can the old pod drain before being removed from LB targets?
• Does readiness become false before shutdown starts?
• Is termination grace longer than expected in-flight request duration?
• Does the cloud LB have its own deregistration delay?
• Are WebSocket/SSE clients expected to reconnect?
• Are controller replicas spread across nodes/zones?

6. Readiness, Liveness, and Startup Probes

Probe design can make or break safe deployment.

For NGINX controller or reverse proxy:

• readiness should reflect ability to serve configured traffic,
• liveness should not restart during normal reload latency,
• startup should handle cold config generation,
• metrics endpoint should not be mistaken for readiness,
• health endpoint should not bypass real config failure.

For Java/JAX-RS upstream:

• readiness must become false before app stops accepting traffic,
• readiness should include critical dependency only if failure should remove the pod from serving,
• liveness should not fail because one downstream dependency is slow,
• shutdown must stop accepting new requests before closing old ones.

7. Blue-Green Deployment

Blue-green means two complete versions exist side by side.

client
  -> load balancer / NGINX / Ingress
       -> blue backend  v1  current live
       -> green backend v2  candidate

Cutover changes routing from blue to green.

Advantages:

• clear rollback to blue,
• easy pre-warm green,
• useful for large compatibility changes,
• good for database/API migration when both versions must be validated.

Risks:

• expensive duplicate capacity,
• state divergence,
• sticky session confusion,
• cache inconsistency,
• background job duplication,
• database backward-compatibility issue,
• TLS/hostname/cookie mismatch.

NGINX-style conceptual example:

upstream quote_order_blue {
    server quote-order-v1:8080;
}

upstream quote_order_green {
    server quote-order-v2:8080;
}

server {
    listen 443 ssl;
    server_name quote.example.internal;

    location /api/quote-order/ {
        proxy_pass http://quote_order_green;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

In Kubernetes, blue-green may be implemented by:

• changing Service selector,
• changing Ingress backend service,
• using separate Ingress objects,
• changing cloud load balancer target,
• using a progressive delivery controller.

Internal verification checklist:

• Which object performs the cutover?
• Is rollback a single revert or multiple coordinated changes?
• Does green have identical TLS, auth, CORS, headers, and observability?
• Are database migrations backward-compatible?
• Are clients pinned by cookie/session to old version?

8. Canary Deployment

Canary means exposing a small portion of traffic to a new version.

95% -> stable
 5% -> canary

Canary can be based on:

• percentage weight,
• header,
• cookie,
• source IP,
• tenant,
• user segment,
• path,
• hostname,
• environment.

Canary is useful when:

• risk is uncertain,
• traffic volume is enough to observe signal,
• metrics can distinguish stable vs canary,
• rollback can be fast,
• backend behavior is backward-compatible.

Canary is dangerous when:

• request flows require multi-step consistency,
• state is not compatible across versions,
• only some endpoints are canaried but others are not,
• metrics cannot distinguish version,
• sticky sessions are required but not configured,
• async side effects are not isolated.

Ingress canary concept:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: quote-order-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  ingressClassName: nginx
  rules:
    - host: quote.example.internal
      http:
        paths:
          - path: /api/quote-order
            pathType: Prefix
            backend:
              service:
                name: quote-order-v2
                port:
                  number: 8080

This is implementation-specific. Confirm exact annotation support in the ingress controller used internally.

9. Header-Based Canary

Header-based canary is useful for internal testing and targeted verification.

Example use cases:

• QA sends X-Canary: true,
• one internal tenant tests new backend,
• synthetic monitoring validates new version,
• selected traffic from a test client is routed to candidate.

Conceptual example:

map $http_x_canary $quote_order_upstream {
    default quote_order_stable;
    "true"  quote_order_canary;
}

location /api/quote-order/ {
    proxy_pass http://$quote_order_upstream;
}

Risk:

• headers can be spoofed by external clients unless stripped or controlled,
• variables in proxy_pass can change DNS/resolution behavior,
• observability must capture canary marker,
• cache key must include canary dimension if caching exists.

Internal verification checklist:

• Who is allowed to set canary headers?
• Are external client-supplied control headers stripped first?
• Is canary routing visible in logs?
• Does Java/JAX-RS backend log version/commit?
• Does synthetic monitoring exercise canary path?

10. Cookie-Based Canary and Sticky Sessions

Cookie-based routing can be useful when a user flow must stay on one version.

Common patterns:

first request
  -> assign cookie version=v2
next requests
  -> route consistently to v2

Good for:

• browser flows,
• session-bound behavior,
• multi-step quote creation,
• checkout-like flows,
• progressive frontend/backend compatibility testing.

Risks:

• cookie domain/path mismatch,
• SameSite/Secure issues,
• sticky cookie conflicts with application session cookie,
• rollback leaves users pinned to bad version,
• mobile/API clients may not preserve cookies,
• tenant-level consistency may be more important than user-level consistency.

Senior review question:

Is the unit of safe stickiness user, tenant, session, quote, order, or request?

For quote/order systems, request-level canary may be unsafe if a workflow spans multiple endpoints and state transitions.

11. Shadow Traffic and Traffic Mirroring

Shadow traffic means copying production requests to a candidate backend while the client still receives response from stable backend.

client
  -> NGINX
       -> stable backend -> response to client
       -> shadow backend -> response ignored

Useful for:

• validating read-only behavior,
• warming cache,
• performance comparison,
• detecting compatibility issues,
• observing candidate latency under real traffic shape.

Dangerous for:

• write requests,
• non-idempotent operations,
• quote/order creation,
• payment/order submission,
• side effects like Kafka publish, email, audit record, inventory reservation.

Shadow traffic rule:

Never mirror side-effecting requests unless the shadow backend is fully isolated and cannot mutate production state.

For Java/JAX-RS backend, validate:

• database writes disabled or isolated,
• Kafka producers disabled or routed to test topic,
• downstream integrations stubbed or isolated,
• idempotency keys not reused against real systems,
• audit/compliance logs not polluted,
• auth tokens and PII handling are compliant.

12. Weighted Traffic and Progressive Expansion

Weighted rollout should be evidence-based, not ritual-based.

Example expansion:

0%  -> deploy only, no live traffic
1%  -> smoke production traffic
5%  -> observe basic health
10% -> observe edge cases
25% -> observe scaling behavior
50% -> compare stable/canary
100% -> complete rollout

Better gate criteria:

• no significant increase in 5xx,
• no increase in 499/504,
• no latency regression above threshold,
• no spike in Java exceptions,
• no database pool saturation,
• no Kafka lag regression,
• no suspicious business metric drop,
• no tenant-specific error spike,
• no security/auth anomaly.

Bad gate criteria:

• "pod is running",
• "deployment succeeded",
• "no one complained",
• "CPU looks okay",
• "we waited five minutes".

13. Feature Flags and NGINX Routing

Feature flags and NGINX routing solve different problems.

Common anti-pattern:

NGINX sends 10% to v2
feature flag enables 50% of tenants inside v2
result: actual exposure is unclear

Preferred approach:

• define exposure unit,
• define owner of routing decision,
• log both route version and feature flag state,
• avoid conflicting traffic controls,
• ensure rollback path for both route and flag.

14. Backward-Compatible Routing

Routing changes must respect API compatibility.

Safe deployment requires compatibility across:

• URL path,
• HTTP method,
• request body schema,
• response body schema,
• status code behavior,
• headers,
• cookies,
• authentication scheme,
• CORS behavior,
• redirect behavior,
• idempotency behavior,
• timeout expectations.

Backward compatibility examples:

15. Draining Connections

Connection draining is the process of removing an instance from new traffic while allowing existing requests to finish.

Important layers:

cloud load balancer target draining
  -> Kubernetes endpoint removal
  -> NGINX worker graceful shutdown
  -> upstream Java pod graceful shutdown
  -> database / downstream request completion

Problems happen when one layer drains faster than another.

Examples:

• LB sends traffic to terminating ingress pod,
• Ingress routes to terminating Java pod,
• Java pod closes socket while NGINX is reading response,
• NGINX logs 502/504,
• client sees intermittent failure during rollout.

Verification questions:

• Does readiness fail before application shutdown?
• Does the Service remove endpoint quickly enough?
• Is terminationGracePeriodSeconds long enough?
• Is LB deregistration delay longer than request duration?
• Do clients retry safely?
• Are long-lived connections expected to reconnect?

16. Long-Lived Connections During Rollout

WebSocket, SSE, and long polling complicate deployment.

They may last:

• minutes,
• hours,
• until client disconnect,
• until load balancer idle timeout,
• until backend restart.

Rollout questions:

• Are clients reconnect-capable?
• Is reconnect jittered?
• Does reconnect preserve session state?
• Does backend support resume?
• Does NGINX timeout align with LB timeout?
• Does deployment drain or abruptly terminate long connections?

For real-time endpoints, "zero downtime" may actually mean:

existing connections may drop
but clients reconnect quickly
and no durable state is lost

That must be documented explicitly.

17. Production Change Window

Not every change needs a formal change window, but traffic-layer changes need risk classification.

High-risk NGINX/Ingress changes:

• TLS certificate replacement,
• wildcard host routing,
• default backend change,
• auth boundary change,
• major timeout change,
• body size change for critical API,
• rate limit change,
• rewrite rule change,
• CORS change,
• public endpoint exposure,
• cross-tenant routing change,
• controller upgrade,
• cloud load balancer migration.

Low-risk changes may include:

• adding log field,
• adding internal-only test route,
• increasing observability sampling,
• adding synthetic-only canary path,
• adding non-public health endpoint.

Even low-risk changes need rollback.

18. Blast Radius Control

Blast radius is the maximum damage a bad change can cause.

Reduce blast radius by scoping change by:

• environment,
• namespace,
• hostname,
• path,
• tenant,
• region,
• availability zone,
• ingress class,
• controller instance,
• load balancer,
• percentage weight,
• client segment.

Bad pattern:

one global ConfigMap change affects every Ingress in the cluster

Safer pattern:

one namespace-specific Ingress change affects one service path

Principal-level question:

Can this traffic-layer change accidentally affect services that are not part of this PR?

19. Rollback Strategy

Rollback must be planned before rollout.

A good rollback has:

• single owner,
• clear trigger condition,
• known command or Git revert,
• known expected recovery time,
• validation after rollback,
• awareness of irreversible side effects.

Rollback options:

Rollback is not always enough.

If a bad route caused writes to wrong backend, rollback stops new damage but does not repair state.

20. Observability Gates for Rollout

Rollout needs objective gates.

Minimum NGINX/Ingress signals:

• request rate,
• 2xx/3xx/4xx/5xx distribution,
• 499 count,
• 502 count,
• 503 count,
• 504 count,
• request time,
• upstream response time,
• upstream connect failures,
• upstream address distribution,
• TLS handshake errors,
• request body size rejection,
• rate limit hit count,
• auth failure rate.

Backend signals:

• Java exception rate,
• thread pool utilization,
• HTTP server request latency,
• database connection pool usage,
• GC pause,
• downstream timeout,
• Kafka lag,
• business operation success/failure.

Business signals for quote/order systems:

• quote creation success,
• quote update success,
• order submission success,
• pricing call success,
• catalog lookup success,
• external integration error rate,
• tenant-specific failures.

21. Example Rollout Decision Table

22. Safe Deployment Checklist

Before rollout:

• final config is rendered,
• syntax validation passes,
• policy validation passes,
• route ownership is clear,
• affected host/path/tenant is documented,
• timeout/body/header/rate limit changes are reviewed,
• TLS and secret references are valid,
• smoke test plan exists,
• rollback plan exists,
• dashboards are open,
• on-call/SRE/platform contact is known.

During rollout:

• apply to smallest safe scope,
• verify controller accepted config,
• run smoke test through real public/internal route,
• watch access/error logs,
• compare stable vs candidate metrics,
• pause on unclear signal,
• do not expand exposure without evidence.

After rollout:

• verify no delayed failures,
• check long-running endpoints,
• check tenant-specific metrics,
• check error budget impact,
• update runbook if needed,
• record post-change evidence.

23. PR Review Checklist

For NGINX/Ingress rollout PRs, ask:

Scope

• What host/path/namespace/service is affected?
• Is this global or local config?
• Could another service be affected indirectly?

Compatibility

• Does public URL change?
• Does backend base path change?
• Does rewrite preserve expected JAX-RS path?
• Are old and new backend APIs compatible?

Safety

• Is there a canary/blue-green path?
• Is rollback one command or one revert?
• Is the change safe for long-lived connections?
• Are database and downstream side effects compatible?

Observability

• Can we separate stable and canary traffic?
• Are upstream timing fields logged?
• Are 499/502/503/504 visible?
• Is request ID propagated to Java logs?

Security

• Are identity headers protected?
• Are snippets safe or forbidden?
• Does CORS/auth/rate limit change?
• Does TLS termination point change?

Performance

• Does the change affect buffering, compression, keepalive, or timeouts?
• Could it increase connection count or memory/disk temp usage?
• Does it affect large uploads/downloads?

24. Internal Verification Checklist

Verify in internal CSG/team environment:

• Which progressive delivery mechanism is standard: canary annotation, Argo Rollouts, Flagger, service mesh, gateway, or manual process?
• Which NGINX/Ingrress controller implementation is used?
• Are canary annotations enabled and governed?
• Are snippets enabled or disabled?
• Is there a global ConfigMap that affects all services?
• Who owns controller upgrades?
• Who owns TLS certificate rollout?
• Who owns cloud load balancer changes?
• What is the rollback SLA for bad ingress/config change?
• Are dashboards available per host/path/service/version?
• Can logs distinguish stable vs canary traffic?
• Are synthetic checks routed through the same path as real clients?
• Are long-lived connections considered in change review?
• Are production change windows required for global traffic-layer changes?

25. Failure Modes

26. Practical Production Rollout Template

Change:
  - What is changing?
  - Which host/path/service/tenant is affected?
  - Which environment first?

Risk:
  - Routing risk:
  - TLS risk:
  - Auth risk:
  - Timeout risk:
  - Body/header risk:
  - Streaming risk:
  - Blast radius:

Validation:
  - Rendered config checked:
  - Syntax validation:
  - Policy validation:
  - Smoke test:
  - Synthetic check:

Rollout:
  - Initial exposure:
  - Expansion steps:
  - Gate metrics:
  - Owner watching dashboard:

Rollback:
  - Trigger:
  - Command/revert:
  - Expected recovery:
  - Post-rollback validation:

27. Senior Engineer Heuristics

Use these heuristics during design/review:

1. If a change touches global ConfigMap, assume cluster-wide blast radius until proven otherwise.
2. If a change touches auth, assume security impact until proven otherwise.
3. If a change touches rewrite/path, test with real client-visible URLs, not only backend URLs.
4. If a change touches timeout, reason through the entire timeout chain.
5. If a change touches TLS, verify certificate chain, SNI, and termination point.
6. If a change touches rate limit, check false positives and tenant concentration.
7. If a change touches streaming, test with buffering and long-lived connection behavior.
8. If a rollout cannot be observed, it should not be expanded.
9. If rollback requires multiple teams and no runbook, the rollout risk is higher than it looks.
10. If the route controls quote/order state transitions, request-level canary may be unsafe.

28. Closing Mental Model

NGINX progressive delivery is not only about rollout technique. It is about controlling uncertainty.

A mature traffic-layer rollout has:

• explicit scope,
• staged exposure,
• compatibility awareness,
• measurable health gates,
• fast rollback,
• minimal blast radius,
• clear ownership,
• production evidence.

Final takeaway:

In enterprise Java/JAX-RS systems, NGINX changes should be reviewed like production control-plane changes, not like harmless config edits.