This part does not assume CSG uses AWS PrivateLink, VPC Endpoint,
Azure Private Endpoint, Azure Private Link, ECR, ACR, CloudWatch,
Azure Monitor, VPN, Direct Connect, ExpressRoute, Transit Gateway,
Azure Virtual WAN, private DNS zone, split-horizon DNS, service mesh,
or any specific hybrid connectivity model.

Treat every platform and network detail as an internal verification item.

A production JAX-RS service is not only a Java process.
It is a participant in a platform graph:

image registry -> CI/CD -> cluster -> pod identity -> private endpoint -> DNS -> TLS -> monitoring -> incident workflow

When this graph breaks, the Java code may be correct but the service is still unavailable.

Can the pod pull the image?
Can the pod resolve private DNS names?
Can the pod reach private endpoints?
Can cloud services authenticate the workload identity?
Can telemetry leave the cluster?
Can on-prem systems reach the service path?
Can the service validate TLS certificates?
Can we prove where the request or connection failed?

"The application is failing because the code is wrong."

A production request can fail before Java sees it.
A dependency call can fail before the SDK sends a valid request.
A deployment can fail before a container starts.
An incident can be caused by DNS, TLS, registry auth, routing, firewall,
identity binding, or observability pipeline failure.

Private endpoint is not only routing.
It usually requires correct DNS, network policy, firewall, identity, and TLS.

Is traffic using public internet, NAT gateway, gateway endpoint, or interface endpoint?
Is private DNS enabled for the endpoint?
Does the pod subnet route to the endpoint ENI?
Does the endpoint security group allow source traffic?
Does IAM allow the action after network succeeds?
Does the SDK endpoint override point to the intended service endpoint?

Is the service exposed through public endpoint, private endpoint, or both?
Is the private DNS zone linked to the VNet?
Does AKS use Azure CNI, overlay, or another network model?
Are NSG/firewall/UDR rules allowing traffic?
Is ACR reachable privately by the node/pod path?
Is Azure Monitor ingestion available from the cluster network?

Deploy by immutable digest when possible.
Do not rely only on mutable tags for regulated or high-risk production rollout.

containers:
  - name: quote-order-api
    image: registry.example.com/csg/quote-order-api@sha256:abc123...
    imagePullPolicy: IfNotPresent

[ ] Which registry is used: ECR, ACR, internal registry, or another registry?
[ ] Is image promotion tag-based, digest-based, or both?
[ ] Are images scanned before promotion?
[ ] Are images signed?
[ ] Are base images governed?
[ ] Can cluster nodes pull via private network?
[ ] Is imagePullSecret or workload/node identity used?
[ ] Is rollback image still available?
[ ] Is registry access observable during deployment failure?

Application is healthy != telemetry is healthy.
Telemetry is flowing != telemetry is useful.

If telemetry path is broken, mean time to detection and mean time to recovery increase.
If telemetry labels are inconsistent, dashboard queries silently miss affected pods.
If telemetry is too noisy, real incidents drown in irrelevant alerts.

[ ] Where do logs go?
[ ] Where do metrics go?
[ ] Where do traces go?
[ ] Is OpenTelemetry collector used?
[ ] Is telemetry egress public or private?
[ ] Are service.name, environment, cluster, namespace, and tenant labels standardized?
[ ] Are telemetry endpoints reachable from all node pools/subnets?
[ ] Are ingestion failures monitored?
[ ] Are alerts tied to SLOs or only infrastructure symptoms?

Does this service have a dependency path outside the cluster/cloud region?
If yes, what is the routing, DNS, TLS, firewall, identity, timeout, retry, and runbook model?

The JVM may cache DNS lookups.
If DNS changes during failover, a running process may not immediately observe it,
depending on JVM/networkaddress.cache.ttl/security configuration and runtime behavior.

kubectl exec -n <namespace> <pod> -- nslookup <host>
kubectl exec -n <namespace> <pod> -- dig <host>
kubectl exec -n <namespace> <pod> -- getent hosts <host>
kubectl exec -n <namespace> <pod> -- cat /etc/resolv.conf

Log dependency hostnames at startup.
Log resolved endpoint family only when safe and non-sensitive.
Avoid logging secrets, tokens, signed URLs, or full connection strings.
Expose dependency readiness only if it does not create cascading startup failure.

# From local or debug container if openssl exists
openssl s_client -connect example.internal:443 -servername example.internal -showcerts

# For Java-level debugging in non-production reproduction only
-Djavax.net.debug=ssl,handshake

[ ] Which truststore is used by the JVM?
[ ] Are internal CA certificates injected into the image or mounted at runtime?
[ ] Is mTLS used for service-to-service traffic?
[ ] Where does TLS terminate inbound traffic?
[ ] Is TLS re-encrypted between gateway/ingress/service?
[ ] Are certificates rotated automatically?
[ ] Are certificate expiration alerts configured?
[ ] Is SNI required by downstream endpoints?

The pod resolves storage.company.internal to a private IP.
The TLS certificate is issued for storage.cloudprovider.com.
The app connects using storage.company.internal.
TLS hostname validation fails.

The SDK expects account.blob.core.windows.net or bucket.s3.region.amazonaws.com.
Internal DNS overrides the name to a private endpoint.
The private IP is correct, but firewall blocks the subnet.
The app reports timeout.

Do not "fix" TLS by disabling certificate validation.
Fix DNS name, certificate, trust chain, endpoint configuration, or SDK endpoint usage.

// Do not approve this in production PRs.
// Trust-all TLS clients turn network misconfiguration into security exposure.

request latency != dependency latency
HTTP 500 != network timeout
auth failure != network failure
DNS failure != TLS failure
connection timeout != response timeout

dependency.name
dependency.operation
failure.boundary = dns | connect | tls | auth | authorization | timeout | throttling | protocol | application
retryable = true | false

1. Identify dependency name, host, port, protocol, and operation.
2. Check whether failure is DNS, connect, TLS, auth, authorization, timeout, or application response.
3. Exec into same namespace/pod class when safe.
4. Resolve DNS from inside cluster.
5. Test TCP connectivity.
6. Test TLS handshake with SNI.
7. Check network policies, security groups, NSG, firewall, route table.
8. Check workload identity/IAM/RBAC/resource policy.
9. Check dependency-side logs/audit trail.
10. Check recent deployment/config/DNS/cert/network changes.
11. Decide rollback, config fix, route fix, credential fix, or dependency escalation.

kubectl get pods -n <namespace>
kubectl describe pod -n <namespace> <pod>
kubectl logs -n <namespace> <pod> --previous
kubectl exec -n <namespace> <pod> -- getent hosts <host>
kubectl exec -n <namespace> <pod> -- nc -vz <host> 443
kubectl exec -n <namespace> <pod> -- curl -vk https://<host>/health

ImagePullBackOff
ErrImagePull
unauthorized: authentication required
manifest unknown
x509 certificate signed by unknown authority
connection timeout to registry

1. Confirm image reference and tag/digest.
2. Confirm image exists in registry.
3. Confirm namespace has correct pull secret or node/workload identity.
4. Confirm registry policy allows cluster identity.
5. Confirm private DNS resolves registry endpoint.
6. Confirm network path from node/subnet to registry.
7. Confirm image architecture matches node architecture.
8. Confirm admission controller/signature/scanning policy did not block pull.
9. Confirm rollback image still exists.

Can this image be pulled by production clusters before rollout begins?
Is the tag immutable or is digest pinned?
Is there a rollback artifact?
Is the image signed/scanned according to policy?

1. Check whether app is producing telemetry locally.
2. Check agent/sidecar/collector logs.
3. Check collector queue/drop metrics.
4. Check egress/private endpoint to observability backend.
5. Check credentials/workspace/project key.
6. Check service.name/environment labels.
7. Check sampling config.
8. Check metric cardinality/relabeling/drop rules.
9. Check alert query changed or labels drifted.
10. Add incident note that observability gap affected MTTR.

[ ] What systems are reached outside cloud/cluster?
[ ] Is dependency synchronous or asynchronous?
[ ] Is route public, private, VPN, Direct Connect, ExpressRoute, or partner link?
[ ] Are source IPs allowlisted?
[ ] Is NAT involved?
[ ] Is DNS cloud-owned, on-prem-owned, or split?
[ ] Who owns certificate lifecycle?
[ ] Are maintenance windows known?
[ ] What timeout/retry budget protects the service?
[ ] What fallback exists if on-prem system is unavailable?
[ ] Is customer impact documented?

Hybrid dependencies are often slower and less elastic than cloud-native dependencies.
Do not apply aggressive retry or high concurrency without explicit capacity agreement.

Hardcode resolved IP or rely on indefinite JVM DNS cache.

Use stable DNS names, controlled TTL, verified private DNS, and dependency reconnect behavior.

Disable certificate and hostname validation.

Fix truststore, certificate SAN, SNI, DNS, or endpoint selection.

Deployment assumes image pull will work because CI pushed successfully.

Verify registry access from cluster, image immutability, scan status, and rollback artifact.

Telemetry failure is invisible until incident.

Monitor collector health, ingestion failure, label quality, and alert query coverage.

Endpoint exists, but workloads still resolve public hostname or wrong private zone.

Treat private endpoint, DNS, route, firewall, identity, and TLS as one deployable capability.

[ ] Does the change alter DNS names, private zones, endpoint hostnames, or service discovery?
[ ] Does the change alter TLS certificate, truststore, mTLS, or SNI behavior?
[ ] Does the change alter registry path, image tag, digest, signing, or scan policy?
[ ] Does the change alter telemetry destination, service name, labels, or collector config?
[ ] Does the change introduce public egress where private path is expected?
[ ] Does the change introduce private endpoint dependency without DNS verification?
[ ] Does the change affect on-prem/partner allowlist or routing?
[ ] Does the change include rollback strategy?
[ ] Does the change include runbook or operational note?
[ ] Can we validate the path from inside the cluster before production traffic uses it?

[ ] TLS validation is disabled.
[ ] Private IPs are hardcoded.
[ ] Secrets/connection strings are logged.
[ ] No rollback artifact exists for deployment change.
[ ] Monitoring labels drift from dashboard/alert expectations.
[ ] Hybrid dependency has no timeout/retry/fallback model.

Runtime/platform:
[ ] Which cloud(s) are used: AWS, Azure, both, on-prem, hybrid?
[ ] Which clusters run the service: EKS, AKS, OpenShift, other Kubernetes, VM, app server?
[ ] Is traffic public, private, or hybrid?

Private endpoints:
[ ] Are AWS VPC Endpoints/PrivateLink used?
[ ] Are Azure Private Endpoints/Private Link used?
[ ] Which dependencies require private access?
[ ] Who owns private DNS zones?
[ ] How are endpoint policies/firewalls reviewed?

Registry:
[ ] Which registry is used: ECR, ACR, internal?
[ ] Are image tags immutable?
[ ] Are deployments digest-pinned?
[ ] Is image signing enforced?
[ ] Is vulnerability scanning a hard gate?

Monitoring:
[ ] Which logging backend is used?
[ ] Which metrics backend is used?
[ ] Which tracing backend is used?
[ ] Is OpenTelemetry collector used?
[ ] Are telemetry ingestion failures alerted?

Hybrid:
[ ] Does service call on-prem or partner systems?
[ ] What network path is used?
[ ] Are source IPs allowlisted?
[ ] Is DNS split-horizon?
[ ] What is the escalation path when hybrid link fails?

DNS/TLS:
[ ] Is JVM DNS cache policy defined?
[ ] Are private DNS zones linked to service networks?
[ ] Are internal CA certs managed centrally?
[ ] Are certificate expiry alerts configured?
[ ] Is mTLS used anywhere?

Application availability is an emergent property of code + runtime + network + identity + DNS + TLS + registry + telemetry + operations.

A Java/JAX-RS service can be perfectly implemented and still fail because the platform graph is broken.

Does the code compile?
Does the endpoint work locally?
Does the unit test pass?

Can production pull it?
Can production resolve it?
Can production reach it?
Can production trust it?
Can production authenticate it?
Can production observe it?
Can production roll it back?
Can production explain customer impact when it fails?