This part does not assume CSG uses Kubernetes, EKS, AKS, AWS Pod Identity, IRSA,
Azure Workload Identity, private endpoints, PrivateLink, or any specific cloud pattern.
Treat all runtime, identity, and networking details as internal verification items.

JAX-RS resource/service code
-> SDK client
-> credential provider chain
-> pod service account / workload identity
-> cloud IAM / RBAC / trust policy
-> DNS resolution
-> cluster egress path
-> private endpoint / public endpoint
-> TLS validation
-> cloud service authorization
-> cloud service quota/throttling
-> audit log / telemetry

Service identity answers:
"Is this workload allowed to call this cloud API?"

User/tenant authorization answers:
"Is this request allowed to cause this workload to call that cloud API for this tenant/resource?"

export quote PDF
upload attachment
read tenant-specific catalog config
write audit evidence
load pricing/rules configuration
fetch secret for downstream system
publish document to object storage

Which service account is attached to the pod?
Which cloud identity is bound to that service account?
Which permissions does that cloud identity have?
Which endpoint does the SDK call?
Does DNS resolve to public or private address?
Does the cluster allow egress to that address?
Does the remote cloud service allow this identity?
How are errors classified and logged?

Kubernetes ServiceAccount
-> workload identity binding / annotation / association
-> cloud IAM role / managed identity / federated identity credential
-> cloud service authorization

Do not run business services with the namespace default ServiceAccount unless explicitly justified.
Do not mount Kubernetes API tokens if the app does not need them.
Do not grant broad Kubernetes RBAC just because the service needs cloud access.
Do not use long-lived static cloud keys inside Kubernetes secrets unless there is a strong exception.
Use one workload identity per meaningful permission boundary.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: quote-order-api
  namespace: quote-order
  annotations:
    # Cloud-specific identity binding may appear here.
    # Exact annotation depends on platform and must be verified internally.
    example.cloud/role: "quote-order-api-role"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-order-api
spec:
  template:
    spec:
      serviceAccountName: quote-order-api
      automountServiceAccountToken: true
      containers:
        - name: app
          image: example/quote-order-api:1.0.0

Allow a Kubernetes workload to obtain AWS credentials without embedding static AWS keys.

Pod
-> Kubernetes ServiceAccount
-> identity association / trust relationship
-> IAM role
-> temporary AWS credentials
-> AWS SDK call

The AWS SDK credential provider chain resolves credentials.
The application code should not hardcode access keys.
The application code should not know whether the credentials came from IRSA,
EKS Pod Identity, node role, environment variables, or local developer config.

Is the pod using a workload-specific ServiceAccount?
Is the IAM role scoped to only the required AWS APIs and resources?
Is the trust relationship limited to the correct cluster/namespace/service account?
Is the SDK using temporary credentials rather than static keys?
Are region/account assumptions explicit?
Is CloudTrail useful enough to attribute calls to the workload?

Check whether the cluster uses IRSA, EKS Pod Identity, static keys, node IAM role, or another platform abstraction.
Check whether SDK clients use the default credential provider chain or custom credential provider.
Check IAM role names, trust policy, and resource policy.
Check whether calls go through VPC endpoints or public AWS endpoints.
Check CloudTrail attribution for the workload.

Cluster identity:
How AKS itself manages Azure resources.

Workload identity:
How an application running in a pod authenticates to Azure services.

Pod
-> Kubernetes ServiceAccount
-> federated identity / workload identity binding
-> Microsoft Entra identity
-> Azure RBAC / service permission
-> Azure SDK call

Azure Identity library resolves a credential.
Application code calls Azure SDK clients.
The app should not store client secrets or certificates unless explicitly required.

Check whether AKS Workload Identity is enabled.
Check service account annotations and federated identity credential.
Check Azure RBAC role assignment scope.
Check whether the service uses DefaultAzureCredential or custom credential wiring.
Check whether private DNS zones are linked to the cluster VNet.

The cloud service is reachable through a private IP inside a VPC/VNet,
not through the public internet path.

Developer laptop DNS -> public endpoint
Pod in cluster DNS -> private endpoint IP
CI runner DNS -> maybe public, maybe blocked
On-prem resolver -> maybe split-horizon route

Cloud service configured for private access
Private endpoint created and approved
Private DNS record exists
VPC/VNet DNS resolver can answer the record
Cluster/pod DNS forwards correctly
Route table sends traffic to the endpoint
Firewall/SG/NSG allows traffic
Kubernetes NetworkPolicy allows egress
TLS certificate still matches hostname
Cloud IAM/RBAC authorizes the identity

Private endpoint says: "network path is private."
IAM/RBAC says: "this identity may call the service."
Application authorization says: "this user/tenant may trigger this operation."

What hostname is the SDK calling?
What IP does it resolve to inside the pod?
Does it resolve differently from node, pod, CI, and laptop?
Is the result public or private?
Is there a private DNS zone/record?
Is DNS cached by JVM, OS, sidecar, or node resolver?

nslookup <service-hostname>
dig <service-hostname>
curl -v https://<service-hostname>/
openssl s_client -connect <service-hostname>:443 -servername <service-hostname>

Check CoreDNS configuration.
Check private DNS zones and links.
Check split-horizon DNS behavior.
Check JVM DNS TTL settings if incidents show stale endpoint resolution.
Check whether sidecars/proxies override DNS or outbound routing.

Application startup
-> load config
-> build cloud SDK clients once
-> inject clients into services
-> use per-request context only for authorization/correlation
-> call cloud SDK with bounded timeout/retry
-> map errors into domain/technical API errors

public final class ObjectStorageService {
    private final CloudStorageClient client;
    private final StorageConfig config;

    public ObjectStorageService(CloudStorageClient client, StorageConfig config) {
        this.client = client;
        this.config = config;
    }

    public StoredObjectRef uploadQuoteDocument(
            TenantId tenantId,
            QuoteId quoteId,
            InputStream content,
            long contentLength,
            CorrelationContext correlation
    ) {
        // 1. Validate tenant authorization before cloud call.
        // 2. Build tenant-safe object key.
        // 3. Attach correlation metadata where allowed.
        // 4. Apply bounded timeout/retry in the client layer.
        // 5. Map cloud errors into stable service errors.
        return client.putObject(
                config.bucketOrContainer(),
                buildObjectKey(tenantId, quoteId),
                content,
                contentLength,
                correlation
        );
    }
}

// Anti-pattern: client created per request, credential behavior hidden, no timeout ownership.
public Response upload(InputStream input) {
    var client = CloudClientBuilder.defaultClient();
    client.putObject("prod-bucket", "some/key", input);
    return Response.ok().build();
}

client lifecycle unclear
connection reuse lost
credential source not reviewable
hardcoded environment/resource
no tenant boundary
no timeout/retry decision
hard to test
hard to observe

Pod starts
-> app calls Key Vault / Secrets Manager / object storage immediately
-> transient cloud/DNS issue happens
-> pod fails startup
-> rollout stalls
-> old replicas are terminated
-> outage expands

Startup should validate local configuration and fail on impossible local state.
Readiness should represent whether the service can safely receive traffic.
External dependency checks should be bounded, cached, and classified.
Critical vs optional dependencies must be explicit.

Check startup dependency calls.
Check readiness/liveness/startup probe semantics.
Check whether cloud dependency failure removes pods from service.
Check whether rollout strategy protects against dependency incidents.

operation name
cloud provider
cloud service
region
account/subscription/environment
endpoint type: public/private/unknown
result: success/error/throttled/timeout/auth_denied
error class
duration histogram
retry count
correlation ID
trace ID/span ID

object key
full URL with query string
customer name
raw tenant ID if high-cardinality policy forbids it
secret name if sensitive
full exception message containing request IDs or identifiers

{
  "event": "cloud.storage.put.completed",
  "provider": "aws-or-azure",
  "service": "object-storage",
  "endpointType": "private",
  "operation": "putObject",
  "durationMs": 187,
  "result": "success",
  "retryCount": 1,
  "correlationId": "...",
  "traceId": "..."
}

Uploaded file to https://bucket.example.com/tenant-123/customer-secret-contract.pdf?sig=...

Which endpoint/action triggered the call?
Which tenant/user/resource was involved?
Which cloud service should be called?
Which bucket/container/secret/config/resource is expected?

kubectl get pod <pod> -n <ns> -o yaml | grep -A5 serviceAccountName
kubectl get serviceaccount <sa> -n <ns> -o yaml
kubectl auth can-i --as=system:serviceaccount:<ns>:<sa> get pods -n <ns>

AWS: check IAM role association/trust policy/pod identity association.
Azure: check workload identity annotation/federated credential/managed identity client ID.

Enable SDK credential provider debug logs in a safe non-sensitive way.
Check whether environment variables override workload identity.
Check whether local config accidentally leaked into image or pod.

kubectl exec -n <ns> <pod> -- nslookup <cloud-service-hostname>
kubectl exec -n <ns> <pod> -- sh -c 'getent hosts <cloud-service-hostname>'

kubectl exec -n <ns> <pod> -- curl -v https://<cloud-service-hostname>/
kubectl exec -n <ns> <pod> -- openssl s_client -connect <host>:443 -servername <host>

Check cloud audit logs.
Check denied action/resource.
Check resource policy.
Check role assignment scope.
Check tenant/account/subscription mismatch.

Was cloud 403 mapped to API 500?
Was timeout retried too aggressively?
Was error logged with correlation ID?
Was customer-facing response safe and stable?

[ ] Does the app use a workload-specific Kubernetes ServiceAccount?
[ ] Is cloud identity obtained through platform-standard workload identity?
[ ] Are static credentials avoided?
[ ] Is cloud resource name configured, not hardcoded?
[ ] Is tenant/user authorization checked before the cloud operation?
[ ] Is object/secret/config key construction tenant-safe?
[ ] Are timeout, retry, and circuit breaker policies explicit?
[ ] Are SDK clients reused and lifecycle-managed?
[ ] Are cloud errors mapped into stable application errors?
[ ] Are logs redacted and correlated?
[ ] Are metrics/traces emitted for cloud calls?
[ ] Does the code work with private endpoints and restricted egress?
[ ] Is local development credential behavior documented?
[ ] Are tests covering auth denied, timeout, not found, throttling, and retry exhaustion?

Runtime and cluster
[ ] Are services deployed to Kubernetes? Which clusters/environments?
[ ] EKS, AKS, both, on-prem Kubernetes, or another platform?
[ ] Which namespace and ServiceAccount convention is used?
[ ] Is default ServiceAccount forbidden for production workloads?

Cloud identity
[ ] AWS IRSA, EKS Pod Identity, Azure Workload Identity, managed identity, static secrets, or platform abstraction?
[ ] Where is identity association declared: Terraform, Helm, Kustomize, GitOps repo, platform portal?
[ ] Who owns IAM/RBAC role assignment?
[ ] How are permission changes reviewed?

SDK behavior
[ ] Which SDKs are used by Java services?
[ ] Do SDK clients use default credential chain or custom providers?
[ ] Are clients singleton/lifecycle-managed?
[ ] Are timeouts/retries standardized?

Private networking
[ ] Are cloud services reached through public endpoints, private endpoints, or service endpoints?
[ ] Which DNS zones/resolvers are involved?
[ ] Are NetworkPolicies used for egress?
[ ] Are outbound proxies or service mesh sidecars involved?

Observability and operations
[ ] Are cloud calls traced?
[ ] Are auth/network/DNS errors distinguishable in logs?
[ ] Are cloud audit logs accessible during incidents?
[ ] Is there a runbook for pod-to-cloud access failures?

Do not ask only: "Why does the SDK fail?"
Ask:
- Which identity is being used?
- Which endpoint is being called?
- Which network path is taken?
- Which policy denies or allows it?
- Which tenant/user action caused it?
- Which timeout/retry behavior amplifies it?
- Which telemetry proves the answer?

Kubernetes ServiceAccounts:
https://kubernetes.io/docs/concepts/security/service-accounts/
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

AWS EKS Pod Identity:
https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

Azure AKS Workload Identity:
https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster