This part does not assume CSG uses Azure AKS, Microsoft Entra Workload ID,
managed identity, Azure Key Vault CSI driver, External Secrets Operator,
Azure Disk CSI, Azure Files CSI, Azure Blob CSI, Azure Load Balancer,
Application Gateway Ingress Controller, NGINX Ingress, Azure Monitor,
Managed Prometheus, Managed Grafana, Azure CNI, kubenet, private clusters,
or any specific Azure subscription/network topology.

Treat all AKS/Azure details as internal verification items.

AKS is Kubernetes plus Azure identity, Azure networking, Azure storage,
Azure load balancing, Azure Monitor, Azure RBAC, Azure subscription/resource group
boundaries, and Azure failure modes.

A workload can be a valid Kubernetes Deployment and still fail in AKS because
managed identity, federated credentials, Key Vault permissions, subnet routes,
private DNS, NSG rules, ingress controller, or node pool configuration are wrong.

Which identity does this Pod use?
Which Azure principal is allowed to read Key Vault, Blob, App Configuration, or ACR?
Is the ServiceAccount federated to a managed identity?
Can the Pod resolve and reach private endpoints?
Which load balancer or ingress controller receives traffic?
Where does TLS terminate?
Which storage class backs the PVC?
Which node pool runs the Pod?
Can we debug this from Kubernetes events, Azure activity logs, Entra sign-in/audit logs,
Key Vault diagnostics, load balancer health, and Azure Monitor?

Pod
Deployment
Service
Ingress
ConfigMap
Secret
ServiceAccount
RBAC
StorageClass
PVC
Node scheduling

Microsoft Entra ID
managed identity
Workload Identity / OIDC federation
Azure RBAC
VNet
subnet
NSG
route table
private endpoint
private DNS zone
Azure Load Balancer
Application Gateway
Azure Disk
Azure Files
Azure Blob
Azure Key Vault
Azure App Configuration
Azure Container Registry
Azure Monitor
VM Scale Sets
node pools
resource groups
subscriptions

Pod -> Kubernetes ServiceAccount -> federated identity / managed identity -> Azure token -> Azure API

kubectl get sa -n <ns> <service-account> -o yaml
kubectl describe pod -n <ns> <pod>
az aks show --resource-group <rg> --name <cluster> --query "oidcIssuerProfile"
az identity show --resource-group <rg> --name <identity>
az ad app federated-credential list --id <app-or-client-id>

apiVersion: v1
kind: ServiceAccount
metadata:
  name: quote-order-api
  namespace: quote-order
  labels:
    azure.workload.identity/use: "true"
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"

import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.security.keyvault.secrets.SecretClientBuilder;

var credential = new DefaultAzureCredentialBuilder().build();

var client = new SecretClientBuilder()
    .vaultUrl("https://<vault-name>.vault.azure.net/")
    .credential(credential)
    .buildClient();

Which credential does DefaultAzureCredential resolve inside AKS?
Is Workload Identity enabled on the cluster?
Is the ServiceAccount annotated correctly?
Is the federated credential subject correct?
Does the identity have Azure RBAC role assignment or Key Vault access policy?
Are local dev credentials clearly separated from cluster credentials?
Does the app fail fast if it resolves the wrong identity?

printenv | sort | grep -E 'AZURE|IDENTITY|TENANT|CLIENT'

az account show

Can the Pod read exactly the intended Key Vault secret?
Can it not read another service's secret?
Do Azure activity/diagnostic logs show the intended identity?

management group
subscription
resource group
resource
Key Vault data plane
storage account/container
ACR registry

Contributor at subscription scope for application Pod identity.

Key Vault Secrets User on one vault
Storage Blob Data Contributor on one container/account if required
AcrPull on one ACR for node/kubelet identity if required
Monitoring Metrics Publisher only if explicitly needed

Is role assignment scoped narrowly?
Is data-plane permission handled separately from management-plane permission?
Can this workload list all secrets or only read named secrets?
Does the identity have write/delete where read is enough?
Is Key Vault firewall/private endpoint compatible with AKS subnet?
Is secret access logged?

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: quote-order-keyvault
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "<managed-identity-client-id>"
    keyvaultName: "<keyvault-name>"
    tenantId: "<tenant-id>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret

Does mounted file update automatically?
Does the app reload it?
Is restart required?
Can old and new secret work during rotation?
Are secrets redacted from logs and config dumps?
Is Key Vault access private or public?

transactional state -> PostgreSQL / database
large files -> Blob Storage / object storage
temporary files -> emptyDir with size limits
shared files -> Azure Files only when required
block volume -> avoid for stateless APIs unless justified

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: import-export-share
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile-csi
  resources:
    requests:
      storage: 100Gi

Disk is attached to node/zone.
RWO semantics constrain scheduling and failover.
StorageClass defines SKU, replication, expansion, reclaim behavior.
Backups/snapshots do not replace application-level consistency.

kubectl get storageclass
kubectl describe pvc -n <ns> <pvc>
kubectl describe pod -n <ns> <pod>
kubectl get volumeattachment
kubectl logs -n kube-system -l app=csi-azuredisk-controller

multi-pod shared import/export directory
reports or generated files that need shared access
legacy file-oriented integration

high-QPS request path
database-like storage
distributed lock substitute
large numbers of tiny synchronous file operations

kubectl describe pvc -n <ns> <pvc>
kubectl describe pod -n <ns> <pod>
kubectl logs -n kube-system -l app=csi-azurefile-controller

kubectl get ingressclass
kubectl get ingress -A
kubectl get svc -A --field-selector spec.type=LoadBalancer
kubectl get pods -A | grep -Ei 'ingress|gateway|nginx|agic'

public vs internal frontend
TLS termination point
WAF policy
host/path routing
path rewrite
health probe path
backend protocol
private DNS
NSG rules
idle timeout
client IP preservation

kubectl describe ingress -n <ns> <ingress>
kubectl get svc -n <ns> <service> -o yaml
kubectl get endpointslice -n <ns>
kubectl describe pod -n <ns> <pod>
kubectl logs -n <ingress-namespace> <ingress-controller-pod>
az network application-gateway show --resource-group <rg> --name <gateway>
az network lb show --resource-group <rg> --name <lb>

Compare Kubernetes readiness, ingress/controller health, Azure backend health,
and application business readiness.

VNet
subnets
route tables
NSG
Azure CNI / kubenet / overlay
private cluster endpoint
private DNS zones
private endpoints
NAT gateway / egress path
firewall/proxy
service endpoint/private endpoint policy

Do Pods receive VNet-routable IPs?
Which subnet hosts nodes?
Which subnet hosts private endpoints?
Can Pods reach Key Vault/Storage/App Configuration privately?
Does DNS resolve public or private endpoint IP?
Does NSG allow node/pod -> dependency?
Is outbound forced through firewall/proxy?
Is the cluster private?

The app times out calling Key Vault or Storage.
IAM is correct.
Root cause is private DNS or NSG/routing, not the Java SDK.

nslookup <vault-name>.vault.azure.net
nslookup <storage-account>.blob.core.windows.net
curl -vk https://<vault-name>.vault.azure.net/

DNS response must match expected private/public path.
HTTP 401/403 may prove network path works but identity/authorization fails.
Timeout usually points to network/routing/firewall.

kubelet/node identity -> Azure Container Registry -> image layers -> container runtime

kubectl describe pod -n <ns> <pod>
az acr repository show-tags --name <acr-name> --repository <repo>
az role assignment list --assignee <identity-client-or-principal-id> --scope <acr-resource-id>

Is image tag immutable?
Is rollback image retained?
Is ACR access private if required?
Is image scanned and signed if platform requires it?
Is node/kubelet identity granted only AcrPull?

system vs user node pool
VM size
CPU/memory ratio
ephemeral OS disk
availability zones
node image version
upgrade surge
cluster autoscaler
labels/taints
spot node pool
GPU/specialized node pool
max pods per node
subnet IP capacity

JVM memory sizing vs container limit
CPU throttling
JIT warmup
startup probe delay
node drain graceful termination
OOMKilled diagnosis
thread count vs memory stack
observability agent overhead

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-order-api
spec:
  template:
    spec:
      serviceAccountName: quote-order-api
      nodeSelector:
        workload: api
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: quote-order-api
      containers:
        - name: app
          image: <acr-name>.azurecr.io/quote-order-api:<tag>
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              memory: "2Gi"

Kubernetes events
Pod logs
container restart/OOM metrics
node metrics
application metrics
application traces
ingress/load balancer metrics
Application Gateway backend health if used
Key Vault diagnostic logs
Azure activity logs
Azure Monitor / Managed Prometheus / Grafana if enabled

DNS
Application Gateway / Azure LB
ingress controller
backend health
NSG
service/endpoints
pod readiness
TLS certificate
path rewrite

resolved identity
federated credential
role assignment
Key Vault access model
private endpoint/DNS
firewall/NSG
Azure diagnostic logs

Pod can authenticate, but as the wrong Azure principal.

ServiceAccount annotation wrong
federated credential subject mismatch
local credential fallback in non-cluster environment
old aad-pod-identity pattern mixed with Workload Identity

Prove identity from Azure logs and resource access.
Do not assume annotation equals effective principal.

403 from Key Vault -> identity/authorization/access policy problem
Timeout -> DNS/private endpoint/NSG/firewall/routing problem
SecretProviderClass mount failure -> CSI/identity/Key Vault path problem

readiness path mismatch
wrong service port
path rewrite issue
backend protocol mismatch
NSG blocks gateway subnet
application returns 401 on health probe

Azure Disk zone mismatch
Azure Files private endpoint/DNS failure
CSI driver issue
quota/SKU unavailable
identity permission issue

Pods Pending or nodes cannot allocate Pod IPs.
Autoscaler adds nodes but workloads still fail scheduling.

Azure CNI IP allocation consumes subnet capacity
subnet too small
max pods per node too high
node pool expansion not planned

AcrPull missing
private endpoint DNS wrong
registry firewall
image tag does not exist
cross-subscription role assignment missing

kubectl get pod -n <ns> <pod> -o wide
kubectl describe pod -n <ns> <pod>
kubectl logs -n <ns> <pod> --previous
kubectl get events -n <ns> --sort-by=.lastTimestamp

image pull
config/secret mount
ServiceAccount
SecretProviderClass
volume mount
node scheduling
OOMKilled
startup probe

kubectl get sa -n <ns> <sa> -o yaml
kubectl exec -n <ns> <pod> -- printenv | sort | grep -E 'AZURE|CLIENT|TENANT|IDENTITY'
az role assignment list --assignee <principal-id>
az keyvault show --name <vault>
az keyvault secret show --vault-name <vault> --name <secret>

identity resolved
federated credential
role assignment scope
Key Vault access model
private endpoint path
DNS resolution
firewall/NSG

kubectl get ingressclass
kubectl describe ingress -n <ns> <ingress>
kubectl get svc -n <ns> <svc> -o yaml
kubectl get endpointslice -n <ns>
kubectl logs -n <ingress-ns> <controller-pod>
az network application-gateway show-backend-health --resource-group <rg> --name <gateway>

host/path rule
ingress class
service selector
target/backend health
NSG
TLS certificate
health probe path

kubectl get storageclass
kubectl describe pvc -n <ns> <pvc>
kubectl describe pod -n <ns> <pod>
kubectl get volumeattachment
kubectl logs -n kube-system -l app=csi-azuredisk-controller
kubectl logs -n kube-system -l app=csi-azurefile-controller

CSI driver
StorageClass
quota
zone
private endpoint
DNS
identity

ServiceAccount is explicit.
Workload identity/managed identity mapping is documented.
No static Azure secrets are embedded.
Role assignments are least-privilege and scoped narrowly.
Key Vault access model is clear.
Secret rotation behavior is defined.
Config/secret source of truth is clear.
Ingress controller is known and intentional.
Public/internal exposure is intentional.
TLS termination point is explicit.
Health probe path matches readiness semantics.
Application Gateway/Azure LB/Ingress annotations match platform standard.
NSG/routing/private endpoint path is understood.
StorageClass/PVC semantics match workload.
No durable state hides in container filesystem.
Node pool selection is intentional.
Resource requests/limits align with JVM sizing.
Topology spread / zone resilience is considered.
Image tag/digest rollback path exists.
ACR access uses least privilege.
Observability covers app, Pod, ingress, Azure identity, Key Vault, storage, and network.
Runbook includes Azure-side debugging, not only kubectl.

Is the workload actually deployed on AKS?
Which Azure subscriptions/resource groups/environments are used?
Which regions are used?
Which cluster names and namespaces map to quote/order services?
Is Microsoft Entra Workload ID enabled?
Is managed identity used directly or through Workload Identity federation?
Which ServiceAccounts map to which managed identities?
Are Azure SDKs using DefaultAzureCredential or custom credential wiring?
Are static service principal secrets used anywhere?
Which Key Vaults store application secrets?
Is Key Vault access through RBAC or access policies?
Is Key Vault accessed through private endpoint?
How are secrets mounted/synced/read?
Is Azure App Configuration used?
Which StorageClasses exist?
Is Azure Disk used by application Pods?
Is Azure Files used for shared storage?
Is Blob Storage accessed directly by app?
Which ingress controller is used: AGIC, NGINX, App Routing, or other?
Is Azure API Management in front of AKS?
Where does TLS terminate?
Which DNS zones are public/private?
Which VNets/subnets host nodes and private endpoints?
Which NSG rules control frontend -> node/pod and pod -> dependency?
Is outbound forced through firewall/proxy?
Is the cluster private?
Which node pools run Java/JAX-RS APIs?
How are node image upgrades/drains handled?
Is cluster autoscaler enabled?
How is ACR access granted?
Where are Azure Monitor metrics/logs/traces reviewed?
Are Key Vault diagnostic logs enabled?
What is the incident runbook for identity failure, Key Vault 403, private endpoint timeout, backend unhealthy, image pull, and PVC mount failure?

Kubernetes plane:
  Deployment, Pod, Service, Ingress, PVC, ServiceAccount

Azure identity plane:
  Microsoft Entra ID, managed identity, federated credential, Azure RBAC,
  Key Vault access, diagnostic logs

Azure network plane:
  VNet, subnet, NSG, route table, private endpoint, private DNS, firewall

Azure operations plane:
  node pools, CSI drivers, ingress controllers, Azure Monitor, ACR, storage,
  Key Vault, Application Gateway, Load Balancer

kubectl get pods

client -> DNS -> gateway/LB -> ingress -> service -> pod -> JVM -> JAX-RS -> managed identity -> Azure API / DB / Kafka