This part does not assume CSG uses Amazon EKS, IRSA, EKS Pod Identity,
AWS Load Balancer Controller, EBS CSI, EFS CSI, External Secrets Operator,
Secrets Store CSI, AWS Secrets Manager, Systems Manager Parameter Store,
Karpenter, Managed Node Groups, Fargate, CloudWatch, Prometheus, Grafana,
or any specific AWS account/network topology.

Treat all AWS/EKS details as internal verification items.

EKS is Kubernetes plus AWS identity, AWS networking, AWS storage, AWS load
balancing, AWS observability, AWS IAM, AWS account boundaries, and AWS failure modes.

A service can be correct as a Kubernetes workload and still fail in EKS because
IAM, VPC, DNS, node capacity, security groups, target groups, or cloud add-ons
are wrong.

Which identity does this Pod use?
Which IAM permissions does that identity have?
Can the Pod reach STS, Secrets Manager, SSM, S3, RDS, Kafka, or internal endpoints?
Which load balancer terminates TLS?
Which target group health check decides readiness?
Which storage system backs the PVC?
Which node type runs the Pod?
Which AWS account, VPC, subnet, route table, and security group are in path?
Can we debug this from Kubernetes events, controller logs, AWS APIs, and CloudTrail?

Pod
Deployment
Service
Ingress
ConfigMap
Secret
ServiceAccount
RBAC
StorageClass
PVC
Node scheduling

IAM
STS
EKS Pod Identity or IRSA
VPC CNI
subnets
security groups
route tables
NAT gateways
VPC endpoints
ALB/NLB
target groups
EBS/EFS CSI drivers
ECR
CloudWatch
CloudTrail
managed node groups
Fargate profiles
AWS account boundaries

Pod -> Kubernetes ServiceAccount -> AWS IAM role -> temporary credentials -> AWS API

kubectl get sa -n <ns> <service-account> -o yaml
kubectl describe pod -n <ns> <pod>
aws eks list-pod-identity-associations --cluster-name <cluster>
aws eks describe-addon --cluster-name <cluster> --addon-name eks-pod-identity-agent

apiVersion: v1
kind: ServiceAccount
metadata:
  name: quote-order-api
  namespace: quote-order
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/quote-order-api-role

aws eks list-pod-identity-associations \
  --cluster-name <cluster> \
  --namespace quote-order \
  --service-account quote-order-api

aws sts get-caller-identity

Success proves the Pod has some AWS identity.
It does not prove the identity is the intended one.
It does not prove the permissions are least-privilege.
It does not prove all required service calls will work.

read config from SSM Parameter Store
read secret from Secrets Manager
read/write object to S3
emit metrics/logs
publish events
call KMS decrypt
access ECR indirectly through node/pull identity
read from SQS/SNS/MSK-related APIs

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue",
    "ssm:GetParameter",
    "kms:Decrypt"
  ],
  "Resource": [
    "arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:quote-order/*",
    "arn:aws:ssm:ap-southeast-1:123456789012:parameter/quote-order/*",
    "arn:aws:kms:ap-southeast-1:123456789012:key/<key-id>"
  ]
}

Is permission scoped to namespace/service/environment?
Is it tied to a ServiceAccount, not node role?
Can one compromised Pod access another service's secrets?
Does policy allow wildcard read across all secrets?
Does KMS policy also allow the role?
Does CloudTrail show the correct role as caller?
Are cross-account calls intentional and documented?

kubectl get pod -n <ns> <pod> -o jsonpath='{.spec.serviceAccountName}'
kubectl get sa -n <ns> <sa> -o yaml
kubectl logs -n kube-system -l app.kubernetes.io/name=eks-pod-identity-agent
aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=<role-name>

// Prefer provider chain from SDK runtime environment.
// Do not commit access keys or inject static long-lived credentials.
var s3 = S3Client.builder()
    .region(Region.AP_SOUTHEAST_1)
    .build();

transactional state -> PostgreSQL / database
large binary object -> S3 / object storage
temporary scratch -> emptyDir with size discipline
shared file system -> EFS only if semantics require it
block volume -> usually not for stateless API Pods

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: report-export-workspace
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 20Gi

provisioning dependency
attach/mount lifecycle
backup/retention questions
performance characteristics
security permissions
cleanup responsibility

EBS volume is attached to a node/AZ.
A Pod scheduled to the wrong zone may not mount it.
RWO workloads need careful disruption planning.
StorageClass parameters affect type, encryption, IOPS, throughput, expansion.
Snapshots/backups are not automatic application correctness.

kubectl get storageclass
kubectl describe pvc -n <ns> <pvc>
kubectl describe pv <pv>
kubectl describe pod -n <ns> <pod>
kubectl logs -n kube-system -l app=ebs-csi-controller

shared report/export/import directory
multi-pod read/write shared files
legacy component that expects filesystem semantics

high-throughput low-latency database storage
coordination/locking substitute without careful semantics
hot path per-request small file operations

kubectl describe pvc -n <ns> <pvc>
kubectl describe pod -n <ns> <pod>
kubectl logs -n kube-system -l app=efs-csi-controller
nslookup <efs-dns-name>

DNS -> ALB/NLB -> Kubernetes Service/Ingress -> Pod

internet-facing vs internal
TLS termination point
target type: instance vs ip
health check path
listener rules
path rewrite behavior
security groups
WAF integration
idle timeout
HTTP/2 behavior
access logs

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: quote-order-api
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health/ready
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
spec:
  ingressClassName: alb
  rules:
    - host: quote-order.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: quote-order-api
                port:
                  number: 8080

kubectl get ingress -n <ns>
kubectl describe ingress -n <ns> <ingress>
kubectl get svc -n <ns>
kubectl get endpointslice -n <ns>
kubectl get pods -n <ns> -o wide
kubectl logs -n kube-system deployment/aws-load-balancer-controller
aws elbv2 describe-load-balancers
aws elbv2 describe-target-health --target-group-arn <arn>

Always compare three health concepts:
1. Kubernetes readiness
2. AWS target group health
3. Application business readiness

Where is the source of truth?
How is it rotated?
How does the app detect rotation?
Is secret value logged during config dump?
Is the secret injected as env var or mounted file?
Does restart pick up a new secret?
Does runtime reload pick up a new secret?
Can old and new credential work during rotation window?

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: quote-order-db-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: quote-order-db-secret
  data:
    - secretKey: password
      remoteRef:
        key: /quote-order/prod/db
        property: password

managed node group
self-managed node group
Fargate profile
Karpenter-provisioned node
specialized node pool

CPU throttling
memory pressure
OOMKilled
ephemeral storage pressure
image pull latency
JIT warmup
node drain behavior
spot interruption
AZ spread
ENI/IP exhaustion
max Pods per node
node security group
kernel/network behavior

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-order-api
spec:
  template:
    spec:
      serviceAccountName: quote-order-api
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: quote-order-api
      containers:
        - name: app
          image: <account>.dkr.ecr.<region>.amazonaws.com/quote-order-api:<tag>
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              memory: "2Gi"

heap
metaspace
thread stacks
direct buffers
JIT/code cache
native libraries
TLS/compression buffers
observability agent overhead

VPC
subnets
route tables
NAT gateway
VPC endpoints
security groups
NACLs
VPC CNI behavior
pod IP allocation
DNS resolver path
private hosted zones
cross-account/cross-VPC connectivity

Do Pods get VPC IPs?
Which subnet do Pods run in?
Can Pods reach private RDS/MSK/internal APIs?
Can Pods reach AWS APIs without public internet?
Are VPC endpoints used for STS, S3, Secrets Manager, SSM, ECR, CloudWatch?
Is DNS split-horizon involved?
Do security groups allow pod/node/LB paths?
Is network policy enforced by the CNI or another plugin?

ECR API
ECR Docker registry
S3
STS
Secrets Manager
SSM
CloudWatch Logs
KMS

Application startup fails while reading secret.
The exception says timeout.
Root cause is not the Java SDK.
Root cause is no route to Secrets Manager endpoint or wrong proxy/no_proxy.

kubelet on node -> ECR auth/token -> ECR registry -> image layers -> container runtime

kubectl describe pod -n <ns> <pod>
aws ecr describe-images --repository-name <repo> --image-ids imageTag=<tag>

Is the tag immutable?
Is digest pinning used where required?
Is image signed/scanned?
Is rollback image still available?
Is image pull path private and reliable?

Kubernetes events
Pod logs
application metrics
application traces
node metrics
container restart/OOM metrics
ALB/NLB metrics
target group health
CloudTrail IAM events
controller logs
EBS/EFS metrics if used

Route 53
ALB/NLB listener
target group
security group
ingress controller
service/endpoints
node/pod network
DNS
TLS

IAM policy
role association
STS
credential provider chain
endpoint/network
KMS policy
resource policy
CloudTrail

AccessDeniedException in one endpoint only.

Do not broaden to wildcard.
Map exact call -> action -> resource -> KMS/resource policy -> role.
Add least-privilege permission with test evidence.

wrong health path
wrong port
security group blocks LB
app requires Host header
health endpoint returns 401/403
path rewrite mismatch
startup too slow

no NAT gateway
missing VPC endpoint
endpoint security group blocks traffic
DNS private hosted zone issue
proxy/no_proxy incorrect

PV bound to zone A
Pod scheduled to zone B
storage class WaitForFirstConsumer not used
node affinity not respected

VPC CNI IP exhaustion
subnet too small
ENI limit reached
max pods per node reached

shared ingress group
host/path overlap
annotation conflict
team boundary not enforced

kubectl get pod -n <ns> <pod> -o wide
kubectl describe pod -n <ns> <pod>
kubectl logs -n <ns> <pod> --previous
kubectl get events -n <ns> --sort-by=.lastTimestamp

image pull
config/secret mount
service account
volume mount
node scheduling
OOMKilled
startup probe

kubectl exec -n <ns> <pod> -- env | sort
kubectl exec -n <ns> <pod> -- aws sts get-caller-identity
kubectl get sa -n <ns> <sa> -o yaml
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole

role identity
policy action/resource
KMS/resource policy
SDK credential chain
STS endpoint
proxy/no_proxy
CloudTrail caller

kubectl describe ingress -n <ns> <ingress>
kubectl get svc -n <ns> <svc> -o yaml
kubectl get endpointslice -n <ns>
kubectl logs -n kube-system deployment/aws-load-balancer-controller
aws elbv2 describe-target-health --target-group-arn <arn>

target group health
listener rule
security groups
readiness endpoint
service selector
pod labels
container port

kubectl describe pvc -n <ns> <pvc>
kubectl describe pod -n <ns> <pod>
kubectl get storageclass
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-efs-csi-driver

CSI driver installed
IAM permission
storage class
AZ/node affinity
security groups for EFS
volume attachment

ServiceAccount is explicit.
IAM role is least-privilege.
No static AWS credentials are embedded.
IRSA/Pod Identity association is documented.
KMS/resource policies are compatible with IAM role.
Config/secret source of truth is clear.
Secret rotation behavior is defined.
Ingress/Service annotations match platform standard.
ALB/NLB is internal/external intentionally.
TLS termination is explicit.
Health check path matches readiness semantics.
Security groups allow exactly required paths.
StorageClass/PVC semantics match workload.
No durable state is hidden in container filesystem.
Node scheduling constraints are intentional.
Resource requests/limits align with JVM sizing.
Topology spread / AZ resilience is considered.
Image tag/digest rollback path exists.
Observability covers app, Pod, LB, IAM, and AWS API calls.
Runbook includes AWS-side debugging, not only kubectl.

Is the workload actually deployed on EKS?
Which AWS accounts/environments are used?
Which regions are used?
Which cluster names and namespaces map to quote/order services?
Is IRSA or EKS Pod Identity used?
Are AWS SDKs using default credential chain?
Which IAM roles map to which Kubernetes ServiceAccounts?
Are node roles still over-privileged?
Is IMDS restricted from Pods?
Are any Pods using hostNetwork?
Which AWS Load Balancer Controller version is installed?
Are ALB/NLB annotations standardized by platform team?
Is ingress shared across teams?
Which TLS termination model is used?
Are WAF/API Gateway/CloudFront in front of ALB?
Which StorageClasses exist?
Is EBS used by application Pods?
Is EFS used for shared storage?
Are S3 buckets accessed directly by app?
Are VPC endpoints used for AWS APIs?
Is traffic private-only or internet-facing?
Which subnets host nodes and load balancers?
Which security groups control LB -> node/pod and pod -> dependency?
Is VPC CNI using security groups for pods?
Is Karpenter, Managed Node Groups, Fargate, or self-managed nodes used?
How are node upgrades/drains handled?
How are external secrets synced or mounted?
Which secrets/config services are source of truth?
How is secret rotation tested?
Where are ALB/NLB metrics and access logs stored?
Where are CloudTrail events reviewed?
What is the incident runbook for AccessDenied, target unhealthy, image pull, PVC mount, and AWS API timeout?

Kubernetes plane:
  Deployment, Pod, Service, Ingress, PVC, ServiceAccount

AWS identity plane:
  IAM role, policy, STS, CloudTrail, KMS, resource policy

AWS network plane:
  VPC, subnet, route table, NAT, endpoint, SG, NACL, DNS

AWS operations plane:
  add-ons, nodes, controllers, CloudWatch, target groups, storage, ECR

kubectl get pods

client -> DNS -> LB -> target group -> service -> pod -> JVM -> JAX-RS -> IAM role -> AWS API / DB / Kafka