Azure AKS Platform Engineering
Azure AKS Architecture, Identity, Networking, and Operations
Mengoperasikan JAX-RS workloads pada Azure Kubernetes Service: AKS Standard and Automatic, managed control plane, system and user node pools, Microsoft Entra access, managed identities, Workload ID, Azure CNI Overlay/Cilium, private clusters, application routing, storage CSI, Azure Monitor, upgrades, security, reliability, backup, cost, and incident response.
Part 050 — Azure AKS Architecture, Identity, Networking, and Operations
Azure Kubernetes Service mengelola Kubernetes control plane, tetapi production reliability tetap bergantung pada node pools, managed identities, Microsoft Entra access, virtual-network topology, Pod IP strategy, egress, private DNS, Azure Load Balancer, ingress/application routing, CSI storage, observability, upgrade channels, OS lifecycle, quotas, and application behavior. AKS Automatic memperluas platform-managed defaults for compute, networking, ingress, and monitoring; AKS Standard provides wider control. “Managed” means ownership is shifted and constrained, not removed.
Daftar Isi
- Target kompetensi
- Scope dan baseline
- Boundary dengan Kubernetes generik
- Current AKS operating models
- Shared responsibility model
- Mental model AKS architecture
- Managed control plane
- Control-plane and node-resource ownership
- Node resource group
- Subscription and resource-group boundaries
- Kubernetes API access
- Public API server
- Authorized IP ranges
- Private cluster
- API Server VNet Integration boundary
- Private DNS
- Hub-spoke connectivity
- AKS Standard
- AKS Automatic
- Automatic managed defaults
- Automatic suitability boundary
- Migration and coexistence
- Node-pool mental model
- System node pools
- User node pools
- System-pool availability
- Virtual Machine Scale Sets
- Node-pool modes
- Multiple node pools
- Node-pool immutability and recreation
- Availability zones
- Zone-redundant node pools
- Regional node-pool boundary
- VM sizes
- CPU architecture
- Spot node pools
- GPU and specialized pools
- Windows node pools boundary
- Azure Linux and Ubuntu
- Azure Linux 2 retirement
- Node images
- Node image upgrades
- Virtual nodes boundary
- Node labels and taints
- System workload isolation
- Cluster Autoscaler
- Autoscaler profiles
- Scale-down and disruption
- Capacity and quota
- Authentication and authorization model
- Microsoft Entra authentication
- Kubernetes RBAC
- Azure RBAC for Kubernetes authorization
- Local accounts boundary
- Admin kubeconfig risk
- Human-access lifecycle
- CI/CD identities
- Cluster managed identity
- System-assigned managed identity
- User-assigned managed identity
- Kubelet identity
- Control-plane identity versus kubelet identity
- Managed-identity role assignments
- Workload identity mental model
- Microsoft Entra Workload ID
- OIDC issuer
- Federated identity credential
- Kubernetes ServiceAccount
- Azure Identity SDK
- DefaultAzureCredential
- Workload identity mutation and labels
- Workload identity versus legacy pod-managed identity
- Least-privilege Azure RBAC
- Cross-subscription access
- Credential caching and rotation
- Network architecture choices
- Azure CNI overview
- Azure CNI Overlay
- Azure CNI Powered by Cilium
- Azure CNI flat networking
- Kubenet boundary
- Automatic networking defaults
- Pod and node address spaces
- Service CIDR
- DNS service IP
- Subnet sizing
- Pod IP exhaustion
maxPods- Overlay routing and SNAT
- VNet-routable Pod IPs
- Dual-stack networking
- Cilium eBPF data plane
- NetworkPolicy
- Cilium Network Policy
- Azure Network Policy Manager retirement
- Security groups and NSGs
- Application Security Groups boundary
- Private Link and private endpoints
- User-defined routes
- Azure Firewall egress
- Outbound types
- Azure Load Balancer egress
- Managed NAT Gateway
- User-assigned NAT Gateway boundary
- Egress FQDN requirements
- Restricted-egress clusters
- SNAT port exhaustion
- LocalDNS boundary
- CoreDNS
- Custom DNS and hybrid resolution
- Load-balancing options
- Azure Load Balancer
- Public and internal Services
- Backend pools and health probes
- Source IP and external traffic policy
- Application routing add-on
- Managed NGINX ingress
- Application routing and community ingress-nginx retirement
- Azure DNS integration
- Gateway API boundary
- Application Gateway for Containers boundary
- Application Gateway Ingress Controller boundary
- NGINX and other ingress controllers
- Web Application Firewall boundary
- TLS and Key Vault certificate boundary
- Deregistration and graceful termination
- Storage mental model
- Azure Disk CSI
- Disk zone affinity
- Disk performance tiers
- Azure Files CSI
- NFS and SMB
- Azure Blob CSI boundary
- Ephemeral OS and temporary disks
- StorageClass governance
- Volume expansion
- Volume snapshots
- Stateful workload placement
- Backup and restore
- Azure Backup for AKS
- External database backups
- Key Vault and secrets
- Azure Key Vault provider for Secrets Store CSI
- Secret rotation
- Kubernetes Secret boundary
- Observability mental model
- Azure Monitor
- Container insights
- Managed service for Prometheus
- Azure Managed Grafana
- Control-plane resource logs
- Activity logs
- Diagnostic settings
- OpenTelemetry
- Network observability
- Application logs
- Cost and cardinality
- Security posture
- Private API and network segmentation
- Microsoft Entra and RBAC
- Pod Security Admission
- Azure Policy for Kubernetes boundary
- Defender for Containers boundary
- Azure Container Registry
- ACR pull identity
- Image scanning and provenance
- Encryption
- Disk encryption sets boundary
- Confidential containers boundary
- Multi-subscription and landing-zone architecture
- Multi-tenancy
- Reliability mental model
- Availability zones and zone resilience
- Control-plane versus data-plane availability
- System-pool resilience
- Application topology spread
- Node replacement and PDBs
- Regional outage
- Multi-region deployment models
- Failure-domain headroom
- Cluster recreation
- Upgrade lifecycle
- Supported Kubernetes versions
- Control-plane and node version skew
- Manual upgrades
- Automatic upgrade channels
- Node OS upgrade channels
- Planned maintenance
- Surge upgrades
- PDB and drain behavior
- Upgrade prechecks
- API deprecation
- Blue-green node-pool upgrade
- Blue-green cluster migration
- Rollback boundary
- Cost model
- Control-plane tier and Automatic charges
- VM and node-pool cost
- Spot and reservations
- Load balancer and public-IP cost
- NAT Gateway and Firewall cost
- Storage and backup cost
- Log Analytics and metrics cost
- Cross-zone and egress cost
- Rightsizing and autoscaling
- Cost allocation
- JAX-RS workload integration
- Java Azure SDK identity
- Azure SDK client lifecycle
- Key Vault, Blob, Service Bus, and Cosmos access
- Private endpoints and DNS
- Health and load balancers
- Graceful termination
- Connection pools and node churn
- Failure-model matrix
- Debugging playbook
- Testing strategy
- Architecture patterns
- Anti-patterns
- PR review checklist
- Trade-off yang harus dipahami senior engineer
- Internal verification checklist
- Latihan verifikasi
- Ringkasan
- Referensi resmi
Target kompetensi
Setelah menyelesaikan part ini, Anda harus mampu:
- membedakan AKS Standard and AKS Automatic operating models;
- menjelaskan managed control plane, customer subscription resources, and node resource group ownership;
- memilih public endpoint, authorized ranges, private cluster, or API Server VNet Integration;
- mendesain system and user node pools across zones, VM types, OS, architecture, Spot, and specialized workloads;
- membedakan Microsoft Entra authentication, Kubernetes RBAC, Azure RBAC for Kubernetes, local admin, cluster identity, kubelet identity, and workload identity;
- mengimplementasikan Microsoft Entra Workload ID for Java Azure SDK clients;
- memilih Azure CNI Overlay, Azure CNI Powered by Cilium, or VNet-routable networking;
- merencanakan node/Pod/Service CIDRs, subnet capacity,
maxPods, outbound type, NAT, Firewall, and private endpoints; - memahami Cilium network policy and Azure NPM retirement;
- memilih Azure Load Balancer, application routing add-on, managed NGINX, Application Gateway, or another controller;
- memilih Azure Disk, Azure Files, Blob, or external managed data services;
- mengintegrasikan Azure Monitor, Container insights, managed Prometheus, Grafana, diagnostic settings, and OpenTelemetry;
- menjalankan Kubernetes/node-image upgrades with channels, maintenance windows, surge, PDB, and API-deprecation gates;
- merencanakan zone resilience, multi-region, backup, and cluster recreation;
- menganalisis cost from nodes, NAT/Firewall, load balancers, storage, and observability;
- mendiagnosis identity, private DNS, IP exhaustion, image pull, LoadBalancer, disk, and upgrade incidents.
Scope dan baseline
Baseline:
- generic Kubernetes and networking from Parts 046–048;
- Java/JAX-RS workloads;
- Azure subscription, resource groups, VNet, Microsoft Entra, ACR, Azure Monitor, and managed identity concepts;
- IaC/GitOps as desired source of truth;
- Linux node pools as primary example;
- AKS Standard or Automatic.
Part ini tidak mengasumsikan:
- current CSG deployment is AKS;
- AKS Automatic;
- public/private cluster;
- Azure CNI mode;
- Cilium;
- application routing add-on;
- Azure Policy/Defender;
- Azure Disk/Files;
- exact region, subscription, VNet, VM size, Kubernetes version, or pricing tier;
- Helm, Terraform, Bicep, ARM, or Azure CLI as source of truth.
Boundary dengan Kubernetes generik
This part focuses on Azure implementation:
- AKS control plane and node resource ownership;
- Entra and managed identities;
- Azure CNI;
- Azure Load Balancer and routing add-ons;
- Azure storage and monitoring;
- AKS lifecycle and platform constraints.
Generic Kubernetes manifests and lifecycle remain Parts 046–047.
Current AKS operating models
AKS Standard
Platform team configures:
- node pools and autoscaling;
- CNI/network data plane;
- ingress;
- monitoring;
- upgrades;
- egress;
- policies;
- add-ons.
AKS Automatic
AKS applies production-oriented managed defaults and automatically manages more compute/platform behavior.
Current official documentation positions Automatic as the recommended production-ready default for many workloads, but workloads requiring unsupported customizations remain candidates for Standard.
Shared responsibility model
| Layer | Azure-managed baseline | Customer responsibility |
|---|---|---|
| Control plane | API server/etcd/control components | access model, logs, upgrades/channels |
| Cluster identity | managed identity capability | role assignments and least privilege |
| Nodes | managed lifecycle features vary by mode | pools, sizing, workloads, PDB, security |
| Network | Azure primitives/integrations | address plan, CNI, egress, policy, DNS |
| Storage | Azure services/CSI | class, topology, performance, backup |
| Observability | Azure Monitor integrations | enablement, retention, alerts, cost |
| Application | none | correctness, auth, resilience, data recovery |
Mental model AKS architecture
Managed control plane
Azure operates control-plane components.
Customer does not manage etcd/control-plane VMs.
Availability and SLA options depend on AKS mode/tier and region; verify current service terms.
Control-plane and node-resource ownership
Control plane is provider-managed.
Nodes, load balancers, disks, public IPs, and related resources are created in customer subscription, commonly in a managed node resource group.
Node resource group
AKS creates/manages infrastructure resources in a node resource group by default.
Manual modification of AKS-managed resources can be overwritten or break lifecycle.
Use supported AKS APIs.
Subscription and resource-group boundaries
Design:
- platform subscription;
- workload subscriptions;
- hub/spoke VNets;
- shared ACR/Key Vault/monitoring;
- role assignments;
- policy;
- quotas.
Kubernetes API access
Access path combines:
- network reachability;
- Microsoft Entra/IAM token;
- Kubernetes or Azure RBAC;
- local account policy.
Public API server
Public endpoint must use:
- Entra auth;
- authorized IP ranges where possible;
- no shared admin credentials;
- audit/resource logs.
Authorized IP ranges
Ranges can restrict public endpoint.
Maintain CI, VPN, NAT, and admin egress changes.
They do not apply to every private access mode identically.
Private cluster
Private AKS uses private networking for API server/node communication and private endpoint/DNS architecture.
Operators need VNet connectivity and private DNS resolution.
API Server VNet Integration boundary
API Server VNet Integration provides VNet-integrated API connectivity with different architecture from Private Link-based private cluster.
Feature and migration support are version/region dependent.
Choose with current docs and network team.
Private DNS
Private clusters depend on private DNS zones/links/resolvers.
Hub-spoke custom DNS must forward AKS private zone correctly.
Hub-spoke connectivity
Validate:
- routes;
- peering;
- DNS;
- firewall;
- UDR;
- return path;
- private endpoints;
- on-prem access.
AKS Standard
Benefits:
- broad VM/network/add-on customization;
- bring your VNet/subnets;
- choose CNI/data plane;
- custom ingress/egress;
- explicit upgrade strategy.
Costs:
- more platform decisions and lifecycle ownership.
AKS Automatic
Automatic provides managed defaults such as:
- automatically allocated/scaled compute;
- production-tuned policies;
- managed networking baseline;
- application routing;
- monitoring baseline;
- security defaults.
Exact managed components and restrictions evolve.
Automatic managed defaults
Current docs describe Automatic with defaults including Azure CNI Overlay powered by Cilium, managed egress/NAT behavior, application routing, LocalDNS, and preconfigured monitoring.
Do not reproduce these manually without checking ownership.
Automatic suitability boundary
Verify support for:
- custom CNI;
- Windows/GPU/special nodes;
- host networking;
- privileged DaemonSets;
- custom ingress;
- storage;
- policy;
- regulated network path;
- fixed capacity/reservations.
Migration and coexistence
Moving between Standard/Automatic or rebuilding can affect:
- node provisioning;
- network CIDRs;
- egress IPs;
- ingress classes;
- policy;
- monitoring;
- storage;
- identity;
- pricing.
Use blue-green cluster migration where in-place conversion is unsupported/risky.
Node-pool mental model
Node pool groups nodes with common:
- VM size;
- OS;
- mode;
- zones;
- autoscaling;
- taints/labels;
- max Pods;
- upgrade behavior.
System node pools
System mode hosts critical system Pods.
It has AKS-specific requirements and should use reliable capacity.
Do not schedule ordinary heavy workloads there.
User node pools
User pools host application workloads.
They can use different:
- VM sizes;
- OS;
- Spot;
- zones;
- taints;
- autoscaling.
System-pool availability
Use multiple system nodes and zones according to production reliability guidance and region support.
Protect CoreDNS, metrics, policy, CSI, and ingress dependencies.
Virtual Machine Scale Sets
AKS node pools commonly use VMSS for node lifecycle and scaling.
Do not manually scale/mutate VMSS outside supported AKS operations.
Node-pool modes
System and User affect scheduling rules and platform expectations.
At least one valid system pool is required.
Multiple node pools
Create pools by operational requirement, not one per microservice.
Useful dimensions:
- system/application;
- Spot/baseline;
- CPU/memory/GPU;
- Linux/Windows;
- architecture;
- compliance.
Node-pool immutability and recreation
Some pool properties cannot change in place.
Use new pool, cordon/drain, migrate, remove old pool.
Plan PDB and storage topology.
Availability zones
Zone support depends on region/VM size/storage.
AKS cluster/node pools created without zones may require recreation to gain zonal resilience.
Zone-redundant node pools
Spread nodes and application Pods across zones.
A zonal node pool plus no Pod topology spread can still co-locate replicas.
Regional node-pool boundary
Some configurations use regional/zone-balanced behavior.
Check actual VMSS zone settings and failure semantics.
VM sizes
Select from:
- CPU/memory ratio;
- network/storage throughput;
- ephemeral disk;
- architecture;
- quota/capacity;
- price;
- accelerated networking.
Benchmark Java workload.
CPU architecture
Use ARM64 where supported and images/native agents are multi-arch.
Separate architecture pools with labels/affinity.
Spot node pools
Spot pools are interruptible and user-mode.
Use:
- taint;
- toleration;
- multiple replicas;
- PDB;
- idempotency;
- no sole system capacity.
GPU and specialized pools
Need drivers/device plugins, compatible images, quotas, taints, and expensive idle-capacity governance.
Windows node pools boundary
Windows has separate networking, OS, storage, and version constraints.
Keep Linux system pool.
Azure Linux and Ubuntu
Choose supported node OS SKU and patch lifecycle.
Custom node packages are not a durable configuration mechanism.
Azure Linux 2 retirement
As of July 2026, Azure Linux 2.0 support and security updates ended on November 30, 2025, and node images were removed starting March 31, 2026.
Existing designs must migrate to Azure Linux 3 or another supported OS SKU.
Node images
Node image contains:
- OS;
- container runtime;
- kernel;
- packages;
- Kubernetes node components.
Track node image version separately from Kubernetes minor version.
Node image upgrades
Node image-only upgrade can deliver security/package fixes without changing Kubernetes minor.
Still drains/reimages nodes.
Virtual nodes boundary
Virtual nodes/ACI can burst selected Pods without VM nodes, with significant feature/network/storage limits.
Do not treat as transparent node pool.
Node labels and taints
Use AKS-supported labels/taints.
System-reserved labels may be immutable.
System workload isolation
Use taints, priorities, resources, and separate system pools.
Cluster Autoscaler
AKS integrates Kubernetes Cluster Autoscaler per node pool.
It adds/removes VMSS instances based on unschedulable Pods and utilization.
Autoscaler profiles
Profiles are cluster-wide and affect scale-down timing, utilization, delays, and unready handling.
Changing one value affects every autoscaled pool.
Scale-down and disruption
Scale-down must respect Pods and constraints but can be blocked by:
- PDB;
- local storage;
- system Pods;
- affinity;
- unmanaged Pods;
- safe-to-evict annotations.
Capacity and quota
Azure quotas include:
- regional vCPU;
- VM family;
- public IP;
- load balancer;
- disk;
- network;
- subscription/resource limits.
Autoscaling cannot exceed quota or regional capacity.
Authentication and authorization model
Network access
→ Microsoft Entra authentication
→ Azure RBAC or Kubernetes RBAC authorization
→ admission
Microsoft Entra authentication
AKS integrates with Entra for users/groups and tokens.
Use group-based access and conditional access/MFA where applicable.
Kubernetes RBAC
Native roles and bindings authorize Kubernetes API resources.
Azure RBAC for Kubernetes authorization
Azure RBAC can authorize Kubernetes actions using Azure role assignments and AKS integration.
Understand propagation latency and supported role scopes.
Local accounts boundary
Disable local accounts where Entra-only operation and break-glass design permit.
Local admin kubeconfig is highly privileged.
Admin kubeconfig risk
Admin credentials can bypass normal Entra group governance.
Protect, audit, rotate, and restrict retrieval.
Human-access lifecycle
Use Privileged Identity Management/eligible roles and groups where available.
Separate read, deploy, and admin.
CI/CD identities
Use workload federation/managed identity/service principal with limited Azure and Kubernetes rights.
Avoid client secrets.
Cluster managed identity
AKS cluster identity lets control plane manage Azure resources.
It is separate from application workload identity.
System-assigned managed identity
Lifecycle tied to AKS resource.
Simple, but identity ID changes if cluster recreated.
User-assigned managed identity
Independent Azure resource.
Useful for stable lifecycle, pre-created roles, and controlled sharing.
Sharing one identity across clusters increases blast radius.
Kubelet identity
Nodes/kubelet use a managed identity for operations such as pulling from ACR and managing certain node tasks.
Do not grant application data access to kubelet identity.
Control-plane identity versus kubelet identity
| Identity | Purpose |
|---|---|
| Cluster/control-plane managed identity | AKS manages Azure infrastructure |
| Kubelet identity | node-level Azure operations and image pull |
| Workload identity | application Pod accesses Azure APIs |
| Human/CI identity | cluster administration/deployment |
Managed-identity role assignments
Role assignment can take time to propagate.
IaC should order identity creation, role assignment, and cluster/workload rollout.
Workload identity mental model
Microsoft Entra Workload ID
Workload ID is the current Pod-to-Azure identity mechanism.
It uses OIDC federation and short-lived tokens.
OIDC issuer
AKS exposes an issuer URL when feature enabled.
Federated credential trusts issuer, subject, and audience.
Federated identity credential
Defines trust:
issuer + subject(system:serviceaccount:namespace:name) + audience
Scope exactly.
Kubernetes ServiceAccount
Use one ServiceAccount per workload capability.
Annotation/client-id configuration depends on setup.
Azure Identity SDK
Use supported Azure Identity library that understands Workload ID.
Old SDKs may require upgrade/migration sidecar; do not standardize the sidecar as permanent.
DefaultAzureCredential
DefaultAzureCredential can use workload identity when environment/token file is injected.
In production, verify provider order and avoid accidental fallback to developer credentials or node metadata.
Workload identity mutation and labels
Pods typically need the workload-identity enablement label and a matching ServiceAccount/federated credential.
Verify exact required annotations/labels for installed version.
Workload identity versus legacy pod-managed identity
Legacy AAD Pod Identity/pod-managed identity uses node/NMI/MIC-style mechanisms and has migration complexity.
Prefer Entra Workload ID for new workloads.
Least-privilege Azure RBAC
Scope roles to:
- one Key Vault/secret;
- storage account/container;
- Service Bus namespace/queue;
- Cosmos account/database;
- resource group only if required.
Avoid Contributor.
Cross-subscription access
Managed identity can receive roles in another subscription/tenant subject to Entra and Azure RBAC policy.
Model tenant boundaries and federation.
Credential caching and rotation
Azure SDK caches tokens and refreshes.
Do not persist access tokens.
Handle Entra endpoint/network/clock failures.
Network architecture choices
AKS networking includes:
- CNI/IPAM;
- data plane;
- NetworkPolicy;
- VNet/subnets;
- service/load balancer;
- egress;
- DNS;
- private endpoints.
Azure CNI overview
Azure CNI variants assign and route Pod addresses differently.
Choose before cluster creation because migration can require new cluster/pool.
Azure CNI Overlay
Pods receive addresses from an overlay CIDR separate from VNet node subnet.
Benefits:
- reduced VNet IP consumption;
- scalable address plan;
- simpler subnet sizing.
Egress to VNet uses node translation/routing semantics.
Azure CNI Powered by Cilium
Uses Cilium/eBPF data plane with Azure CNI IPAM options.
Benefits can include scalable service routing, policy, and observability.
Verify supported Kubernetes versions/features.
Azure CNI flat networking
Pods receive VNet-routable addresses through Azure CNI models.
Benefits:
- direct VNet reachability;
- NSG/infrastructure integration.
Costs:
- subnet IP consumption;
- planning;
- route/security complexity.
Kubenet boundary
Kubenet is older/basic networking with node-based routing/NAT.
For new production designs, evaluate current Azure guidance and CNI roadmap rather than selecting from familiarity.
Automatic networking defaults
AKS Automatic currently uses Azure CNI Overlay powered by Cilium and provider-managed networking defaults.
Do not add conflicting CNI/egress components.
Pod and node address spaces
Plan non-overlapping:
- VNet/subnets;
- Pod CIDR;
- Service CIDR;
- on-prem/hub;
- peered VNets;
- private endpoints;
- future clusters.
Service CIDR
Service virtual IP range must not overlap connected networks.
Changing often requires cluster recreation.
DNS service IP
Choose an IP inside Service CIDR according to AKS requirements.
Subnet sizing
Account for:
- nodes;
- upgrade surge;
- autoscaler max;
- flat Pod IPs if used;
- load balancers/private endpoints;
- other resources;
- one-zone failure.
Pod IP exhaustion
In flat networking, subnet exhaustion blocks Pod scheduling.
Overlay reduces VNet Pod IP consumption but Pod CIDR still needs capacity.
maxPods
Affects per-node address allocation and scheduling density.
Too high can exhaust IPs and node CPU/memory.
Overlay routing and SNAT
Overlay Pod traffic to outside cluster is typically translated through node/egress path.
This affects source IP and NetworkPolicy/NSG analysis.
VNet-routable Pod IPs
Direct Pod addresses simplify some reachability but increase subnet and route dependence.
Dual-stack networking
Requires cluster/network/LB/application/dependency support.
Test IPv6 egress and private endpoints.
Cilium eBPF data plane
eBPF replaces parts of service/network-policy dataplane.
Use Cilium tooling/metrics for diagnosis in addition to Azure/Kubernetes logs.
NetworkPolicy
Standard Kubernetes policy requires selected engine.
AKS supports policy options according to networking mode/version.
Cilium Network Policy
Cilium can enforce standard NetworkPolicy and Cilium extensions.
Extensions increase platform coupling.
Azure Network Policy Manager retirement
Microsoft announced Linux Azure NPM support ending September 30, 2028.
New/modern designs should plan Cilium Network Policy and migrate before deadline.
Security groups and NSGs
NSGs apply at subnet/NIC infrastructure boundaries.
They do not replace Pod NetworkPolicy.
Overlay Pod source can appear as node addresses externally.
Application Security Groups boundary
ASGs group Azure NICs, not arbitrary Kubernetes ServiceAccounts.
Use only where networking mode/integration supports intended resources.
Private Link and private endpoints
Use private endpoints for Azure PaaS services.
Need:
- DNS zone links;
- routes;
- NSG/firewall;
- service-specific policies;
- cross-region resolution.
User-defined routes
UDRs send egress through firewall/NVA.
AKS route ownership and outbound type must align.
Asymmetric routing breaks traffic.
Azure Firewall egress
Provides centralized filtering, FQDN/application rules, and logging.
Costs and SNAT capacity must be modeled.
Outbound types
AKS supports different outbound models depending mode/version:
- load balancer;
- managed NAT gateway;
- user-assigned NAT gateway;
- user-defined routing;
- none/blocked scenarios;
- Automatic-managed defaults.
Choose deliberately.
Azure Load Balancer egress
Outbound rules can use Load Balancer public IP/SNAT.
Port capacity depends on configuration and node count.
Managed NAT Gateway
Provides managed outbound IP and SNAT scale.
Automatic clusters may provision managed egress according to current defaults.
User-assigned NAT Gateway boundary
Attach existing NAT to subnet for controlled addresses/ownership where supported.
Egress FQDN requirements
Restricted clusters need documented Azure endpoints for:
- control plane/node operations;
- MCR/ACR;
- Entra;
- monitoring;
- extension/add-ons;
- workload identity;
- OS updates.
Required list changes with features.
Restricted-egress clusters
Use Azure Firewall/UDR/private endpoints with explicit validation.
Blocking required FQDNs can stop upgrades, identity, monitoring, or node operations.
SNAT port exhaustion
Symptoms:
- intermittent outbound timeout;
- destination-specific failures;
- high connection churn.
Mitigate:
- connection pooling;
- NAT Gateway;
- more egress IPs;
- Private Link;
- destination distribution;
- monitoring.
LocalDNS boundary
AKS Automatic includes LocalDNS according to current docs.
Standard clusters may use different DNS caching architecture.
Treat it as a node-local dependency.
CoreDNS
CoreDNS remains Kubernetes service discovery.
Protect replicas/system pools/resources.
Custom DNS and hybrid resolution
Azure custom DNS must resolve:
- cluster-private API;
- Azure private endpoints;
- cluster Services;
- on-prem domains.
Conditional forwarding and private resolver design are critical.
Load-balancing options
| Requirement | Common Azure mechanism |
|---|---|
| Kubernetes L4 Service | Azure Load Balancer |
| Managed NGINX Ingress | AKS application routing add-on |
| Azure-native L7/WAF | Application Gateway/AGC family |
| Custom proxy | NGINX/Envoy/HAProxy controller |
| Global edge | Front Door/Traffic Manager plus regional path |
Azure Load Balancer
AKS Service LoadBalancer integrates with Azure Load Balancer.
Public/internal configuration uses annotations and Azure resources.
Public and internal Services
Internal Services use private frontend IP/subnet configuration.
Public Services require public IP and exposure policy.
Backend pools and health probes
AKS manages LB rules/backends/probes from Service.
Health probe path/protocol must align with application or node target mode.
Source IP and external traffic policy
externalTrafficPolicy: Local can preserve source IP but requires local healthy endpoints and changes availability.
Application routing add-on
AKS application routing add-on manages one or more NGINX ingress controllers and integrates with Azure DNS according to configuration.
Microsoft currently recommends it for AKS ingress use cases.
Managed NGINX ingress
Provider manages lifecycle/configuration surface of the add-on.
Application teams still own Ingress objects, TLS, DNS intent, routes, and backend health.
Application routing and community ingress-nginx retirement
The application routing add-on is an AKS-managed product capability even though it uses NGINX ingress technology.
Do not equate its support lifecycle with a self-installed retired community kubernetes/ingress-nginx deployment.
Verify Microsoft service support and migration notices.
Azure DNS integration
Application routing can manage records for configured public/private Azure DNS zones.
Use least-privilege identity and ownership boundaries.
Gateway API boundary
AKS/ingress implementations evolve toward Gateway API.
Check exact controller, CRD, conformance, and provider support.
Application Gateway for Containers boundary
AGC is Azure-native L7 application delivery with Gateway API/Ingress integrations according to current service support.
Evaluate WAF, private/public, performance, identity, and migration.
Application Gateway Ingress Controller boundary
AGIC integrates AKS with Application Gateway.
Its architecture differs from AGC and NGINX.
Do not mix resource ownership.
NGINX and other ingress controllers
Self-managed controllers require:
- lifecycle;
- security;
- LoadBalancer;
- class;
- metrics;
- migration;
- community support.
Community ingress-nginx is retired; use a maintained option.
Web Application Firewall boundary
Front Door/Application Gateway/other products provide WAF.
WAF does not replace application security.
TLS and Key Vault certificate boundary
Certificates can live in Kubernetes Secret, Key Vault-integrated path, or edge service.
Define rotation and trust.
Deregistration and graceful termination
Align:
Pod readiness
→ Azure LB/ingress backend update
→ connection drain
→ Java shutdown
→ termination grace
Storage mental model
Azure Disk CSI
Provides block storage attached to nodes.
Suitable for single-node attached filesystems and stateful applications.
Disk zone affinity
Zonal disks constrain Pod scheduling to compatible zone.
Use topology-aware binding and multi-zone replicas where application supports.
Disk performance tiers
Select size/type/IOPS/throughput from workload.
Disk scaling may affect cost and performance independently.
Azure Files CSI
Shared SMB/NFS file storage.
Consider latency, identity, permissions, throughput, and protocol.
NFS and SMB
Linux/Windows support and mount options differ.
Test locking and case semantics.
Azure Blob CSI boundary
Blob CSI exposes object data through mount semantics with limitations.
Prefer Blob SDK for true object workflows.
Ephemeral OS and temporary disks
Ephemeral disks are fast and node-local but disappear on reimage/replacement.
Use only for reconstructable data.
StorageClass governance
Platform defines:
- CSI driver;
- disk/file SKU;
- reclaim;
- expansion;
- binding mode;
- zone;
- encryption;
- mount options;
- tags.
Volume expansion
Verify driver/filesystem support and Pod restart requirements.
Volume snapshots
CSI snapshot resources integrate with Azure snapshots according to driver/support.
Application consistency remains separate.
Stateful workload placement
Combine zone, disk, PDB, node pools, backup, and replication.
Backup and restore
Back up:
- Kubernetes objects;
- persistent volumes;
- external PaaS data;
- Key Vault/identity config;
- DNS;
- IaC.
Azure Backup for AKS
Azure Backup can protect AKS resources and persistent volumes according to supported coverage.
Verify region, storage type, extension, identity, vault, and restore behavior.
External database backups
Azure Database for PostgreSQL, Cosmos DB, SQL, etc. use service-native backup/replication.
AKS backup does not automatically include them.
Key Vault and secrets
Prefer workload identity and direct SDK retrieval or CSI provider rather than static secrets.
Azure Key Vault provider for Secrets Store CSI
Mounts Key Vault objects into Pods using managed/workload identity.
Rotation updates mounted files according to configured behavior.
Secret rotation
Application must reload files/credentials and reconnect.
Kubernetes Secret boundary
Kubernetes Secret remains API data protected by RBAC/etcd controls, not a vault.
Observability mental model
Azure Monitor
Azure Monitor is the umbrella for metrics, logs, alerts, dashboards, and integrations.
Container insights
Collects container inventory/performance/logs according to data-collection rules.
Control ingestion volume.
Managed service for Prometheus
Managed Prometheus collects/stores Prometheus metrics.
Use recording/alert rules and cardinality governance.
Azure Managed Grafana
Visualizes Prometheus and other data sources with Entra access.
Control-plane resource logs
Enable relevant AKS control-plane categories through diagnostic settings.
Categories/availability vary by mode/tier/version.
Activity logs
Azure Activity Log captures Azure resource-management operations.
Kubernetes audit/resource logs are separate.
Diagnostic settings
Route logs/metrics to:
- Log Analytics;
- Storage;
- Event Hub;
- partner destinations.
Set retention and access.
OpenTelemetry
Use Azure Monitor OpenTelemetry distribution or vendor-neutral collectors according to organization standard.
Avoid double collection.
Network observability
Cilium/Azure network observability features can expose flows and DNS/service data.
Feature maturity and cost vary.
Application logs
Use structured stdout with cluster/namespace/workload/Pod/image/trace fields.
Cost and cardinality
Log Analytics and Prometheus costs rise with volume/cardinality.
Filter debug logs and high-cardinality labels.
Security posture
Layers:
- tenant/subscription;
- Entra;
- Azure RBAC;
- Kubernetes RBAC;
- network;
- Pod security;
- identity;
- image;
- Key Vault;
- Defender/Policy;
- audit.
Private API and network segmentation
Private API reduces public exposure but requires secure admin connectivity.
Use network policy and private endpoints for workloads.
Microsoft Entra and RBAC
Centralize human identity, MFA/PIM, groups, and revocation.
Avoid permanent cluster-admin groups.
Pod Security Admission
Enforce restricted/baseline standards per namespace.
Azure Policy for Kubernetes boundary
Azure Policy can audit/enforce cluster/workload rules through add-ons/integrations.
Admission dependency and policy rollout can block workloads.
Defender for Containers boundary
Defender can provide posture and runtime/vulnerability signals depending plan and agents.
It is not a substitute for controls and incident ownership.
Azure Container Registry
ACR stores images close to AKS.
Use private endpoints/firewall, retention, immutable digests, and geo-replication where needed.
ACR pull identity
Kubelet identity needs AcrPull or equivalent access.
Application workload identity should not receive image-pull role unless needed.
Image scanning and provenance
Use Defender/ACR ecosystem and admission policies for vulnerabilities/signatures.
Encryption
Cover:
- managed disks/files;
- Key Vault;
- TLS;
- Log Analytics/storage;
- backups;
- customer-managed keys where required.
Disk encryption sets boundary
Customer-managed-key disk encryption adds key availability/role complexity.
Confidential containers boundary
Confidential nodes/containers protect specific threat models with hardware isolation.
Compatibility and cost must be evaluated.
Multi-subscription and landing-zone architecture
Separate platform, production, non-production, shared networking, and security as required.
Use Azure Policy and management groups.
Multi-tenancy
For hostile/high-regulation tenants, use separate clusters/subscriptions.
Namespace is logical isolation only.
Reliability mental model
Availability requires:
AKS control plane
+ system pool
+ user-node capacity
+ CNI/IP
+ DNS
+ LB/ingress
+ storage
+ Azure dependencies
+ workload replicas
Availability zones and zone resilience
Create zone-redundant pools where region supports.
Distribute Pods with topology spread.
Control-plane versus data-plane availability
Control plane can be healthy while node pools/IP/DNS/LB fail.
System-pool resilience
If system pool fails, CoreDNS/policy/CSI/metrics can fail cluster-wide.
Use reliable multi-node/zone capacity.
Application topology spread
Spread at zone and hostname levels.
Ensure every zone has node capacity and disk/network compatibility.
Node replacement and PDBs
Upgrades, autoscaler, repairs, and Spot terminate nodes.
PDBs plus replicas and surge capacity determine progress.
Regional outage
One AKS cluster is regional and cannot survive full regional loss.
Multi-region deployment models
Use active/passive, active/active, or stamps/cells.
Need Front Door/Traffic Manager/global DNS, data replication, identity, artifacts, and failover runbook.
Failure-domain headroom
Remaining zones/pools must handle critical traffic after loss.
Cluster recreation
Versioned IaC and GitOps plus external backups make clusters replaceable.
Upgrade lifecycle
Upgrade includes:
- Kubernetes control plane;
- node pools;
- node images/OS;
- add-ons/extensions;
- APIs;
- workloads.
Supported Kubernetes versions
AKS support varies by region and changes over time.
Use AKS release tracker/API, not hard-coded assumptions.
Control-plane and node version skew
AKS enforces supported skew and upgrade paths.
Upgrade control plane and pools according to current policy.
Manual upgrades
Explicitly select target version and node pools.
Validate prechecks and quota.
Automatic upgrade channels
Channels can automate Kubernetes upgrades.
Choose from stability/compliance appetite and maintenance windows.
Automation does not prove application compatibility.
Node OS upgrade channels
Node OS channels control security/node-image updates separately.
Understand reimage/drain behavior.
Planned maintenance
Configure allowed maintenance windows for control plane/node OS where supported.
Emergency security action can still occur outside ideal window.
Surge upgrades
AKS adds surge nodes during pool upgrade.
Requires subnet IP, VM quota/capacity, and downstream capacity.
PDB and drain behavior
PDB can block upgrade.
AKS force/timeout behavior depends on operation settings.
Review unready Pods and max surge.
Upgrade prechecks
AKS prechecks can catch quota, subnet, PDB, and compatibility issues.
Still run application tests.
API deprecation
Scan manifests, controllers, webhooks, and audit logs.
Blue-green node-pool upgrade
Create new pool with desired OS/version/settings, shift workloads, drain old pool.
Good for immutable properties and rollback.
Blue-green cluster migration
Use for network-mode, private-cluster, Automatic/Standard, or major architecture change.
Rollback boundary
AKS control-plane rollback support is limited compared with application rollback and may not be available for every upgrade path.
Assume forward-fix or blue-green unless current docs explicitly support rollback.
Cost model
Cost includes more than nodes.
Control-plane tier and Automatic charges
AKS offers modes/tiers with different management/SLA/Automatic pricing.
Use current Azure pricing; avoid permanent assumptions.
VM and node-pool cost
Cost drivers:
- VM SKU;
- OS;
- disk;
- idle system pools;
- max autoscaler;
- surge;
- zones;
- reservations/savings.
Spot and reservations
Spot lowers interruptible cost.
Reservations/savings plans reduce stable baseline.
Load balancer and public-IP cost
Track rules, data processing, public IPs, Application Gateway/AGC/Front Door, and shared ingress.
NAT Gateway and Firewall cost
Hourly/data/SNAT architecture can dominate egress.
Storage and backup cost
Managed disks, snapshots, Files, Blob, backup vault, cross-region replication.
Log Analytics and metrics cost
Container logs, audit, Prometheus samples, Application Insights, and long retention.
Cross-zone and egress cost
Cross-region/internet/firewall and some zone paths can add cost.
Rightsizing and autoscaling
Accurate Pod requests enable bin-packing.
Automatic/Cluster Autoscaler still needs workload requests and PDBs.
Cost allocation
Use Azure tags at infrastructure level and Kubernetes labels/cost tooling for shared clusters.
JAX-RS workload integration
ServiceAccount
One Azure workload identity per application capability.
Network
Private endpoints and DNS for Azure PaaS.
Health
LB/ingress readiness aligned with Java lifecycle.
Observability
OpenTelemetry plus platform metadata.
Java Azure SDK identity
Use supported azure-identity and workload identity.
Do not inject client secret for normal AKS workload.
Azure SDK client lifecycle
Reuse clients.
Configure:
- endpoint;
- credential;
- retry;
- timeout;
- connection pool;
- diagnostics;
- shutdown.
Key Vault, Blob, Service Bus, and Cosmos access
Each service requires:
- data-plane RBAC;
- private DNS/network;
- quota;
- retry/idempotency;
- SDK semantics;
- cost.
Private endpoints and DNS
A private endpoint without correct private DNS can silently resolve public address or fail.
Test from the Pod.
Health and load balancers
Azure LB and ingress health can differ from Kubernetes readiness.
Map every probe and backend mode.
Graceful termination
Align:
- Pod readiness;
- Load Balancer/ingress backend update;
preStop;- Java drain;
- termination grace;
- long connections.
Connection pools and node churn
Rollouts/upgrades create connection churn to PostgreSQL, Redis, Service Bus, and other services.
Use jitter and pool limits.
Failure-model matrix
| Failure | Impact | Detection | Response |
|---|---|---|---|
| Public API unrestricted | attack surface | AKS config | private/authorized ranges |
| Private cluster DNS missing | admin/node failure | DNS tests | zone links/forwarding |
| Node resource group edited manually | lifecycle drift | activity log | supported AKS API |
| System pool on Spot/single node | cluster add-on outage | placement | reliable system pool |
| Azure Linux 2 retained after removal | cannot scale/security risk | OS inventory | migrate to Azure Linux 3 |
| Cluster identity gets app data roles | privilege expansion | role audit | workload identity |
| Kubelet identity gets Key Vault/Blob app rights | node compromise impact | RBAC review | separate identities |
| Federated credential wildcard subject | cross-workload access | Entra audit | exact subject |
| Old Azure Identity SDK | Workload ID failure | SDK logs | supported version |
DefaultAzureCredential uses wrong provider | unexpected privilege | identity telemetry | deterministic config |
| Flat CNI subnet exhausted | Pods cannot start | IP metrics/events | overlay/larger subnet |
| Overlay CIDR overlaps hub/on-prem | routing failure | design review | non-overlap/rebuild |
| Azure NPM retirement ignored | unsupported policy | inventory | migrate Cilium before 2028 |
| UDR/firewall blocks required AKS FQDN | upgrade/identity failure | flow logs | egress matrix |
| SNAT ports exhausted | intermittent outbound failure | NAT metrics | NAT/pooling/private endpoint |
| Application routing confused with self-managed ingress-nginx | wrong lifecycle decision | add-on inventory | provider-managed support model |
| LB health path differs from readiness | dropped traffic | probe logs | align health |
| Azure Disk zone mismatch | Pod Pending | scheduler/CSI | topology-aware binding |
| Key Vault CSI rotates file but app does not reload | stale credential | version/connection | reload/reconnect |
| Diagnostic settings disabled | weak incident evidence | policy | enable/retain |
| Log Analytics cardinality grows | cost/ingestion issue | billing | DCR/filter/labels |
| Surge upgrade lacks VM quota/IP | upgrade stuck | precheck/events | quota/headroom |
| PDB blocks node image update | stale nodes | operation error | replicas/PDB |
| Automatic channel upgrades incompatible app | outage | release tests | staging/maintenance/gates |
| One zone contains all application Pods | zone outage | placement | topology spread |
| AKS backup assumed to include PaaS DB | incomplete restore | drill | service-native backup |
Debugging playbook
Nodes are not ready or pool cannot scale
Check:
- regional/family vCPU quota;
- VM SKU capacity;
- subnet/IP;
- route/NSG/firewall;
- node image/OS support;
- cluster/kubelet identity roles;
- private DNS/API;
- CNI;
- VMSS provisioning errors.
Pods are Pending despite autoscaler
Check requests, node selectors/taints, zones, max Pods/IP, disk zone, Spot availability, autoscaler max, quota, and system-reserved capacity.
Workload ID returns 401/credential unavailable
Check:
- cluster OIDC/Workload ID enabled;
- Pod label;
- ServiceAccount annotation/client ID;
- federated credential issuer/subject/audience;
- Azure Identity version;
- role assignment propagation;
- Entra/network/clock;
- wrong tenant;
- fallback provider.
ACR image pull fails
Check kubelet identity AcrPull, ACR firewall/private endpoint/DNS, image name/tag/digest, architecture, and node egress.
Private Key Vault/Storage access fails
From Pod, verify DNS resolves private endpoint, route/firewall, Workload ID token, data-plane role, and service firewall.
Service LoadBalancer remains pending
Check Azure LB quota, subnet/public IP annotations, permissions, node resource group policy, controller events, and unsupported configuration.
Application routing returns 502/503
Check IngressClass/add-on, controller Pods, backend Service/EndpointSlices, readiness, TLS Secret, DNS, and timeout.
Azure Disk cannot attach
Check zone, CSI driver, identity, disk state, attachment limit, StorageClass, encryption, and node pool.
Upgrade fails
Check prechecks, PDB, max surge, subnet IPs, VM quota, unsupported node OS, deprecated APIs, admission webhooks, and maintenance settings.
DNS works on one node only
Check CoreDNS/LocalDNS, Cilium, node route/NSG, custom DNS, private-zone links, and node health.
Testing strategy
Cluster foundation tests
- public/private API path;
- Entra and RBAC;
- private DNS;
- system-pool failure;
- zone spread;
- quota;
- IaC recreation.
Identity tests
- workload positive/denied actions;
- wrong ServiceAccount;
- wrong namespace;
- token rotation;
- role propagation;
- no kubelet/cluster-identity fallback;
- private Entra path.
Network tests
- overlay/flat IP capacity;
- Cilium NetworkPolicy;
- UDR/Firewall;
- NAT/SNAT;
- private endpoints;
- DNS;
- LoadBalancer source IP;
- dual-stack where used.
Node-pool tests
- autoscale up/down;
- Spot eviction;
- system/user isolation;
- node image upgrade;
- zone loss;
- ARM64;
- blue-green pool migration.
Ingress tests
- application routing;
- public/private DNS;
- TLS;
- client IP;
- path/host;
- drain;
- controller upgrade;
- migration from self-managed controller.
Storage and backup tests
- disk/files provisioning;
- zone scheduling;
- expansion;
- snapshots;
- Key Vault CSI rotation;
- AKS Backup restore;
- external DB restore.
Upgrade tests
- channels in non-prod;
- prechecks;
- surge/quota;
- PDB;
- API scan;
- node OS;
- application mixed version;
- rollback/forward-fix.
Architecture patterns
Private AKS in hub-spoke
Private API, custom/private DNS, Azure Firewall/UDR, private endpoints, controlled admin path.
System/user pool isolation
Reliable zonal system pool plus application-specific user pools.
Entra Workload ID per ServiceAccount
No client secrets and no application roles on cluster/kubelet identities.
Azure CNI Overlay with Cilium
Reduces VNet IP pressure and provides modern policy/data plane for suitable workloads.
Managed application routing
Provider-managed NGINX ingress with explicit service support ownership.
Replaceable regional cluster
IaC/GitOps plus PaaS databases, Azure Backup, and tested recreation.
Multi-region stamp
Independent regional AKS and data plane behind global routing.
Anti-patterns
- treat AKS managed control plane as fully managed application;
- leave API server public without ranges;
- manually edit node resource-group VMSS/LB;
- put applications on system pool without policy;
- use Spot as only system capacity;
- retain Azure Linux 2 after support/image removal;
- grant application data roles to cluster or kubelet identity;
- use client secrets in Pods instead of Workload ID;
- wildcard federated credential subjects;
- use outdated Azure Identity SDK;
- choose flat CNI without subnet growth model;
- overlap Pod/Service CIDR with hub/on-prem;
- ignore Azure NPM retirement;
- block egress without AKS/add-on FQDN matrix;
- assume private endpoint works without private DNS;
- equate AKS application routing add-on with unsupported self-managed ingress-nginx;
- assume Azure LB balances each HTTP/2 request;
- use Azure Disk without zone-aware scheduling;
- assume AKS Backup covers external PostgreSQL/Cosmos/Blob;
- enable all logs indefinitely;
- use auto-upgrade channel without staging and maintenance gates;
- define impossible PDB during upgrades;
- claim zone resilience without topology spread and spare capacity.
PR review checklist
Cluster and access
- Standard or Automatic decision?
- API public/private/integration mode?
- authorized ranges/private DNS?
- Entra auth?
- Kubernetes or Azure RBAC?
- local accounts/admin kubeconfig?
- control-plane logs?
- IaC ownership?
Node pools
- system/user separation?
- multiple system nodes/zones?
- VM size/architecture/OS supported?
- no Azure Linux 2?
- Spot tainted and optional?
- max Pods/subnet capacity?
- autoscaler profile?
- PDB/surge/quota?
- blue-green path for immutable change?
Identity/security
- cluster and kubelet identities separated?
- Workload ID enabled?
- exact federated subject/audience?
- one ServiceAccount/capability?
- supported Azure Identity SDK?
- least-privilege data-plane roles?
- ACR pull role only on kubelet?
- Pod Security/Policy/Defender boundaries?
Networking/edge
- Azure CNI mode/data plane?
- Pod/Service/VNet CIDR non-overlap?
- Cilium policy?
- Azure NPM migration?
- outbound type/NAT/Firewall?
- required FQDN/private endpoints?
- ingress/application routing ownership?
- LB source IP/health/drain?
- private DNS tested from Pod?
Storage/observability
- Disk/Files/Blob semantics?
- zone binding?
- encryption/Key Vault?
- snapshots/backup?
- Azure Monitor/Container insights?
- managed Prometheus/Grafana?
- diagnostic categories/retention?
- cardinality/cost?
Reliability/upgrades
- zone spread and failure headroom?
- system-pool resilience?
- supported Kubernetes/OS versions?
- upgrade channels/windows?
- max surge and quota?
- API-deprecation gate?
- AKS Backup plus external data?
- multi-region/restore drill?
Trade-off yang harus dipahami senior engineer
| Decision | Benefit | Cost/risk |
|---|---|---|
| AKS Standard | customization/control | platform operations |
| AKS Automatic | managed production defaults | supported-feature constraints |
| System/user pools | isolation | extra baseline cost |
| Spot pool | cost | interruption/capacity |
| Azure Linux 3 | Azure-optimized supported OS | migration/testing |
| Ubuntu | ecosystem familiarity | different lifecycle |
| Entra + Azure RBAC | centralized Azure governance | propagation/role mapping |
| Kubernetes RBAC | native granularity | separate governance plane |
| System-assigned identity | simple lifecycle | changes on recreation |
| User-assigned identity | stable/reusable | role/lifecycle complexity |
| Workload ID | no secrets/short-lived tokens | OIDC/SDK/config |
| Azure CNI Overlay | lower VNet IP use | translated Pod networking |
| Flat Azure CNI | VNet-routable Pods | subnet consumption |
| Cilium | eBPF/policy/scale | operational/tool coupling |
| Private cluster | lower API exposure | DNS/admin complexity |
| Azure Firewall | centralized egress policy | cost/latency/SNAT |
| NAT Gateway | scalable stable egress | cost/no L7 policy |
| Managed application routing | reduced ingress lifecycle | provider feature boundary |
| Self-managed ingress | flexibility | security/upgrade burden |
| Azure Disk | performant block | zonal attachment |
| Azure Files | shared storage | latency/protocol cost |
| Automatic upgrades | security currency | compatibility risk |
| Blue-green pool/cluster | rollback/isolation | duplicate cost |
Internal verification checklist
Tenant/subscription/cluster
- Entra tenant and subscriptions.
- management groups/policies.
- AKS cluster mode/tier/version.
- resource groups/node resource group.
- public/private/API integration.
- authorized ranges.
- local accounts.
- Entra/RBAC model.
- IaC/GitOps source.
Node pools
- system/user pools.
- VMSS/VM sizes.
- zones.
- OS SKU and node image.
- no Azure Linux 2.
- architecture.
- Spot/GPU/Windows.
- autoscaler profile.
- quotas/reservations/headroom.
Identity
- cluster managed identity.
- kubelet identity.
- ACR role.
- Workload ID/OIDC.
- ServiceAccounts.
- federated credentials.
- Azure Identity SDK versions.
- role assignments/scopes.
- PIM/break-glass.
Networking
- VNet/subnets/CIDRs.
- Azure CNI mode.
- Cilium/data plane.
- max Pods/IP capacity.
- Service/DNS CIDR.
- NetworkPolicy engine.
- NSGs/UDRs/Firewall.
- outbound type/NAT/SNAT.
- private endpoints/DNS.
- CoreDNS/LocalDNS.
Edge/storage
- Azure LB Services.
- application routing add-on.
- other ingress/Gateway.
- public/private DNS.
- TLS/WAF/source IP.
- Azure Disk/Files/Blob CSI.
- StorageClasses/zones.
- Key Vault CSI.
- backup/snapshots.
Operations
- Azure Monitor/Container insights.
- managed Prometheus/Grafana.
- diagnostic settings.
- Defender/Policy.
- upgrade and node-OS channels.
- maintenance windows.
- API/OS lifecycle.
- restore/multi-region tests.
- FinOps/budgets.
Latihan verifikasi
- Draw the AKS control plane, node resource group, VNet, system/user pools, LB, private endpoint, identity, and monitoring paths.
- Migrate one Java workload from client-secret authentication to Microsoft Entra Workload ID and prove least privilege.
- Calculate subnet/IP capacity for Azure CNI flat versus Overlay under max autoscaling, surge upgrade, and zone failure.
- Build a private AKS cluster path through hub DNS and private endpoints; test resolution from admin and Pod.
- Apply Cilium default-deny policy and restore only DNS, Entra, monitoring, Key Vault, and application flows.
- Compare Azure Load Balancer and application routing add-on for source IP, health, HTTP/2, and drain.
- Upgrade or replace an Azure Linux 2 test pool with Azure Linux 3 through blue-green node-pool migration.
- Trigger Spot eviction and system/user pool failure while measuring JAX-RS availability.
- Run node-image/Kubernetes upgrade with max surge, quota pressure, and PDB constraints.
- Restore AKS resources and Azure Disk data to a recreated cluster while separately restoring external databases.
Ringkasan
- AKS manages Kubernetes control plane; customer remains responsible for workloads, node/network policy, identity, data, and operations.
- AKS Standard maximizes configuration control; Automatic provides broader provider-managed defaults.
- Node pools are the core data-plane lifecycle and failure boundary.
- Reliable clusters isolate system workloads from application/Spot capacity and spread both across zones.
- Azure Linux 2 is no longer a supported 2026 production node-image choice; migrate to Azure Linux 3 or another supported OS.
- Microsoft Entra authentication and Kubernetes/Azure RBAC govern human access.
- Cluster identity, kubelet identity, and application Workload ID must remain separate.
- Entra Workload ID uses OIDC federation and supported Azure Identity SDKs, not static client secrets.
- Azure CNI Overlay reduces VNet IP demand; Azure CNI Powered by Cilium provides an eBPF data plane and modern policy option.
- Network CIDRs, subnet capacity, max Pods, outbound type, SNAT, private endpoints, and DNS are architectural constraints.
- Azure NPM for Linux has a 2028 retirement deadline; Cilium migration should be planned early.
- AKS application routing add-on is provider-managed and should not be treated as a self-installed retired community ingress-nginx deployment.
- Azure Disk is zonal block storage; Azure Files is shared file storage; Blob remains object storage.
- Azure Monitor, Container insights, managed Prometheus, Grafana, and diagnostic settings provide complementary signals.
- Upgrade lifecycle spans Kubernetes, node images/OS, add-ons, policies, and workloads.
- Surge upgrades need VM quota, subnet capacity, PDB compatibility, and downstream headroom.
- AKS Backup does not replace service-native backups for external databases.
- Zone resilience requires actual Pod spread and capacity after a zone/pool loss.
- Exact tenant, subscription, cluster mode, CNI, identity, routing, storage, and lifecycle remain Internal verification checklist.
Referensi resmi
- What is Azure Kubernetes Service
- AKS Best Practices
- AKS Automatic
- AKS Reliability Best Practices
- Reliability in AKS
- AKS System Node Pools
- AKS Node Pools
- AKS Cluster Autoscaler
- AKS Managed Identity Overview
- Microsoft Entra Workload ID
- AKS Access and Identity
- Private AKS Clusters
- AKS Networking Concepts
- Azure CNI Overview
- Azure CNI Overlay
- Azure CNI Powered by Cilium
- AKS Network Best Practices
- AKS Network Policies
- AKS Network Policy Best Practices
- AKS Outbound Network Requirements
- AKS Application Routing Add-on
- AKS Storage Concepts
- Azure Disk CSI on AKS
- Azure Files CSI on AKS
- Azure Key Vault Provider for Secrets Store CSI
- Monitor AKS
- Azure Monitor Kubernetes Best Practices
- AKS Supported Kubernetes Versions
- AKS Upgrade Options
- Upgrade an AKS Cluster
- AKS Zone Resiliency
- AKS Multi-region Deployment Models
- AKS Cost Best Practices
You just completed lesson 50 in final stretch. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.