Part 007 — DNS in AWS and Azure

Fokus part ini adalah memahami DNS sebagai control point kritis dalam sistem enterprise cloud. Banyak masalah yang tampak seperti aplikasi lambat, database down, SDK timeout, atau private endpoint unreachable sebenarnya adalah masalah name resolution: record salah, private zone tidak ter-link, CoreDNS forward salah, TTL terlalu panjang, split-horizon tidak konsisten, atau resolver hybrid tidak tahu domain yang harus diteruskan.

DNS bukan sekadar “nama menjadi IP”. Dalam production, DNS menentukan arah traffic, private/public path, failover behavior, blast radius, observability, dan keamanan akses dependency.

Untuk backend Java/JAX-RS di EKS/AKS, DNS ada di beberapa layer:

Application code
  ↓ resolves endpoint hostname
JVM / OS resolver
  ↓
Pod DNS config
  ↓
Kubernetes CoreDNS
  ↓
Cloud VPC/VNet resolver
  ↓
Route 53 / Azure DNS / Private DNS Zone / corporate DNS
  ↓
A / AAAA / CNAME / Alias / private endpoint record
  ↓
Network path to target service

Jika salah satu layer salah, symptom di aplikasi sering hanya terlihat sebagai:

• UnknownHostException
• ConnectTimeoutException
• SSLHandshakeException
• AWS SDK timeout
• Azure SDK timeout
• HTTP 502/503/504
• database connection refused atau timeout
• broker unreachable
• secret/config retrieval failure

1. Core mental model

DNS adalah distributed naming system. Aplikasi jarang memanggil IP langsung. Aplikasi memanggil nama:

api.internal.company.local
postgres.prod.internal
kafka-bootstrap.prod.internal
my-bucket.s3.ap-southeast-1.amazonaws.com
myvault.vault.azure.net
my-service.default.svc.cluster.local

Resolver mengubah nama tersebut menjadi alamat yang bisa dipakai network stack.

Namun dalam cloud, satu nama bisa resolve ke IP berbeda tergantung tempat query dilakukan:

Inilah kenapa DNS harus dibaca bersama network path. Nama yang sama tidak selalu berarti jalur yang sama.

Mental model praktis

Hostname resolution = namespace + resolver path + record + source context

Untuk setiap hostname production, tanyakan:

1. Nama ini dimiliki siapa?
2. Zone-nya public atau private?
3. Resolver mana yang menjawab?
4. Dari pod, resolve ke IP apa?
5. Dari node, resolve ke IP apa?
6. Dari on-prem, resolve ke IP apa?
7. Apakah IP hasil resolve private atau public?
8. Apakah TTL terlalu panjang untuk failover?
9. Apakah ada split-horizon behavior?
10. Apakah Java service melakukan caching DNS terlalu lama?

2. Kenapa DNS menjadi risk point dalam cloud architecture?

DNS dipakai untuk menyembunyikan perubahan infrastruktur. Load balancer, private endpoint, database failover, service discovery, dan regional failover biasanya diekspos lewat nama, bukan IP.

Itu berguna, tetapi menimbulkan risiko:

DNS adalah salah satu dependency “diam”. Saat sehat, tidak terlihat. Saat rusak, semua layer di atasnya ikut terlihat rusak.

3. Lifecycle DNS dalam enterprise system

DNS record production harus punya lifecycle, bukan dibuat manual lalu dilupakan.

1. Domain ownership defined
2. Public/private zone created
3. Zone associated with VPC/VNet or exposed via resolver forwarding
4. Records created through IaC or controlled process
5. Application endpoint configured
6. TLS certificate aligned with hostname
7. Health check/failover configured if needed
8. Logs/metrics/queries observable
9. TTL reviewed for operational behavior
10. Record retired safely when service is removed

Lifecycle yang buruk biasanya terlihat dari:

• record manual tanpa owner;
• nama service masih resolve setelah service pensiun;
• DNS menunjuk load balancer lama;
• private zone tidak terdokumentasi;
• wildcard terlalu luas;
• environment naming tidak konsisten;
• certificate CN/SAN tidak cocok dengan hostname;
• failover DNS tidak pernah dites.

4. Public DNS vs private DNS

Public DNS

Public DNS bisa di-resolve dari internet. Biasanya dipakai untuk:

• public API endpoint;
• developer portal;
• public website;
• public CDN/front door;
• public API gateway;
• public load balancer.

Contoh:

api.company.com -> public ALB / Azure Front Door / Application Gateway / API Gateway

Public DNS bukan berarti backend harus public. Public DNS bisa menunjuk edge/gateway yang kemudian meneruskan ke private backend.

Private DNS

Private DNS hanya bisa di-resolve dari network tertentu, seperti VPC, VNet, peered network, atau on-prem via forwarding.

Contoh:

orders-db.prod.internal -> 10.20.42.15
kafka-bootstrap.prod.internal -> 10.20.80.10
myvault.vault.azure.net -> 10.40.12.7 via Private Endpoint DNS

Private DNS dipakai untuk:

• database private endpoint;
• internal API;
• service-to-service traffic;
• private object storage endpoint;
• private registry;
• hybrid connectivity;
• internal admin endpoint.

Split-horizon DNS

Split-horizon berarti nama yang sama memiliki jawaban berbeda tergantung dari mana query dilakukan.

api.company.com
  from internet -> 203.0.113.10
  from corporate network -> 10.10.20.5
  from cloud VPC/VNet -> 10.20.30.8

Split-horizon berguna, tetapi berbahaya jika tidak terdokumentasi. Debugging harus selalu menyebut query source.

5. AWS DNS implementation

5.1 Route 53

Amazon Route 53 digunakan untuk DNS hosting, routing policy, health check, dan domain registration. Dalam enterprise backend, yang paling sering relevan:

• public hosted zone;
• private hosted zone;
• record set;
• alias record;
• routing policy;
• health check;
• Route 53 Resolver;
• resolver inbound/outbound endpoint;
• forwarding rule.

5.2 Private hosted zone

Private hosted zone adalah DNS zone yang hanya berlaku untuk VPC yang diasosiasikan.

Private Hosted Zone: prod.internal
Associated VPCs:
  - vpc-prod-app
  - vpc-prod-shared
Records:
  orders-db.prod.internal -> 10.20.10.15
  redis.prod.internal     -> 10.20.11.20

Failure umum:

• private hosted zone belum diasosiasikan ke VPC EKS;
• VPC peering ada, tetapi DNS resolution antar VPC tidak aktif atau zone tidak di-share;
• record ada di hosted zone berbeda;
• public hosted zone dan private hosted zone punya nama sama, tetapi jawaban tidak sesuai ekspektasi;
• record menunjuk IP lama.

5.3 Route 53 VPC Resolver

Di VPC, AWS menyediakan resolver default. Resource dalam VPC menggunakan resolver ini untuk:

• public DNS;
• VPC-specific DNS;
• Route 53 private hosted zone;
• endpoint-specific private DNS jika diaktifkan.

Dalam pod EKS, resolusi biasanya melewati CoreDNS terlebih dahulu, lalu diteruskan ke resolver VPC untuk nama di luar cluster.

Pod
  -> CoreDNS
    -> VPC Resolver
      -> Route 53 / public DNS / private hosted zone / endpoint DNS

5.4 Route 53 Resolver inbound/outbound endpoint

Untuk hybrid DNS:

• Inbound endpoint memungkinkan on-prem resolver bertanya ke resolver AWS untuk private hosted zone.
• Outbound endpoint memungkinkan resolver AWS meneruskan query domain tertentu ke DNS on-prem.

Contoh:

On-prem app resolving db.prod.aws.internal
  -> corporate DNS
  -> Route 53 Resolver inbound endpoint
  -> private hosted zone
  -> private IP

EKS pod resolving legacy-db.corp.local
  -> CoreDNS
  -> VPC Resolver
  -> Route 53 outbound resolver rule
  -> corporate DNS
  -> on-prem IP

5.5 Alias record

Alias record Route 53 dapat menunjuk ke AWS resource seperti load balancer. Ini berbeda dari CNAME karena dapat digunakan di zone apex dan memahami AWS target tertentu.

Contoh:

api.company.com -> Alias to ALB DNS name

Gunakan alias untuk AWS load balancer jika sesuai, bukan hardcode IP load balancer.

6. Azure DNS implementation

6.1 Azure DNS

Azure DNS dapat meng-host public DNS zone. Untuk private name resolution, Azure menyediakan Private DNS Zone.

Azure DNS sering muncul dalam architecture sebagai:

• public DNS zone;
• Private DNS Zone;
• Private Endpoint DNS integration;
• VNet link;
• Azure DNS Private Resolver;
• forwarding ruleset;
• hub-spoke DNS pattern.

6.2 Azure Private DNS Zone

Private DNS Zone memungkinkan nama privat di-resolve dari VNet yang di-link.

Private DNS Zone: prod.internal
VNet links:
  - vnet-prod-aks
  - vnet-prod-shared
Records:
  orders-db.prod.internal -> 10.40.10.15
  redis.prod.internal     -> 10.40.11.20

Failure umum:

• Private DNS Zone belum di-link ke VNet tempat AKS berada;
• record private endpoint tidak dibuat atau tertimpa;
• hub-spoke VNet tidak punya DNS forwarding yang benar;
• peered VNet tidak otomatis berarti DNS zone bisa di-resolve;
• custom DNS server tidak forward ke Azure resolver/private resolver.

6.3 Private Endpoint DNS

Azure Private Endpoint memberi private IP di VNet untuk mengakses Azure PaaS service. Agar aplikasi memakai private IP, DNS harus resolve hostname service ke private endpoint IP.

Contoh mental model:

myvault.vault.azure.net
  public default answer -> public Azure endpoint
  from linked VNet with Private DNS -> 10.40.12.7

Private endpoint tanpa DNS yang benar sering menghasilkan false sense of security: network sudah private endpoint, tetapi aplikasi masih resolve ke public endpoint.

6.4 Azure DNS Private Resolver

Dalam desain enterprise/hub-spoke/hybrid, Azure DNS Private Resolver dapat dipakai untuk forwarding DNS antara Azure dan on-prem tanpa mengelola VM DNS forwarder sendiri.

Pattern umum:

On-prem DNS
  -> Azure DNS Private Resolver inbound endpoint
  -> Private DNS Zone

AKS workload
  -> CoreDNS
  -> VNet DNS
  -> Azure DNS Private Resolver outbound ruleset
  -> on-prem DNS

7. Kubernetes CoreDNS interaction

Kubernetes menyediakan DNS untuk Services dan Pods. Di cluster modern, CoreDNS biasanya menjadi cluster DNS.

Service Kubernetes mendapat nama:

<service>.<namespace>.svc.<cluster-domain>

Contoh:

quote-service.ordering.svc.cluster.local
postgres-proxy.data.svc.cluster.local

Pod biasanya memiliki /etc/resolv.conf yang menunjuk ke cluster DNS service.

Contoh alur:

Java pod resolves quote-service.ordering.svc.cluster.local
  -> CoreDNS answers ClusterIP

Untuk domain eksternal atau private cloud:

Java pod resolves myvault.vault.azure.net
  -> CoreDNS forwards upstream
  -> Azure VNet resolver / custom DNS
  -> Private DNS Zone
  -> private endpoint IP

Java pod resolves secretsmanager.ap-southeast-1.amazonaws.com
  -> CoreDNS forwards upstream
  -> VPC Resolver
  -> AWS DNS / VPC endpoint private DNS if enabled
  -> private IP or public AWS endpoint depending setup

CoreDNS failure modes

8. DNS and Java/JAX-RS runtime behavior

Java service biasanya tidak memanggil DNS secara eksplisit. DNS terjadi saat membuka connection:

• HTTP client call;
• JDBC connection;
• Kafka client bootstrap;
• RabbitMQ connection;
• Redis connection;
• AWS SDK client;
• Azure SDK client;
• OpenTelemetry exporter;
• config/secret client.

DNS cache di JVM

JVM dapat melakukan DNS caching. Ini penting untuk failover.

Jika IP dependency berubah tetapi JVM cache terlalu lama, aplikasi tetap mencoba IP lama. Jika TTL terlalu pendek dan traffic tinggi, DNS query rate bisa meningkat.

Hal yang harus dicek:

• networkaddress.cache.ttl
• networkaddress.cache.negative.ttl
• container base image DNS behavior
• client library connection pooling
• apakah client reconnect saat IP berubah

Endpoint config di application properties

Anti-pattern:

object.storage.endpoint=http://10.20.1.10:9000

Lebih baik:

object.storage.endpoint=https://object-storage.prod.internal

Tetapi DNS name harus punya owner, TTL, private/public behavior, certificate alignment, dan runbook.

9. Dampak ke PostgreSQL, Kafka, RabbitMQ, Redis, Camunda, dan NGINX

PostgreSQL

Database hostname bisa menunjuk:

• RDS endpoint;
• Aurora cluster endpoint;
• Azure PostgreSQL private endpoint;
• PgBouncer/proxy service;
• internal DNS CNAME.

Failure umum:

• failover database mengubah target, tetapi app cache IP lama;
• private DNS tidak link ke VPC/VNet;
• hostname resolve public endpoint;
• certificate CN/SAN tidak cocok jika TLS verification aktif.

Kafka

Kafka sangat sensitif terhadap DNS karena broker dapat meng-advertise hostname sendiri.

Checklist:

• bootstrap server resolve dari pod;
• advertised listeners benar;
• internal vs external listener tidak tertukar;
• broker hostname resolve ke private IP;
• DNS TTL dan client metadata refresh dipahami.

RabbitMQ

RabbitMQ cluster dan client connection bergantung pada hostname node/broker.

Checklist:

• broker endpoint private;
• TLS hostname validation;
• load balancer DNS jika digunakan;
• failover behavior;
• connection recovery.

Redis

Redis managed endpoint bisa berubah saat failover.

Checklist:

• primary endpoint vs node endpoint;
• cluster mode discovery;
• DNS cache JVM/client;
• TLS hostname validation;
• private endpoint resolution.

Camunda

Camunda service bisa bergantung pada database, REST endpoint, worker endpoint, dan message broker. DNS failure dapat terlihat sebagai job backlog, external task failure, atau process instance stuck.

NGINX

NGINX juga melakukan DNS resolution. Jika upstream memakai hostname, perhatikan resolver config, TTL, reload behavior, dan DNS failure behavior.

10. DNS in EKS, AKS, on-prem, and hybrid

EKS

EKS workload biasanya:

Pod -> CoreDNS -> VPC Resolver -> Route 53 / private hosted zone / public DNS

Hal penting:

• CoreDNS health;
• VPC DNS attributes;
• private hosted zone association;
• VPC endpoint private DNS enabled;
• security group/NACL tidak memblokir DNS;
• resolver rule untuk on-prem domain.

AKS

AKS workload biasanya:

Pod -> CoreDNS -> VNet DNS/custom DNS -> Azure DNS / Private DNS Zone / Private Resolver

Hal penting:

• Private DNS Zone link ke VNet AKS;
• custom DNS server forwarding ke Azure DNS/Private Resolver;
• private endpoint DNS integration;
• CoreDNS forwarding;
• UDR/firewall tidak memutus DNS path.

On-prem/hybrid

Hybrid DNS membutuhkan rule eksplisit.

On-prem resolving cloud private name
  -> corporate DNS
  -> AWS inbound resolver / Azure private resolver inbound
  -> private zone

Cloud workload resolving on-prem name
  -> cloud resolver
  -> outbound forwarding rule
  -> corporate DNS

Tanpa forwarding, network route mungkin benar tetapi hostname tetap gagal.

11. Mermaid: DNS resolution path from pod to private cloud service

12. Failure modes

13. Production-safe DNS debugging steps

Step 1 — Identify the exact hostname

Do not debug “database DNS”. Debug exact hostname:

orders-db.prod.internal
myvault.vault.azure.net
quote-service.ordering.svc.cluster.local

Step 2 — Query from the same source as the failing workload

A laptop result is not enough. Query from pod.

kubectl exec -it deploy/quote-service -- nslookup orders-db.prod.internal
kubectl exec -it deploy/quote-service -- getent hosts orders-db.prod.internal

Use a debug pod if production policy allows:

kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 -- sleep 3600
kubectl exec -it dnsutils -- nslookup orders-db.prod.internal

Step 3 — Compare pod, node, and resolver answers

kubectl exec -it dnsutils -- cat /etc/resolv.conf
kubectl exec -it dnsutils -- nslookup kubernetes.default.svc.cluster.local
kubectl exec -it dnsutils -- nslookup example.com

If cluster service DNS works but external/private DNS fails, CoreDNS exists but upstream forwarding may be wrong.

Step 4 — Confirm private vs public IP

Ask:

Is the returned IP inside expected VPC/VNet/private CIDR?
Is it a public IP?
Is it the private endpoint IP?
Is it the load balancer IP/DNS expected?

Step 5 — Check zone association/link

AWS:

• private hosted zone associated with the VPC?
• VPC DNS support and DNS hostnames enabled?
• VPC endpoint private DNS enabled?
• resolver rules correct?

Azure:

• Private DNS Zone linked to VNet?
• private endpoint DNS zone group configured?
• custom DNS forwarding to Azure Private Resolver?
• hub-spoke resolver path working?

Step 6 — Check TTL and client caching

Look for mismatch between DNS TTL and runtime behavior:

• JVM DNS cache;
• HTTP client connection pool;
• JDBC pool;
• Kafka metadata refresh;
• Redis client topology refresh;
• NGINX resolver behavior.

Step 7 — Only after DNS is correct, debug network

If hostname resolves to expected IP but connection fails, switch to:

• route table;
• security group/NSG;
• firewall;
• NetworkPolicy;
• private endpoint approval;
• TLS;
• service health.

14. Correctness concerns

DNS correctness means more than “hostname resolves”. It means:

• resolves from the correct source network;
• resolves to expected private/public target;
• target matches environment;
• TLS certificate matches hostname;
• TTL matches failover requirement;
• record lifecycle is owned;
• no stale records;
• no accidental public path;
• no hidden dependency on manual DNS changes;
• rollback behavior understood.

Anti-pattern:

“We tested nslookup from laptop, so DNS is fine.”

Better:

“We tested resolution from the failing pod namespace, node, VPC/VNet, and on-prem source. The hostname resolves to the expected private endpoint IP with TTL 60s. TLS SAN matches the hostname. CoreDNS and upstream resolver metrics are healthy.”

15. Security concerns

DNS can create security bypass.

Rule: do not fix DNS/certificate problems by disabling TLS validation in Java clients.

16. Performance and cost concerns

DNS affects performance indirectly:

• slow resolver increases connection setup latency;
• low TTL increases query volume;
• no local caching increases pressure on CoreDNS/resolver;
• frequent reconnects amplify DNS traffic;
• broken DNS causes retry storm;
• cross-region DNS answer can increase latency and data transfer cost;
• public endpoint answer can route traffic through internet/NAT instead of private path.

For Java services, watch:

• connection pool churn;
• DNS cache TTL;
• HTTP client connection reuse;
• SDK retry behavior;
• CoreDNS CPU and memory;
• resolver query latency;
• NXDOMAIN rate.

17. Observability concerns

DNS observability should include:

• CoreDNS metrics;
• CoreDNS logs if safe and sampled;
• Route 53 Resolver query logs if enabled;
• Azure DNS/Private Resolver diagnostics where available;
• application error logs with hostname;
• dependency metrics grouped by hostname;
• network flow logs;
• load balancer access logs;
• cloud audit logs for DNS record changes.

Application logs should not only say:

Connection timed out

Better:

Dependency call failed: host=orders-db.prod.internal, resolvedIp=10.20.10.15, error=connect_timeout, durationMs=3000, correlationId=...

Be careful: logging resolved IP can be useful for debugging, but check internal security/logging policy.

18. PR review checklist

Use this when reviewing changes that add or modify hostnames.

DNS ownership

• Who owns the domain/zone?
• Is the record managed by IaC?
• Is there a clear owner/team/contact?
• Is the record environment-specific?
• Is deletion/retirement planned?

Public/private behavior

• Is the zone public or private?
• From pod, does it resolve to expected private/public IP?
• From on-prem, does it resolve correctly?
• From peered VPC/VNet, does it resolve correctly?
• Is split-horizon behavior documented?

Cloud-specific checks

• AWS private hosted zone associated with correct VPCs?
• AWS resolver rules/inbound/outbound endpoints correct?
• Azure Private DNS Zone linked to correct VNets?
• Azure Private Endpoint DNS integration correct?
• Custom DNS forwarding documented?

Kubernetes checks

• CoreDNS forwards expected domains correctly?
• NetworkPolicy allows DNS egress?
• Namespace/service names correct?
• No reliance on short names that can resolve ambiguously?

Java/runtime checks

• JVM DNS TTL understood?
• Client connection pool refresh understood?
• SDK endpoint override not hardcoded incorrectly?
• TLS hostname verification remains enabled?

Operations

• TTL appropriate for failover/rollback?
• Dashboard/logging exists?
• Runbook includes DNS validation from pod?
• Rollback plan exists?
• Security/compliance review complete if public exposure changes?

19. Internal verification checklist

Cek hal berikut di internal CSG/team. Jangan mengarang asumsi.

AWS

• Route 53 public hosted zones yang digunakan.
• Route 53 private hosted zones yang digunakan.
• VPC association per private hosted zone.
• Resolver inbound endpoints.
• Resolver outbound endpoints.
• Resolver forwarding rules.
• VPC DNS attributes.
• VPC endpoint private DNS setting.
• Route 53 Resolver query logs jika tersedia.
• DNS record ownership dan IaC source.

Azure

• Azure DNS public zones yang digunakan.
• Azure Private DNS Zones yang digunakan.
• VNet links per Private DNS Zone.
• Private Endpoint DNS zone group.
• Azure DNS Private Resolver usage.
• Hub-spoke DNS forwarding pattern.
• Custom DNS servers per VNet.
• Diagnostic logs untuk DNS jika tersedia.
• DNS record ownership dan IaC source.

Kubernetes

• CoreDNS ConfigMap.
• CoreDNS replicas/resources.
• CoreDNS metrics dashboard.
• DNS-related NetworkPolicy.
• Pod /etc/resolv.conf behavior.
• Cluster domain.
• Known DNS incident notes.

Application/backend

• Hostname config per service.
• AWS SDK endpoint/region config.
• Azure SDK endpoint config.
• JVM DNS cache setting.
• HTTP/JDBC/Kafka/RabbitMQ/Redis client reconnect behavior.
• TLS hostname verification policy.
• Dependency map by hostname.

Operations

• DNS change approval process.
• DNS rollback process.
• TTL standard.
• Incident runbook.
• Monitoring dashboard.
• Contact path: platform, network, SRE, security, backend.

20. Senior engineer heuristics

1. DNS debugging must be done from the failing network location, not from your laptop.
2. A hostname resolving successfully does not prove the answer is correct.
3. Private endpoint without private DNS is incomplete.
4. VPC/VNet peering does not automatically solve DNS.
5. DNS failover does not solve state consistency.
6. Java DNS cache and connection pooling can delay failover.
7. TLS verification failure is a signal, not an inconvenience.
8. Every production DNS record needs an owner and lifecycle.
9. DNS changes are infrastructure changes and should be reviewable.
10. In incident triage, always separate: resolution failure, wrong answer, connection failure, TLS failure, application protocol failure.

21. Mini case study: pod cannot access Key Vault/Secrets Manager

Symptom:

Java service cannot retrieve secret during startup.

Bad diagnosis:

Cloud SDK is broken.

Better diagnostic path:

1. Does pod resolve the secret service hostname?
2. Does it resolve to private endpoint IP or public endpoint?
3. Is the private hosted zone / Private DNS Zone linked?
4. Can pod connect to resolved IP on 443?
5. Does TLS handshake succeed?
6. Does identity token work?
7. Does IAM/RBAC allow read secret?
8. Does SDK retry hide the first error?

This separates DNS, network, TLS, identity, permission, and SDK behavior.

22. References to verify during implementation

Use official provider docs when writing or reviewing actual implementation details:

• AWS Route 53 private hosted zones
• AWS Route 53 VPC Resolver and resolver endpoints
• AWS VPC DNS attributes
• Azure DNS and Azure Private DNS Zone
• Azure Private Endpoint DNS integration
• Azure DNS Private Resolver
• Kubernetes DNS for Services and Pods
• Kubernetes DNS debugging and CoreDNS configuration

Do not assume CSG internal DNS architecture. Confirm it with platform/SRE/network/security team.