Part 049 — Availability and Resilience

Target pembaca: Senior Java/JAX-RS backend engineer yang harus menilai apakah service production tetap aman saat dependency cloud, Kubernetes, database, broker, Redis, DNS, identity, atau network mengalami gangguan.

Availability bukan sekadar “service up”.

Resilience bukan sekadar “punya retry”.

Dalam enterprise backend system, availability adalah kemampuan sistem menjalankan fungsi bisnis yang disepakati saat dibutuhkan. Resilience adalah kemampuan sistem menyerap failure, membatasi blast radius, pulih secara terkendali, dan tetap memberikan outcome yang aman.

Untuk Quote & Order style system, availability tidak boleh hanya dibaca sebagai HTTP 200.

Contoh:

• API GET /quotes/{id} masih bisa melayani read request, tetapi POST /orders gagal karena downstream order orchestration mati;
• service hidup, tetapi Kafka consumer lag membuat status order tertunda 45 menit;
• pod healthy, tetapi tidak bisa membaca secret baru karena Key Vault/Secrets Manager permission berubah;
• ingress sehat, tetapi DNS private endpoint salah resolve ke public endpoint;
• database HA aktif, tetapi connection pool Java tidak reconnect dengan benar setelah failover;
• Redis failover sukses, tetapi aplikasi menulis ulang cache stale yang merusak user journey;
• retry policy terlalu agresif sehingga outage kecil menjadi retry storm.

Part ini membangun mental model availability dan resilience yang bisa dipakai untuk review PR, ADR, incident triage, dan production readiness.

1. Konsep inti

Availability menjawab:

Apakah fungsi bisnis yang dijanjikan masih bisa digunakan saat dibutuhkan?

Resilience menjawab:

Saat failure terjadi, apakah sistem tetap bounded, observable, recoverable, dan tidak memperburuk keadaan?

Availability adalah target.

Resilience adalah kemampuan desain dan operasi untuk mencapai target itu.

availability target
  -> SLO/SLA/error budget
  -> architecture redundancy
  -> dependency isolation
  -> timeout/retry/bulkhead
  -> failover/fallback
  -> observability
  -> runbook
  -> test evidence

Prinsip dasarnya:

1. Failure pasti terjadi.
2. Tidak semua failure perlu dipulihkan dengan cara yang sama.
3. Tidak semua dependency memiliki criticality yang sama.
4. Redundancy tanpa test bukan resilience.
5. Retry tanpa budget bisa menjadi amplifier.
6. Failover tanpa data consistency model bisa menciptakan corruption.
7. Availability harus didefinisikan per capability bisnis, bukan per pod.

2. Availability vs reliability vs resilience

Contoh pada quote/order flow:

3. Kenapa konsep ini ada

Cloud memberikan building block availability: region, availability zone, managed HA, load balancer, autoscaling, health check, managed database, replicated storage, global DNS, backup, and monitoring.

Tetapi cloud tidak otomatis membuat aplikasi resilient.

Aplikasi masih bisa gagal karena:

• hanya deploy di satu AZ;
• pod replica ada banyak, tetapi semua bergantung pada satu database primary;
• health check hanya mengecek /health lokal, bukan readiness dependency yang penting;
• retry SDK default tidak sesuai latency budget API;
• timeout antar layer tidak selaras;
• queue consumer tidak punya backpressure;
• database failover tidak diuji dengan connection pool Java;
• secret rotation memutus semua pod sekaligus;
• NAT gateway menjadi single egress bottleneck;
• private endpoint DNS salah;
• autoscaling terlambat;
• log ingestion terlalu mahal sehingga retention dipotong dan incident sulit dianalisis.

Availability dan resilience ada untuk mengubah failure dari kejadian acak menjadi kejadian yang sudah dimodelkan.

4. Mental model: failure domain

Failure domain adalah boundary tempat failure bisa terjadi secara bersama.

single pod
  < node
  < node pool / node group
  < availability zone
  < region
  < cloud provider service
  < identity provider
  < network hub
  < shared platform component
  < external enterprise dependency

Jika semua replica berada di failure domain yang sama, replica itu tidak memberi availability yang berarti.

Contoh buruk:

3 replicas
all pods on same node
same AZ
same subnet
same node group
same secret mount
same database endpoint
same NAT gateway path

Secara YAML terlihat redundant. Secara failure domain tidak.

Contoh lebih baik:

3+ replicas
spread across nodes
spread across AZs
PodDisruptionBudget configured
readiness probe correct
DB Multi-AZ enabled
broker replicated
Redis failover configured
external dependency timeout bounded
fallback behavior explicit

5. Availability Zone

Availability Zone adalah lokasi terisolasi dalam region yang dirancang untuk mengurangi correlated failure.

Dalam desain production, AZ dipakai untuk:

• menyebar node Kubernetes;
• menyebar load balancer target;
• menyebar managed database standby;
• menyebar broker nodes;
• menyebar Redis replica;
• menyebar NAT gateway jika egress per-AZ dipakai;
• mengurangi dampak power/network failure lokal.

Namun AZ bukan solusi untuk:

• salah konfigurasi IAM/RBAC;
• bug aplikasi;
• schema migration buruk;
• secret rotation salah;
• DNS record salah;
• cloud regional outage;
• data corruption yang direplikasi;
• cost explosion;
• retry storm.

6. Multi-AZ

Multi-AZ berarti workload dirancang berjalan melewati lebih dari satu Availability Zone.

Untuk Java/JAX-RS service di Kubernetes, Multi-AZ biasanya berarti:

EKS/AKS node pool across AZs
  + pod anti-affinity / topology spread
  + PDB
  + load balancer targets in multiple AZs
  + database HA
  + broker replication
  + Redis failover
  + per-AZ capacity awareness

Multi-AZ harus dicek per layer.

Anti-pattern:

Load balancer Multi-AZ
but all pods scheduled in one AZ

Atau:

Pods spread across AZs
but database single-AZ

Atau:

Database Multi-AZ
but application connection pool cannot recover after DB failover

7. Region failure

Region failure lebih jarang tetapi lebih besar dampaknya.

Multi-region bukan default jawaban untuk semua workload karena:

• kompleksitas data replication tinggi;
• consistency trade-off sulit;
• cost besar;
• operational burden tinggi;
• DNS failover butuh test;
• identity/network policy harus sinkron;
• runbook lebih sulit;
• split-brain risk nyata.

Pertanyaan sebelum multi-region:

1. Apakah Multi-AZ cukup untuk SLO bisnis?
2. Berapa RTO/RPO yang benar-benar dibutuhkan?
3. Capability mana yang harus tetap aktif saat region gagal?
4. Apakah semua dependency juga multi-region?
5. Apakah data model mendukung replication/failover?
6. Apakah team punya runbook dan test evidence?
7. Apakah cost-nya diterima bisnis?

Untuk Quote & Order, tidak semua capability harus sama criticality-nya.

8. SLO, SLA, dan error budget

SLA biasanya komitmen formal kepada customer.

SLO adalah target operasional yang dipakai engineering.

Error budget adalah ruang failure yang masih diterima dalam periode tertentu.

SLA: contractual promise
SLO: engineering target
SLI: measured signal
error budget: allowed unreliability

Contoh:

Capability: Submit Order API
SLI: percentage of valid submit requests completed successfully under 2s
SLO: 99.9% over rolling 30 days
Error budget: 0.1% bad events over 30 days

SLO yang buruk:

All services must be 100% available.

Kenapa buruk:

• tidak realistis;
• tidak memberi trade-off;
• tidak membedakan criticality;
• tidak membantu prioritization;
• mendorong overengineering.

SLO yang lebih berguna:

Quote read API: 99.95% monthly availability, p95 < 300ms
Order submission API: 99.9% monthly availability, p95 < 800ms, duplicate rate = 0
Async order event processing: 99% processed under 5 minutes
Export generation: 99% completed under 30 minutes

9. Correctness lebih penting daripada availability semu

Untuk workflow quote/order, “available tapi salah” sering lebih buruk daripada “temporarily unavailable”.

Contoh:

• order dibuat dua kali;
• quote price stale dipakai untuk checkout;
• discount eligibility salah;
• order status maju tanpa downstream confirmation;
• event replay membuat side effect ganda;
• cache stale menampilkan state yang melanggar compliance;
• failover membaca replica yang belum sync.

Availability harus tunduk pada correctness boundary.

If degraded mode cannot preserve correctness, fail closed.
If fallback may expose wrong commercial state, do not fallback silently.
If retry can duplicate side effects, require idempotency.

10. AWS view

Di AWS, resilience biasanya dibangun dengan kombinasi:

• multiple Availability Zones;
• Elastic Load Balancing;
• Auto Scaling;
• EKS managed node groups/Karpenter;
• RDS Multi-AZ;
• Aurora replication jika digunakan;
• MSK broker replication;
• ElastiCache replication/failover;
• S3 durability dan cross-region replication jika diperlukan;
• Route 53 health checks/failover jika digunakan;
• VPC endpoints untuk private dependency;
• CloudWatch metrics/alarms;
• AWS Health dan service quota awareness;
• IAM/STS resilience consideration;
• Well-Architected Reliability review.

AWS-specific concerns untuk backend engineer:

11. Azure view

Di Azure, resilience biasanya dibangun dengan kombinasi:

• Availability Zones;
• zone-redundant services jika tersedia;
• Azure Load Balancer/Application Gateway;
• AKS node pools across zones;
• Azure Database for PostgreSQL Flexible Server HA;
• Azure Cache for Redis replication/failover capabilities;
• Azure Event Hubs/Kafka-compatible endpoint jika digunakan;
• Azure Storage redundancy options;
• Azure Private Endpoint dan Private DNS;
• Azure Monitor dan Log Analytics;
• Azure Service Health;
• Azure Policy untuk governance;
• Azure Well-Architected Reliability review.

Azure-specific concerns untuk backend engineer:

12. EKS/AKS impact

Kubernetes memberi primitives untuk resilience, tetapi tidak otomatis benar.

Yang perlu diperiksa:

Readiness probe untuk production sebaiknya merepresentasikan kesiapan menerima traffic, bukan hanya JVM hidup.

Namun readiness juga tidak boleh terlalu agresif sampai dependency minor membuat semua pod keluar dari load balancer.

bad readiness:
  checks every downstream dependency including optional analytics service

better readiness:
  checks local app readiness + critical dependency required for safe request handling
  optional dependency handled with degraded mode

13. Java/JAX-RS impact

Di Java backend, availability sering runtuh bukan karena cloud mati, tetapi karena resource runtime habis.

Critical runtime resources:

• HTTP server thread pool;
• worker executor;
• JDBC connection pool;
• Kafka/RabbitMQ producer connection;
• Redis client connection;
• cloud SDK HTTP connection pool;
• DNS resolver behavior;
• TLS handshake overhead;
• heap dan GC;
• CPU throttling;
• log appender backpressure;
• metrics/tracing exporter.

Contoh failure amplification:

S3/Blob latency rises
  -> cloud SDK calls wait too long
  -> HTTP worker threads blocked
  -> inbound API latency rises
  -> clients retry
  -> more concurrent requests
  -> DB pool exhausted
  -> unrelated endpoints fail

Resilience pattern:

bounded timeout
  + bounded retry
  + circuit breaker
  + bulkhead executor
  + fallback only if correct
  + idempotency for side effects
  + metrics per dependency

14. Timeout chain

Setiap layer punya timeout.

client timeout
  > API gateway timeout
  > load balancer idle timeout
  > ingress/proxy timeout
  > app request timeout
  > downstream HTTP timeout
  > DB query timeout
  > SDK attempt timeout

Timeout chain yang buruk:

client waits 30s
API gateway waits 29s
Java app waits 60s for downstream
DB query waits unlimited

Akibatnya:

• client sudah disconnect;
• app masih bekerja;
• thread tetap tertahan;
• retry dari client datang;
• load bertambah;
• circuit breaker terlambat.

Timeout chain yang lebih sehat:

client timeout: 5s
API gateway timeout: 6s
ingress timeout: 5.5s
app request budget: 4.5s
downstream call budget: 1s-2s
DB query timeout: bounded
SDK attempt timeout: bounded

Rule:

Every dependency call must have a timeout.
Every timeout must fit inside caller's budget.
Every retry must fit inside total timeout.

15. Retry storm

Retry adalah obat yang bisa menjadi racun.

Retry membantu saat failure bersifat transient.

Retry merusak saat dependency sedang overloaded atau side effect tidak idempotent.

Retry storm pattern:

small dependency latency spike
  -> clients retry
  -> app retries
  -> SDK retries
  -> queue redelivery retries
  -> dependency receives 5x-20x traffic
  -> outage becomes larger

Guardrail:

• exponential backoff;
• jitter;
• max attempts rendah;
• retry hanya untuk transient error;
• no retry untuk validation/business error;
• retry budget;
• circuit breaker;
• client-side rate limiting;
• idempotency key;
• observability per attempt, bukan hanya final failure.

Untuk order submission:

POST /orders must use idempotency key if client or server retry can happen.

16. Circuit breaker

Circuit breaker mencegah aplikasi terus memanggil dependency yang sedang gagal.

State umum:

CLOSED -> calls allowed
OPEN -> calls fail fast
HALF_OPEN -> limited trial calls

Gunanya:

• mengurangi pressure ke dependency;
• melindungi thread pool;
• memberikan failure cepat;
• memungkinkan fallback/degraded response;
• mempercepat recovery dependency.

Bahaya:

• threshold terlalu sensitif menyebabkan false open;
• fallback salah menyebabkan data salah;
• circuit breaker global membuat satu tenant memengaruhi tenant lain;
• tidak ada metric per state;
• half-open terlalu agresif.

Review question:

What happens to this endpoint when dependency X is down for 5 minutes?

Jawaban yang buruk:

The client will retry.

Jawaban yang baik:

The call times out after 800ms, retries once with jitter if error is transient, then circuit opens after threshold. Non-critical feature is degraded. Critical write path fails closed with idempotent error response. Metrics and alert identify dependency X saturation.

17. Bulkhead

Bulkhead memisahkan resource agar failure satu dependency tidak menghabiskan seluruh kapasitas service.

Contoh:

separate thread pool for object storage upload
separate connection pool for reporting DB
separate queue consumer concurrency per event type
separate rate limit per tenant/customer
separate circuit breaker per downstream

Tanpa bulkhead:

slow object storage call consumes all HTTP worker threads
  -> quote read API fails
  -> order submit API fails
  -> health check fails

Dengan bulkhead:

object storage path degraded
core quote/order API still available

Bulkhead cocok untuk:

• file upload/download;
• report/export;
• optional integration;
• slow third-party API;
• cloud SDK call yang tidak boleh memblokir core flow;
• per-tenant noisy neighbor control.

18. Graceful degradation

Graceful degradation adalah kemampuan menurunkan fungsi tanpa menjatuhkan sistem penuh.

Contoh aman:

• disable export sementara;
• tampilkan cached read-only catalog dengan label freshness jika bisnis mengizinkan;
• queue async work untuk diproses nanti;
• hide optional recommendation;
• return partial metadata tanpa file preview;
• pause non-critical consumers;
• rate limit heavy tenants.

Contoh berbahaya:

• gunakan stale price untuk order final;
• submit order tanpa inventory/reservation confirmation jika wajib;
• fallback ke allow-all authorization;
• fallback ke default secret;
• continue dengan config kosong;
• silently drop events.

Rule:

Degrade only where correctness, security, and compliance remain valid.

19. Dependency criticality classification

Tidak semua dependency sama.

Contoh untuk Java/JAX-RS Quote & Order:

20. Blast radius

Blast radius adalah seberapa luas dampak failure.

Pertanyaan:

If this component fails, what else fails with it?

Blast radius bisa muncul dari:

• shared database;
• shared Redis;
• shared Kafka cluster;
• shared ingress;
• shared NAT gateway;
• shared IAM role;
• shared Key Vault/Secrets Manager path;
• shared config namespace;
• shared thread pool;
• shared retry policy;
• shared tenant partition;
• shared CI/CD pipeline.

Cara mengecilkan blast radius:

• separate critical and non-critical workloads;
• isolate tenant/customer jika perlu;
• separate node pool untuk workload berbeda;
• separate queue/topic untuk event critical;
• separate circuit breaker per dependency;
• separate IAM role per workload;
• separate secret path per service;
• rate limit per caller;
• quota per tenant;
• environment isolation;
• canary/blue-green deployment.

21. Autoscaling is not instant resilience

Autoscaling membantu, tetapi bukan obat semua overload.

HPA/Cluster Autoscaler/Karpenter/AKS node autoscaling punya delay:

metric collection
  -> scaling decision
  -> pod scheduling
  -> node provisioning if needed
  -> image pull
  -> JVM startup
  -> readiness
  -> load balancer target registration

Jika traffic spike lebih cepat dari proses ini, service tetap bisa overload.

Guardrail:

• maintain headroom;
• pre-scale untuk known campaign/batch;
• tune JVM startup;
• keep image small;
• avoid cold path heavy initialization;
• ensure quota/IP capacity;
• use queue buffering untuk async workload;
• rate limit traffic;
• use load shedding.

22. Load shedding

Load shedding adalah menolak sebagian request secara sadar agar sistem tetap hidup.

Contoh:

• return 429 untuk traffic non-critical;
• reject heavy export saat DB high load;
• pause optional consumers;
• disable expensive enrichment;
• reject requests tanpa idempotency key pada write endpoint;
• apply per-tenant limit;
• shed traffic sebelum DB pool habis.

Load shedding lebih baik daripada meltdown.

controlled rejection > uncontrolled timeout cascade

23. Health check design

Health check salah bisa membuat outage lebih besar.

Jenis health check:

Anti-pattern:

liveness checks database

Akibat:

database slow
  -> liveness fails
  -> Kubernetes restarts all pods
  -> connection storm to database
  -> outage amplified

Lebih baik:

liveness: JVM/process local health
readiness: ability to safely handle traffic
synthetic: critical end-to-end capability

24. PostgreSQL resilience concern

Managed PostgreSQL HA tidak cukup jika aplikasi tidak siap.

Periksa:

• JDBC connection pool timeout;
• max lifetime lebih pendek dari infrastructure connection lifetime;
• validation query;
• DNS caching JVM;
• retry transaction boundary;
• idempotency untuk write;
• migration safety;
• read replica lag;
• failover test evidence;
• connection storm saat DB pulih;
• pool size vs DB max connections.

Failure mode:

DB failover completes in provider
but Java service holds stale connections
  -> requests fail until pool refresh/restart

Pattern:

bounded pool
connection validation
short connection acquisition timeout
transaction retry only when safe
idempotency for side-effecting operation
failover drill

25. Kafka/RabbitMQ resilience concern

Messaging resilience bukan hanya broker up.

Periksa:

• producer retry and idempotence;
• consumer retry strategy;
• dead-letter queue;
• poison message handling;
• partition/queue design;
• consumer lag alert;
• offset commit timing;
• ordering requirement;
• duplicate handling;
• replay safety;
• broker maintenance plan;
• network/TLS/auth stability.

Anti-pattern:

consumer retries poison message forever
  -> partition blocked
  -> business flow stuck

Pattern:

bounded retry
DLQ
idempotent consumer
observable lag
replay runbook
business compensation if needed

26. Redis resilience concern

Redis can improve availability or destroy it, depending on usage.

Redis cache failure should often degrade, not bring the whole system down.

But Redis used as distributed lock/session/rate limit can become critical.

Periksa:

• usage type: cache, lock, session, rate limit, dedup;
• fail-open vs fail-closed decision;
• TTL policy;
• eviction policy;
• hot key;
• reconnect behavior;
• cluster topology refresh;
• failover handling;
• stale data risk;
• cache stampede protection.

Example:

Redis cache down
quote read can fallback to DB with rate limit

But:

Redis lock down
order submission cannot safely proceed if lock protects duplicate order creation

27. Secret/config resilience concern

Secret/config services are often overlooked as availability dependencies.

Bad pattern:

every request fetches secret/config from Key Vault/Secrets Manager/AppConfig

Risks:

• latency spike;
• throttling;
• dependency outage affects all requests;
• cost increase;
• startup storm;
• token failure.

Better pattern:

fetch at startup or controlled reload
cache with TTL
observe refresh failures
fail closed for missing critical secret
safe defaults for non-critical config
rollout config gradually

Secret rotation resilience:

• support overlapping old/new secret during rotation;
• refresh without full fleet restart if possible;
• validate new secret before rollout;
• alert on authentication failure spike;
• have rollback plan.

28. DNS resilience concern

DNS failure appears as random network failure.

Periksa:

• private DNS zone association;
• Route 53 private hosted zone association;
• CoreDNS health;
• DNS TTL;
• split-horizon behavior;
• DNS resolver forwarding;
• private endpoint records;
• JVM DNS cache TTL;
• failover record behavior;
• negative caching.

Failure mode:

private endpoint is created
but private DNS zone not linked to VNet/VPC resolver path
  -> pod resolves public IP
  -> firewall blocks
  -> SDK timeout

29. Identity resilience concern

Identity failure often looks like application failure.

Examples:

• STS throttling;
• workload identity token projection failure;
• managed identity endpoint unreachable;
• role assignment removed;
• IAM trust policy changed;
• service account annotation wrong;
• token audience mismatch;
• credential refresh fails;
• clock skew.

Pattern:

credential acquisition should be observable
credential refresh should not happen per request unnecessarily
AccessDenied should be distinguishable from network timeout
identity changes should be deployed with rollback path

30. Observability for resilience

Minimum signals:

Dashboards should be capability-oriented, not only pod-oriented.

Submit Order capability dashboard
  -> ingress rate/error/latency
  -> app endpoint latency
  -> DB pool
  -> Kafka publish success
  -> order event lag
  -> dependency circuit state
  -> cloud provider errors

31. Testing resilience

Resilience must be tested.

Test examples:

• kill one pod;
• drain one node;
• remove one AZ worth of nodes in staging;
• force DB failover;
• block Redis temporarily;
• introduce dependency latency;
• expire/rotate secret;
• simulate AccessDenied;
• break private DNS record in non-prod;
• throttle object storage call;
• pause Kafka consumer;
• fill queue with poison message;
• simulate load spike;
• run restore/failover drill.

Evidence should include:

• expected behavior;
• actual behavior;
• metrics/logs/traces;
• time to detect;
• time to mitigate;
• data correctness result;
• follow-up actions.

32. Failure-mode matrix

33. PR review checklist

Saat mereview PR/ADR yang menyentuh availability/resilience, tanyakan:

Capability

• Capability bisnis apa yang dipengaruhi?
• Apakah ini read, write, async, file, admin, atau operational flow?
• Apa SLO/SLA capability ini?
• Apa correctness boundary-nya?

Dependency

• Dependency baru apa yang ditambahkan?
• Apakah synchronous atau asynchronous?
• Apakah dependency critical atau degradable?
• Apa failure mode dependency tersebut?
• Apa fallback/degradation behavior?

Timeout and retry

• Apakah semua call punya timeout?
• Apakah retry bounded?
• Apakah retry memakai backoff dan jitter?
• Apakah idempotency tersedia untuk write?
• Apakah retry budget masuk ke request budget?

Isolation

• Apakah thread pool/connection pool terisolasi?
• Apakah satu tenant bisa menghabiskan kapasitas semua tenant?
• Apakah optional feature bisa menjatuhkan core flow?
• Apakah deployment spread across nodes/AZs?

Failover

• Apa yang terjadi saat DB failover?
• Apa yang terjadi saat broker failover?
• Apa yang terjadi saat Redis failover?
• Apa yang terjadi saat private endpoint/DNS bermasalah?
• Apakah failover pernah diuji?

Observability

• Metric apa yang menunjukkan degradation?
• Log error code apa yang dibutuhkan?
• Trace span apa yang perlu ada?
• Alert apa yang memicu incident response?
• Dashboard apa yang dipakai saat triage?

Recovery

• Apakah rollback aman?
• Apakah data bisa dipulihkan?
• Apakah replay event aman?
• Apakah manual remediation dibutuhkan?
• Siapa owner runbook?

34. Internal verification checklist

Gunakan checklist ini untuk CSG/team verification. Jangan mengasumsikan detail internal tanpa konfirmasi.

Availability target

• SLO/SLA per capability tersedia.
• Error budget didefinisikan.
• Criticality per API/event/job diketahui.
• Customer impact model tersedia.

AWS/Azure platform

• Region dan AZ yang digunakan jelas.
• Multi-AZ strategy terdokumentasi.
• Zone redundancy support diverifikasi per service.
• Service quota dimonitor.
• Cloud provider health alert tersedia.

EKS/AKS

• Node group/node pool tersebar antar AZ/zone.
• Topology spread/anti-affinity sesuai kebutuhan.
• PDB tersedia untuk workload critical.
• HPA dan cluster autoscaling berjalan.
• Scaling quota/IP capacity cukup.
• Readiness/liveness/startup probe benar.

Load balancer and ingress

• Health check path benar.
• Target tersebar antar zone.
• Timeout chain terdokumentasi.
• TLS termination chain jelas.
• 502/503/504 dashboard tersedia.

Data and messaging

• PostgreSQL HA/failover behavior diuji.
• JDBC pool failover behavior diuji.
• Kafka/RabbitMQ lag alert tersedia.
• DLQ/retry policy jelas.
• Redis failover behavior diuji.
• Cache fallback correctness diverifikasi.

Identity/config/secret

• Workload identity runtime diverifikasi.
• Credential refresh observable.
• Secret/config cache strategy jelas.
• Secret rotation tested.
• AccessDenied triage path tersedia.

Observability

• Dashboard per capability tersedia.
• Dependency latency/error metrics tersedia.
• Retry/circuit breaker metrics tersedia.
• Queue lag metrics tersedia.
• Synthetic checks tersedia untuk user journey critical.

Runbook and test evidence

• Failover runbook tersedia.
• DB failover drill pernah dilakukan.
• Node drain/upgrade drill pernah dilakukan.
• Dependency outage simulation pernah dilakukan.
• RCA history dipelajari.

35. Senior engineer heuristics

• Jangan percaya replica count sampai tahu failure domain-nya.
• Jangan percaya health check sampai tahu apa yang dicek.
• Jangan percaya retry sampai tahu max attempts, backoff, jitter, dan idempotency.
• Jangan percaya fallback sampai tahu correctness boundary.
• Jangan percaya managed HA sampai failover sudah diuji dari aplikasi.
• Jangan percaya dashboard sampai bisa menjawab “dependency mana yang gagal?” dalam incident.
• Jangan percaya Multi-AZ sampai database, broker, cache, NAT, LB, pod, dan node juga dicek.
• Jangan percaya “cloud provider handles it” untuk application-level correctness.

36. Kesimpulan

Availability adalah target bisnis-operasional.

Resilience adalah kualitas desain dan operasi yang membuat target itu realistis.

Untuk senior Java/JAX-RS engineer, fokus terpenting bukan menghafal semua fitur HA AWS/Azure, tetapi memahami:

• capability mana yang harus tetap tersedia;
• failure domain mana yang bisa jatuh bersama;
• dependency mana yang critical;
• timeout/retry mana yang bisa memperbesar outage;
• correctness boundary mana yang tidak boleh dilanggar;
• observability mana yang membuktikan kondisi sistem;
• runbook mana yang benar-benar pernah diuji.

Cloud menyediakan building block. Reliability tetap harus didesain, diuji, dan dioperasikan.