Part 050 — Backup, Restore, and Disaster Recovery

Target pembaca: Senior Java/JAX-RS backend engineer yang perlu memastikan data, service, dan dependency cloud bisa dipulihkan secara nyata saat terjadi data loss, regional disruption, deployment accident, security incident, atau platform failure.

Backup bukan DR.

Replication bukan DR.

Snapshot bukan bukti recovery.

Disaster recovery adalah kemampuan organisasi memulihkan capability bisnis dalam batas waktu dan batas kehilangan data yang disetujui.

Untuk enterprise Quote & Order style system, DR tidak boleh dipahami sebagai “database punya backup”. Sistem biasanya memiliki:

PostgreSQL state
Kafka/RabbitMQ events
Redis cache/session/lock state
Camunda/process state
object storage documents
config and secrets
container images
Kubernetes manifests
IaC state
DNS records
identity/RBAC
audit logs
observability data
external integrations

Jika satu bagian dipulihkan tanpa urutan dependency yang benar, sistem bisa hidup tetapi salah.

Part ini membangun mental model backup, restore, dan disaster recovery untuk backend engineer yang bekerja di AWS/Azure, EKS/AKS, PostgreSQL, Kafka/RabbitMQ, Redis, Camunda, object storage, GitOps/IaC, dan hybrid/on-prem environment.

1. Konsep inti

Backup adalah salinan data/configuration pada titik waktu tertentu.

Restore adalah proses mengembalikan data/configuration dari backup.

Disaster Recovery adalah strategi dan eksekusi untuk memulihkan workload/capability setelah gangguan besar.

backup   = copy exists
restore  = copy can be used
DR       = business capability can be recovered within RTO/RPO

Pertanyaan inti DR:

1. Apa yang harus dipulihkan?
2. Dalam urutan apa?
3. Dalam waktu berapa lama?
4. Dengan kehilangan data maksimal berapa?
5. Siapa yang memutuskan aktivasi DR?
6. Siapa yang menjalankan runbook?
7. Bagaimana correctness diverifikasi setelah recovery?
8. Bagaimana customer impact dikomunikasikan?
9. Bagaimana kembali ke primary environment?
10. Bukti test apa yang tersedia?

2. RPO dan RTO

RPO adalah Recovery Point Objective.

RPO = berapa banyak data loss maksimal yang dapat diterima

RTO adalah Recovery Time Objective.

RTO = berapa lama maksimal capability boleh tidak tersedia sebelum dipulihkan

Contoh:

RPO 5 minutes
RTO 60 minutes

Artinya:

• sistem boleh kehilangan maksimal sekitar 5 menit data;
• sistem harus kembali menjalankan capability dalam 60 menit.

RPO/RTO harus per capability, bukan global.

Jangan menerima RPO/RTO tanpa menanyakan biaya dan kompleksitasnya.

Lower RPO/RTO = higher cost + higher complexity + more testing burden

3. Kenapa backup saja tidak cukup

Backup hanya menjawab: “apakah salinan pernah dibuat?”

DR harus menjawab:

• apakah backup valid?
• apakah backup bisa didekripsi?
• apakah key masih tersedia?
• apakah backup konsisten antar service?
• apakah restore pernah diuji?
• apakah restore memenuhi RTO?
• apakah data loss memenuhi RPO?
• apakah aplikasi bisa start dengan data hasil restore?
• apakah external integrations aman setelah restore?
• apakah DNS dan identity sudah menunjuk ke environment yang benar?
• apakah event replay tidak menggandakan side effect?

Anti-pattern:

We have daily backups, so DR is covered.

Masalah:

• daily backup mungkin RPO 24 jam;
• restore bisa butuh 8 jam;
• secret/key untuk decrypt mungkin tidak tersedia;
• app schema sudah lebih baru dari backup;
• Kafka offset/event state tidak sinkron dengan database;
• object storage document tidak match dengan DB metadata;
• DNS cutover belum ada;
• team belum pernah latihan.

4. Disaster categories

DR tidak hanya untuk region outage.

Kategori disaster:

Setiap kategori butuh recovery playbook berbeda.

5. Backup taxonomy

Jenis backup:

Backup harus diklasifikasikan berdasarkan:

• criticality;
• retention;
• encryption;
• immutability;
• restore scope;
• restore speed;
• owner;
• test frequency.

6. Restore taxonomy

Jenis restore:

Setiap jenis restore punya risk berbeda.

Contoh:

Database restore to 10:00
but object storage still contains files uploaded until 10:30

Hasilnya:

• DB metadata tidak mengenal sebagian object;
• object orphaned;
• user melihat inconsistency;
• cleanup/reconciliation diperlukan.

7. Data consistency concern

DR paling sulit bukan “membuat service hidup”.

Yang sulit adalah menjaga konsistensi antar state.

State sources:

PostgreSQL transactional state
Camunda/process engine state
Kafka/RabbitMQ event stream
Redis transient state
S3/Blob documents
external system state
search/read models
cache/materialized view

Consistency questions:

• Apakah DB dan event stream dipulihkan ke titik waktu yang konsisten?
• Apakah object storage version selaras dengan DB metadata?
• Apakah external system sudah menerima side effect sebelum disaster?
• Apakah order submission bisa di-retry tanpa duplicate?
• Apakah event replay aman?
• Apakah Camunda/process state cocok dengan event/outbox state?
• Apakah cache harus dibuang setelah restore?
• Apakah read model harus rebuild?

Rule:

After restore, assume cache/read models are suspicious until reconciled.

8. Backup and DR in AWS

AWS building blocks yang umum relevan:

• Amazon RDS automated backups;
• RDS snapshots;
• Point-in-time recovery;
• Aurora backup/replica jika digunakan;
• AWS Backup;
• S3 versioning;
• S3 lifecycle/replication/Object Lock jika digunakan;
• EBS snapshots;
• EFS backup/replication jika digunakan;
• MSK topic retention/cluster backup pattern;
• ECR replication/lifecycle;
• Route 53 DNS failover/cutover;
• CloudFormation/Terraform state backup;
• CloudTrail archival;
• KMS key policy/key availability;
• IAM role/policy as code.

AWS-specific review questions:

9. Backup and DR in Azure

Azure building blocks yang umum relevan:

• Azure Backup;
• Azure Database for PostgreSQL Flexible Server backups/PITR;
• zone-redundant HA jika tersedia;
• geo-redundant backup jika digunakan;
• Azure Blob versioning/soft delete/immutability/replication options;
• Azure Site Recovery untuk VM jika relevan;
• Azure Key Vault soft delete/purge protection;
• Azure Container Registry geo-replication jika digunakan;
• Azure Traffic Manager/Front Door/DNS untuk failover/cutover jika digunakan;
• Azure Monitor/Activity Log retention;
• Bicep/Terraform/GitOps state;
• Managed identity/RBAC role assignment as code.

Azure-specific review questions:

10. Kubernetes DR: EKS and AKS

Untuk Kubernetes, jangan mulai dari backup etcd kecuali platform memang mengelola cluster backup tersebut.

Untuk workload application, DR yang lebih maintainable biasanya:

IaC recreates cluster/platform resources
GitOps reconciles namespaces/workloads/secrets refs/ingress/policies
container registry provides images
secret/config manager provides runtime values
persistent data restored from managed data services
DNS/load balancer cutover points traffic to recovered environment

Yang harus dipulihkan:

• cluster infrastructure;
• node groups/node pools;
• ingress controllers;
• service accounts/workload identity;
• namespaces;
• network policies;
• config maps/secrets references;
• external secret/csi provider config;
• deployments/statefulsets/jobs;
• HPA/PDB;
• storage classes/PVCs jika ada;
• monitoring agents;
• GitOps controller;
• image pull permissions.

Kubernetes manifest di Git bukan cukup jika:

• image registry tidak tersedia;
• secret manager tidak tersedia;
• workload identity role missing;
• private endpoint DNS missing;
• storage class berbeda di DR region;
• ingress annotation berbeda per cloud;
• certificate tidak tersedia;
• external dependency endpoint masih menunjuk primary region.

11. PostgreSQL backup and restore

PostgreSQL adalah pusat state banyak enterprise backend.

Review area:

• automated backup aktif;
• PITR retention;
• snapshot schedule;
• logical backup jika diperlukan;
• encryption key;
• HA/failover mode;
• read replica/geo replica jika digunakan;
• maintenance window;
• schema migration compatibility;
• extension compatibility;
• connection string/DNS after restore;
• application user/role permission;
• connection pool recovery;
• restore test frequency.

Restore sequence example:

1. Freeze write traffic or activate incident mode.
2. Identify target restore point before corruption/outage.
3. Restore PostgreSQL to new instance/environment.
4. Validate schema version and migration state.
5. Validate critical business tables.
6. Reconnect application using controlled config/DNS change.
7. Rebuild cache/read models if needed.
8. Resume consumers in controlled order.
9. Run reconciliation checks.
10. Reopen traffic gradually.

Danger:

Restore DB while Kafka consumers keep processing old events

Could cause:

• duplicate writes;
• missing state transitions;
• out-of-order process state;
• inconsistent external side effects.

12. Kafka/RabbitMQ DR

Messaging DR depends on semantics.

Questions:

• Is the broker source of truth or transport?
• Are messages replayable from retention?
• Are consumers idempotent?
• Are producer side effects deduplicated?
• Is ordering required?
• Is there a DLQ?
• Are offsets part of recovery?
• Is the event schema backward compatible?
• Can external events be re-fetched?
• What happens to in-flight messages?

Kafka restore/recovery patterns:

retention-based replay
mirror/replicate topics if multi-region
rebuild read models from compacted/event topics
pause consumers during DB restore
resume by consumer group/partition carefully

RabbitMQ recovery patterns:

quorum queue / mirrored pattern if used
DLQ inspection
manual requeue with idempotency
publisher confirms
consumer ack after durable business write

Do not assume:

broker HA = business event recovery

You need replay and idempotency evidence.

13. Redis backup and restore

Redis DR depends on usage.

Rule:

Do not restore transient state unless you know its semantics.

For cache:

flush and rebuild may be safer than restore stale cache

For idempotency:

RPO must match duplicate-prevention requirement

14. Object storage DR

Object storage often stores:

• uploaded documents;
• generated PDFs;
• exports;
• imports;
• binary attachments;
• reconciliation files;
• audit evidence;
• archive data.

Review:

• versioning;
• soft delete;
• retention;
• lifecycle;
• replication;
• encryption key;
• object lock/immutability if required;
• metadata consistency with database;
• presigned URL/SAS behavior after failover;
• private endpoint/DNS in DR environment;
• large file multipart cleanup;
• orphaned object reconciliation.

Failure example:

DB restored to 09:55
Blob/S3 bucket still has files until 10:15

Required:

reconciliation job identifies objects without DB metadata
policy decides keep, quarantine, or delete

15. Camunda/process state DR

For process-driven systems, DR must preserve state machine correctness.

Questions:

• Where is Camunda/process state stored?
• Is process DB included in backup/PITR?
• Are external task locks safe after restore?
• Are timers/jobs duplicated?
• Are process events replayable?
• Are external side effects idempotent?
• Is there compensation logic?
• Are stuck process instances detectable?
• Is there a manual repair procedure?

Danger:

process state restored to before external order submission
external order system already received request
process retries submission
duplicate order created

Pattern:

idempotency key
external correlation ID
outbox pattern
reconciliation workflow
manual compensation path

16. Config, secret, and key recovery

A recovered service cannot start if secrets/keys/config are missing.

Required:

• secret manager available;
• secret versions available;
• access policy/RBAC restored;
• workload identity restored;
• KMS/Key Vault key available;
• certificate available;
• config labels/environments correct;
• feature flags safe;
• expired credentials handled;
• rotation paused/resumed intentionally.

Key risk:

backup exists but encryption key is deleted or inaccessible

This makes backup useless.

For production:

• protect KMS/Key Vault keys;
• enable soft delete/purge protection where applicable;
• document key ownership;
• test restore with encrypted data;
• avoid manual-only key recovery knowledge.

17. DNS cutover and failback

DR usually requires routing traffic to recovered environment.

DNS cutover considerations:

• public DNS TTL;
• private DNS TTL;
• split-horizon DNS;
• Route 53 or Azure DNS ownership;
• Traffic Manager/Front Door/Route 53 failover if used;
• API Gateway/APIM backend target;
• load balancer hostname;
• certificate/SNI;
• client DNS caching;
• mobile/corporate proxy caching;
• on-prem resolver forwarding;
• rollback/failback.

DNS cutover is not instant.

Even with low TTL, real clients and proxies may cache longer.

Runbook must include:

pre-cutover validation
traffic drain/freeze
DNS update
monitoring during propagation
old endpoint behavior
failback decision

18. Dependency recovery order

Recovery order matters.

Example order for cloud Kubernetes backend:

1. Incident command and decision authority
2. Freeze writes if needed
3. Network foundation / private connectivity
4. Identity/RBAC/workload identity
5. Key/secret/config services
6. Container registry/image availability
7. Data stores: PostgreSQL/object storage/broker/Redis/process DB
8. Kubernetes cluster/node pools/add-ons
9. Ingress/load balancer/API gateway
10. Application workloads
11. Consumers/jobs in controlled order
12. Observability/alerts
13. Synthetic checks
14. DNS/cutover
15. Customer validation
16. Reconciliation
17. Failback plan

Bad order:

start consumers before database restore validation

Bad order:

cut over DNS before private endpoint and secret access validated

Bad order:

restore app before registry/identity works

19. DR strategy patterns

Common strategies:

Do not choose strategy by prestige.

Choose by:

• business criticality;
• RPO/RTO;
• data consistency;
• operational maturity;
• cost tolerance;
• test frequency;
• team expertise.

For many enterprise systems, hybrid approach is common:

critical APIs: warm standby or active-passive
reporting/export: backup/restore
object storage: replication/versioning
DB: PITR + replica/geo-restore depending RPO/RTO
observability: replicated retention/export

20. Application-level DR behavior

A Java/JAX-RS service should be DR-aware in behavior, not necessarily cloud-aware in business logic.

Needed properties:

• idempotency for side-effecting APIs;
• correlation IDs across retries/replays;
• bounded timeouts;
• safe startup validation;
• explicit degraded mode;
• config-driven endpoints;
• no hardcoded region/account/subscription;
• no static credentials;
• safe cache invalidation;
• replay-safe consumers;
• schema migration compatibility;
• clear error responses during incident mode.

Bad:

// endpoint hardcoded to primary region storage URL
String storageEndpoint = "https://primary-region-only.example";

Better:

endpoint comes from environment/config service
validated at startup
observable in dependency health dashboard
change controlled through GitOps/config pipeline

21. Idempotency in DR

DR often involves retry, replay, duplicate delivery, and uncertain external state.

Idempotency is mandatory for:

• order submission;
• quote finalization;
• payment-like external calls;
• fulfillment callouts;
• document upload finalization;
• event consumers with side effects;
• process engine external tasks.

Pattern:

idempotency key
  + request hash
  + persisted result
  + external correlation ID
  + duplicate detection window

During recovery:

same idempotency key -> same business outcome, not new side effect

Without idempotency, DR replay can create data corruption.

22. Restore drill

A restore drill validates that recovery works.

Minimum drill types:

A drill must produce evidence:

• date;
• environment;
• scope;
• restore point;
• steps executed;
• time per step;
• result;
• issues found;
• RTO/RPO achieved;
• data validation;
• sign-off;
• follow-up actions.

23. DR evidence

For regulated/enterprise environments, saying “we can restore” is not enough.

Evidence examples:

• backup policy screenshot/export;
• automated backup configuration;
• PITR configuration;
• restore drill log;
• checksum/reconciliation result;
• synthetic test result after restore;
• runbook version;
• incident simulation report;
• approval/sign-off;
• IaC/GitOps commit reference;
• audit logs;
• key access validation;
• DR exercise RCA.

Evidence should be versioned and discoverable.

24. Security and compliance in DR

DR environment must not become a weaker security environment.

Review:

• same or stricter IAM/RBAC;
• least privilege for DR operators;
• break-glass access audited;
• encryption keys protected;
• network private access preserved;
• no public exposure introduced during restore;
• WAF/API policy restored;
• audit logs retained;
• PII handling unchanged;
• data residency constraints respected;
• object storage policies restored;
• secrets not copied into tickets/chat manually.

Anti-pattern:

DR restore uses shared admin credential and temporary public database access

This may recover availability but create security incident.

25. Cost and capacity in DR

DR has cost trade-offs.

Cost drivers:

• standby compute;
• replicated database;
• cross-region data transfer;
• object replication;
• log retention;
• idle load balancers;
• standby NAT/firewall;
• duplicate monitoring stack;
• backup storage;
• restore testing environments;
• license costs.

Capacity questions:

• Can DR environment handle full production load?
• Is it intentionally scaled down?
• How fast can it scale up?
• Are quotas pre-approved?
• Are subnet IP ranges large enough?
• Are database/broker/cache sizes sufficient?
• Are image pulls throttled?
• Is warm standby tested under load?

Common trap:

DR environment exists but cannot scale because quota/subnet/IP/DB tier not ready.

26. Observability during DR

During DR, observability often breaks first.

Ensure:

• logs from DR environment go to known workspace/log group;
• dashboards can filter environment/region;
• alerts do not point only to primary region;
• synthetic checks target DR endpoint after cutover;
• traces preserve environment tag;
• audit logs are retained;
• runbook links to correct dashboard;
• on-call knows which metrics define recovery success.

Recovery success is not:

pods are running

Recovery success is:

critical business synthetic checks pass
error rate acceptable
latency within degraded target
data reconciliation clean
no duplicate side effects detected

27. Failure-mode matrix

28. DR runbook template

A practical DR runbook should include:

# Disaster Recovery Runbook — <Capability/System>

## Scope
- Capability:
- Environment:
- Region/account/subscription:
- Owner:

## Activation criteria
- When to activate:
- Who can approve:
- Communication channel:

## RPO/RTO
- RPO:
- RTO:
- Measurement method:

## Dependencies
- Network:
- Identity:
- Secrets/keys:
- Database:
- Broker:
- Object storage:
- Kubernetes:
- DNS/API gateway:
- Observability:

## Pre-checks
- Current incident state:
- Freeze writes required:
- Backup restore point:
- Customer impact:

## Recovery steps
1. ...
2. ...

## Validation
- Data validation:
- Synthetic checks:
- Log/metric checks:
- Security checks:

## Cutover
- DNS/gateway change:
- Expected propagation:
- Rollback:

## Reconciliation
- Events:
- Object storage:
- External systems:

## Failback
- Criteria:
- Steps:
- Risks:

## Evidence
- Links:
- Timestamp:
- Sign-off:

29. PR review checklist

Saat mereview PR/ADR yang menyentuh backup/DR, tanyakan:

Data

• Data baru apa yang dibuat?
• Apakah data itu source of truth atau derived?
• Apakah perlu backup?
• Apa retention requirement?
• Apakah ada PII/compliance requirement?
• Apakah encryption key dependency jelas?

Restore

• Bagaimana data dipulihkan?
• Restore scope apa yang didukung?
• Apakah restore pernah diuji?
• Apakah RTO/RPO diketahui?
• Apakah ada reconciliation step?

Events

• Apakah event replay aman?
• Apakah consumer idempotent?
• Apakah offset/ack timing benar?
• Apakah DLQ tersedia?
• Apakah poison message bisa diisolasi?

Object storage

• Apakah object versioning/soft delete diperlukan?
• Apakah DB metadata dan object storage bisa direkonsiliasi?
• Apakah presigned URL/SAS tetap aman setelah restore?

Kubernetes/cloud

• Apakah resource bisa dibuat ulang dari IaC/GitOps?
• Apakah image tersedia di registry target?
• Apakah workload identity restored?
• Apakah private endpoint/DNS restored?
• Apakah secrets/config restored?

Security

• Apakah DR environment punya security baseline sama?
• Apakah break-glass access diaudit?
• Apakah backup immutable atau terlindungi dari deletion?
• Apakah key management aman?

Operations

• Apakah runbook ada?
• Apakah owner jelas?
• Apakah activation criteria jelas?
• Apakah test evidence tersedia?
• Apakah failback plan ada?

30. Internal verification checklist

Gunakan checklist ini untuk CSG/team verification. Jangan mengasumsikan detail internal tanpa konfirmasi.

RPO/RTO and scope

• RPO/RTO per capability terdokumentasi.
• DR scope per service diketahui.
• Critical vs non-critical capability jelas.
• Activation authority jelas.
• Customer communication path jelas.

AWS/Azure backup configuration

• RDS/Azure PostgreSQL backup aktif.
• PITR retention sesuai requirement.
• Snapshot policy jelas.
• Object storage versioning/soft delete/replication sesuai requirement.
• Registry replication/cache strategy jelas.
• Cloud audit logs retained.
• KMS/Key Vault recovery protection aktif.

Kubernetes/EKS/AKS

• Cluster can be recreated from IaC.
• Workloads can be recreated from GitOps.
• Ingress/load balancer config versioned.
• Workload identity config versioned.
• Secrets/config references documented.
• DR namespace/resource quota ready.
• Image pull works in DR environment.

Data stores

• PostgreSQL restore drill performed.
• DB failover/restore app reconnect behavior tested.
• Kafka/RabbitMQ replay strategy documented.
• DLQ recovery documented.
• Redis usage classified: cache/session/lock/idempotency.
• Object storage reconciliation process exists.
• Camunda/process state recovery documented if relevant.

Network/DNS/private connectivity

• Private endpoint/VPC endpoint available in DR path.
• Private DNS zone/hosted zone associations documented.
• DNS cutover runbook exists.
• Firewall/NSG/SG rules ready.
• Hybrid connectivity route documented.
• Certificate/SNI setup available.

Security/compliance

• Backup encryption verified.
• Restore with key access tested.
• DR access least privilege.
• Break-glass account audited.
• PII/data residency constraints reviewed.
• Immutable/locked backup requirement reviewed.
• Secret handling during DR is safe.

Observability and evidence

• DR dashboard exists or can filter DR environment.
• Synthetic checks for critical capability exist.
• DR alerts routed correctly.
• Restore drill evidence stored.
• RTO/RPO measurement captured.
• Post-DR reconciliation evidence defined.
• Failback evidence template exists.

31. Senior engineer heuristics

• Backup that is never restored is hope, not capability.
• RPO/RTO without test evidence is a wish.
• Replication can replicate corruption.
• Cache restore is often less safe than cache rebuild.
• Event replay without idempotency is data corruption risk.
• DR environment with weaker security is a second incident.
• DNS cutover without validation is not a recovery plan.
• Database restore without object/event reconciliation is incomplete.
• Kubernetes manifests without registry, identity, secrets, and private DNS are not enough.
• Failback is part of DR, not an afterthought.

32. Kesimpulan

Disaster recovery is not a vendor feature. It is an end-to-end operating capability.

AWS and Azure provide backup, replication, managed HA, DNS, identity, storage, monitoring, and automation primitives. But production recovery depends on how those primitives are combined with application correctness, idempotency, data consistency, runbooks, observability, security, and tested operational execution.

Untuk senior Java/JAX-RS backend engineer, pertanyaan paling penting bukan hanya:

Do we have backup?

Pertanyaan yang lebih benar:

Can we restore the right business capability, with acceptable data loss, within acceptable time, without violating correctness, security, or compliance — and have we tested it?

Jika jawabannya belum berbasis evidence, DR masih belum selesai.