Backup, Restore, Disaster Recovery, and Data Resilience Model

1. Core Idea

Enterprise data model harus bisa dipulihkan.

Dalam CPQ / Quote / Order / Billing / Fulfillment / Inventory, kehilangan atau korupsi data dapat berdampak langsung ke:

• quote pipeline,
• order execution,
• product activation,
• billing correctness,
• invoice dispute,
• customer service,
• financial reporting,
• compliance evidence,
• audit history,
• integration traceability.

Backup bukan hanya file cadangan. Disaster recovery bukan hanya infrastructure failover.

Mental model:

Data resilience means the system can restore correct business truth, not merely restart servers.

Restore harus mempertimbangkan database, event broker, cache, search index, object storage, file feed, archive, analytics, external system mapping, and in-flight workflows.

2. Why Backup/Restore Modelling Matters

Tanpa model recovery yang jelas:

• database restore mengembalikan order tetapi event broker offset sudah maju,
• outbox event hilang setelah restore,
• search index menampilkan data yang tidak ada di DB,
• invoice file export dikirim dua kali setelah recovery,
• usage feed diproses ulang dan duplicate charge dibuat,
• tenant restore mencampur data tenant lain,
• audit record hilang tetapi business data ada,
• PITR mengembalikan old state tetapi external billing sudah menerima new state,
• restore dari backup membangkitkan data yang sudah dipurge/anonymized,
• DR environment tidak punya latest reference data/config,
• RPO/RTO tidak realistis untuk quote-to-cash flow.

Data recovery harus didesain dengan distributed consistency awareness.

3. RPO and RTO

Example:

RPO = 15 minutes
RTO = 2 hours

But RPO/RTO can differ by data domain:

Do not assign one blanket value without business validation.

4. Recovery Scope

Recovery can be scoped.

Each scope has different risk.

Entity/tenant restore is much harder than full DB restore if data is highly connected.

5. Backup Inventory

A complete backup model identifies all stores.

Stores:

• PostgreSQL databases,
• object storage,
• event broker topics/queues,
• Redis/cache if persistence required,
• search index,
• workflow engine DB,
• analytics warehouse,
• file feed storage,
• audit log storage,
• secrets/config references,
• reference data/config service,
• archive store,
• external ID mapping store.

Backup registry:

backup_asset
- asset_type
- asset_name
- owner_group
- backup_frequency
- retention_policy
- restore_method
- criticality

If a store contains source-of-truth or unrebuildable evidence, it must be in backup plan.

6. Backup Record Model

Fields:

backup_record
- id
- backup_type
- asset_type
- asset_name
- environment
- tenant_id nullable
- started_at
- completed_at
- status
- backup_location
- backup_size
- checksum
- encryption_status
- retention_until
- created_by

Backup metadata supports:

• audit,
• restore selection,
• compliance evidence,
• failed backup alert,
• retention cleanup.

Backup without metadata is hard to trust.

7. Restore Record Model

Restore should be auditable.

Fields:

restore_record
- id
- restore_type
- source_backup_id
- target_environment
- target_asset
- tenant_id nullable
- requested_by
- approved_by
- reason_code
- status
- started_at
- completed_at
- validation_status
- rollback_strategy

Restore should require approval in production.

Restore can be more dangerous than deletion because it can overwrite correct current data.

8. Restore Validation

After restore, validate:

• row counts,
• critical business invariants,
• latest committed timestamp,
• outbox/inbox consistency,
• event offset alignment,
• workflow/domain consistency,
• integration message state,
• data quality checks,
• tenant isolation,
• search/projection consistency,
• audit presence,
• purge/anonymization reapplication,
• external system reconciliation.

Restore complete means business correctness verified, not only database online.

9. Point-in-Time Recovery

PITR restores to specific timestamp.

Risks:

• external systems may have state after timestamp,
• events after timestamp may have been published,
• customers may have received confirmations,
• invoices/files may have been sent,
• payment processed externally,
• audit/access logs may differ,
• cache/search may reflect later state.

PITR requires reconciliation plan:

DB restored to 10:00
Events published 10:00-10:15 may need replay/suppression/compensation
External billing messages sent 10:00-10:15 need status check

Do not do PITR without distributed impact analysis.

10. Outbox and Restore

After database restore, outbox state may be inconsistent with broker.

Cases:

Mitigation:

• idempotent event consumers,
• event_id stable,
• outbox publish audit,
• broker offset comparison,
• reconciliation by business state,
• replay with deduplication.

11. Inbox and Restore

Consumer inbox prevents duplicate processing.

After restore:

• inbox may lose processed records,
• broker may redeliver old events,
• consumer may process side effects again.

Mitigation:

• business-level idempotency constraints,
• external side-effect idempotency keys,
• dedup tables restored consistently,
• replay mode controls,
• reconciliation before reprocessing.

Inbox alone is not enough if inbox backup is older than external side effects.

12. Search and Projection Rebuild

Search index and read models are usually rebuildable.

After DB restore:

• invalidate search index,
• rebuild projection from source DB/events,
• reset checkpoint,
• track rebuild status,
• prevent stale projection from being served as fresh,
• reapply security/masking/purge rules.

Projection freshness should be visible during recovery.

13. Redis/Cache Recovery

Cache usually should be treated as disposable.

After restore/failover:

• flush affected cache regions,
• warm critical cache if needed,
• ensure tenant/environment namespace correct,
• invalidate permission/security-sensitive cache,
• do not trust old cached state after PITR.

If Redis stores durable state, it must be backup/restore governed like database.

Avoid durable business truth in Redis unless intentionally designed.

14. File and Object Storage Recovery

Files include:

• usage feed,
• export file,
• invoice PDF,
• contract document,
• audit evidence,
• snapshot archive,
• external payload reference.

Backup/restore must preserve:

• file object,
• metadata,
• checksum,
• access control,
• encryption,
• retention/legal hold,
• database reference consistency.

Database restored but object missing means broken evidence.

15. External System Reconciliation After Recovery

After restore/failover, reconcile with external systems:

• billing system,
• ERP,
• OSS,
• CRM,
• payment provider,
• workflow engine,
• identity provider,
• partner systems.

Examples:

Local invoice status restored to DRAFT, but ERP has invoice issued.
Local product inactive, but OSS service active.
Local billing trigger pending, but billing already activated.

Recovery runbook should include external status checks.

16. Tenant Restore

Tenant restore in shared environment is difficult.

Challenges:

• tenant rows across many tables,
• shared reference data,
• cross-tenant metadata,
• global sequences/business numbers,
• shared event topics,
• shared search indexes,
• shared object storage,
• tenant-specific encryption keys,
• tenant-specific integrations.

Tenant restore model:

tenant_restore_request
- tenant_id
- restore_point
- scope
- status
- approved_by
- validation_result

Tenant restore must not overwrite other tenants or leak data.

17. Disaster Recovery Topology

DR topology may be:

• active-active,
• active-passive,
• warm standby,
• cold standby,
• backup-only,
• regional failover.

Data model should know:

primary_region
dr_region
replication_mode
failover_status
last_replication_lag

DR affects:

• tenant routing,
• event broker,
• integrations,
• data residency,
• external callbacks,
• DNS/endpoints,
• support tooling,
• reporting freshness.

18. Failover and Split-Brain

Split-brain happens when two sites accept writes independently without coordination.

Risks:

• duplicate order numbers,
• conflicting quote updates,
• duplicate billing triggers,
• product inventory divergence,
• double resource allocation.

Mitigation:

• clear primary writer,
• fencing tokens,
• region leadership,
• write suspension during failover,
• idempotency keys,
• conflict resolution model,
• reconciliation after failback.

Do not design active-active writes casually for financial/order data.

19. Backup Retention and Legal Hold

Backup retention must align with:

• retention policy,
• legal hold,
• privacy deletion,
• audit requirement,
• customer contract,
• data residency.

If data is purged/anonymized in live system, backup restore may reintroduce it.

Need process:

• deletion/anonymization ledger,
• reapply purge after restore,
• backup expiry policy,
• legal hold exception,
• restore validation against deletion ledger.

20. Restore Drill

Backups are unproven until restored.

Restore drill should record:

restore_drill
- id
- drill_type
- asset_name
- environment
- backup_id
- started_at
- completed_at
- rto_actual
- rpo_actual
- validation_status
- issues_found

Drill validates:

• backup exists,
• restore works,
• docs/runbook correct,
• access credentials work,
• data validation passes,
• RTO/RPO realistic.

21. PostgreSQL Physical Design

Backup record:

create table backup_record (
  id uuid primary key,
  backup_type text not null,
  asset_type text not null,
  asset_name text not null,
  environment text not null,
  tenant_id uuid,
  started_at timestamptz not null,
  completed_at timestamptz,
  status text not null,
  backup_location text,
  backup_size_bytes bigint,
  checksum text,
  encryption_status text,
  retention_until timestamptz,
  created_by text
);

Restore record:

create table restore_record (
  id uuid primary key,
  restore_type text not null,
  source_backup_id uuid references backup_record(id),
  target_environment text not null,
  target_asset text not null,
  tenant_id uuid,
  requested_by text not null,
  approved_by text,
  reason_code text not null,
  status text not null,
  started_at timestamptz,
  completed_at timestamptz,
  validation_status text,
  rollback_strategy text,
  created_at timestamptz not null
);

Restore validation result:

create table restore_validation_result (
  id uuid primary key,
  restore_record_id uuid not null references restore_record(id),
  validation_code text not null,
  status text not null,
  severity text not null,
  message text,
  checked_at timestamptz not null
);

Restore drill:

create table restore_drill (
  id uuid primary key,
  drill_type text not null,
  asset_name text not null,
  environment text not null,
  backup_id uuid references backup_record(id),
  started_at timestamptz not null,
  completed_at timestamptz,
  rto_actual_seconds integer,
  rpo_actual_seconds integer,
  validation_status text,
  issues_found text
);

Indexes:

create index idx_backup_asset_time
on backup_record (asset_type, asset_name, started_at desc);

create index idx_backup_status_time
on backup_record (status, started_at desc);

create index idx_restore_status_time
on restore_record (status, created_at desc);

create index idx_restore_validation_restore
on restore_validation_result (restore_record_id, severity, status);

22. Java/JAX-RS Backend Implications

Internal/admin APIs:

GET /backup-records
GET /restore-records
POST /restore-requests
POST /restore-records/{id}/validate
GET /restore-drills
POST /restore-drills
GET /recovery-status

Restore orchestration should:

• require authorization,
• validate target environment,
• prevent accidental prod overwrite,
• check legal hold/privacy ledger,
• run validation suite,
• publish recovery events,
• update support/incident record,
• trigger projection/search rebuild where required.

Production restore should be treated as high-risk operational workflow.

23. Recovery Events

Events:

• BackupCompleted
• BackupFailed
• RestoreRequested
• RestoreStarted
• RestoreCompleted
• RestoreValidationFailed
• FailoverStarted
• FailoverCompleted
• ProjectionRebuildStarted
• ProjectionRebuildCompleted
• PostRestoreReconciliationRequired

These events support operational visibility.

Do not include sensitive backup locations in broadly consumed events if restricted.

24. Data Quality Checks After Restore

Examples:

-- Outbox pending too old after restore
select event_type, count(*), min(created_at)
from outbox_event
where status in ('PENDING', 'FAILED')
group by event_type;

-- Product active without active charge after recovery
select pi.id
from product_instance pi
left join recurring_charge rc
  on rc.product_instance_id = pi.id
 and rc.status = 'ACTIVE'
where pi.status = 'ACTIVE'
  and rc.id is null;

-- Restore validation failures
select validation_code, severity, count(*)
from restore_validation_result
where restore_record_id = :restore_id
  and status = 'FAILED'
group by validation_code, severity;

The actual validation suite should be domain-specific.

25. Security and Access

Backup/restore data is sensitive.

Controls:

• encryption at rest,
• encrypted transfer,
• access control,
• restore approval,
• audit restore/download,
• tenant isolation,
• data residency,
• legal hold,
• secrets excluded or handled via vault,
• backup location secrecy,
• retention enforcement.

Backup copies can be a major data leakage risk.

26. Observability

Monitor:

• backup success/failure,
• backup age,
• backup duration,
• backup size anomaly,
• checksum validation failure,
• restore drill failures,
• replication lag,
• DR readiness,
• projection rebuild lag,
• post-restore reconciliation mismatch,
• legal hold/purge restore conflicts.

Example:

-- Backup older than expected
select asset_type, asset_name, max(completed_at) as latest_backup
from backup_record
where status = 'COMPLETED'
group by asset_type, asset_name
having max(completed_at) < now() - interval '1 day';

27. Failure Modes

28. PR Review Checklist

When reviewing resilience-related changes, ask:

• Is this data source-of-truth or rebuildable?
• Is it included in backup inventory?
• What RPO/RTO applies?
• How is restore validated?
• Does restore affect outbox/inbox/events?
• Does restore affect external systems?
• Does cache/search/projection need rebuild?
• Does object storage/file evidence need backup?
• Does tenant-level restore need support?
• Does legal hold/privacy purge apply?
• Is restore audited and approved?
• Are restore drills performed?
• Are data quality checks available after recovery?
• Is DR topology impacted?

29. Internal Verification Checklist

Verify these in the internal CSG/team context:

• Backup policy per database/service/data store.
• RPO/RTO commitments per product/customer/deployment.
• PostgreSQL PITR setup and restore procedure.
• Whether restore drills are performed.
• Whether event broker retention/replay is part of DR.
• Whether outbox/inbox recovery behavior is documented.
• Whether search/read models are rebuildable.
• Whether object storage/file feeds/invoice documents are backed up.
• Whether tenant-level restore is supported.
• Whether data residency affects backup/DR region.
• Whether deletion/anonymization is reapplied after restore.
• Whether incidents mention restore failure, inconsistent recovery, or DR gap.

30. Summary

Backup and DR are data modelling concerns because recovery must restore business truth.

A strong model must define:

• RPO/RTO,
• backup inventory,
• backup record,
• restore record,
• restore validation,
• PITR impact,
• outbox/inbox recovery,
• projection/search rebuild,
• cache invalidation,
• object storage consistency,
• external reconciliation,
• tenant restore,
• DR topology,
• legal hold/privacy restore handling,
• restore drill,
• observability.

The key principle:

A system is not resilient because it has backups. It is resilient when those backups can be restored into a business-consistent, externally reconciled, privacy-safe, and operationally verified state.