Manual Operations, Data Repair, and Break-Glass Model

1. Core Idea

Production systems membutuhkan operasi manual, tetapi operasi manual harus terkendali.

Dalam CPQ / Quote / Order / Billing / Fulfillment / Inventory, manual operation bisa berupa:

• retry stuck order,
• resolve fulfillment fallout,
• fix wrong billing account,
• correct product inventory state,
• re-trigger billing charge,
• repair external ID mapping,
• reprocess file row,
• replay event,
• manually cancel order,
• apply billing adjustment,
• unlock stuck workflow,
• override serviceability,
• perform emergency access/break-glass.

Mental model:

Manual operation is a domain command with stronger audit, approval, evidence, and verification requirements—not a random SQL update.

2. Why Manual Operation Modelling Matters

Tanpa model manual operation:

• support memperbaiki data via SQL tanpa audit,
• correction melewati invariants,
• duplicate charge dibuat saat re-trigger,
• order status diubah tanpa item/task consistency,
• product inventory diperbaiki tetapi billing tidak ikut,
• root cause tidak dicatat,
• approval tidak ada untuk sensitive repair,
• repair tidak diverifikasi,
• rollback impossible,
• break-glass access dipakai tanpa reason,
• incident postmortem tidak tahu siapa melakukan apa.

Manual operation harus aman, traceable, and reversible where possible.

3. Manual Operation Categories

Each category needs different authority and safety controls.

4. Manual Operation Is Not CRUD

Bad:

update product_order set status = 'COMPLETED' where id = '...';

Better:

CompleteOrderRepairCommand
- order_id
- reason_code
- incident_reference
- evidence
- approved_by

Command validates:

• order item terminal states,
• fulfillment proof,
• billing impact,
• audit requirement,
• compensation needs,
• authority.

Manual repair should use domain service wherever possible.

5. Data Repair Case

A repair case represents controlled correction.

Fields:

data_repair_case
- id
- repair_number
- repair_type
- target_type
- target_id
- severity
- status
- reason_code
- incident_reference
- requested_by
- approved_by
- assigned_to
- owner_group
- created_at
- completed_at

Statuses:

DRAFT
REQUESTED
APPROVAL_REQUIRED
APPROVED
REJECTED
IN_PROGRESS
APPLIED
VERIFICATION_PENDING
VERIFIED
FAILED
ROLLED_BACK
CANCELLED

6. Repair Evidence

Repair requires evidence.

Examples:

• customer ticket,
• incident number,
• downstream billing response,
• OSS service status,
• screenshot/reference,
• reconciliation result,
• audit timeline,
• external system ID,
• approval decision,
• SQL query evidence,
• before/after snapshot.

Fields:

repair_evidence
- repair_case_id
- evidence_type
- evidence_reference
- evidence_hash
- captured_at
- captured_by

Do not store sensitive raw evidence without classification.

7. Before/After Snapshot

Every risky repair should capture before/after.

repair_change
- repair_case_id
- entity_type
- entity_id
- field_path
- old_value
- new_value
- value_type
- applied_at

For complex repair, store snapshot reference:

before_snapshot_id
after_snapshot_id

This supports audit and rollback.

8. Maker-Checker for Repair

Sensitive repairs should require maker-checker.

Examples:

• billing amount correction,
• invoice status correction,
• product activation/termination correction,
• payment status correction,
• customer/account merge,
• external mapping correction,
• break-glass access.

Rules:

requester != approver
applier != verifier

or stricter:

requester, approver, applier, verifier must satisfy role/authority policy.

9. Break-Glass Access

Break-glass is emergency privileged access.

Use cases:

• production incident,
• customer outage,
• critical billing block,
• security response,
• urgent data repair.

Break-glass must be:

• time-bound,
• reasoned,
• approved or post-approved depending emergency policy,
• scoped,
• heavily audited,
• monitored,
• reviewed after use.

Fields:

break_glass_session
- id
- actor_id
- scope_type
- scope_id
- reason_code
- incident_reference
- approved_by
- started_at
- expires_at
- ended_at
- status

Never make break-glass permanent admin access.

10. Operational Command Model

Operational commands are admin/support actions.

Examples:

RetryOutboxEvent
ReplayIntegrationMessage
ResolveDataQualityIssue
ApplyBillingCorrection
RecalculateQuotePrice
RebuildProjection
ResyncExternalStatus
ReopenOrderItem
ForceCancelWorkflowTask

Command record:

operational_command
- id
- command_type
- target_type
- target_id
- status
- requested_by
- approved_by
- executed_by
- reason_code
- correlation_id
- created_at
- completed_at

Operational commands should be audited like business commands.

11. Safe Retry Operation

Retry operation should know:

• what failed,
• whether error is retryable,
• idempotency key,
• previous attempts,
• downstream status,
• max retry policy,
• current entity state.

Example:

Retry billing trigger only if:
  product active
  no active charge exists
  previous trigger failed or timed out
  idempotency key reused

Retry should not blindly "send again".

12. Replay Operation

Replay event/file row/message should define:

• replay source,
• replay target consumer/process,
• replay mode,
• side effects enabled/disabled,
• idempotency behavior,
• reason,
• approval if risky.

Replay modes:

PROJECTION_REBUILD
SIDE_EFFECT_RETRY
VALIDATION_ONLY
DRY_RUN

Never replay production event into side-effect consumer without idempotency and approval.

13. Override Operation

Override bypasses or modifies a rule with control.

Examples:

• serviceability override,
• credit hold override,
• approval threshold override,
• backdate override,
• billing readiness override.

Override fields:

override_id
override_type
target_type
target_id
rule_code
approved_by
expires_at
reason_code
status

Overrides should expire or be reviewed.

Override is not removing validation permanently.

14. Data Correction vs Business Adjustment

Important distinction:

Example:

Wrong product activation date due to system bug:
  data correction.

Customer overbilled:
  credit adjustment, not just invoice edit.

Do not "fix" issued invoice by mutating history unless policy allows. Usually create adjustment/credit.

15. Repair and Immutable Records

Immutable records require correction strategy.

Examples:

• accepted quote,
• approved decision,
• issued invoice,
• completed order,
• audit record,
• published catalog version.

Possible repair:

• correction record,
• adjustment,
• reversal,
• superseding version,
• annotation,
• compensation,
• new event.

Do not update immutable historical artifacts directly.

16. Repair Workflow

Generic workflow:

Detect issue
  -> create repair case
  -> capture evidence
  -> assess impact
  -> choose repair strategy
  -> approval/maker-checker
  -> dry run if possible
  -> apply repair
  -> write audit/change history
  -> run validation/reconciliation
  -> verify
  -> close case
  -> record root cause/prevention

Repair without verification is incomplete.

17. Dry Run

Dry run evaluates expected changes without applying.

Fields:

repair_dry_run
- repair_case_id
- predicted_changes
- affected_entities
- validation_result
- risk_level
- executed_at

Useful for:

• bulk repair,
• billing correction,
• migration exception,
• product inventory correction,
• mapping merge/split,
• replay operation.

18. Repair Verification

Verification checks repair success.

Examples:

• data quality rule passes,
• reconciliation mismatch resolved,
• downstream ack received,
• product-charge consistency restored,
• support timeline updated,
• customer impact cleared,
• no duplicate charge/order created.

Fields:

repair_verification
- repair_case_id
- verification_type
- status
- verified_by
- verified_at
- evidence_reference

19. Rollback and Forward Fix

Repair may fail.

Options:

• rollback previous value,
• apply compensating correction,
• create new adjustment,
• cancel repair,
• escalate to engineering,
• open incident.

Repair case should document rollback strategy before applying risky change.

For immutable/financial records, rollback may not be allowed. Use forward fix.

20. Root Cause and Preventive Action

Repair is not complete until root cause is addressed.

Fields:

root_cause_code
root_cause_description
preventive_action
linked_defect
linked_runbook_update
linked_validation_rule

Examples:

• missing validation,
• event lost,
• mapping error,
• race condition,
• migration bug,
• external system defect,
• manual mistake.

Preventive action might be:

• add DB constraint,
• add validation,
• add reconciliation rule,
• fix event consumer,
• update runbook,
• add monitoring.

21. PostgreSQL Physical Design

Repair case:

create table data_repair_case (
  id uuid primary key,
  repair_number text not null unique,
  repair_type text not null,
  target_type text not null,
  target_id uuid,
  severity text not null,
  status text not null,
  reason_code text,
  incident_reference text,
  requested_by text not null,
  approved_by text,
  assigned_to text,
  owner_group text,
  root_cause_code text,
  preventive_action text,
  created_at timestamptz not null,
  completed_at timestamptz,
  updated_at timestamptz not null
);

Repair change:

create table data_repair_change (
  id uuid primary key,
  repair_case_id uuid not null references data_repair_case(id),
  entity_type text not null,
  entity_id uuid not null,
  field_path text,
  old_value text,
  new_value text,
  value_type text,
  before_snapshot_id uuid,
  after_snapshot_id uuid,
  applied_at timestamptz,
  applied_by text
);

Break-glass session:

create table break_glass_session (
  id uuid primary key,
  actor_id text not null,
  scope_type text not null,
  scope_id uuid,
  reason_code text not null,
  incident_reference text,
  approved_by text,
  status text not null,
  started_at timestamptz not null,
  expires_at timestamptz not null,
  ended_at timestamptz,
  created_at timestamptz not null
);

Operational command:

create table operational_command (
  id uuid primary key,
  command_type text not null,
  target_type text,
  target_id uuid,
  status text not null,
  requested_by text not null,
  approved_by text,
  executed_by text,
  reason_code text,
  correlation_id text,
  created_at timestamptz not null,
  completed_at timestamptz,
  error_code text,
  error_message text
);

Indexes:

create index idx_repair_case_status_severity
on data_repair_case (status, severity, created_at);

create index idx_repair_case_target
on data_repair_case (target_type, target_id);

create index idx_repair_change_entity
on data_repair_change (entity_type, entity_id);

create index idx_break_glass_actor_time
on break_glass_session (actor_id, started_at desc);

create index idx_operational_command_target
on operational_command (target_type, target_id, created_at desc);

22. Java/JAX-RS Backend Implications

Internal/support APIs:

POST /internal/data-repair-cases
POST /internal/data-repair-cases/{id}/approve
POST /internal/data-repair-cases/{id}/dry-run
POST /internal/data-repair-cases/{id}/apply
POST /internal/data-repair-cases/{id}/verify
POST /internal/break-glass-sessions
POST /internal/operational-commands

Service rules:

• enforce authority,
• require reason,
• require approval for high-risk repair,
• capture before snapshot,
• apply via domain command if possible,
• write audit,
• publish events if business state changed,
• run validation/reconciliation after repair.

23. MyBatis/JPA/JDBC Implications

Manual repairs should avoid unmanaged SQL when possible.

If SQL is necessary:

• script reviewed,
• transaction bounded,
• before/after captured,
• dry run count,
• rollback/forward plan,
• run in maintenance window if needed,
• audit repair case,
• verify after.

JDBC/MyBatis may be appropriate for controlled bulk repair. JPA may be safer for domain invariants in small repairs.

Choose based on risk and invariant needs.

24. Security and Privacy

Manual operations are privileged.

Controls:

• least privilege,
• just-in-time access,
• break-glass expiry,
• approval,
• session recording if applicable,
• audit,
• sensitive data masking,
• export/download restriction,
• post-use review,
• anomaly detection.

Break-glass can become insider risk if not monitored.

25. Observability

Monitor:

• open repair cases by severity,
• repair SLA breaches,
• break-glass sessions active,
• break-glass sessions expired but not closed,
• operational command failures,
• repeated repairs for same root cause,
• unverified repairs,
• repair without linked incident/reason,
• high-risk repair without approval.

Example:

-- Active break-glass sessions past expiry
select id, actor_id, scope_type, scope_id, expires_at
from break_glass_session
where status = 'ACTIVE'
  and expires_at < now();

-- Repairs applied but not verified
select id, repair_number, repair_type, target_type, target_id
from data_repair_case
where status = 'APPLIED'
  and completed_at is null;

26. Failure Modes

27. PR Review Checklist

When reviewing manual operation features, ask:

• Is this a domain command or raw mutation?
• What authority is required?
• Is approval/maker-checker needed?
• Is reason code required?
• Is incident/reference required?
• Is before/after snapshot captured?
• Is dry run supported?
• Is repair idempotent?
• Are immutable records protected?
• Is audit written?
• Are events emitted if state changes?
• Is validation/reconciliation run after repair?
• Is rollback or forward-fix strategy defined?
• Are sensitive evidence fields protected?
• Is break-glass time-bound and audited?

28. Internal Verification Checklist

Verify these in the internal CSG/team context:

• Existing production support/manual operation process.
• Whether support tools use domain APIs or direct SQL.
• Whether break-glass access exists and how it is audited.
• Whether repair cases are tracked.
• Whether high-risk repairs require approval.
• Whether before/after snapshots are captured.
• Whether manual billing/product/order corrections are allowed.
• Whether data repair links to incident/root cause.
• Whether repair verification/reconciliation is required.
• Whether operational commands for replay/retry exist.
• Whether incidents mention silent manual fix, missing audit, repeated repair, or break-glass governance issue.

29. Summary

Manual operations are unavoidable, but they must be modelled.

A strong model must define:

• repair case,
• operational command,
• break-glass session,
• reason/evidence,
• approval/maker-checker,
• before/after snapshot,
• dry run,
• idempotent retry/replay,
• immutable record correction strategy,
• verification,
• rollback/forward fix,
• root cause,
• preventive action,
• security and audit.

The key principle:

The safest production repair is not the fastest SQL update. It is a controlled, auditable domain operation with evidence, approval, validation, and verification.