Observability, Operational Metadata, and Incident Correlation Model

1. Core Idea

Enterprise data model harus membantu engineer dan support menjawab:

Apa yang terjadi, kapan, oleh siapa/apa, di service mana, dengan correlation ID apa, event mana yang dikirim, downstream mana yang gagal, dan business object mana yang terdampak?

Observability bukan hanya metrics/logs/traces. Untuk CPQ / Quote / Order / Billing / Telco BSS/OSS, observability juga membutuhkan operational metadata yang tersimpan dalam data model.

Mental model:

Logs explain runtime. Traces explain request path. Metrics explain aggregate health. Operational metadata explains business process state and recovery evidence.

Tanpa metadata ini, incident review hanya menjadi tebak-tebakan dari log yang sudah rotate, dashboard yang tidak punya business context, atau ticket support yang tidak lengkap.

2. Why Observability Data Modelling Matters

Quote-to-cash flow panjang dan distributed:

Quote accepted
  -> approval completed
  -> order created
  -> order decomposed
  -> fulfillment started
  -> service activated
  -> product inventory updated
  -> billing charge activated
  -> invoice generated

Ketika flow gagal, engineer perlu tahu:

• apakah command diterima?
• apakah DB transaction commit?
• apakah outbox event dibuat?
• apakah event published?
• apakah consumer menerima?
• apakah downstream API dipanggil?
• apakah response downstream sukses?
• apakah retry sedang berjalan?
• apakah task stuck?
• apakah compensation dibuat?
• apakah customer terkena impact?

Jika data model tidak menyimpan operational metadata, debugging akan bergantung pada best-effort logs.

3. Operational Metadata vs Business Data

Business data:

quote.status = ACCEPTED
order.status = IN_PROGRESS
product_instance.status = ACTIVE

Operational metadata:

correlation_id
last_processed_event_id
workflow_instance_id
external_request_id
retry_count
last_error_code
last_error_at
processing_owner
last_reconciled_at
projection_freshness_status

Operational metadata tidak menggantikan business data. Ia menjelaskan bagaimana data berubah dan bagaimana memulihkan proses.

4. Correlation ID

Correlation ID mengikat seluruh flow.

Example:

X-Correlation-Id: corr-20260712-quote-123

Dipakai di:

• API request,
• command audit,
• domain event,
• outbox row,
• inbox row,
• workflow process,
• integration message,
• audit event,
• logs,
• metrics labels where safe,
• support timeline.

Fields:

correlation_id
causation_id
request_id
trace_id
span_id

Correlation ID harus dipropagate, bukan digenerate ulang di setiap service kecuali belum ada.

5. Causation ID

Causation ID menjawab:

Record ini disebabkan oleh command/event apa?

Example:

AcceptQuoteCommand command_id = CMD-1
QuoteAccepted event causation_id = CMD-1
QuoteConversionRequested event causation_id = QuoteAccepted.event_id
ProductOrderCreated event causation_id = QuoteConversionRequested.event_id

Correlation groups whole flow.

Causation forms chain.

Both are useful.

6. Business Key

Business key membantu manusia mencari flow.

Examples:

quote_number
order_number
customer_number
billing_account_number
invoice_number
product_instance_number
workflow_business_key
external_order_reference

Operational metadata should carry business key where useful.

Example:

integration_message.business_entity_type = ORDER
integration_message.business_entity_id = order-id
integration_message.business_entity_number = O-10001

Support biasanya punya order number, bukan UUID internal.

7. Processing Status

Async processing should have explicit status.

Examples:

PENDING
PROCESSING
SENT
ACKNOWLEDGED
COMPLETED
FAILED
RETRY_PENDING
DEAD_LETTERED
MANUAL_INTERVENTION
CANCELLED

Status should be domain-specific enough to be actionable.

Bad:

status = ERROR

Better:

status = RETRY_PENDING
last_error_code = DOWNSTREAM_TIMEOUT
next_retry_at = ...
owner_group = integration-ops

8. Operational Error Model

Operational error should store structured data.

Fields:

error_code
error_category
error_message
error_detail_reference
retryable
failure_stage
failed_at
failed_by_component
downstream_system
http_status
external_error_code

Categories:

• validation,
• authorization,
• state conflict,
• downstream timeout,
• downstream rejection,
• schema mismatch,
• duplicate,
• transient infrastructure,
• permanent business rule,
• data quality mismatch,
• unknown.

Structured error codes allow dashboards and automated routing.

9. Retry Metadata

Retry metadata:

retry_count
max_retry_count
next_retry_at
last_retry_at
retry_policy
last_error_code
last_error_message

This belongs in:

• outbox,
• inbox,
• integration message,
• fulfillment task,
• saga step,
• rating job,
• data quality repair,
• external sync task.

Retry without persisted metadata is invisible and hard to recover.

10. Dead Letter Metadata

Dead-letter should not be a black hole.

Fields:

dead_letter_reason
dead_lettered_at
dead_letter_queue
owner_group
repair_status
replay_allowed
replay_attempt_count
last_replayed_at

Dead letter item should link to:

• business entity,
• source event/message,
• correlation ID,
• payload reference/hash,
• failure reason,
• repair case if any.

11. Workflow Metadata

If workflow engine exists, domain tables should store workflow references.

Fields:

workflow_engine
workflow_definition_key
workflow_definition_version
workflow_instance_id
workflow_business_key
workflow_task_id
workflow_status
workflow_started_at
workflow_completed_at

Do not make workflow variables the only source of business state.

Domain state and workflow state must be reconcilable.

12. External System Metadata

Integration with external systems needs trace.

Fields:

external_system
external_request_id
external_correlation_id
external_entity_id
external_status
external_status_checked_at
last_sync_at
last_acknowledged_at
payload_reference
response_reference

Example:

order_id = local O-10001
external_order_id = billing-system-order-7788

Support and reconciliation depend on this mapping.

13. Support Timeline Model

Support timeline presents business and operational events together.

Timeline item fields:

timeline_item
- id
- subject_type
- subject_id
- occurred_at
- item_type
- title
- summary
- actor_display
- source_system
- source_reference_type
- source_reference_id
- severity
- correlation_id

Timeline can combine:

• audit event,
• status history,
• approval decision,
• saga step,
• outbox/inbox failure,
• fulfillment fallout,
• billing trigger,
• product inventory update,
• data quality issue.

Timeline is a projection. Source evidence remains in domain/audit tables.

14. Incident Correlation

Incident data model links business objects to incidents.

Fields:

incident_reference
- id
- incident_number
- incident_system
- severity
- status
- started_at
- resolved_at
- summary

Business impact link:

incident_business_impact
- incident_id
- entity_type
- entity_id
- customer_id
- account_id
- impact_type
- impact_severity
- detected_at

Impact types:

• quote blocked,
• order stuck,
• billing delayed,
• duplicate billing,
• product not activated,
• product active not billed,
• data leak,
• reporting incorrect.

15. Runbook Metadata

For operational issues, link rule/failure to runbook.

Fields:

runbook_reference
- issue_code
- owner_group
- runbook_url
- escalation_policy
- repair_command
- dashboard_url

Example:

issue_code = OUTBOX_STUCK
owner_group = platform
runbook = ...

A data issue without runbook becomes tribal knowledge.

16. Operational Ownership

Each operational status/error should have owner.

Fields:

owner_group
assigned_to
escalation_group
support_level
oncall_service

Examples:

• pricing error -> pricing service owner,
• billing acknowledgement missing -> billing integration owner,
• fulfillment fallout -> fulfillment ops,
• product-charge mismatch -> billing ops,
• projection lag -> data platform,
• permission cache issue -> IAM/platform.

Owner metadata enables action.

17. Health State Model

For important processes, store health.

process_health
- process_name
- entity_type
- entity_id
- health_status
- last_success_at
- last_failure_at
- failure_code
- owner_group
- checked_at

Health statuses:

HEALTHY
DEGRADED
FAILED
STALE
UNKNOWN

Examples:

• quote conversion health,
• order fulfillment health,
• billing trigger health,
• product-charge reconciliation health,
• projection health.

18. Watermark and Checkpoint

Batch/stream processing needs checkpoint.

Fields:

checkpoint
- process_name
- source_name
- last_processed_id
- last_processed_time
- last_event_id
- last_offset
- status
- updated_at

This supports:

• resume after failure,
• lag monitoring,
• replay,
• backfill,
• projection rebuild,
• incident review.

19. Operational Metadata in Core Tables

Some operational metadata belongs in core tables.

Example order:

correlation_id
source_system
external_reference
last_reconciled_at
last_event_id
processing_status

But avoid bloating core table with every technical detail.

Detailed operational history can live in separate tables:

order_processing_attempt
integration_message
saga_step
data_quality_result

20. Observability and Metrics Dimensions

Metrics should include useful dimensions, but avoid high-cardinality explosion.

Good dimensions:

• service,
• operation,
• status,
• error_code,
• domain_area,
• owner_group,
• event_type,
• queue_name.

Risky dimensions:

• customer_id,
• quote_id,
• order_id,
• full correlation_id,
• raw external ID.

High-cardinality values are better in logs/traces/tables, not broad metrics labels.

21. PostgreSQL Physical Design

Operational event table:

create table operational_event (
  id uuid primary key,
  event_type text not null,
  severity text not null,
  entity_type text,
  entity_id uuid,
  entity_number text,
  source_system text,
  component text,
  status text,
  error_code text,
  error_message text,
  retryable boolean,
  owner_group text,
  correlation_id text,
  causation_id text,
  occurred_at timestamptz not null,
  metadata jsonb
);

Process checkpoint:

create table process_checkpoint (
  process_name text primary key,
  source_name text not null,
  last_processed_id text,
  last_processed_time timestamptz,
  last_event_id uuid,
  last_offset text,
  status text not null,
  error_code text,
  error_message text,
  updated_at timestamptz not null
);

Support timeline projection:

create table support_timeline_item (
  id uuid primary key,
  subject_type text not null,
  subject_id uuid not null,
  occurred_at timestamptz not null,
  item_type text not null,
  title text not null,
  summary text,
  actor_display text,
  source_system text,
  source_reference_type text,
  source_reference_id uuid,
  severity text,
  correlation_id text
);

Indexes:

create index idx_operational_event_entity_time
on operational_event (entity_type, entity_id, occurred_at desc);

create index idx_operational_event_correlation
on operational_event (correlation_id);

create index idx_operational_event_error_time
on operational_event (error_code, occurred_at desc);

create index idx_timeline_subject_time
on support_timeline_item (subject_type, subject_id, occurred_at desc);

22. Java/JAX-RS Backend Implications

Request filter responsibilities:

• read or generate correlation ID,
• attach to request context,
• add to response header,
• propagate to downstream calls,
• include in logs,
• include in outbox/audit.

Service responsibilities:

• persist correlation ID in command/audit/event,
• use structured error codes,
• write operational event for important failures,
• update process status/checkpoint,
• expose status endpoints.

Example endpoints:

GET /orders/{id}/timeline
GET /orders/{id}/operational-status
GET /correlations/{correlationId}
GET /operational-events?entityType=ORDER&entityId=...

Internal-only endpoints may expose deeper operational evidence.

23. Error Response and Correlation

Every error response should include correlation ID.

{
  "errorCode": "ORDER_STATE_CONFLICT",
  "message": "Order cannot be cancelled in its current state",
  "correlationId": "corr-123"
}

Support can ask customer for correlation ID, then trace API logs, audit, outbox, workflow, and operational events.

24. Observability for Events

For event-driven flows, observe:

• event created,
• event pending,
• event published,
• event consumed,
• event processed,
• event failed,
• event dead-lettered,
• projection updated.

Each stage should be traceable by:

event_id
event_type
aggregate_id
correlation_id
subscriber_name

Without this, "event lost" investigations become difficult.

25. Observability for External Integration

Track:

• outbound request created,
• sent timestamp,
• response timestamp,
• ack status,
• retry count,
• external error,
• external ID mapping,
• callback received,
• reconciliation status.

Example:

Integration to billing:
  local charge id
  external billing charge id
  message id
  ack status
  last sync time

This supports incident review and manual repair.

26. Observability for Data Quality

Data quality should produce operational metadata:

• rule code,
• severity,
• owner group,
• first detected,
• current age,
• repair status,
• root cause,
• linked incident,
• verification result.

Data quality without observability is just a hidden report.

27. Sensitive Observability Data

Observability data can leak sensitive content.

Avoid logging/persisting:

• raw payment data,
• secrets/tokens,
• full PII payload,
• margin/cost unless protected,
• full address in broad operational event,
• raw external payload in error message.

Store reference/hash for sensitive payload.

Operational metadata must follow data classification rules.

28. Incident Review Data Package

For serious incident, useful data package includes:

• affected business entities,
• timeline,
• command audit,
• status history,
• outbox/inbox records,
• integration messages,
• workflow state,
• data quality results,
• error codes,
• retry attempts,
• compensation actions,
• repair actions,
• customer impact,
• financial impact estimate.

Designing these models upfront makes postmortem factual.

29. Failure Modes

30. PR Review Checklist

When reviewing production-critical flow changes, ask:

• Is correlation ID propagated?
• Is causation ID captured for events?
• Is business key stored for support?
• Is operational status queryable?
• Are retry/error fields structured?
• Is downstream external reference stored?
• Is workflow/process ID linked?
• Is support timeline updated?
• Are operational events safe and not leaking sensitive data?
• Is owner group/runbook defined for failures?
• Are checkpoints/watermarks tracked?
• Are metrics/logs/traces aligned with entity IDs and correlation?
• Can incident review reconstruct what happened?
• Are high-cardinality metrics avoided?

31. Internal Verification Checklist

Verify these in the internal CSG/team context:

• Standard correlation ID header and propagation.
• Whether logs, audit, events, and integration messages share correlation ID.
• Whether causation ID/event chain is used.
• Whether support timeline exists.
• Whether workflow IDs are linked to domain entities.
• Whether outbox/inbox/integration message dashboards exist.
• Whether operational events are persisted or only logged.
• Whether dead-letter queues have owner/runbook.
• Whether data quality results link to incident/repair workflow.
• Whether sensitive data is redacted from logs/operational metadata.
• Whether support can search by quote/order/customer/correlation ID.
• Whether incidents mention missing traceability, missing owner, or unclear impact.

32. Summary

Observability data modelling makes production behavior explainable.

A strong model must define:

• correlation ID,
• causation ID,
• business key,
• operational status,
• structured error code,
• retry metadata,
• dead-letter metadata,
• workflow linkage,
• external system references,
• support timeline,
• incident impact,
• runbook/owner,
• checkpoint/watermark,
• sensitive observability controls.

The key principle:

If engineers and support cannot reconstruct a quote-to-cash flow from durable data, the system is not observable enough for enterprise production.