Part 006 — Authoritative Data, Snapshots, Projections, and Cross-System Lineage

Positioning

Enterprise Quote-to-Order systems copy, transform, enrich, and project data across many boundaries.

Tanpa explicit ownership, organisasi akan memiliki beberapa “truth” untuk:

• product;
• customer;
• price;
• quote;
• order;
• inventory;
• and billing.

Masalahnya bukan bahwa data diduplikasi.

Masalahnya adalah tidak ada yang tahu:

• copy mana authoritative;
• copy mana historical snapshot;
• copy mana projection;
• dan bagaimana divergence diselesaikan.

Core thesis: setiap business fact harus memiliki authoritative owner, provenance, temporal semantics, dan correction policy. Copying data is acceptable; ambiguous ownership is not.

1. Source of Truth

A source of truth is the authoritative owner for a business fact.

It answers:

Who defines the meaning?
Who may change it?
Which version is current?
How is history preserved?
How are conflicts resolved?

2. System of Record

System of Record often means the system responsible for official persistence.

It may differ from:

• computation owner;
• interaction system;
• analytics platform;
• and historical archive.

Use terms carefully.

3. Authority Is Per Fact

One entity can contain facts owned by different systems.

Example Quote:

• customer legal name -> customer master;
• selected offering -> quote snapshot from catalog;
• calculated price -> pricing/quote;
• approval decision -> approval domain;
• accepted time -> quote/agreement.

Do not assign authority only at table level.

4. Data Ownership Dimensions

For each fact, define:

• semantic owner;
• write owner;
• lifecycle owner;
• operational owner;
• stewardship owner;
• and access owner.

These roles may differ.

5. Semantic Owner

Defines meaning and business policy.

Example:

• Catalog owner defines ProductOffering semantics.

6. Write Owner

System allowed to mutate authoritative value.

Example:

• Quote domain writes quote state.

7. Lifecycle Owner

Controls creation, transition, retirement, and correction.

8. Operational Owner

Responsible for runtime health and incident response.

9. Data Steward

Responsible for:

• quality;
• governance;
• and correction process.

10. Access Owner

Controls:

• visibility;
• privacy;
• and authorization.

11. Source-of-Truth Matrix

Validate actual internal ownership.

12. Copying Is Not Ownership

A service may store a field for:

• performance;
• audit;
• resilience;
• and presentation.

That does not grant authority to modify its meaning.

13. Reference

A reference points to authoritative data.

Example:

productOfferingId = OFFER-123
version = 7

Benefit:

• less duplication.

Risk:

• future reads may not reproduce historical state.

14. Snapshot

A snapshot copies data at a specific time.

Example:

• offering name;
• selected characteristics;
• price breakdown;
• terms;
• customer display information.

Benefit:

• historical reproducibility.

Risk:

• stale data;
• correction complexity.

15. Projection

A projection is derived for query or integration.

Examples:

• customer order timeline;
• quote summary index;
• support dashboard;
• analytics table.

Projection should be rebuildable where possible.

16. Cache

Cache improves access.

It should have:

• source;
• key;
• freshness;
• invalidation;
• and fallback.

A cache should not silently accept business writes.

17. Replica

A replica copies authoritative storage for:

• availability;
• read scaling;
• or disaster recovery.

Replication lag affects read semantics.

18. Materialized View

A materialized view provides a precomputed read model.

It may combine facts from multiple owners.

It does not become owner of those facts.

19. Derived Fact

A derived fact is calculated from authoritative inputs.

Example:

• quote total;
• margin;
• order progress percentage.

Need:

• calculation definition;
• version;
• and refresh policy.

20. Historical Fact

Historical facts should not be overwritten by current master data.

Example:

• customer legal name on accepted agreement may need historical preservation.

Correction policy may still exist.

21. Master Data

Common master data:

• customer;
• product;
• geography;
• currency;
• organization;
• and reference codes.

Master data is not necessarily transaction truth.

22. Transaction Data

Examples:

• quote;
• order;
• approval;
• acceptance;
• fulfillment event;
• invoice.

Transaction data needs immutable history and lineage.

23. Reference Data

Examples:

• currency codes;
• country codes;
• reason codes;
• units.

Reference data can still be versioned and governed.

24. Event as Data Source

An event stream may be:

• integration contract;
• audit source;
• or authoritative history.

Clarify whether events can reconstruct state.

25. Database as Data Source

Database persistence alone does not define semantic ownership.

A shared database can obscure which service owns a fact.

26. API as Authority Boundary

An API can expose authoritative operations.

But if another service writes the same table directly, the boundary is false.

27. Shared Database Risk

Risks:

• hidden consumers;
• unauthorized writes;
• schema coupling;
• and ambiguous correction.

28. Read-Only Shared Access

Read-only access still creates:

• schema dependency;
• bypassed semantics;
• and migration risk.

Prefer stable APIs/events/views where appropriate.

29. Data Lineage

Lineage explains how a fact moved and transformed.

Example:

Catalog Offering v7
-> Quote Item revision 3
-> Accepted Quote revision 3
-> Product Order Item
-> Service Order Item
-> Inventory Product
-> Billing Charge

30. Lineage Dimensions

Track:

• source entity;
• source version;
• target entity;
• transformation;
• time;
• actor/process;
• and correlation.

31. Business Lineage

Business lineage explains commercial continuity.

Example:

• this billed charge came from this accepted quote item.

32. Technical Lineage

Technical lineage explains data processing.

Example:

• event A produced projection B through processor version C.

33. Document Lineage

Generated proposal should link to:

• quote ID;
• revision;
• template version;
• generation time;
• and checksum where needed.

34. Price Lineage

A quoted price should preserve:

• price definition;
• rule version;
• adjustment;
• context;
• currency;
• and calculation time.

35. Approval Lineage

Approval should link to:

• quote revision;
• scope;
• triggering condition;
• policy version;
• and approver authority.

36. Order Lineage

Order item should link to:

• accepted quote item;
• transformation version;
• and source revision.

37. Fulfillment Lineage

Child service/resource orders should link to:

• source product order item;
• decomposition rule;
• and parent process.

38. Inventory Lineage

Installed product should link to:

• order item;
• fulfillment result;
• and actual technical identifiers.

39. Billing Lineage

Billing charge should link to:

• commercial charge;
• order/inventory trigger;
• effective date;
• and billing account.

40. Lineage Key

A lineage key should be stable.

Possible identifiers:

• correlation ID;
• quote item ID;
• order item ID;
• external reference;
• decomposition key.

One generic correlation ID is not enough for all semantics.

41. Correlation versus Causation

Correlation ID

Groups related processing.

Causation ID

Identifies which command/event caused another event.

Aggregate ID

Identifies domain entity.

Use all where valuable.

42. Provenance

Provenance describes where data came from and why it exists.

Possible fields:

• source system;
• source entity;
• source version;
• captured time;
• actor;
• calculation/rule version.

43. Provenance versus Audit

Provenance

Origin and transformation.

Audit

Who changed what and when.

Related but distinct.

44. Snapshot Provenance

A snapshot should include:

sourceId
sourceVersion
capturedAt
sourceSystem
snapshotSchemaVersion

45. Snapshot Granularity

Possible approaches:

• full object;
• selected commercial fields;
• immutable document;
• event payload;
• normalized snapshot tables.

Choose based on reproduction need.

46. Deep Copy Risk

Copying an entire external model creates:

• large payload;
• irrelevant fields;
• privacy risk;
• and compatibility burden.

Snapshot only what is required, with provenance.

47. Reference-Only Risk

Reference-only quote can become unreproducible when:

• catalog changes;
• customer name changes;
• pricing rule retires;
• and external data is deleted.

48. Hybrid Reference + Snapshot

Common pattern:

• keep stable source ID/version;
• copy display and commercially relevant facts.

This supports both lineage and history.

49. Mutable Snapshot Smell

If a quote snapshot is updated automatically whenever catalog changes, it is not a true historical snapshot.

50. Staleness

A copy is stale when source changes.

Staleness is not always a defect.

Historical snapshot is intentionally stale.

Projection may be temporarily stale.

Cache should have freshness policy.

51. Staleness Classification

Intended historical staleness

Accepted quote snapshot.

Acceptable eventual staleness

Support projection.

Unsafe staleness

Eligibility or price reused beyond validity.

52. Freshness Policy

Define:

• maximum age;
• invalidation event;
• refresh trigger;
• and behavior when source unavailable.

53. Data Validity

Data validity may depend on:

• effective period;
• version;
• approval;
• and context.

Fresh data can still be invalid for the current transaction.

54. Reconciliation

Reconciliation compares expected and actual state.

Examples:

• quote price versus order price;
• order completion versus inventory;
• inventory activation versus billing;
• event count versus projection.

55. Reconciliation Types

• point-in-time;
• continuous;
• scheduled batch;
• event-driven;
• manual;
• and exception-based.

56. Reconciliation Record

Should include:

• compared entities;
• expected;
• actual;
• difference;
• severity;
• owner;
• and resolution.

57. Conflict

A conflict occurs when multiple systems claim different values.

Need policy:

• authority wins;
• latest valid version;
• manual adjudication;
• merge;
• or compensation.

58. Last-Write-Wins Risk

Last-write-wins ignores:

• business authority;
• clock skew;
• and semantic validity.

Avoid for critical commercial data.

59. Correction

A correction changes authoritative data due to error.

It should preserve:

• previous value;
• reason;
• actor;
• time;
• and downstream impact.

60. Correction versus Amendment

Correction

Fixes incorrect recorded fact.

Amendment

Creates a new valid business agreement/change.

Do not use correction to bypass commercial process.

61. Backdated Correction

A backdated correction affects temporal history.

Need:

• effective time;
• processing time;
• impacted transactions;
• and recalculation policy.

62. Data Repair

Operational repair may:

• rebuild projection;
• replay event;
• correct orphan link;
• or re-run synchronization.

Use explicit tools and audit.

63. Direct Database Repair Risk

Risks:

• bypassed invariants;
• missing events;
• stale caches;
• and untracked downstream effects.

64. Rebuildable Projection

A projection is safer when it can be rebuilt from authoritative history.

Need:

• deterministic processor;
• versioning;
• and replay strategy.

65. Projection Drift

Projection drift occurs when read model differs from source.

Detect through:

• checksums;
• counts;
• version markers;
• and reconciliation.

66. Eventual Consistency

Eventual consistency should define:

• expected delay;
• user-visible behavior;
• and recovery.

“Eventually” without bound is not an operational policy.

67. Consistency Contract

Example:

Product-order status projection may lag authoritative order state by up to 30 seconds under normal conditions.

Internal targets must be verified.

68. Read-Your-Writes

After creating quote/order, user may expect immediate visibility.

Options:

• return authoritative result;
• query command side;
• session overlay;
• or wait for projection.

69. Monotonic Read

User should not see progress move backward due to replica lag.

Need routing or version awareness.

70. Version Token

A response may include version or event position.

Clients can use it to detect stale projection.

71. Data Ownership and Microservices

Each service should ideally own:

• persistence;
• semantics;
• and mutation of its domain state.

Shared libraries do not create shared ownership.

72. Data Ownership and Teams

Ownership includes:

• code;
• schema;
• contracts;
• quality;
• incidents;
• and correction.

A database owned by platform team but semantics owned elsewhere needs clear collaboration.

73. Data Product Thinking

A domain may publish reliable data products with:

• contract;
• quality;
• owner;
• freshness;
• and documentation.

Useful for analytics and integration.

74. Analytics Boundary

Analytics should consume governed data.

It should not become operational authority because dashboard looks more complete.

75. Data Warehouse

Warehouse can integrate lifecycle facts.

But warehouse latency and transformations mean it is usually not command authority.

76. Reporting Projection

Reports may denormalize:

• quote;
• order;
• customer;
• and billing.

Keep lineage back to authoritative sources.

77. Search Index

Search index is a projection.

It can be rebuilt.

Never update domain state by editing index.

78. Customer Portal View

Portal may combine:

• quote;
• order;
• fulfillment;
• inventory;
• billing.

This is a composite projection, not one aggregate.

79. API Composition

A BFF or API composer may join data.

Need:

• freshness;
• partial failure;
• source attribution;
• and authorization.

80. Canonical Data Hub Risk

Centralizing all data into a hub may create:

• second source of truth;
• delayed semantics;
• and write-back ambiguity.

81. Master Data Synchronization

For customer/product master:

• define change events;
• version;
• deletion/tombstone;
• and conflict.

82. Deletion

Deletion policy depends on:

• legal retention;
• historical evidence;
• privacy;
• and referential integrity.

Hard delete may be unsafe for commercial records.

83. Tombstone

A tombstone indicates entity no longer active/available.

Useful for event consumers and caches.

84. Redaction

Privacy may require redaction while preserving:

• transaction lineage;
• audit;
• and legal evidence.

85. Data Retention

Define retention for:

• drafts;
• quotes;
• approvals;
• agreements;
• orders;
• events;
• logs;
• and idempotency records.

86. Archival

Archived data should remain:

• discoverable;
• explainable;
• and linked.

Archive is not data loss.

87. Schema Evolution

Schema changes need:

• backward compatibility;
• defaults;
• migration;
• and consumer handling.

88. Semantic Evolution

Even without schema change, meaning may change.

Example:

• completed used to mean technical completion;
• now means customer acceptance.

This is breaking.

89. Data Contract

A data contract should define:

• owner;
• schema;
• semantics;
• quality;
• freshness;
• versioning;
• and deprecation.

90. Contract Tests

Test:

• structure;
• required fields;
• enum handling;
• semantic examples;
• and version compatibility.

91. Data Quality Dimensions

• completeness;
• accuracy;
• consistency;
• timeliness;
• uniqueness;
• validity;
• and traceability.

92. Data Quality Rule

Example:

Every accepted quote item has a source offering ID, source version, and accepted price breakdown.

93. Data Quality Alert

Alert should include:

• violated rule;
• affected entities;
• severity;
• owner;
• and remediation.

94. Duplicate Data

Duplicate can mean:

• accidental duplicate entity;
• legitimate snapshot;
• replica;
• or repeated event.

Classify before deleting.

95. Identity Resolution

Multiple systems may refer to same party/product.

Need:

• matching policy;
• namespace;
• and confidence.

Do not merge on name only.

96. Cross-Tenant Data

Tenant boundaries must apply to:

• authoritative data;
• snapshots;
• projections;
• logs;
• search;
• and analytics.

97. Tenant Leakage via Projection

A secure source can still leak through:

• shared cache key;
• search index;
• report;
• or support dashboard.

98. Field Sensitivity

Classify fields:

• public;
• customer-visible;
• internal;
• confidential;
• regulated.

Apply through all copies.

99. Price and Margin Sensitivity

Quoted price may be customer-visible.

Cost and margin may be restricted.

Do not snapshot all pricing internals into customer-facing payload.

100. Data Lineage Diagram

Each arrow should identify transformation and owner.

101. Data Ownership Smells

• multiple writers;
• no correction owner;
• read model edited manually;
• snapshot refreshed silently;
• external ID reused without namespace;
• and current master data overwrites history.

102. Lineage Smells

• order item cannot identify quote item;
• billed charge cannot identify commercial price;
• inventory product cannot identify fulfillment result;
• generated document lacks revision;
• and manual repair breaks links.

103. Projection Smells

• projection used for command validation;
• stale state not visible;
• no rebuild;
• no source version;
• and no ownership.

104. Reconciliation Smells

• batch produces spreadsheet only;
• mismatch has no owner;
• same mismatch repeats;
• no severity;
• and correction bypasses domain commands.

105. Data Governance Smells

• glossary separate from schemas;
• retention unknown;
• sensitivity missing;
• and contract changes unannounced.

106. Source-of-Truth Workshop

Participants:

• domain owners;
• engineering;
• data;
• support;
• operations;
• integration;
• and security.

Outputs:

• business-fact matrix;
• ownership;
• copy classification;
• lineage;
• correction policy;
• and gaps.

107. Business Fact Inventory Template

## Fact

## Meaning

## Authoritative Owner

## Write Owner

## Lifecycle Owner

## Copies

## Snapshot / Projection / Cache

## Freshness

## Correction

## Retention

## Sensitivity

108. Lineage Record Template

Source context:
Source entity:
Source ID/version:
Transformation:
Target context:
Target entity:
Target ID/version:
Occurred at:
Processor/rule version:
Correlation:

109. Reconciliation Rule Template

Expected source:
Actual source:
Join key:
Comparison:
Tolerance:
Frequency:
Severity:
Owner:
Repair:

110. Correction Command Template

Command:
Target fact:
Current value:
Corrected value:
Reason:
Evidence:
Effective time:
Actor:
Authority:
Downstream actions:
Audit:

111. Worked Example: Catalog to Quote

Catalog owns:

• offering definition.

Quote stores:

• offering ID/version;
• selected characteristic snapshot;
• display name;
• commercial terms.

Catalog change does not mutate accepted quote.

112. Worked Example: Customer Name Change

Customer master changes legal name.

Open draft quote may refresh.

Accepted agreement may preserve historical name and reference current party.

Policy must distinguish display correction from legal amendment.

113. Worked Example: Price Reproduction

To reproduce quote price, preserve:

• catalog version;
• price rule version;
• customer context;
• quantity;
• term;
• adjustment;
• currency;
• and rounding policy.

Total alone is insufficient.

114. Worked Example: Order Projection Lag

Order accepted at 10:00:00.

Portal projection updates at 10:00:15.

Immediately after command, UI may:

• use command result;
• show pending synchronization;
• or poll by version token.

115. Worked Example: Inventory Mismatch

Order completed.

No inventory product exists.

Reconciliation detects:

• missing inventory;
• order item reference;
• fulfillment completion;
• severity;
• owner.

Repair command creates or replays inventory update with audit.

116. Worked Example: Billing Leakage

Inventory active.

No billing charge.

Lineage identifies:

• order item;
• commercial charge;
• activation event;
• missing billing acknowledgement.

Revenue assurance can act without manual cross-database guessing.

117. Worked Example: Shared Database

Quote and Order services write the same customer snapshot table.

Ambiguity:

• who corrects?
• which schema version?
• which lifecycle?

Better:

• context-owned snapshot;
• shared customer reference;
• explicit event/API.

118. Worked Example: Data Correction

Support finds wrong site code on order.

Do not update order row directly.

Use correction flow:

• validate authority;
• preserve old value;
• assess fulfillment impact;
• publish event;
• and reconcile downstream.

119. Senior Engineer Operating Model

Identify facts, not tables

Ask what meaning is owned.

Classify every copy

Reference, snapshot, projection, cache, or replica.

Preserve provenance

Source, version, time, transformation.

Design correction

Before production incident.

Make consistency explicit

Freshness and lag are product behavior.

Enable reconciliation

Expected versus actual.

Protect history

Do not refresh accepted snapshots silently.

Secure every copy

Tenant and field sensitivity.

120. Internal Verification Checklist

Authority

• Who owns catalog facts?
• Who owns quoted price?
• Who owns acceptance?
• Who owns order intent?
• Who owns installed product?
• Who owns billed charge?

Copies

• Which fields are references?
• Which are snapshots?
• Which are projections?
• Which are caches?
• Are they labeled explicitly?

Lineage

• Can quote item map to order item?
• Can order item map to fulfillment?
• Can inventory map to order?
• Can billed charge map to quoted charge?
• Are transformation versions stored?

Freshness

• What lag is expected?
• How is stale data shown?
• What invalidates cache?
• What is intentionally historical?

Correction

• Who can correct each fact?
• Are explicit repair commands available?
• Are backdated corrections supported?
• How are downstream copies updated?

Reconciliation

• What reconciliation jobs exist?
• Who owns mismatches?
• What are severity thresholds?
• Are recurring mismatches tracked?

Security

• Is tenant scope preserved?
• Do projections enforce authorization?
• Are cost/margin fields protected?
• What retention and redaction policies apply?

121. Practical Exercises

Exercise 1 — Fact ownership

List 30 business facts and assign authority.

Exercise 2 — Copy classification

Classify every duplicate field as reference, snapshot, projection, cache, or replica.

Exercise 3 — End-to-end lineage

Trace one quote item to billing charge.

Exercise 4 — Reconciliation design

Create a rule for order-to-inventory mismatch.

Exercise 5 — Correction workflow

Replace one direct data fix with an audited command.

Exercise 6 — Projection lag

Design UI behavior for read-model delay.

122. Part Completion Checklist

You are done if you can:

• define source of truth per business fact;
• distinguish authority from storage;
• classify copies correctly;
• design snapshots with provenance;
• preserve quote-to-billing lineage;
• define freshness and consistency contracts;
• design reconciliation;
• distinguish correction from amendment;
• protect tenant and field sensitivity across projections;
• and create an internal data-ownership verification backlog.

123. Key Takeaways

1. Authority is assigned per business fact.
2. Storage does not automatically imply ownership.
3. References, snapshots, projections, caches, and replicas have different semantics.
4. Historical snapshots may be intentionally stale.
5. Provenance explains origin; audit explains change.
6. Lineage preserves commercial continuity.
7. Eventual consistency needs a bounded expectation.
8. Reconciliation is a production capability.
9. Direct database repair is risky.
10. Internal ownership and correction policy must be verified.

124. References

Conceptual baseline:

• Domain-Driven Design ownership, bounded contexts, aggregates, and anti-corruption layers.
• General enterprise master-data, lineage, data-contract, reconciliation, and temporal-data practices.
• Distributed systems projections, caches, replicas, event streams, and eventual consistency.
• CPQ, Quote-to-Order, Inventory, and Billing lifecycle concepts.

These references do not define internal CSG data ownership or persistence architecture.