Data Contract, Schema Registry, and Compatibility Governance Model

1. Core Idea

Data contract adalah kesepakatan eksplisit antara producer dan consumer tentang bentuk, arti, kualitas, kompatibilitas, dan lifecycle data.

Dalam enterprise CPQ / Quote / Order / Billing / Telco BSS/OSS, data contract bisa berupa:

• REST API schema,
• event schema,
• Kafka/RabbitMQ message payload,
• file/feed layout,
• database view contract,
• analytics dataset,
• report metric definition,
• external integration payload,
• TM Forum API extension contract.

Mental model:

A schema says what fields exist. A data contract says what fields mean, who owns them, who consumes them, how they evolve, and what quality/compatibility guarantees exist.

2. Why Data Contracts Matter

Tanpa contract governance:

• producer menghapus field dan consumer production gagal,
• enum baru membuat client crash,
• field tetap ada tapi meaning berubah diam-diam,
• nullable menjadi required tanpa notice,
• event payload membawa PII baru tanpa review,
• analytics dataset berubah grain,
• API response berubah format date/time,
• external feed menambah column dan parser gagal,
• dashboard KPI berubah tanpa metric version,
• deprecation tidak pernah selesai,
• consumer tidak diketahui saat producer ingin migrate.

Data contract mengurangi surprise dalam distributed enterprise systems.

3. Contract Types

Each contract needs owner and compatibility policy.

4. Schema vs Semantics

Schema-level change:

field added
field removed
type changed
nullable changed
enum added

Semantic change:

status COMPLETED now means billing complete, not fulfillment complete
amount now includes tax
validUntil changed from inclusive to exclusive
createdAt timezone changed
quote total now excludes discounts

Semantic changes can break consumers even when schema is unchanged.

Data contract must document semantics.

5. Contract Registry

Contract registry stores metadata.

Fields:

data_contract
- id
- contract_name
- contract_type
- owner_group
- producer_system
- status
- current_version
- compatibility_policy
- data_classification
- documentation_url

Version:

data_contract_version
- id
- contract_id
- version
- schema_definition
- semantic_description
- created_at
- effective_from
- deprecated_at
- retired_at

Consumer registry:

data_contract_consumer
- contract_id
- consumer_system
- consumer_owner_group
- criticality
- version_used
- contact_channel

6. Schema Registry

Schema registry stores machine-readable schemas.

Common schema types:

• OpenAPI,
• JSON Schema,
• Avro,
• Protobuf,
• AsyncAPI,
• CSV schema,
• SQL view definition,
• table/dataset schema.

Schema registry should support:

• versioning,
• compatibility check,
• ownership,
• validation,
• CI integration,
• deprecation metadata,
• consumer discovery.

Actual tooling may vary. The modelling principle remains.

7. Compatibility Policies

Common policies:

Event streams often require backward compatibility because consumers deploy independently.

8. Breaking vs Non-Breaking Changes

Usually safe:

• add optional field,
• add field with default,
• add new event type,
• add new endpoint,
• add nullable column to dataset,
• add enum value if consumers tolerate unknown.

Usually breaking:

• remove field,
• rename field,
• change type,
• change format,
• change required/nullable,
• change field meaning,
• change enum meaning,
• change grain of dataset,
• reorder CSV columns if parser depends on order,
• change timezone/date convention,
• change money precision/scale.

Treat enum additions carefully. Some consumers crash on unknown enum.

9. Contract Versioning

Versioning strategies:

API

/v1/quotes
/v2/quotes

or media/header versioning.

Event

eventType = QuoteAccepted
eventVersion = 2

or topic per version.

Dataset

fact_invoice_line_v2
schema_version column

File

header record includes schema_version

Rules:

• version when contract changes,
• document migration path,
• maintain compatibility window,
• monitor consumer adoption,
• retire old version only when safe.

10. Producer-Consumer Registry

Producer must know consumers for impact analysis.

Consumer fields:

consumer_system
consumer_owner_group
consumer_contact
version_used
criticality
usage_type
last_seen_at
migration_status

Usage types:

• command processing,
• projection,
• analytics,
• notification,
• support dashboard,
• billing integration,
• compliance export.

Critical consumers need stronger coordination.

11. Consumer-Driven Contracts

Consumers can define expectations.

Example:

order-service expects QuoteAccepted:
  quoteId required
  quoteVersion required
  customerId required
  billingAccountId required
  status = ACCEPTED

Contract tests verify producer still satisfies consumer expectations.

This is powerful for microservices.

But contract expectations should be governed, not scattered randomly.

12. Contract Testing

Contract tests should cover:

• required fields,
• type/format,
• enum tolerance,
• optional field behavior,
• default values,
• date/time/money serialization,
• error response shape,
• event version,
• compatibility between old/new versions,
• sensitive field absence/presence rules.

Run in CI before producer deploy.

For event consumers, test with sample payloads for all supported versions.

13. Sample Payload Registry

Store example payloads.

Fields:

contract_sample
- contract_id
- version
- sample_name
- sample_payload
- scenario
- valid
- created_at

Examples:

• QuoteAccepted standard,
• QuoteAccepted with optional billing account,
• QuoteAccepted for amendment,
• OrderCreated with bundle,
• BillingAck failure,
• Usage file with late event.

Samples are useful for docs, tests, onboarding, and incident reproduction.

14. Data Quality Contract

Data contract should include quality expectations.

Examples:

quoteId: required
quoteVersion: required
billingAccountId: required if billable quote
currency: ISO 3-letter code
totalAmount: decimal string, scale 4
occurredAt: UTC ISO-8601

Dataset contract quality:

no duplicate invoice_line_id
amount not null
currency not null
ingested_at within SLA

Contract is incomplete without quality guarantees.

15. Freshness and SLA Contract

For datasets/projections:

refresh_frequency
freshness_sla
source_watermark
expected_latency
availability_sla

Example:

fact_invoice_line is refreshed daily by 04:00 UTC.
operational order projection freshness <= 2 minutes.

Consumers need to know if data is safe for operational decisions.

16. Security and Classification Contract

Contract should state:

• data classification,
• PII fields,
• restricted fields,
• allowed consumers,
• masking requirements,
• export allowed,
• retention requirement,
• encryption/tokenization requirement.

Example:

QuoteApproved event must not contain margin/cost.
Invoice dataset contains billing data and is finance-restricted.

Contract review should include security/privacy.

17. Deprecation Model

Fields:

deprecated_at
replacement_contract
replacement_field
deprecation_reason
consumer_migration_deadline
retired_at

Deprecation process:

1. Mark deprecated.
2. Notify consumers.
3. Monitor usage.
4. Support migration.
5. Stop producer usage if safe.
6. Retire/remove after deadline.

Do not deprecate silently.

18. Runtime Contract Usage Tracking

For APIs/events/datasets, track actual usage.

Examples:

• API client calls version v1,
• event consumer processes v1,
• dataset queried by dashboard,
• file feed consumed by external system.

Fields:

contract_usage
- contract_id
- version
- consumer_system
- last_seen_at
- usage_count
- environment

Usage tracking supports retirement decisions.

19. Schema Evolution Workflow

Workflow:

Propose change
  -> classify breaking/non-breaking
  -> run compatibility check
  -> run contract tests
  -> identify consumers
  -> review security/classification
  -> publish new version
  -> deploy producer
  -> monitor consumers
  -> deprecate old version
  -> retire old version

For high-critical contracts, require approval from consumer owners.

20. API Contract Governance

API contract governance includes:

• OpenAPI version,
• error model,
• auth model,
• idempotency semantics,
• pagination,
• filtering,
• enum behavior,
• money/date/time conventions,
• backward compatibility rules,
• deprecation policy.

API schema alone does not document lifecycle/business semantics sufficiently.

21. Event Contract Governance

Event contract governance includes:

• event name,
• event version,
• event meaning,
• producer,
• topic/exchange,
• routing/partition key,
• required fields,
• aggregate version semantics,
• correlation/causation fields,
• ordering guarantee,
• idempotency expectation,
• replay behavior,
• retention,
• sensitive fields.

Consumers should not depend on unspecified fields.

22. File/Feed Contract Governance

Batch/file feeds need contract too.

Fields:

• file name pattern,
• schema version,
• delimiter/encoding,
• header/trailer,
• column order,
• nullable rules,
• date/time format,
• numeric precision,
• row-level error handling,
• idempotency key,
• retry/replay,
• checksum,
• delivery SLA.

Example usage feed can break rating if column semantics change.

23. Dataset Contract Governance

Analytics datasets need contract:

• grain,
• primary key,
• dimensions,
• measures,
• time basis,
• refresh SLA,
• source lineage,
• quality rules,
• owner,
• classification,
• backward compatibility policy.

Changing fact grain is a breaking change even if columns remain same.

24. Contract and Reference Data

Reference data changes can break contract.

Examples:

• new status code,
• retired cancellation reason,
• UOM code renamed,
• currency precision changed,
• external mapping changed.

Contract should state:

• unknown enum handling,
• code set version,
• allowed values source,
• deprecation policy,
• compatibility expectations.

25. PostgreSQL Physical Design

Contract table:

create table data_contract (
  id uuid primary key,
  contract_name text not null unique,
  contract_type text not null,
  owner_group text not null,
  producer_system text,
  status text not null,
  current_version text,
  compatibility_policy text not null,
  data_classification text,
  documentation_url text,
  created_at timestamptz not null,
  updated_at timestamptz not null
);

Contract version:

create table data_contract_version (
  id uuid primary key,
  contract_id uuid not null references data_contract(id),
  version text not null,
  schema_definition jsonb,
  semantic_description text,
  schema_hash text,
  effective_from timestamptz,
  deprecated_at timestamptz,
  retired_at timestamptz,
  status text not null,
  created_at timestamptz not null,
  unique (contract_id, version)
);

Consumer registry:

create table data_contract_consumer (
  id uuid primary key,
  contract_id uuid not null references data_contract(id),
  consumer_system text not null,
  consumer_owner_group text,
  version_used text,
  criticality text not null,
  usage_type text,
  contact_channel text,
  migration_status text,
  last_seen_at timestamptz,
  created_at timestamptz not null,
  unique (contract_id, consumer_system, usage_type)
);

Sample payload:

create table data_contract_sample (
  id uuid primary key,
  contract_id uuid not null references data_contract(id),
  version text not null,
  sample_name text not null,
  scenario text,
  sample_payload jsonb,
  valid boolean not null default true,
  created_at timestamptz not null
);

Indexes:

create index idx_contract_type_status
on data_contract (contract_type, status);

create index idx_contract_consumer_contract
on data_contract_consumer (contract_id, criticality);

create index idx_contract_consumer_last_seen
on data_contract_consumer (last_seen_at);

create index idx_contract_version_status
on data_contract_version (contract_id, status);

26. Java/JAX-RS Backend Implications

Possible internal APIs:

GET /data-contracts
GET /data-contracts/{contractName}
GET /data-contracts/{contractName}/versions
POST /data-contracts/{contractName}/compatibility-check
GET /data-contracts/{contractName}/consumers
POST /data-contracts/{contractName}/samples

CI/CD integration:

• validate schema compatibility,
• run contract tests,
• publish contract version,
• block breaking change without approval,
• generate docs,
• notify consumers.

27. Contract Validation in Runtime

Producer can validate outgoing messages.

Consumer can validate incoming messages.

Runtime validation options:

• strict reject,
• quarantine/DLQ,
• accept with warning,
• schema version routing,
• compatibility adapter.

For external feeds, invalid rows should be persisted as validation errors, not silently dropped.

28. Contract Drift Detection

Contract drift happens when implementation differs from documented contract.

Detect with:

• schema generation comparison,
• runtime payload validation,
• consumer tests,
• sample payload tests,
• production payload sampling,
• OpenAPI diff,
• event schema hash,
• dataset schema check.

Drift should be treated as production risk.

29. Observability

Monitor:

• contract validation failures,
• unknown event version,
• unsupported schema version,
• consumer using deprecated version,
• breaking change attempts blocked,
• field-level quality violations,
• payload missing required fields,
• API clients still calling retired endpoint,
• dataset freshness SLA breach.

Example checks:

-- Consumers still on deprecated contract version
select dc.contract_name, c.consumer_system, c.version_used, c.last_seen_at
from data_contract_consumer c
join data_contract dc on dc.id = c.contract_id
join data_contract_version v
  on v.contract_id = dc.id
 and v.version = c.version_used
where v.deprecated_at is not null
  and v.retired_at is null;

-- Contract versions without schema hash
select dc.contract_name, v.version
from data_contract_version v
join data_contract dc on dc.id = v.contract_id
where v.schema_hash is null;

30. Failure Modes

31. PR Review Checklist

When reviewing contract changes, ask:

• What contract type is changing?
• Is this schema or semantic change?
• Is it backward compatible?
• Are consumers known?
• Are critical consumers affected?
• Is version bump required?
• Is OpenAPI/AsyncAPI/schema updated?
• Are samples updated?
• Are contract tests updated?
• Are error/enum/date/money semantics changed?
• Are sensitive fields added?
• Is deprecation needed?
• Is usage tracking available?
• Is migration plan defined?
• Are docs and owner metadata updated?

32. Internal Verification Checklist

Verify these in the internal CSG/team context:

• Existing schema registry or contract catalog.
• API contract governance process.
• Event schema/versioning process.
• Whether AsyncAPI/OpenAPI is used.
• Whether Kafka/RabbitMQ payload schemas are validated.
• Whether consumer registry exists.
• Whether contract tests run in CI.
• Whether deprecated API/event versions are usage-tracked.
• Whether file/feed contracts exist for usage/billing/OSS integrations.
• Whether analytics dataset contracts define grain/freshness.
• Whether security review is required for new sensitive fields.
• Whether incidents mention broken consumer, unknown enum, dataset grain change, or undocumented payload change.

33. Summary

Data contracts make distributed data change safe.

A strong model must define:

• contract registry,
• schema registry,
• contract version,
• producer,
• consumer registry,
• compatibility policy,
• semantic documentation,
• sample payloads,
• contract tests,
• quality rules,
• freshness SLA,
• security classification,
• deprecation workflow,
• usage tracking,
• runtime validation,
• drift detection,
• impact analysis.

The key principle:

In enterprise systems, compatibility is a data model concern. Every API, event, file, dataset, and metric consumed by others needs an owned, versioned, tested, and governed contract.