Test Data, Synthetic Data, and Data Model Verification Model

1. Core Idea

Data model yang baik harus bisa diuji.

Dalam enterprise CPQ / Quote / Order / Billing / Fulfillment / Inventory, correctness bukan hanya function/unit test. Data model harus diverifikasi melalui:

• invariant tests,
• state transition tests,
• database constraint tests,
• migration/backfill tests,
• event contract tests,
• idempotency/concurrency tests,
• reconciliation tests,
• reporting metric tests,
• privacy/security tests,
• synthetic data generation,
• production-like volume tests.

Mental model:

If you cannot create realistic safe test data and verify invariants automatically, your data model correctness will depend on luck and manual QA.

2. Why Test Data Modelling Matters

Tanpa test data strategy:

• tests hanya happy path,
• quote/order/billing edge cases tidak tercover,
• migration lolos di dev kecil tetapi gagal di production,
• event consumer gagal karena optional field null,
• invoice total mismatch tidak tertangkap,
• privacy leak muncul karena test pakai production dump,
• cache key tenant leak tidak diuji,
• duplicate order retry tidak diuji,
• reporting KPI salah karena bundle/test data tidak realistic,
• serviceability/pricing/billing cases tidak bisa direproduksi.

Enterprise test data harus merepresentasikan domain complexity tanpa membocorkan data customer.

3. Types of Test Data

Use different data for different goals.

4. Avoid Raw Production Data in Lower Environments

Production data can contain:

• PII,
• billing data,
• contract terms,
• margin/cost,
• site/location,
• service/resource IDs,
• usage data,
• audit evidence,
• payment references.

Risks:

• privacy breach,
• customer confidential leak,
• broad developer access,
• data residency violation,
• uncontrolled export,
• logs/screenshots leak.

Prefer synthetic data. If production-derived data is needed, use governed masking/anonymization.

5. Synthetic Data Requirements

Synthetic data should be:

• realistic enough for business rules,
• internally consistent,
• privacy safe,
• deterministic where useful,
• parameterized by tenant/region/product,
• able to generate edge cases,
• able to scale volume,
• versioned with model/schema,
• documented.

Example generated domain:

Customer
  -> Account
  -> Billing Account
  -> Site
  -> Quote
  -> Quote Items
  -> Approval
  -> Order
  -> Fulfillment
  -> Product Inventory
  -> Charge
  -> Invoice

Random data without relational consistency is not enough.

6. Golden Dataset

Golden dataset contains known scenarios and expected results.

Examples:

• simple quote accepted and converted to order,
• quote with discount approval,
• bundle order decomposition,
• modify existing product,
• disconnect product and terminate charge,
• usage rating with allowance,
• invoice with discount/tax/credit,
• order fallout then retry success,
• active product without charge mismatch,
• stale event ignored,
• duplicate convert request idempotent.

Golden dataset should be stable and reviewed.

7. Edge Case Dataset

Edge cases:

• quote expires at timezone boundary,
• billing account changes mid-cycle,
• product upgraded with proration,
• late usage event,
• duplicate usage event,
• order cancellation during fulfillment,
• approval invalidated by material quote change,
• product active but service inactive,
• bundle parent/child status mismatch,
• future-dated cancellation,
• backdated activation,
• retired reason code in historical record,
• tenant boundary case,
• external ID collision across systems.

Top engineers design test data for failure modes, not only happy path.

8. Test Data Metadata

Test data should be traceable.

Fields:

test_dataset
- id
- dataset_code
- purpose
- domain_area
- version
- generated_by
- generated_at
- data_classification
- safe_for_lower_env

Scenario:

test_scenario
- id
- dataset_id
- scenario_code
- description
- expected_outcome
- tags

This supports reuse and governance.

9. Fixture Design

Good fixtures are:

• small,
• deterministic,
• named by business scenario,
• easy to understand,
• explicit about expected output,
• not over-shared across unrelated tests,
• not dependent on test execution order.

Bad fixture:

one giant SQL seed file used by every test

Better:

QuoteWithApprovedDiscountFixture
OrderWithFalloutFixture
ActiveProductWithRecurringChargeFixture

10. Invariant Testing

Test business invariants.

Examples:

Quote total equals sum item totals.
Order cannot complete with open mandatory items.
Accepted quote is immutable.
One order per accepted quote version.
Active product has valid product offering version.
Issued invoice total equals invoice lines.
Current effective-dated rows do not overlap.

Use both:

• application service tests,
• database constraint tests,
• reconciliation tests.

11. State Machine Testing

Test allowed and forbidden transitions.

Example quote:

DRAFT -> PRICED allowed
PRICED -> SUBMITTED allowed
SUBMITTED -> APPROVED allowed
APPROVED -> ACCEPTED allowed
ACCEPTED -> DRAFT forbidden
EXPIRED -> ACCEPTED forbidden unless special policy

State machine tests should include:

• actor permission,
• expected version,
• material change,
• approval validity,
• audit/event emission.

12. Database Constraint Testing

Test constraints with real database.

Examples:

• duplicate source quote/order fails,
• current effective-dated duplicate fails,
• quantity <= 0 fails,
• missing mandatory FK fails,
• invalid status check fails,
• idempotency key unique,
• tenant-scoped uniqueness works,
• cross-tenant same business number allowed if designed.

Do not mock away constraints.

13. Migration and Backfill Testing

Migration tests should use realistic old data.

Test:

• expand/contract compatibility,
• old app reads new schema if applicable,
• new app reads old data,
• backfill idempotency,
• partial backfill resume,
• exception handling,
• shadow read comparison,
• rollback/forward fix,
• performance on volume,
• indexes/query plans.

Migration correctness depends on existing data shape, not ideal schema.

14. Event Contract Testing

Test event producers and consumers.

Producer tests:

• emits required fields,
• event version correct,
• aggregate version included,
• correlation/causation included,
• payload does not contain restricted fields,
• schema compatible.

Consumer tests:

• handles duplicate event,
• handles old version,
• handles new optional field,
• rejects unsupported version,
• ignores stale aggregate version,
• processes null optional fields safely.

15. Idempotency Testing

Test duplicate/retry behavior.

Scenarios:

• same idempotency key same payload returns same result,
• same key different payload returns conflict,
• duplicate event skipped,
• duplicate file row skipped,
• duplicate usage event not billed twice,
• duplicate quote conversion creates one order,
• retry billing trigger creates one charge,
• replay projection does not create side effect.

Idempotency must be tested at database/integration level.

16. Concurrency Testing

Use real DB/concurrent threads/processes where possible.

Scenarios:

• two quote accept commands,
• two quote convert commands,
• approval and quote revision race,
• cancel order vs fulfillment complete,
• two rating workers consuming allowance,
• two workers claiming same task,
• duplicate external callback,
• cache stale read during update.

Expected result should be deterministic and safe.

17. Reconciliation Testing

Create known mismatches and verify detection.

Examples:

• accepted quote without order,
• active product without charge,
• terminated product with active charge,
• fulfilled order item without product instance,
• invoice total mismatch,
• projection behind source version,
• outbox event missing,
• external mapping duplicate.

Test not only detection but also severity, owner, and repair status.

18. Reporting/KPI Testing

Metric tests need expected values.

Scenarios:

• quote conversion rate with revisions,
• MRR with active/suspended/terminated subscriptions,
• churn with cancellation effective date,
• installed base with bundle parent/child,
• invoice revenue excluding voided invoices,
• order backlog by status,
• fallout aging.

Do not test dashboard only visually. Test metric logic and grain.

19. Privacy/Security Testing

Test:

• unauthorized user cannot see margin/cost,
• tenant A cannot see tenant B data,
• masked fields are masked,
• event payload excludes restricted fields,
• export requires approval,
• purge/anonymization updates search/read models,
• break-glass access is audited,
• lower env test dataset has no real PII.

Security is part of data model verification.

20. Synthetic Data Generator Model

Generator configuration:

synthetic_data_generation_run
- id
- generator_name
- generator_version
- dataset_code
- tenant_id
- volume_profile
- scenario_profile
- seed
- status
- started_at
- completed_at

Why seed matters:

• reproducible bugs,
• deterministic tests,
• compare expected results.

Profiles:

small
integration
performance
migration
edge-case
reporting

21. Data Masking Model

If using production-derived masked data:

Fields:

data_masking_run
- id
- source_environment
- target_environment
- masking_policy_version
- status
- started_at
- completed_at
- validation_status

Masking rules:

masking_rule
- entity_type
- field_path
- strategy
- deterministic
- preserves_format
- owner_group

Strategies:

• null,
• random replacement,
• deterministic token,
• hash,
• format-preserving fake,
• range bucketing,
• aggregation.

Validate that masked data contains no real sensitive values.

22. Test Data Lifecycle

Test data also needs lifecycle.

• created,
• used,
• refreshed,
• expired,
• purged,
• archived if golden,
• regenerated for schema version.

Test data can become stale when:

• schema changes,
• rule changes,
• reference data changes,
• event contract changes,
• product catalog changes,
• pricing model changes.

Version test data with model/contracts.

23. PostgreSQL Physical Design

Test dataset registry:

create table test_dataset (
  id uuid primary key,
  dataset_code text not null unique,
  purpose text not null,
  domain_area text,
  version text not null,
  data_classification text not null,
  safe_for_lower_env boolean not null default true,
  generated_by text,
  generated_at timestamptz,
  created_at timestamptz not null
);

Test scenario:

create table test_scenario (
  id uuid primary key,
  dataset_id uuid not null references test_dataset(id),
  scenario_code text not null,
  description text,
  expected_outcome jsonb,
  tags text[],
  unique (dataset_id, scenario_code)
);

Synthetic data generation run:

create table synthetic_data_generation_run (
  id uuid primary key,
  generator_name text not null,
  generator_version text not null,
  dataset_code text not null,
  tenant_id uuid,
  volume_profile text,
  scenario_profile text,
  seed text,
  status text not null,
  started_at timestamptz not null,
  completed_at timestamptz,
  row_count bigint,
  error_message text
);

Masking run:

create table data_masking_run (
  id uuid primary key,
  source_environment text not null,
  target_environment text not null,
  masking_policy_version text not null,
  status text not null,
  started_at timestamptz not null,
  completed_at timestamptz,
  validation_status text,
  error_message text
);

24. Java/JAX-RS Backend Implications

Test support APIs may exist in non-prod only:

POST /test-data/generate
POST /test-data/reset
GET /test-data/datasets
POST /test-data/scenarios/{scenarioCode}/load

Rules:

• disabled in production unless specifically controlled,
• tenant/environment scoped,
• no real sensitive data,
• deterministic seed support,
• cleanup support,
• audit if used in shared environments.

Application code should expose domain builders/test fixtures separately from production endpoints.

25. CI/CD Integration

Data model verification in CI:

• schema migration test,
• rollback/forward migration test,
• DB constraint test,
• contract compatibility test,
• event payload test,
• generated OpenAPI diff,
• seed golden dataset,
• run invariant test suite,
• run reconciliation tests,
• run security/masking tests.

For heavier volume tests, run nightly/pre-release.

26. Production-Like Testing

Production-like testing needs:

• realistic volume,
• realistic distribution,
• realistic tenant count,
• realistic quote/order item depth,
• realistic usage volume,
• realistic status mix,
• realistic history/audit size,
• realistic index/table bloat,
• realistic batch files,
• realistic failure cases.

Small dev DB hides:

• slow query,
• migration lock,
• memory issue,
• batch timeout,
• pagination problem,
• deadlock frequency.

27. Observability

Monitor test data quality:

• synthetic generator failures,
• stale golden dataset,
• missing scenario coverage,
• contract sample outdated,
• masked data validation failures,
• production data detected in lower env,
• invariant test failures,
• migration test failures,
• reconciliation test failures.

Example:

-- Golden datasets older than model version policy
select dataset_code, version, generated_at
from test_dataset
where purpose = 'GOLDEN'
  and generated_at < now() - interval '90 days';

-- Masking run failed validation
select id, source_environment, target_environment, validation_status
from data_masking_run
where validation_status = 'FAILED';

28. Failure Modes

29. PR Review Checklist

When reviewing data model changes, ask:

• Are fixtures updated?
• Are golden scenarios affected?
• Is synthetic generator updated?
• Are DB constraint tests updated?
• Are state machine tests updated?
• Are migration/backfill tests included?
• Are event/API contract samples updated?
• Are reconciliation rules tested?
• Are KPI/reporting expected values updated?
• Are privacy/security tests updated?
• Are tenant isolation cases covered?
• Is volume/performance test needed?
• Is old data compatibility tested?
• Is test data safe for lower environments?

30. Internal Verification Checklist

Verify these in the internal CSG/team context:

• Existing test data strategy.
• Whether lower environments use synthetic, masked, or production data.
• Whether golden datasets exist for quote/order/billing flows.
• Whether data model invariants are tested.
• Whether DB constraints are tested with real PostgreSQL.
• Whether event/API contract samples exist.
• Whether migration/backfill tests use realistic data.
• Whether reconciliation rules have known mismatch tests.
• Whether reporting/KPI tests exist.
• Whether tenant isolation/security tests exist.
• Whether data masking validation is automated.
• Whether incidents mention lack of realistic test data, migration surprise, privacy leak in lower env, or missed edge case.

31. Summary

Data model verification requires deliberate test data design.

A strong model must define:

• fixture data,
• synthetic data,
• golden dataset,
• edge-case dataset,
• volume dataset,
• contract samples,
• invariant tests,
• state machine tests,
• DB constraint tests,
• migration/backfill tests,
• event/idempotency/concurrency tests,
• reconciliation tests,
• reporting/KPI tests,
• privacy/security tests,
• masking validation,
• test data lifecycle.

The key principle:

A complex enterprise data model is only as trustworthy as the scenarios used to test it. Production correctness requires realistic, safe, repeatable, and edge-case-rich test data.