Part 025 — XML Pipeline Architecture

Goal: mampu merancang XML processing pipeline Java yang aman, deterministic, observable, replayable, dan tahan terhadap payload besar, schema evolution, transform failure, partner variation, dan kebutuhan audit/regulatory.

Di part sebelumnya kita membahas parser, XSD, XPath, XQuery, XSLT, binding, serialization, diagnostics, dan performance sebagai kemampuan terpisah. Di production, semua kemampuan itu jarang berdiri sendiri. Biasanya XML masuk sebagai flow:

receive document -> identify contract -> parse safely -> validate -> extract metadata -> enrich -> transform -> route -> persist -> audit -> notify downstream

Itulah XML pipeline.

Pipeline yang baik bukan hanya “bisa memproses XML”. Pipeline yang baik memiliki boundary yang jelas:

• payload mana yang diterima;
• contract mana yang dipakai;
• policy keamanan mana yang aktif;
• kapan validasi terjadi;
• transform asset versi berapa yang digunakan;
• output apa yang dihasilkan;
• error apa yang retryable;
• bukti audit apa yang disimpan;
• bagaimana replay dilakukan tanpa mengubah hasil secara diam-diam.

Mental model utama:

XML pipeline = controlled sequence of irreversible and reversible decisions over an XML artifact.

Yang berbahaya bukan hanya error parsing. Yang lebih berbahaya adalah keputusan diam-diam: schema auto-download, transform berubah tanpa versioning, XPath salah namespace, error dipotong, payload tersimpan tanpa hash, atau replay menggunakan asset versi baru.

1. Kaufman Deconstruction

Skill ini bisa dipecah menjadi beberapa sub-skill.

Practice target untuk 20 jam:

Build one production-like XML pipeline that can ingest, validate, transform, audit, fail safely, and replay a real XML contract.

Bukan targetnya:

Learn every XML API.

Targetnya:

Own the lifecycle of an XML artifact from intake to evidence.

2. Reference Pipeline Shape

Pipeline dasar:

Dalam sistem besar, pipeline ini sering dipisah menjadi beberapa stage asynchronous:

Satu pipeline monolitik synchronous terlihat sederhana, tetapi sulit dioperasikan saat:

• payload besar;
• validasi lama;
• transform butuh enrichment;
• downstream sering gagal;
• ada manual review;
• replay dibutuhkan;
• partner mengirim data buruk tapi harus dipertahankan sebagai evidence.

Rule:

If a stage can fail independently, observe it and persist its state independently.

3. Pipeline State Model

XML pipeline production-grade perlu state eksplisit. Jangan hanya try-catch lalu log.

Contoh lifecycle:

State harus merepresentasikan keputusan bisnis-operasional, bukan detail implementasi terlalu rendah.

Buruk:

STARTED, STEP_1_DONE, STEP_2_DONE, STEP_3_DONE

Lebih baik:

RECEIVED, CONTRACT_IDENTIFIED, STRUCTURALLY_VALID, TRANSFORMED, OUTPUT_VALID, DISPATCHED, QUARANTINED

Karena state ini bisa dipakai oleh:

• operator;
• audit reviewer;
• retry scheduler;
• manual review UI;
• SLA dashboard;
• replay tooling;
• incident analysis.

4. Artifact Model

Pipeline tidak hanya menghasilkan satu output. Ia menghasilkan artifact.

A useful artifact manifest:

{
  "pipelineRunId": "run-20260702-000123",
  "documentId": "doc-abc",
  "inputHash": "sha256:...",
  "inputContentType": "application/xml",
  "detectedContract": "partner-order-v3",
  "schemaBundleVersion": "orders-schema-3.2.1",
  "transformBundleVersion": "partner-to-canonical-5.4.0",
  "securityPolicyVersion": "xml-policy-2026.07",
  "enrichmentSnapshotId": "snapshot-789",
  "outputHash": "sha256:...",
  "startedAt": "2026-07-02T10:00:00Z",
  "finishedAt": "2026-07-02T10:00:03Z",
  "result": "OUTPUT_VALID"
}

In regulated systems, manifest ini lebih penting daripada log biasa.

Log menjawab:

What happened around that time?

Manifest menjawab:

Exactly what input, contract, code, policy, and asset version produced this output?

5. Ingress Boundary

Ingress adalah titik paling berisiko. Semua asumsi buruk masuk dari sini.

Minimum checks:

Ingress should not perform heavy semantic work. It should do enough to make the artifact safe and traceable.

A common mistake:

String xml = requestBodyAsString();
Document doc = parse(xml);

This hides:

• byte size;
• charset;
• content-type;
• raw hash;
• storage durability;
• parse limits;
• evidence trail.

Better mental model:

The raw XML is an artifact first, a Java object later.

6. Contract Resolution

A pipeline cannot validate or transform safely until it knows which contract applies.

Contract can be resolved from:

Production rule:

schemaLocation is a hint, not authority.

Recommended approach:

A contract registry entry may contain:

contractId: partner-order-v3
partner: acme
rootQName: "{urn:acme:order:v3}OrderRequest"
schemaBundle: orders-schema-3.2.1
inputValidationMode: strict
transformBundle: acme-order-to-canonical-5.4.0
outputContract: canonical-order-v2
securityPolicy: xml-partner-default-2026.07
routingPolicy: order-ingest

This is better than hardcoding schema paths inside code.

7. Secure Parse Stage

The parse stage should not have business logic. Its job:

1. apply parser security policy;
2. reject unsafe constructs;
3. expose a safe processing representation;
4. collect location-aware diagnostics;
5. avoid accidental network/filesystem access.

Example secure parser profile:

final class XmlSecurityProfile {
    boolean disallowDoctype = true;
    boolean externalGeneralEntities = false;
    boolean externalParameterEntities = false;
    boolean loadExternalDtd = false;
    boolean xincludeAware = false;
    boolean expandEntityReferences = false;
    int maxElementDepth = 128;
    int maxAttributesPerElement = 256;
    long maxInputBytes = 50 * 1024 * 1024;
}

The exact features depend on parser/API, but architecture should make policy explicit.

Bad:

DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(input);

Better:

DocumentBuilderFactory factory = SecureXmlFactories.newDocumentBuilderFactory(policy);
DocumentBuilder builder = factory.newDocumentBuilder();
builder.setErrorHandler(diagnostics.errorHandler());
Document document = builder.parse(source);

The key is not the helper method. The key is policy centralization.

Every XML entry point must use the same default-deny policy.

8. Validation Stage

Validation should produce structured evidence, not just throw an exception.

Minimal validation result:

public record ValidationIssue(
    Severity severity,
    String code,
    String message,
    Integer line,
    Integer column,
    String systemId,
    String schemaVersion,
    String contractId
) {}

public record ValidationReport(
    String documentId,
    String contractId,
    String schemaBundleVersion,
    boolean valid,
    List<ValidationIssue> issues
) {}

Do not collapse all validation failures to:

Invalid XML

That creates operational blindness.

Validation stage options:

Recommended pipeline:

Why validate output? Because valid input does not guarantee valid output.

Transform can:

• omit required fields;
• create invalid enum;
• break namespace;
• round decimal incorrectly;
• generate illegal date;
• duplicate identifiers;
• violate ordering.

9. Metadata Extraction Stage

Metadata extraction answers operational questions:

• What document is this?
• Who sent it?
• What version is it?
• Which business entity does it affect?
• Can we deduplicate it?
• Where should it route?
• What should appear in search/UI?

Example metadata:

{
  "documentId": "doc-abc",
  "partnerId": "acme",
  "contractId": "partner-order-v3",
  "businessKey": "ORDER-10001",
  "submittedAt": "2026-07-02T10:00:00Z",
  "customerId": "C-123",
  "documentType": "ORDER_REQUEST",
  "declaredVersion": "3.1"
}

Implementation choices:

A common pattern:

public interface MetadataExtractor {
    ExtractedMetadata extract(InputArtifact artifact, Contract contract);
}

For large XML, avoid building a full DOM just to get three routing fields.

If metadata is in the envelope/header, extract it streaming.

10. Semantic Precheck Stage

XSD checks grammar. It does not fully check business meaning.

Examples of semantic rules:

Keep semantic precheck separate from XSD validation.

Design principle:

XSD protects the shape. Semantic validation protects business truth.

Do not bury business validation inside XSLT unless it is intentionally governed as transformation logic. Hidden business rules in stylesheets are hard to audit.

11. Enrichment Stage

Enrichment adds external data needed for transformation/routing.

Examples:

• partner configuration;
• reference data;
• product catalog mapping;
• tax codes;
• customer status;
• jurisdiction mapping;
• canonical code table;
• regulatory rule version.

Enrichment is dangerous because it introduces time variance.

Same input XML can produce different output if reference data changes.

Production-grade enrichment must decide:

Example:

public record EnrichmentContext(
    String referenceDataVersion,
    Map<String, String> codeMappings,
    Instant loadedAt,
    String snapshotHash
) {}

Avoid this:

String taxCode = taxService.getCurrentTaxCode(country);

inside a transformation that must be replayable.

Better:

EnrichmentContext ctx = enrichmentSnapshotService.loadFor(contract, processingDate);
transformer.setParameter("taxCodeMapVersion", ctx.referenceDataVersion());

12. Transformation Stage

Transformation is where XML pipelines become difficult.

A transformation stage should have:

• input contract;
• output contract;
• versioned transform asset;
• parameters;
• resolver policy;
• deterministic output rules;
• error listener;
• output validation;
• test coverage;
• audit manifest.

Example manifest:

transformRun:
  inputContract: partner-order-v3
  outputContract: canonical-order-v2
  transformBundle: acme-order-to-canonical-5.4.0
  processor: Saxon-HE
  parameters:
    partnerId: acme
    referenceDataVersion: tax-map-2026.07.01
  resultHash: sha256:...

Transformation implementation choices:

XSLT often wins when:

• source and target are both XML;
• transform must be declarative;
• mapping must be reviewed by integration specialists;
• output needs deterministic structure;
• tests can be golden-file based.

Java mapper often wins when:

• transformation depends heavily on domain services;
• output is not XML-centric;
• type-safe domain behavior matters;
• debugging in Java code is more important than declarative mapping.

13. Routing Stage

Routing should be explicit and explainable.

Routing can depend on:

• contract ID;
• document type;
• partner;
• business key;
• jurisdiction;
• validation result;
• semantic status;
• output type;
• manual review status;
• downstream availability.

Avoid routing hidden inside transformation.

Bad:

<xsl:if test="/order/country = 'ID'">
  <!-- generate different output and rely on caller to infer route -->
</xsl:if>

Better:

RoutingDecision decision = routingPolicy.decide(metadata, validation, semanticResult, outputContract);

Example decision:

{
  "route": "REGULATORY_ID_ORDER_SUBMISSION",
  "reason": "country=ID and documentType=ORDER_REQUEST",
  "requiredOutputContract": "id-reg-order-v2",
  "retryPolicy": "partner-regulatory-default"
}

Routing decision should be stored because it affects auditability.

14. Persistence Stage

Persistence design determines replay, audit, and debugging quality.

Typical tables/collections:

A simple relational shape:

Do not store only the final output. You will need the intermediate evidence when a partner says:

Your system changed our data incorrectly.

You should be able to answer:

This exact input hash, using schema bundle X and transform bundle Y, produced this exact output hash at this time.

15. Audit Event Model

Audit is not the same as logging.

Logging is diagnostic. Audit is evidence.

Audit event example:

{
  "eventId": "evt-001",
  "pipelineRunId": "run-123",
  "eventType": "INPUT_SCHEMA_VALIDATED",
  "occurredAt": "2026-07-02T10:00:02Z",
  "actor": "system:validation-worker",
  "contractId": "partner-order-v3",
  "assetVersion": "orders-schema-3.2.1",
  "result": "VALID",
  "inputHash": "sha256:...",
  "evidenceRef": "s3://.../validation-report.json"
}

Audit events should be:

• append-only;
• timestamped;
• correlated;
• minimal but sufficient;
• PII-safe;
• linked to durable artifacts;
• stable across deployments.

Do not put huge XML payloads directly into audit events. Store artifacts separately and reference them.

16. Idempotency and Deduplication

XML partner systems often resend payloads.

Reasons:

• timeout after successful processing;
• partner retry;
• batch resubmission;
• duplicate file upload;
• network uncertainty;
• manual resend.

Idempotency key candidates:

Recommended strategy:

technical duplicate = same raw hash from same source
business duplicate = same partner + document type + business key + business version

Example:

public record XmlIdempotencyKey(
    String partnerId,
    String documentType,
    String businessKey,
    String businessVersion,
    String rawSha256
) {}

Pipeline should define behavior:

17. Retry and Quarantine

Not every failure should retry.

Failure classification:

Retry must be stage-aware.

Bad:

retry whole pipeline from raw input every time

Better:

resume from last durable successful stage if input/artifacts/asset versions are unchanged

Quarantine is not trash. Quarantine is controlled storage for unsafe or unprocessable artifacts.

Quarantined items need:

• reason;
• severity;
• raw artifact reference;
• safe preview;
• who can access;
• manual action options;
• retention policy.

18. Stage Isolation and Transaction Boundaries

A pipeline mixes CPU, I/O, persistence, and external calls. One database transaction for the entire flow is usually wrong.

Better:

Each durable stage commits its result and emits the next work item.

Example:

Key invariant:

Never emit next work unless current stage evidence is durable.

If using queue + database, apply outbox pattern when consistency matters.

19. Pipeline Orchestration Options

For enterprise XML, job/workflow-style design is often better than synchronous call stack.

Rule:

If humans, retries, or audit are involved, model the pipeline state explicitly.

20. Observability Model

Metrics should map to pipeline stages.

Recommended metrics:

Structured log fields:

{
  "pipelineRunId": "run-123",
  "documentId": "doc-abc",
  "partnerId": "acme",
  "contractId": "partner-order-v3",
  "stage": "OUTPUT_VALIDATION",
  "schemaBundleVersion": "canonical-order-2.1.0",
  "result": "FAILED",
  "line": 42,
  "column": 17,
  "errorCode": "XSD_ENUM_INVALID"
}

Trace spans:

xml.ingest
xml.contract.resolve
xml.parse
xml.validate.input
xml.extract.metadata
xml.semantic.precheck
xml.enrich
xml.transform
xml.validate.output
xml.route
xml.dispatch

Do not log full XML by default. Use payload hash and artifact reference.

21. Replay Architecture

Replay is not “run it again”. Replay must define what remains fixed.

Replay modes:

Forensic replay requires:

• raw input bytes;
• exact parser/security policy;
• schema bundle version;
• transform bundle version;
• enrichment snapshot;
• runtime version or compatibility notes;
• output serialization rules;
• stable clock/randomness policy.

Replay request example:

{
  "mode": "FORENSIC",
  "pipelineRunId": "run-123",
  "useOriginalAssets": true,
  "dispatchDownstream": false,
  "requestedBy": "audit-user-1",
  "reason": "partner dispute"
}

Replay output should be compared by hash/canonical form.

replay output hash == original output hash

If not equal, system must explain why.

22. Configuration and Asset Versioning

XML pipeline assets include:

• XSD bundles;
• XSLT stylesheets;
• XPath expressions;
• XQuery modules;
• resolver catalogs;
• mapping tables;
• routing rules;
• security profiles;
• partner profiles;
• output serialization config.

Treat them like code.

Minimum governance:

Do not deploy mutable latest.xsd or current-transform.xsl into production without resolved immutable version in manifest.

Bad:

/schema/order.xsd
/xslt/partner-transform.xsl

Better:

/schema/orders-schema-3.2.1/order.xsd
/xslt/acme-order-to-canonical-5.4.0/main.xsl

23. Code Architecture

A clean Java pipeline separates policy, stages, artifacts, and orchestration.

Example package structure:

com.example.xmlpipeline
  artifact/
    InputArtifact.java
    OutputArtifact.java
    ArtifactStore.java
    ArtifactHash.java
  contract/
    Contract.java
    ContractRegistry.java
    SchemaBundle.java
    TransformBundle.java
  security/
    XmlSecurityPolicy.java
    SecureXmlFactories.java
    DenyByDefaultResourceResolver.java
  stage/
    PipelineStage.java
    IngestStage.java
    ContractResolutionStage.java
    ValidationStage.java
    MetadataExtractionStage.java
    EnrichmentStage.java
    TransformationStage.java
    OutputValidationStage.java
    DispatchStage.java
  diagnostics/
    ValidationReport.java
    TransformReport.java
    PipelineDiagnostic.java
  audit/
    AuditEvent.java
    AuditTrail.java
  replay/
    ReplayService.java

Stage interface:

public interface PipelineStage<I, O> {
    StageResult<O> execute(PipelineContext context, I input);
}

Context:

public record PipelineContext(
    String pipelineRunId,
    String documentId,
    Clock clock,
    XmlSecurityPolicy securityPolicy,
    ContractRegistry contractRegistry,
    ArtifactStore artifactStore,
    AuditTrail auditTrail
) {}

Stage result:

public sealed interface StageResult<T> permits StageSuccess, StageRejected, StageRetryableFailure, StageFatalFailure {
    String stageName();
}

This makes failure classification explicit.

24. Example End-to-End Flow

Use case: partner submits order XML.

State changes:

RECEIVED
CONTRACT_IDENTIFIED
STRUCTURALLY_VALID
SEMANTIC_VALID
ENRICHED
TRANSFORMED
OUTPUT_VALID
DISPATCHED
ARCHIVED

Evidence:

raw hash
schema version
validation report
metadata
reference snapshot version
transform bundle version
output hash
dispatch receipt
audit timeline

25. Common Design Mistakes

Mistake 1: Treating XML as String

Bad:

String result = xml.replace("<old>", "<new>");

Why it fails:

• namespace ignorance;
• escaping errors;
• invalid structure;
• accidental replacement;
• no validation;
• impossible audit semantics.

Use parser/transformer.

Mistake 2: Combining Validation, Mapping, and Dispatch

Bad:

public void handle(String xml) {
    Document doc = parse(xml);
    validate(doc);
    String out = transform(doc);
    http.post(out);
}

This has weak recovery. If dispatch fails, what exactly succeeded? Was output persisted? Can it be resent without re-transforming?

Better: durable stage boundaries.

Mistake 3: No Output Validation

Input XSD passes. Transform output is sent. Downstream rejects.

Output validation would catch it before dispatch.

Mistake 4: Mutable Assets

Changing XSLT in place breaks replay and audit.

Use immutable versioned bundles.

Mistake 5: Log-Only Diagnostics

Logs expire, are noisy, and are not structured evidence.

Persist validation/transform reports.

Mistake 6: Retry Everything

Malformed XML will never become valid by retrying.

Classify failure.

26. Production Readiness Checklist

Security

• all parser factories use central secure defaults;
• DTD/external entity behavior is explicit;
• schema/XSLT/XQuery external resource access is controlled;
• max size/depth/entity/attribute limits exist;
• full payload logging is disabled by default;
• quarantine is access-controlled.

Contract

• contract registry maps partner/root/version to schema/transform;
• schema bundles are immutable;
• transform bundles are immutable;
• output contract is explicit;
• schemaLocation is not blindly trusted.

Pipeline

• each failure-prone stage has state;
• durable evidence exists before next stage;
• retry policy is stage-aware;
• idempotency key is defined;
• duplicate behavior is documented;
• manual review path exists where needed.

Audit

• raw input hash stored;
• output hash stored;
• validation reports persisted;
• transform manifest persisted;
• dispatch receipts stored;
• replay mode documented.

Observability

• metrics per contract/stage/result;
• structured logs include run/document/contract IDs;
• traces map to pipeline stages;
• dashboards show backlog, failure, latency, and quarantine;
• alerts distinguish invalid input from system failures.

27. Practice Drill

Build a small but realistic pipeline:

Input: partner invoice XML
Output: canonical invoice XML
Requirements:
- secure parse
- input XSD validation
- metadata extraction: invoiceId, supplierId, amount, currency
- semantic check: amount > 0
- enrichment: supplier code mapping from versioned JSON file
- XSLT transform
- output XSD validation
- artifact manifest
- replay command

Add failure cases:

1. malformed XML;
2. XXE payload;
3. wrong namespace;
4. invalid enum;
5. missing supplier mapping;
6. XSLT bug;
7. invalid output;
8. duplicate input;
9. downstream failure;
10. replay with original assets.

Expected result:

You should be able to explain exactly which stage failed, why, whether retry is safe, and what evidence exists.

28. Mental Model Summary

A production XML pipeline is not just parser + mapper.

It is:

artifact lifecycle + contract governance + secure processing + deterministic transformation + durable evidence + controlled replay

The main invariant:

For every accepted XML artifact, the system can prove what contract was used, what decisions were made, what output was produced, and how to reproduce or explain it.

When you can design that, XML stops being “legacy format” and becomes a controlled integration substrate.

29. What Comes Next

Part 026 moves from internal pipeline architecture to production integration patterns:

• SOAP/XML services;
• batch XML feeds;
• regulatory submission;
• partner envelopes;
• canonical message design;
• schema registry;
• correlation IDs;
• contract negotiation;
• XML in message queues;
• XML over file/object-store pipelines;
• enterprise integration failure modes.