Part 030 — Observability, Operations, and Supportability

Core idea: A large-scale ERP is not production-ready when it works on the happy path. It is production-ready when engineers and support teams can detect, explain, contain, repair, and prove what happened when business processes fail.

ERP operations are different from normal web application operations. A failed page load is annoying. A failed posting batch, duplicated payment, stuck approval, missed shipment, broken period close, or incorrect report can affect financial statements, customer commitments, inventory availability, and audit evidence.

Observability in ERP must cover both:

1. technical system health — latency, errors, CPU, memory, DB, queues, traces;
2. business process health — stuck workflows, unreconciled ledgers, failed postings, duplicate attempts, aging exceptions, projection lag, settlement mismatches.

This part focuses on building a supportable ERP: one where failure is visible, diagnosable, recoverable, and defensible.

1. Kaufman Skill Deconstruction

To become effective at ERP observability and operations, decompose the skill into these sub-skills:

The goal is not to collect every possible signal. The goal is to make important business failures visible early and explainable quickly.

2. Observability vs Monitoring

Monitoring asks: is the system behaving as expected?

Observability asks: can we understand why the system is behaving this way from external signals?

ERP needs both.

2.1 Technical monitoring examples

2.2 Business observability examples

A large ERP can have green servers and still be failing the business.

3. ERP Observability Model

ERP observability should connect technical execution with business lifecycle.

The key design principle:

Every important business command should leave a trail that connects request, decision, state transition, side effect, integration, report projection, and audit evidence.

4. Correlation IDs and Business Identifiers

Technical trace IDs are necessary but not enough. ERP support teams often search by business identifiers.

4.1 Identifier taxonomy

4.2 Logging context

Every meaningful log line should include enough context to reconstruct the flow.

try (MDC.MDCCloseable ignored1 = MDC.putCloseable("tenantId", tenantId);
     MDC.MDCCloseable ignored2 = MDC.putCloseable("companyCode", companyCode);
     MDC.MDCCloseable ignored3 = MDC.putCloseable("correlationId", correlationId);
     MDC.MDCCloseable ignored4 = MDC.putCloseable("documentId", documentId.toString())) {

    purchaseOrderService.approve(command);
}

For asynchronous processing, correlation context must be propagated explicitly through message headers.

headers:
  traceparent: 00-...
  correlation-id: corr-p2p-20260701-00091
  causation-id: evt-po-approved-000128
  tenant-id: tenant-a
  company-code: ID01
  source-document-id: po_01J...

5. Structured Logging for ERP

Free-text logs are useful for humans, but structured logs are necessary for reliable querying, alerting, and investigation.

5.1 ERP log event fields

5.2 What not to log

ERP logs often become privacy and security risks.

Do not log:

• passwords, tokens, API keys;
• full bank account numbers;
• full tax IDs if not required;
• unnecessary personal data;
• raw payment card data;
• full document attachments;
• complete salary or sensitive HR payloads;
• unmasked customer identity in high-volume traces;
• secrets embedded in integration payloads.

Log identifiers and evidence references, not uncontrolled sensitive payloads.

6. Metrics: Technical and Business

Metrics should be low-cardinality, aggregatable, and actionable.

6.1 Technical metric examples

6.2 Business metric examples

6.3 Metric cardinality rule

Do not put high-cardinality values into metric labels.

Bad:

erp.posting.failed.count{documentNumber="INV-2026-001929"}

Better:

erp.posting.failed.count{company="ID01", documentType="SALES_INVOICE", reason="PERIOD_CLOSED"}

Use logs/traces for document-level details. Use metrics for aggregate health.

7. Tracing for ERP Workflows

Distributed tracing is useful for ERP, but only if spans map to meaningful operations.

7.1 Trace shape for purchase order approval

A useful trace should show:

• command handling;
• authorization check;
• workflow step;
• database transaction;
• audit event write;
• outbox write;
• downstream consumer processing.

7.2 Span naming

Bad span names:

process
handle
run
execute

Better span names:

PurchaseOrder.Approve
Authorization.EvaluateApprovalAuthority
Workflow.CompleteTask
Audit.WriteBusinessEvent
Outbox.EnqueueBusinessEvent

Span names should help explain the business operation.

8. Business Health Dashboards

ERP dashboards should not only show infrastructure.

8.1 Dashboard categories

8.2 Finance operations dashboard

8.3 Workflow dashboard

9. Alert Design

ERP alerts should be actionable. A bad alert says “something failed.” A good alert says “business risk exists, here is owner and next action.”

9.1 Alert severity

9.2 Alert examples

alert: ERPPostingPipelineBacklog
condition: erp.posting.pending.count{company="ID01"} > 1000 for 15m
severity: SEV2
owner: finance-platform-oncall
businessImpact: "Operational documents are not reflected in financial ledger."
runbook: "posting-pipeline-backlog.md"

alert: APSubledgerGLReconciliationGap
condition: erp.reconciliation.gap.amount{ledger="AP", company="ID01"} != 0 for 30m
severity: SEV1
owner: finance-control-oncall
businessImpact: "AP control account does not reconcile to AP subledger."
runbook: "ap-gl-reconciliation-gap.md"

9.3 Alert anti-patterns

10. Runbooks

Runbooks convert observability into action.

10.1 ERP runbook template

# Runbook: Posting Pipeline Backlog

## Symptoms
- `erp.posting.pending.count` rising for company X.
- Accounting event consumers lagging.
- Finance reports show projection lag.

## Business Impact
- Operational documents may not be reflected in GL.
- Close process may be delayed.

## First Checks
1. Check consumer error rate.
2. Check DB lock wait and deadlocks.
3. Check outbox publish lag.
4. Check failed accounting events by reason code.
5. Check whether period is closed or config changed.

## Safe Actions
- Pause non-critical batch reports.
- Increase consumer workers if DB is not saturated.
- Retry events in `FAILED_RETRYABLE` state.
- Route `FAILED_TERMINAL` to finance exception queue.

## Unsafe Actions
- Do not manually mark accounting events as posted.
- Do not delete outbox records.
- Do not reopen closed period without finance approval.

## Escalation
- Finance platform on-call.
- Finance controller.
- Database on-call if lock wait > threshold.

## Evidence to Capture
- Batch run ID.
- Event IDs.
- Error reason codes.
- Reconciliation state before and after remediation.

10.2 Good runbook properties

11. Operational Control Plane

Large ERP needs admin tools, but admin tools are dangerous. They must be designed as controlled operations, not ad-hoc database access.

11.1 Required operational tools

11.2 Control-plane design rules

11.3 Posting retry console example

Support tools must encode the legal repair path, not bypass it.

12. Error Taxonomy

Not all errors should be handled the same way.

12.1 ERP error classes

12.2 Error envelope

{
  "errorCode": "ERP-POSTING-0031",
  "errorClass": "CONFIGURATION_ERROR",
  "reasonCode": "MISSING_POSTING_RULE",
  "message": "No posting rule configured for GOODS_RECEIPT in company ID01.",
  "correlationId": "corr-p2p-20260701-00091",
  "documentId": "grn_01J...",
  "documentNumber": "GRN-2026-000128",
  "supportAction": "Route to finance configuration owner and retry after approved config publication."
}

A good error message supports users, support teams, engineers, and auditors.

13. Incident Response for ERP

ERP incidents should be handled with both technical and business discipline.

13.1 Incident phases

13.2 ERP incident triage questions

13.3 Containment actions

Containment should stop further harm while preserving investigation evidence.

14. Supportability by Design

Supportability is not something added after launch. It is designed into every feature.

14.1 Feature supportability checklist

For every new ERP feature, ask:

• How will support find a document by business number?
• How will they see the full lifecycle timeline?
• How will they know which config version was used?
• How will they see emitted events and downstream status?
• How will they distinguish validation error from system error?
• How will they safely retry or repair?
• How will they know whether report/read model caught up?
• What dashboard metric will show this feature is healthy?
• What alert will fire when it is unhealthy?
• What audit evidence proves the action was valid?

14.2 Support case model

Support cases should link to evidence, not copy uncontrolled data into tickets.

15. Batch Operations

ERP batch jobs are operationally critical: posting, MRP, depreciation, dunning, report refresh, exchange rate import, bank statement import, migration, and period close.

15.1 Batch observability

15.2 Batch run state machine

A batch job is not complete merely because the process exits. It is complete when its business output is reconciled.

16. Read Model and Projection Operations

ERP read models and reports need operational visibility.

16.1 Projection signals

16.2 Projection failure handling

Do not let reports silently serve incorrect or stale data without metadata.

17. Production Data Repair

ERP production repair must be controlled. Direct SQL updates are sometimes tempting, but they are rarely defensible unless wrapped in strict process and evidence.

17.1 Repair principles

17.2 Repair request model

A repair without reconciliation is only a mutation, not a completed remediation.

18. Capacity and Workload Operations

ERP traffic is not uniform. It has strong business cycles.

18.1 ERP workload spikes

18.2 Capacity signals

Capacity planning must model business calendar, not just average traffic.

19. SLOs for ERP

Service Level Objectives should reflect business impact.

19.1 Example ERP SLOs

19.2 Error budgets for ERP

For ERP, error budget is not only downtime. It can include:

• delayed postings;
• unreconciled differences;
• stuck workflows;
• stale critical reports;
• duplicate suppressed attempts;
• manual repair volume;
• failed batch retries;
• integration exception backlog.

A system can be “available” but still consume its business error budget.

20. Deployment and Release Observability

ERP release risk is high because small changes in rules, config, or reports can have large downstream effects.

20.1 Release telemetry

20.2 Canary for ERP

A good ERP canary checks not only HTTP health but business paths:

• create draft document in test tenant;
• route approval task;
• post synthetic accounting event in sandbox scope;
• process outbox message;
• update read model;
• verify audit event;
• verify report/checkpoint freshness.

Never use production financial documents as destructive canaries.

21. Multi-Tenant Operations

Multi-tenant ERP operations require tenant-aware observability and containment.

21.1 Tenant-aware signals

21.2 Isolation operations

Tenant containment is an operational feature, not merely an architectural statement.

22. Security Observability

ERP security monitoring must include business controls.

22.1 Security signals

22.2 Control observability

Every important control should produce metrics:

erp.control.denied.count{control="MAKER_CHECKER", process="P2P"}
erp.control.override.count{control="CREDIT_LIMIT", company="ID01"}
erp.emergency_access.active.count{tenant="tenant-a"}
erp.sensitive_export.rows.count{report="VENDOR_BANK_ACCOUNTS"}

Security observability should detect business abuse, not only infrastructure attacks.

23. Audit and Evidence Operations

Audit evidence must be available, queryable, and trustworthy.

23.1 Evidence lookup questions

Support and audit teams should be able to answer:

• Who created this document?
• Who approved it?
• What authority did they have at the time?
• Which config version was used?
• Which state transitions occurred?
• Which accounting entries were generated?
• Which integration messages were sent?
• Which report included this transaction?
• Was the document corrected, reversed, voided, or amended?
• Which support/admin actions touched it?

23.2 Evidence timeline

The timeline should be reconstructable without querying random logs manually.

24. Production Readiness Review

Before launching an ERP capability, perform a production readiness review.

24.1 Review areas

24.2 Launch gate

A capability should not launch unless:

• dashboards exist;
• critical alerts exist;
• runbooks exist;
• support lookup exists;
• audit evidence exists;
• retry/repair path exists;
• reconciliation path exists where applicable;
• business owner knows what “healthy” means;
• on-call knows what to do when unhealthy.

25. Postmortems and Learning Loops

ERP incidents should improve the system.

25.1 Postmortem questions

25.2 Learning loop

A postmortem without test, alert, runbook, or design improvement is incomplete.

26. Source Notes

This material combines ERP-specific operational design with modern Java observability practices.

Relevant baseline references:

• OpenTelemetry Java documentation introduces generating and collecting telemetry such as metrics, logs, and traces using OpenTelemetry Java APIs and SDKs.
• OpenTelemetry Java instrumentation includes zero-code and agent-based instrumentation options for Java applications.
• Spring Boot Actuator observability documents logs, metrics, and traces and integrates with Micrometer Observation.
• Spring Boot tracing auto-configures Micrometer Tracing and supports OpenTelemetry with OTLP.
• Java Flight Recorder is useful for Java performance and production diagnostics.
• OWASP logging guidance remains relevant for designing safe, useful application logs.

The ERP-specific layer is the business observability model: document timeline, workflow health, posting health, reconciliation gaps, control overrides, projection lag, and support repair evidence.

27. Kaufman 20-Hour Practice Plan

Hour 1-3: Define business health signals

Pick one ERP process, such as order-to-cash.

Define metrics for:

• orders submitted;
• orders blocked by credit control;
• shipments pending;
• invoices posted;
• AR settlements;
• failed postings;
• reconciliation gaps;
• report lag.

Hour 4-6: Add correlation model

Design headers/fields for:

• trace ID;
• correlation ID;
• causation ID;
• command ID;
• document ID/number;
• tenant/company;
• batch run ID.

Hour 7-9: Build structured logs

Implement structured logs for:

• command received;
• authorization decision;
• lifecycle transition;
• audit write;
• outbox publish;
• consumer processing;
• retry/failure.

Hour 10-12: Build dashboards

Create dashboard sketches for:

• process owner;
• finance operations;
• integration operations;
• engineering on-call;
• support desk.

Hour 13-15: Write runbooks

Write runbooks for:

• failed posting backlog;
• stuck workflow;
• duplicate payment confirmation;
• stale report projection;
• reconciliation gap.

Hour 16-18: Design operational tools

Specify admin console actions:

• retry posting;
• park message;
• reassign approval;
• rebuild projection;
• open support case;
• capture repair evidence.

Hour 19-20: Simulate incident

Run a tabletop exercise:

• create duplicate payment confirmation;
• break posting rule config;
• delay read model projection;
• produce reconciliation gap;
• walk through detection, triage, containment, repair, reconciliation, postmortem.

28. Design Review Checklist

Use this checklist for ERP observability and operations review.

Signals

• Are business health metrics defined for the capability?
• Are technical metrics sufficient for latency, errors, saturation, and backlog?
• Are logs structured and safe from sensitive data leakage?
• Are traces meaningful at business operation boundaries?
• Are correlation IDs propagated across async messages?

Dashboards and alerts

• Do dashboards exist for business owner, support, integration, finance, and engineering?
• Are alerts actionable with severity, owner, business impact, and runbook?
• Are alert thresholds based on ERP workload and calendar?
• Are stale reports and projection lag visible?
• Are reconciliation gaps visible?

Supportability

• Can support search by document number, user, batch run, and correlation ID?
• Is document lifecycle timeline visible?
• Are workflow tasks, postings, integrations, and audit events connected?
• Are errors classified with reason codes and support actions?
• Are safe retry/repair tools available?

Operations

• Are batch jobs observable with run ID, checkpoint, progress, and reconciliation result?
• Can failed messages be parked, retried, or routed to exception queue safely?
• Can tenant-level containment be applied?
• Are production repairs approved, evidenced, dry-run-capable, and reconciled?
• Are runbooks tested through tabletop exercises?

Incident response

• Can blast radius be identified by tenant/company/process/document type?
• Are containment switches available for risky integrations/processes?
• Is unknown outcome handled differently from safe retry?
• Does postmortem feed tests, metrics, alerts, runbooks, and design improvements?

29. Summary

ERP observability is not only logs, metrics, and traces. It is the ability to understand business truth under failure.

The central ideas:

• A green server dashboard does not mean the ERP business process is healthy.
• Every important command needs correlation across request, workflow, ledger, integration, report, and audit.
• Metrics should include business health: stuck workflows, failed postings, reconciliation gaps, projection lag, duplicate suppression, and close blockers.
• Logs must be structured, safe, and searchable by business identifiers.
• Traces must represent meaningful ERP operations.
• Alerts need owner, severity, business impact, and runbook.
• Support tools must repair through controlled, audited operations, not ad-hoc database mutation.
• Batch, read models, and integrations require first-class operational visibility.
• Incidents should improve invariants, tests, metrics, alerts, runbooks, and design.

A top ERP engineer asks:

When this fails in production, how will we know, how will we explain it, how will we fix it safely, and how will we prove what happened?

That question separates demo-ready ERP from enterprise-grade ERP.