1. What is the operation name?
2. What is the resource or aggregate?
3. What is the correlation ID?
4. What is the actor or system principal?
5. What state transition, command, event, or query occurred?
6. Which dependency was called?
7. What was the latency budget?
8. What decision was made?
9. What invariant was protected?
10. What evidence will exist after the fact?

Error occurred
Something went wrong
Failed to process
Invalid request

CaseSubmitted
EvidenceAttached
RiskScoreCalculated
CaseAssigned
EscalationTriggered
ReviewCompleted
EnforcementActionApproved

actor=user:123
operation=APPROVE_ENFORCEMENT_ACTION
resource=case:ABC-2026-00091
policyVersion=authz-policy-2026.06.12
result=ALLOW
reason=actor has role SENIOR_REVIEWER and case state is PENDING_APPROVAL

service:
  name: case-command-service
  domain: enforcement

context_fields:
  correlation_id: string
  trace_id: string
  actor_id: string?
  tenant_id: string
  case_id: string?
  command_id: string?
  workflow_instance_id: string?
  policy_version: string?

operation_fields:
  operation: string
  outcome: SUCCESS | FAILED | REJECTED | RETRIED | SKIPPED | QUARANTINED
  error_class: string?
  latency_ms: number

sensitive_fields:
  never_log:
    - password
    - token
    - raw_document_content
    - national_id
    - full_address

public record TelemetryContext(
        String correlationId,
        String traceId,
        String tenantId,
        String actorId,
        String caseId,
        String workflowInstanceId,
        String policyVersion
) {
    public Map<String, String> logFields() {
        Map<String, String> fields = new LinkedHashMap<>();
        putIfPresent(fields, "correlation_id", correlationId);
        putIfPresent(fields, "trace_id", traceId);
        putIfPresent(fields, "tenant_id", tenantId);
        putIfPresent(fields, "actor_id", actorId);
        putIfPresent(fields, "case_id", caseId);
        putIfPresent(fields, "workflow_instance_id", workflowInstanceId);
        putIfPresent(fields, "policy_version", policyVersion);
        return Map.copyOf(fields);
    }

    private static void putIfPresent(Map<String, String> map, String key, String value) {
        if (value != null && !value.isBlank()) {
            map.put(key, value);
        }
    }
}

[ ] Every service has a stable service.name.
[ ] Every inbound request receives or creates a correlation ID.
[ ] Every log has operation and outcome.
[ ] Every failure has error_class.
[ ] Every metric has bounded labels.
[ ] Sensitive fields are explicitly blocked.
[ ] Audit fields are not dependent on best-effort logs.
[ ] The contract is tested or linted.

public final class CorrelationIds {
    public static final String HEADER = "X-Correlation-Id";

    private CorrelationIds() {}

    public static String fromInboundHeader(String value) {
        if (value == null || value.isBlank()) {
            return UUID.randomUUID().toString();
        }
        if (value.length() > 128) {
            throw new IllegalArgumentException("Correlation ID is too long");
        }
        return value;
    }
}

public record MessageEnvelope<T>(
        UUID eventId,
        String eventType,
        String correlationId,
        String causationId,
        String tenantId,
        Instant occurredAt,
        int schemaVersion,
        T payload
) {}

{
  "level": "INFO",
  "service": "case-command-service",
  "operation": "case.submit",
  "outcome": "SUCCESS",
  "correlation_id": "8ec9f6d2-8c8a-4e4b-98f8-23f071a71d45",
  "tenant_id": "tenant-01",
  "actor_id": "user-778",
  "case_id": "CASE-2026-0091",
  "workflow_instance_id": "wf-4451",
  "latency_ms": 84,
  "message": "Case submitted"
}

public final class LogEvent {
    private final Map<String, Object> fields = new LinkedHashMap<>();

    private LogEvent(String operation) {
        fields.put("operation", operation);
    }

    public static LogEvent operation(String operation) {
        return new LogEvent(operation);
    }

    public LogEvent field(String key, Object value) {
        if (value != null) {
            fields.put(key, value);
        }
        return this;
    }

    public LogEvent context(TelemetryContext context) {
        fields.putAll(context.logFields());
        return this;
    }

    public Map<String, Object> fields() {
        return Map.copyOf(fields);
    }
}

log.info("case.submitted {}", LogEvent.operation("case.submit")
        .context(context)
        .field("case_id", caseId.value())
        .field("outcome", "SUCCESS")
        .field("latency_ms", elapsed.toMillis())
        .fields());

try {
    transitionCase(command);
} catch (Exception e) {
    log.error("Failed", e);
}

try {
    transitionCase(command);
} catch (InvalidTransitionException e) {
    audit.denied(command, e.reason());
    throw e;
} catch (Exception e) {
    diagnostics.operationFailed("case.transition", command.caseId(), e);
    throw e;
}

public interface CaseMetrics {
    void commandAccepted(String commandType);
    void commandRejected(String commandType, String reasonClass);
    void transitionCompleted(String fromState, String toState);
    void transitionFailed(String fromState, String toState, String errorClass);
    void workflowLagRecorded(Duration lag);
    void deadLetterCreated(String messageType, String reasonClass);
}

case_command_total
case_command_latency_seconds
case_transition_total
case_workflow_lag_seconds
case_dead_letter_total
case_outbox_pending
case_retry_total

command_type=SUBMIT_CASE
outcome=SUCCESS
error_class=VALIDATION
dependency=identity-service
state=UNDER_REVIEW

user_id=user-123
case_id=CASE-2026-0091
email=person@example.com
exception_message=Connection refused to 10.1.2.3
raw_path=/cases/CASE-2026-0091/evidence/EV-123

handle
process
call
run
execute

HTTP POST /cases
case.submit
risk.calculate
document.verify
outbox.persist

service.name
operation.name
tenant.id
case.type
command.type
outcome
error.class
dependency.name
messaging.system
messaging.destination

public final class TracedCaseSubmissionHandler {
    private final Tracer tracer;
    private final SubmitCaseHandler delegate;

    public TracedCaseSubmissionHandler(Tracer tracer, SubmitCaseHandler delegate) {
        this.tracer = tracer;
        this.delegate = delegate;
    }

    public CaseId handle(SubmitCase command) {
        Span span = tracer.spanBuilder("case.submit")
                .setAttribute("command.type", "SubmitCase")
                .setAttribute("tenant.id", command.tenantId().value())
                .startSpan();

        try (Scope ignored = span.makeCurrent()) {
            CaseId id = delegate.handle(command);
            span.setAttribute("outcome", "SUCCESS");
            return id;
        } catch (DomainException e) {
            span.setAttribute("outcome", "REJECTED");
            span.setAttribute("error.class", e.errorClass());
            throw e;
        } catch (RuntimeException e) {
            span.setAttribute("outcome", "FAILED");
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

public record WorkItem<T>(
        String correlationId,
        String traceParent,
        T payload
) {}

public record CaseTimelineEntry(
        CaseId caseId,
        Instant occurredAt,
        String eventType,
        String actorId,
        String source,
        String fromState,
        String toState,
        String reasonCode,
        String correlationId,
        Map<String, String> safeAttributes
) {}

workflow_instance_id
case_id
from_state
to_state
trigger
actor/system
guard_result
action_result
latency_ms
correlation_id
policy_version
failure_class

public enum ErrorClass {
    VALIDATION,
    AUTHENTICATION,
    AUTHORIZATION,
    CONFLICT,
    NOT_FOUND,
    RATE_LIMITED,
    DEPENDENCY_TIMEOUT,
    DEPENDENCY_UNAVAILABLE,
    SERIALIZATION,
    DATA_INTEGRITY,
    INVARIANT_VIOLATION,
    BUG,
    UNKNOWN
}

public interface ClassifiedFailure {
    ErrorClass errorClass();
}

public final class InvalidCaseTransitionException extends RuntimeException implements ClassifiedFailure {
    private final ErrorClass errorClass = ErrorClass.CONFLICT;

    public InvalidCaseTransitionException(String message) {
        super(message);
    }

    @Override
    public ErrorClass errorClass() {
        return errorClass;
    }
}

public record DiagnosticEnvelope<T>(
        UUID messageId,
        String messageType,
        String schemaVersion,
        String producer,
        String correlationId,
        String causationId,
        String tenantId,
        Instant producedAt,
        int attempt,
        T payload
) {}

message_id
message_type
consumer
failure_class
failure_message_sanitized
first_failed_at
last_failed_at
attempt_count
correlation_id
payload_pointer
replay_eligible
quarantine_reason

@GetMapping("/health")
public String health() {
    database.query("select count(*) from huge_table");
    downstream.callExpensiveEndpoint();
    return "ok";
}

liveness: should the orchestrator restart this process?
readiness: should traffic be routed here?
diagnostics: what is degraded and why?

operation
dependency
attempt
max_attempts
backoff_ms
jittered=true|false
error_class
final_outcome

retry_attempt_total{dependency,error_class}
retry_exhausted_total{dependency,error_class}
retry_delay_seconds{dependency}

breaker_name
state=CLOSED|OPEN|HALF_OPEN
state_transition
failure_rate
slow_call_rate
permitted_calls
rejected_calls

bulkhead_name
active_calls
max_concurrent_calls
queue_depth
rejected_calls
wait_time

shed_reason
priority
queue_depth
capacity_limit
request_class

fallback_used_total{operation,fallback_type,reason_class}

cache_request_total{cache_name,outcome=hit|miss|load_success|load_failure}
cache_load_latency_seconds{cache_name}
cache_eviction_total{cache_name,reason}
cache_entry_count{cache_name}
cache_stampede_prevented_total{cache_name}
cache_stale_served_total{cache_name}

cache_name
key_class
key_hash
version
source_version
ttl_ms
age_ms
loader_latency_ms

workflow_instance_started_total{workflow_type}
workflow_transition_total{workflow_type,from_state,to_state,outcome}
workflow_transition_latency_seconds{workflow_type,transition}
workflow_stuck_instance_total{workflow_type,state}
workflow_timer_lag_seconds{workflow_type,timer_type}
workflow_compensation_total{workflow_type,outcome}
workflow_escalation_total{workflow_type,reason}

public record StuckWorkflowRule(
        String workflowType,
        String state,
        Duration maxAge,
        String escalationReason
) {}

public List<StuckWorkflow> detect(Instant now, List<WorkflowInstance> instances) {
    return instances.stream()
            .filter(instance -> instance.stateEnteredAt().plus(maxAgeFor(instance)).isBefore(now))
            .map(instance -> new StuckWorkflow(
                    instance.id(),
                    instance.workflowType(),
                    instance.state(),
                    Duration.between(instance.stateEnteredAt(), now)))
            .toList();
}

Show all transitions for case CASE-2026-0091.
Show all failed guard evaluations for workflow wf-4451.
Show all cases stuck in UNDER_REVIEW for more than 5 days.
Show all escalations triggered by SLA timer between date A and B.
Show all compensation actions after external dependency failure.

public record AuthorizationDecisionEvent(
        String decisionId,
        String actorId,
        String action,
        String resourceType,
        String resourceIdHash,
        String tenantId,
        String policyVersion,
        String result,
        String reasonCode,
        Instant decidedAt,
        String correlationId
) {}

authz_decision_total{action,resource_type,result,reason_code}
authz_policy_error_total{policy_version,error_class}
authz_cache_request_total{outcome}

User john@example.com denied because salary=123456 and investigationFlag=true

actor_id=user-778 action=CASE_VIEW resource_type=CASE result=DENY reason_code=OWNERSHIP_MISMATCH policy_version=2026.06.12

Incident ID:
Time window:
Affected tenants/users/resources:
Primary symptom:
Relevant deployments:
Relevant feature flags:
Top-level metrics:
Representative traces:
Error classes:
Dead-letter/quarantine records:
Workflow timeline:
Audit records:
Data reconciliation result:
Root cause hypothesis:
Confirmed root cause:
Corrective actions:
Preventive telemetry gaps:

Enable debug spans for tenant tenant-01 for 30 minutes.
Capture sanitized validation failure details for command type SubmitCase.
Increase sampling for correlation ID C-123.
Enable dead-letter payload pointer capture for consumer X.

[ ] Scope is narrow.
[ ] Duration is bounded.
[ ] Access is authorized.
[ ] Sensitive fields are still blocked.
[ ] Flag usage is audited.
[ ] Cost impact is understood.

public CaseId submit(TelemetryContext context, SubmitCase command) {
    return handler.handle(context, command);
}

public final class ContextScope implements AutoCloseable {
    private final Map<String, String> previous;

    private ContextScope(Map<String, String> previous) {
        this.previous = previous;
    }

    public static ContextScope open(TelemetryContext context) {
        Map<String, String> previous = captureMdc();
        context.logFields().forEach(MDC::put);
        return new ContextScope(previous);
    }

    @Override
    public void close() {
        restoreMdc(previous);
    }

    private static Map<String, String> captureMdc() {
        Map<String, String> copy = MDC.getCopyOfContextMap();
        return copy == null ? Map.of() : copy;
    }

    private static void restoreMdc(Map<String, String> previous) {
        MDC.clear();
        previous.forEach(MDC::put);
    }
}

try (ContextScope ignored = ContextScope.open(context)) {
    service.handle(command);
}

executor_active_threads
executor_queue_depth
executor_completed_task_total
executor_rejected_task_total
executor_task_latency_seconds
lock_wait_seconds
lock_hold_seconds
semaphore_available_permits
structured_scope_cancelled_total

partition_backlog{partition="17"} 10420
partition_lag_seconds{partition="17"} 912

p95 command latency exceeds SLO for 10 minutes
error budget burn rate too high
outbox pending age exceeds threshold
workflow stuck count increasing
DLQ count increasing for critical consumer
authorization policy evaluation failing

any ERROR log
CPU above 70% for 1 minute
one dependency timeout
one retry happened
cache miss rate changed slightly

what is broken
why it matters
scope of impact
current value
threshold
start time
runbook link
recent deploy link
relevant dashboard
example trace/log query

We have 30 graphs. Nobody knows which one matters.

1. Inventory current logs, metrics, traces, and audit records.
2. Identify the top 10 production questions that are hard to answer.
3. Define common fields: correlation_id, tenant_id, operation, outcome, error_class.
4. Add boundary middleware for correlation context.
5. Standardize error taxonomy.
6. Add metrics facade per domain/service.
7. Add tracing around domain-significant operations.
8. Add workflow/domain timelines where needed.
9. Remove sensitive/noisy logs.
10. Add telemetry tests for critical flows.

log.info("processing case " + caseId);
try {
    service.process(caseId);
    log.info("done");
} catch (Exception e) {
    log.error("error", e);
}

OperationTimer timer = OperationTimer.start();

try (ContextScope ignored = ContextScope.open(context)) {
    service.process(caseId);

    metrics.commandAccepted("ProcessCase");
    log.info("case.process completed operation=case.process outcome=SUCCESS case_id={} latency_ms={}",
            caseId.value(), timer.elapsedMillis());
} catch (ClassifiedFailure e) {
    metrics.commandRejected("ProcessCase", e.errorClass().name());
    log.warn("case.process rejected operation=case.process outcome=REJECTED case_id={} error_class={}",
            caseId.value(), e.errorClass());
    throw e;
} catch (RuntimeException e) {
    metrics.commandRejected("ProcessCase", ErrorClass.UNKNOWN.name());
    log.error("case.process failed operation=case.process outcome=FAILED case_id={}",
            caseId.value(), e);
    throw e;
}

[ ] correlation ID is created when absent;
[ ] inbound correlation ID is propagated;
[ ] sensitive fields are redacted;
[ ] metrics are emitted with bounded labels;
[ ] failure classes are mapped correctly;
[ ] audit record is written for authorization decision;
[ ] workflow transition timeline entry is written;
[ ] dead-letter contains replay metadata;
[ ] fallback emits degraded metric;
[ ] retry emits attempt count.

public final class RecordingCaseMetrics implements CaseMetrics {
    private final List<String> events = new ArrayList<>();

    @Override
    public void commandAccepted(String commandType) {
        events.add("accepted:" + commandType);
    }

    @Override
    public void commandRejected(String commandType, String reasonClass) {
        events.add("rejected:" + commandType + ":" + reasonClass);
    }

    @Override
    public void transitionCompleted(String fromState, String toState) {
        events.add("transition:" + fromState + "->" + toState);
    }

    @Override
    public void transitionFailed(String fromState, String toState, String errorClass) {
        events.add("transition_failed:" + fromState + "->" + toState + ":" + errorClass);
    }

    @Override
    public void workflowLagRecorded(Duration lag) {
        events.add("workflow_lag:" + lag.toMillis());
    }

    @Override
    public void deadLetterCreated(String messageType, String reasonClass) {
        events.add("dead_letter:" + messageType + ":" + reasonClass);
    }

    public boolean contains(String event) {
        return events.contains(event);
    }
}

Context
[ ] Is there a correlation ID at every entry boundary?
[ ] Is trace context propagated across HTTP/messaging/async?
[ ] Are command/event/workflow IDs distinct from correlation IDs?

Logs
[ ] Are logs structured?
[ ] Do logs include operation, outcome, and error_class?
[ ] Are sensitive fields excluded or redacted?
[ ] Are validation rejections not logged as noisy errors?

Metrics
[ ] Are metrics named consistently?
[ ] Are labels bounded?
[ ] Are latency distributions captured?
[ ] Are queue depth, lag, retries, DLQs, and fallback visible?

Traces
[ ] Are domain-significant spans named?
[ ] Are dependency calls visible?
[ ] Is sampling policy appropriate for errors and slow requests?

Workflow/Data
[ ] Can we reconstruct a case timeline?
[ ] Can we explain stuck states?
[ ] Can we link audit records to decisions?

Reliability
[ ] Are retries observable?
[ ] Are circuit breaker transitions observable?
[ ] Are bulkhead rejections observable?
[ ] Are cache stale/fallback behaviors observable?

Operations
[ ] Do alerts map to user impact or risk?
[ ] Does each alert include a runbook?
[ ] Is there an incident evidence package template?

service.name
operation names
context fields
error classes
metric names
sensitive fields
required audit records

transition event
guard result
action result
metric
log event
audit record if needed
stuck detector rule

first attempt timeout
second attempt success
retry exhausted
circuit opened
fallback used

PII in logs
unique IDs in metric labels
ambiguous ERROR logs
silent fallback
catch-and-log-only