Request ini berasal dari mana?
User/tenant mana yang memicunya?
Span mana parent-nya?
Event Kafka ini disebabkan oleh HTTP command apa?
Log ini milik request yang mana?
Kenapa audit trail terputus?
Kenapa trace tampak seperti beberapa request terpisah?

HTTP inbound request
  -> JAX-RS filter/resource method
  -> service/domain layer
  -> database call
  -> outbound HTTP call
  -> Kafka publish
  -> async executor
  -> Kafka consumer in another service
  -> reconciliation job

trace_id
span_id
parent_span_id
correlation_id
causation_id
request_id
tenant_id
user_id / subject
service_name
operation_name
idempotency_key
feature_flag_snapshot
locale / timezone
security principal

Propagate what is required for correctness, security, and observability.
Do not propagate sensitive data blindly.

traceparent: 00-<trace-id>-<span-id>-<flags>
tracestate: vendor-specific state

trace_id -> satu distributed execution graph
span_id  -> satu operation/node dalam graph

X-Correlation-ID: quote-request-8f37...

Cari semua log yang terkait satu external request/business operation.

HTTP command CreateQuote
  correlation_id = c-123
  causation_id   = request-001

Kafka event QuoteCreated
  correlation_id = c-123
  causation_id   = event-quote-created-001

Kafka event PricingRequested
  correlation_id = c-123
  causation_id   = event-quote-created-001

Never assume context crosses boundary automatically.

@Provider
@Priority(Priorities.AUTHENTICATION)
public final class RequestContextFilter implements ContainerRequestFilter, ContainerResponseFilter {

    @Override
    public void filter(ContainerRequestContext request) {
        String correlationId = firstNonBlank(
            request.getHeaderString("X-Correlation-ID"),
            UUID.randomUUID().toString()
        );

        MDC.put("correlation_id", correlationId);
        MDC.put("http_method", request.getMethod());
        MDC.put("http_path", request.getUriInfo().getPath());

        // Do not blindly trust tenant/user headers unless gateway/auth policy says so.
    }

    @Override
    public void filter(ContainerRequestContext request, ContainerResponseContext response) {
        String correlationId = MDC.get("correlation_id");
        if (correlationId != null) {
            response.getHeaders().putSingle("X-Correlation-ID", correlationId);
        }
        MDC.clear();
    }
}

MDC.clear() wajib dilakukan.

public final class OutboundHeaderFactory {
    public Map<String, String> buildHeaders(RequestContext ctx) {
        Map<String, String> headers = new HashMap<>();
        headers.put("X-Correlation-ID", ctx.correlationId());
        headers.put("X-Request-ID", UUID.randomUUID().toString());

        // W3C trace context is usually injected by OpenTelemetry instrumentation.
        // If manual propagation is used, verify exact propagator standard.

        return headers;
    }
}

// BAD: forward all inbound headers blindly
inboundHeaders.forEach(outboundRequest::header);

Authorization header bisa bocor ke downstream yang tidak berhak.
Internal-only header bisa keluar trust boundary.
Spoofed tenant header bisa dipakai oleh service lain.
Large headers bisa menyebabkan 431 Request Header Fields Too Large.

Whitelist propagated headers.

traceparent
tracestate
x-correlation-id
x-causation-id
event-id
event-type
event-version
source-service
tenant-id, jika policy mengizinkan

ProducerRecord<String, byte[]> record = new ProducerRecord<>(topic, key, payload);
record.headers().add("x-correlation-id", correlationId.getBytes(StandardCharsets.UTF_8));
record.headers().add("x-causation-id", commandId.getBytes(StandardCharsets.UTF_8));
record.headers().add("event-id", eventId.getBytes(StandardCharsets.UTF_8));
record.headers().add("source-service", "quote-order-service".getBytes(StandardCharsets.UTF_8));
producer.send(record);

EventPublisher
  -> enrich metadata
  -> inject trace context
  -> enforce event ID
  -> enforce event type/version
  -> publish

public void onMessage(ConsumerRecord<String, byte[]> record) {
    try (MdcScope ignored = MdcScope.fromKafka(record.headers())) {
        String correlationId = Mdc.get("correlation_id");
        log.info("Processing event topic={} partition={} offset={}",
            record.topic(), record.partition(), record.offset());

        handler.handle(record);
    }
}

Read Kafka headers.
Restore correlation_id into MDC.
Restore trace context into OTel context if instrumentation/manual propagation supports it.
Create consumer span.
Clear MDC after processing.
Commit offset only after safe processing boundary.

Consumer logs tidak punya correlation ID.
Trace baru dibuat untuk setiap consumer processing, tidak tersambung dengan producer.
MDC dari message sebelumnya bocor ke message berikutnya.
DLQ event kehilangan metadata original.
Retry topic membuat causation chain putus.

MDC.put("correlation_id", "c-123");
executor.submit(() -> {
    log.info("Running async task"); // correlation_id might be missing
});

public final class ContextAwareExecutor implements Executor {
    private final Executor delegate;

    public ContextAwareExecutor(Executor delegate) {
        this.delegate = delegate;
    }

    @Override
    public void execute(Runnable command) {
        Map<String, String> capturedMdc = MDC.getCopyOfContextMap();
        io.opentelemetry.context.Context capturedOtel = io.opentelemetry.context.Context.current();

        delegate.execute(() -> {
            Map<String, String> previous = MDC.getCopyOfContextMap();
            try (var scope = capturedOtel.makeCurrent()) {
                if (capturedMdc != null) {
                    MDC.setContextMap(capturedMdc);
                } else {
                    MDC.clear();
                }
                command.run();
            } finally {
                MDC.clear();
                if (previous != null) {
                    MDC.setContextMap(previous);
                }
            }
        });
    }
}

Security context propagation ke async task harus dibatasi.

Context leak antar request.
Memory leak jika value besar.
Data user/tenant bocor ke request lain.
Async task kehilangan context.
Test menjadi flaky karena state tersembunyi.

public final class TenantContextHolder {
    private static final ThreadLocal<String> TENANT = new ThreadLocal<>();

    public static void set(String tenant) { TENANT.set(tenant); }
    public static String get() { return TENANT.get(); }
}

Set di boundary yang jelas.
Clear di finally/filter.
Tidak dipakai sebagai pengganti parameter domain.
Tidak dipakai lintas async boundary tanpa wrapper.
Tidak menyimpan object besar.

Use explicit context object for business-sensitive decisions.
Use MDC/OTel context for observability metadata.

tenant tier
routing hint
experiment cohort
region hint

PII
pricing data
large payload
authorization decision
secret
full user profile

Baggage is not a secure data store.
Baggage is not an authorization source.
Baggage should be small, low-cardinality, and policy-approved.

X-Tenant-ID: tenant-a
X-User-ID: admin
X-Correlation-ID: c-123

Identity and tenant context must come from trusted authentication/authorization layer, not arbitrary client headers.

Correlation/request ID, jika divalidasi dan dibatasi panjang/formatnya.

authenticated subject
tenant identity
permission claims
service identity

Generate missing correlation ID.
Validate incoming correlation ID format/length.
Populate MDC.
Ensure response includes correlation ID.
Clear MDC in all outcomes.
Do not decide business authorization unless designed as auth filter.

event_id
correlation_id
causation_id
traceparent, if supported
event_type
event_version
occurred_at
source_service
source_instance, optional
producer_schema_version

Original event goes to retry topic.
Retry publisher creates new event headers from scratch.
DLQ contains payload only, no original metadata.
Operator cannot trace original HTTP request.

Original correlation ID preserved.
Original event ID preserved or referenced.
Retry attempt count added.
Last failure reason added safely.
Original topic/partition/offset captured.
DLQ record keeps diagnostic metadata.

x-original-topic
x-original-partition
x-original-offset
x-original-event-id
x-correlation-id
x-causation-id
x-retry-attempt
x-last-error-code
x-failed-at

public record RequestContext(
    String correlationId,
    String requestId,
    String tenantId,
    String subject,
    Instant receivedAt
) {}

@POST
@Path("/quotes")
public Response createQuote(CreateQuoteRequest request, @Context HttpHeaders headers) {
    RequestContext ctx = requestContextFactory.from(headers);
    QuoteResult result = quoteService.createQuote(ctx, request);
    return Response.status(Status.CREATED).entity(result).build();
}

Business-relevant context is explicit.
Tests can construct context easily.
No hidden dependency on ThreadLocal.
Context crossing async boundary becomes intentional.

product catalog
pricing rules
tax rules
currency
permissions
data partition
feature flags
routing
logging/audit

Tenant used for authorization/data access must come from trusted identity/config boundary.
Tenant used for logs must be sanitized and policy-approved.

Logs have missing correlation_id.
Trace shows multiple roots for one business operation.
Kafka consumer spans are disconnected.
DLQ messages cannot be traced to original request.
Tenant ID appears wrong in logs.
MDC fields from previous request appear in current request.

1. Check inbound HTTP headers.
2. Check request filter extraction.
3. Check MDC populated at resource method.
4. Check outbound HTTP/Kafka headers.
5. Check executor boundary capture/restore.
6. Check consumer header extraction.
7. Check MDC cleanup.
8. Check OTel propagator configuration.
9. Check gateway/proxy header mutation.
10. Check log formatter includes fields.

Send request with known X-Correlation-ID.
Assert response contains same correlation ID.
Assert logs include same correlation ID.
Assert outbound HTTP mock receives same correlation ID.
Assert Kafka record headers include same correlation ID.
Assert consumer logs include same correlation ID.

missing correlation ID generates one
invalid correlation ID rejected or normalized
MDC cleared after request
outbound header allowlist applied
Kafka headers include required metadata
executor wrapper preserves MDC/OTel context
security-sensitive headers are not forwarded

@Test
void clearsMdcAfterRequest() {
    filter.filter(requestContext);
    assertThat(MDC.get("correlation_id")).isEqualTo("c-123");

    filter.filter(requestContext, responseContext);
    assertThat(MDC.getCopyOfContextMap()).isNullOrEmpty();
}

JAX-RS test container
mock downstream HTTP server
Testcontainers Kafka
log capture/test appender

Does inbound filter validate and normalize correlation/request IDs?
Is MDC cleared in all code paths?
Are propagated HTTP headers allowlisted?
Is trace context propagated by OTel instrumentation or manual propagator?
Are Kafka headers populated consistently?
Are retry/DLQ headers preserving diagnostic metadata?
Does executor/thread pool capture and restore context?
Is security/tenant context sourced from trusted boundary?
Are sensitive values excluded from baggage/logs/headers?
Are tests covering missing/invalid/leaked context?

forward all headers
static ThreadLocal without clear
business logic reading tenant from MDC
Kafka event without event_id/correlation_id
async task using user identity indefinitely
MDC fields not included in log pattern

HTTP Context
- What inbound headers are standardized?
- Is W3C Trace Context used?
- Is X-Correlation-ID used separately from trace ID?
- Does gateway generate/normalize request IDs?
- Are spoofed identity/tenant headers stripped?

JAX-RS/Jersey Layer
- Which request/response filters populate MDC?
- Are filters registered explicitly or discovered by scanning?
- Is MDC cleared in response filter/finally block?
- Are ExceptionMappers preserving response correlation headers?

OpenTelemetry
- Is OTel Java agent used or manual SDK instrumentation?
- Which propagators are enabled?
- Are traces, metrics, and logs correlated?
- Is baggage enabled or restricted?

Kafka
- Which headers are mandatory for events?
- Are traceparent/correlation/causation IDs propagated?
- Are retry/DLQ topics preserving original metadata?
- Is event_id required?

Executors
- Are custom executors wrapped for MDC/OTel context?
- Are async jobs allowed to carry user context?
- Is security context propagated, transformed, or dropped?

Tenancy
- Where is tenant resolved?
- Is tenant context trusted from token, gateway, database, or request payload?
- Are tenant IDs logged? If yes, is this policy-approved?

Operations
- Can one production incident timeline be reconstructed from logs/traces/events?
- Are runbooks using correlation ID and trace ID consistently?

Context that affects correctness should be explicit.
Context that affects observability can live in MDC/OTel context.
Context that affects security must come from trusted identity boundary.
Context crossing async boundary must be captured/restored intentionally.
Context crossing service boundary must be allowlisted.
Context in Kafka must survive retry and DLQ.
Context must be cleaned up after use.

Given one customer-impacting issue, can we reconstruct the technical and business path across HTTP, DB, Kafka, jobs, and downstream services?

Inbound request receives context.
Service validates and normalizes context.
Business layer uses explicit safe context.
Observability layer records context.
Outbound integrations propagate allowed context.
Async/event boundaries preserve causality.
Cleanup prevents leakage.

Every externally triggered operation should be traceable without leaking sensitive data or trusting untrusted metadata.