Request ini datang dari siapa?
Masuk endpoint mana?
Memproses tenant mana?
Memanggil downstream apa?
Query apa yang lambat?
Event Kafka apa yang dipublish?
Error ini retryable atau bukan?
Apakah customer terdampak?
Apakah data sensitif bocor ke log?

Code changes behavior.
Log changes diagnosability.
Bad logs increase MTTR.
Leaky logs increase security risk.
Noisy logs increase cost and hide signal.

User failed to create order because customer not found

{
  "level": "WARN",
  "event": "order.create.rejected",
  "reason": "customer_not_found",
  "correlationId": "c-7f3a",
  "tenantId": "tenant-a",
  "customerId": "cust-123",
  "orderId": null,
  "httpMethod": "POST",
  "httpPath": "/orders",
  "status": 404,
  "durationMs": 42
}

- field konsisten
- value bisa dicari
- event name stabil
- sensitive data tidak bocor
- cardinality dikontrol
- correlation context selalu ada

message: "Order creation rejected because customer was not found"
event: "order.create.rejected"
reason: "customer_not_found"
status: 404
correlationId: "..."
tenantId: "..."

Human-readable message explains what happened.
Structured fields explain how to find, aggregate, and correlate it.

log.info("Create order failed for tenant " + tenantId + " user " + userId + " request " + request);

- raw request bisa berisi PII/secret
- sulit dicari per field
- inconsistent naming
- expensive string concatenation jika level disabled
- rentan log injection jika input mentah masuk message

log.warn("Order creation rejected: customer not found",
    kv("event", "order.create.rejected"),
    kv("reason", "customer_not_found"),
    kv("tenantId", tenantId),
    kv("customerId", customerId),
    kv("correlationId", correlationId));

timestamp
level
serviceName
serviceVersion
environment
podName / hostName
threadName
loggerName
event
message
correlationId
requestId
traceId
spanId
tenantId
principalId / subjectId
httpMethod
httpPathTemplate
httpRoute
httpStatus
durationMs
errorCode
exceptionClass

Prefer path template over raw path for metrics/log aggregation.

Good: /quotes/{quoteId}/items/{itemId}
Risky: /quotes/Q-123/items/I-456

One HTTP request to one service = one requestId.

- debugging local service
- matching access log and application log
- support ticket for one API call

One user action / business command / workflow transaction = one correlationId.

POST /quotes
  -> calls pricing service
  -> calls catalog service
  -> writes database
  -> publishes QuoteCreated event

One distributed trace = one traceId.

Event A caused command B.
Command B caused event C.

QuoteSubmitted event
  causationId = commandId of SubmitQuote

OrderCreated event
  causationId = eventId of QuoteSubmitted

Event ini muncul karena request mana?
Retry ini retry dari operasi mana?
Reconciliation job ini memperbaiki kegagalan apa?

requestId
  Unique per inbound service request.

correlationId
  Stable across business flow.

traceId/spanId
  From tracing system.

causationId
  Parent event/command/message/request that caused this operation.

messageId / eventId
  Unique event/message identifier for asynchronous processing.

idempotencyKey
  Caller-provided or generated deduplication key for unsafe/retryable operations.

MDC.put("correlationId", correlationId);
MDC.put("tenantId", tenantId);
try {
    resourceMethod.invoke();
} finally {
    MDC.clear();
}

- setiap log dalam request otomatis punya context
- service layer tidak perlu pass correlationId ke setiap method hanya untuk logging
- compatible dengan JSON log encoder

- MDC berbasis ThreadLocal
- thread pool reuse bisa membawa context lama jika tidak dibersihkan
- async executor kehilangan context jika tidak dipropagate
- Kafka consumer thread dapat mencampur message context jika cleanup buruk

MDC must be set at boundary and cleared at boundary.

- inbound HTTP filter
- Kafka consumer poll/message handler
- scheduled job start/end
- executor task wrapper
- workflow worker task boundary

@Provider
@Priority(Priorities.AUTHENTICATION - 100)
public class CorrelationFilter implements ContainerRequestFilter, ContainerResponseFilter {

    @Override
    public void filter(ContainerRequestContext request) {
        String incomingCorrelationId = request.getHeaderString("X-Correlation-ID");
        String correlationId = isValid(incomingCorrelationId)
            ? incomingCorrelationId
            : generateCorrelationId();

        String requestId = generateRequestId();

        request.setProperty("correlationId", correlationId);
        request.setProperty("requestId", requestId);

        MDC.put("correlationId", correlationId);
        MDC.put("requestId", requestId);
        MDC.put("httpMethod", request.getMethod());
        MDC.put("httpPath", request.getUriInfo().getPath());
    }

    @Override
    public void filter(ContainerRequestContext request, ContainerResponseContext response) {
        try {
            String correlationId = (String) request.getProperty("correlationId");
            String requestId = (String) request.getProperty("requestId");

            response.getHeaders().putSingle("X-Correlation-ID", correlationId);
            response.getHeaders().putSingle("X-Request-ID", requestId);
        } finally {
            MDC.clear();
        }
    }
}

- Header name harus mengikuti standard internal.
- Jangan percaya semua inbound ID mentah tanpa validasi panjang/charset.
- Jangan generate ulang correlation ID jika caller sudah mengirim ID valid.
- Response harus mengembalikan correlation ID agar client/support bisa melaporkan ID tersebut.

PII leak
credential leak
payment/tax/customer data leak
large payload cost
multipart/binary corruption
performance overhead
legal/compliance exposure

Do not log full request/response body in production.

- gunakan allowlist field
- gunakan sampling
- gunakan environment gating
- gunakan redaction
- gunakan temporary flag dengan expiry
- hindari multipart/binary
- audit siapa mengaktifkan payload logging

- customer identity
- address
- pricing detail
- discount rule
- tax data
- contract terms
- product configuration
- commercial entitlement

- local debugging
- temporary deep diagnosis
- low-level protocol detail

- selected decision branch
- non-sensitive intermediate state
- feature flag evaluation summary

- service startup completed
- config profile loaded
- request completed access log
- job started/completed
- event published/consumed summary

- validation/domain rejection yang signifikan
- downstream transient failure yang recovered
- retry exhausted but converted to known error
- degraded mode activated
- suspicious but not confirmed security event

- unexpected exception
- database unavailable
- event publish failed after retry
- job failed permanently
- data inconsistency detected

If everything is ERROR, nothing is actionable.
If failures are INFO, operations will miss incidents.

http.request.completed
http.request.rejected
quote.create.accepted
quote.create.rejected
quote.price.calculated
order.submit.started
order.submit.failed
kafka.event.published
kafka.event.consumed
kafka.event.rejected
database.query.slow
feature_flag.evaluated
job.reconciliation.completed
security.authorization.denied

<noun>.<action>.<outcome>

"Something happened"
"Error"
"Failed"
"Debug"
"Processing"

Quote cannot be submitted because required approval is missing.

{
  "event": "quote.submit.rejected",
  "reason": "approval_missing",
  "quoteId": "Q-123",
  "tenantId": "tenant-a",
  "status": 409,
  "correlationId": "c-1"
}

Apakah rejection ini expected business behavior, abuse signal, integration bug, atau data inconsistency?

event
dependencyName
dependencyType
operation
method
urlTemplate / route
status / errorCode
durationMs
attempt
retryable
circuitBreakerState
correlationId
traceId

{
  "event": "http.client.completed",
  "dependencyName": "catalog-service",
  "operation": "getProductOffering",
  "httpMethod": "GET",
  "route": "/product-offerings/{id}",
  "status": 200,
  "durationMs": 37,
  "attempt": 1,
  "correlationId": "c-123"
}

{
  "event": "http.client.failed",
  "dependencyName": "pricing-service",
  "operation": "calculatePrice",
  "errorType": "timeout",
  "durationMs": 2000,
  "attempt": 3,
  "retryable": true,
  "finalFailure": true,
  "correlationId": "c-123"
}

queryName
repositoryName
operation
rowCount
durationMs
sqlState
constraintName
lockWaitMs
transactionId/correlationId

{
  "event": "database.query.slow",
  "repository": "QuoteRepository",
  "queryName": "findActiveQuoteByCustomer",
  "durationMs": 850,
  "rowCount": 42,
  "tenantId": "tenant-a",
  "correlationId": "c-123"
}

event = kafka.event.published / kafka.event.publish_failed
topic
partition
keyHash atau safeKey
eventType
eventId
schemaVersion
correlationId
causationId
attempt
durationMs

event = kafka.event.consumed / kafka.event.rejected / kafka.event.failed
topic
partition
offset
consumerGroup
eventType
eventId
schemaVersion
correlationId
causationId
processingDurationMs
retryAttempt

tenantId or tenantKey
region / partition if relevant
configurationVersion if tenant-specific config affects behavior
catalogVersion if catalog-driven behavior affects result
pricingRuleVersion if pricing is involved

Quote result changed because tenant-specific catalog version changed.

- tenant identifier yang boleh muncul di log
- apakah perlu hashing/masking
- apakah tenant-specific config version tersedia
- apakah catalog/pricing effective date dilog secara aman

password
secret
access token
refresh token
API key
authorization header
cookie
private key
customer full name
email
phone
address
payment identifier
tax identifier
contract detail
full request body
full quote/order payload

- application logging helper
- JSON encoder/filter
- HTTP logging filter
- exception mapper
- observability exporter
- log ingestion pipeline

Redaction in only one layer is not enough.

- exception message
- third-party library log
- raw request dump
- debug statement
- SQL parameter logging
- stack trace with embedded payload

Expected domain/validation errors should not log full stack trace.
Unexpected technical errors should log stack trace once at boundary.
Retried transient errors may log summary per attempt and full stack only on final failure.

Same exception logged in repository, service, resource, and exception mapper.

- duplicate error count
- noisy alert
- inflated log cost
- confusing incident timeline

Log with full stack at ownership boundary.
Propagate typed exception upward.
ExceptionMapper logs final unhandled technical failure once.

User input contains newline or fake JSON field.

- forged log line
- broken log parser
- misleading audit trail

- structured fields via encoder
- sanitize control characters
- limit length
- avoid raw payload
- validate inbound correlation ID charset/length

requestId
traceId
spanId
userId
customerId
quoteId
orderId
raw path
raw query string
idempotencyKey
payload hash

Log can contain high-cardinality fields for search.
Metrics labels should avoid high-cardinality fields.
Dashboards should aggregate on stable low-cardinality dimensions.

serviceName
environment
endpointTemplate
statusClass
errorCategory
errorCode
dependencyName
eventType
tenantTier if allowed

method
path template
status
duration
request size
response size
caller identity class
correlationId
requestId
traceId

business event
technical decision
dependency result
error taxonomy
state transition

service name
version/build sha
environment
active profile
runtime/container type
JAX-RS implementation if known
registered key features
config source summary
safe default status
external dependencies configured
observability enabled/disabled

{
  "event": "service.startup.completed",
  "serviceName": "quote-service",
  "version": "1.42.0",
  "environment": "prod",
  "profile": "prod-aws",
  "jaxrsImplementation": "jersey",
  "configSources": ["env", "mounted-config", "secret-manager"],
  "observabilityEnabled": true
}

normal rollout
pod eviction
OOMKilled
crash loop
manual restart
node drain

event = service.shutdown.started
reason if known
inflightRequests
consumerPaused
jobStopped
durationMs
shutdownCompleted

[ ] Apakah endpoint baru punya request/correlation context?
[ ] Apakah log event name stabil dan searchable?
[ ] Apakah log punya errorCode/errorCategory untuk failure?
[ ] Apakah log level sesuai operational meaning?
[ ] Apakah expected domain/validation errors tidak spam stack trace?
[ ] Apakah unexpected technical failures punya stack trace di boundary yang tepat?
[ ] Apakah outbound HTTP/DB/Kafka calls punya context yang cukup?
[ ] Apakah tenant ID/config/catalog/pricing version dilog secara aman jika relevan?
[ ] Apakah PII/secret/token/request body tidak bocor?
[ ] Apakah correlation ID dikembalikan di response?
[ ] Apakah MDC dibersihkan pada request/message/job boundary?
[ ] Apakah async executor/Kafka consumer mempertahankan atau membersihkan context dengan benar?
[ ] Apakah log field names konsisten dengan standard internal?
[ ] Apakah high-cardinality fields tidak dipakai sebagai metric labels?
[ ] Apakah log volume wajar untuk traffic production?

Logging stack
[ ] SLF4J/Logback/Log4j2/java.util.logging?
[ ] JSON encoder dipakai atau plain text?
[ ] Apakah platform log ingestion parsing field secara benar?
[ ] Apakah ada standard event naming?

Correlation model
[ ] Header correlation ID standard apa yang dipakai?
[ ] Apakah request ID berbeda dari correlation ID?
[ ] Apakah trace ID dari OpenTelemetry dipakai?
[ ] Apakah causation ID dipakai di event-driven flow?

JAX-RS boundary
[ ] Apakah ada ContainerRequestFilter untuk correlation/MDC?
[ ] Apakah ada ContainerResponseFilter untuk response header dan cleanup?
[ ] Apakah ExceptionMapper menulis structured error log?
[ ] Apakah access log tersedia dari runtime/gateway/container?

Context propagation
[ ] Apakah MDC ikut executor/thread pool?
[ ] Apakah MDC dibersihkan setelah async task?
[ ] Apakah Kafka headers membawa correlation/causation ID?
[ ] Apakah scheduled job punya generated correlation/job run ID?

Security
[ ] Apakah Authorization/Cookie/Set-Cookie/header secret diredact?
[ ] Apakah request/response body logging disabled by default?
[ ] Apakah PII field policy terdokumentasi?
[ ] Apakah audit log dipisah dari application log?

Operations
[ ] Apakah log bisa dicari berdasarkan correlationId?
[ ] Apakah log bisa dicari berdasarkan tenantId jika policy mengizinkan?
[ ] Apakah error log terhubung dengan metrics/traces?
[ ] Apakah startup/shutdown log cukup untuk incident diagnosis?

Incident timeline tidak bisa dirangkai lintas service.
Support punya error screenshot tapi tidak bisa menemukan log.

- inbound filter tidak ada
- gateway tidak meneruskan header
- response tidak mengembalikan ID
- async boundary kehilangan context

- generate ID at edge/service boundary
- propagate outbound
- return ID to caller
- test propagation

Log request A muncul dengan tenant/correlation request B.

- thread reused
- MDC tidak clear di finally
- async task wrapper salah

- clear MDC in finally
- use context-aware executor
- add tests for cleanup

Token, PII, address, pricing detail, atau full payload muncul di log platform.

- raw request logging
- exception message berisi payload
- third-party debug log
- SQL parameter logging

- redaction filter
- disable raw body logging
- classify fields
- scan logs for known sensitive patterns

Log ingestion cost naik tajam.
Useful logs tertutup noise.
Alert menjadi noisy.

- error logged repeatedly inside retry loop
- debug/info log in hot path
- stack trace duplicated at many layers
- high-traffic endpoint logs too much

- log summary at boundary
- sample noisy logs
- reduce per-attempt logging
- aggregate metrics instead of logs

A log line should either help explain a state transition, a failure, a security-relevant event, or an operational lifecycle event.

Use structured fields, not string parsing.
Use stable event names.
Generate requestId per request.
Propagate correlationId across service boundary.
Use causationId across command/event chains.
Use traceId/spanId from tracing system.
Set MDC at boundary.
Clear MDC in finally.
Do not log raw body by default.
Redact secrets and PII.
Do not spam stack traces for expected errors.
Log full stack once for unexpected technical failures.
Prefer route template over raw path.
Control cardinality.
Review logging as production behavior, not cosmetic code.