This part does not assume CSG uses a specific JVM distribution, GC algorithm,
heap percentage, Kubernetes resource policy, Java base image, container runtime,
or performance tuning standard.

Treat JVM flags, container limits, GC policy, memory sizing, CPU limits,
heap dump policy, and production diagnostics as internal verification items.

memory limit = heap size
CPU limit = harmless quota
OOMKilled = Java heap leak
more threads = more throughput
liveness failure = app bug
fast startup locally = fast startup in Kubernetes

heap budget
native memory budget
thread stack budget
metaspace budget
code cache budget
direct buffer budget
GC CPU budget
JIT warmup budget
startup deadline budget

Java heap
metaspace
thread stacks
direct byte buffers
mapped files
JIT code cache
GC structures
JNI/native libraries
class metadata
JVM internal allocations
application native allocations

-Xmx must be smaller than container memory limit.

platform resource policies differ
container base image may use different JVM build
memory limit may be absent in local Docker
CPU limit can distort available processor count
JVM defaults may not match latency goals
native memory may be high for your workload

java -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|MaxRAM|UseContainerSupport|ActiveProcessorCount"
java -XshowSettings:system -version
java -XshowSettings:vm -version

what memory limit Kubernetes set
what CPU request/limit Kubernetes set
what JVM thinks the max heap is
what JVM thinks available processors are

fixed heap size: -Xmx1024m
percentage-based: -XX:MaxRAMPercentage=70

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-Xms512m -Xmx1024m"

predictable
simple to reason about
stable across environments

must align manually with container limit
bad if same manifest runs under different limits
can waste memory in small environments

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:InitialRAMPercentage=25 -XX:MaxRAMPercentage=70"

adapts to memory limit
useful across environments
less duplication between deployment and JVM flags

native memory headroom must still be modeled
percentage may be too aggressive for high-thread/direct-buffer apps
less obvious during review unless documented

Heap sizing must be derived from container memory limit and non-heap headroom,
not copied from another service.

thread stacks
metaspace
code cache
direct buffers
NIO/network buffers
TLS/native crypto
compression libraries
JDBC driver/native allocations
Netty/Grizzly/Tomcat buffers if applicable
mapped files
JVM internal structures

container_memory_limit
  = heap
  + metaspace
  + thread_stack_total
  + direct_buffers
  + code_cache
  + JVM native overhead
  + OS/process overhead
  + safety margin

memory limit: 1024 MiB
heap max:     600-750 MiB
headroom:     274-424 MiB

number of loaded classes
frameworks and reflection
JAX-RS providers/resources
JSON/XML libraries
CDI/HK2 proxies
generated classes
classloader leaks
application server deployment model

java.lang.OutOfMemoryError: Metaspace

process memory grows due to metaspace
kernel kills container before Java throws clear OOME

-XX:MaxMetaspaceSize=256m

Is classloading stable after startup?
Does redeploy/reload happen in-process?
Are dynamic proxies/generated classes used heavily?
Is metaspace monitored?

HTTP request workers
Jersey/Servlet container pools
Kafka consumers
JDBC pool housekeeper
scheduler threads
async executors
resilience library schedulers
OpenTelemetry exporters
GC threads
JIT compiler threads

thread_stack_total = number_of_threads * stack_size

~500 MiB native memory potential

-Xss512k

StackOverflowError
framework recursion failure
deep serialization/deserialization failure

control thread pool sizes
avoid unbounded executors
measure thread count

NIO
HTTP clients
server connectors
Kafka clients
PostgreSQL/JDBC internals
TLS stacks
Netty-based libraries
file streaming
compression

java.lang.OutOfMemoryError: Direct buffer memory
container OOMKilled
latency spikes during streaming
memory not visible in heap histogram

-XX:MaxDirectMemorySize=128m

If service streams files, handles multipart, uses Kafka, or heavy HTTP clients,
non-heap buffer budget must be considered.

long-running service throughput
warmup behavior
CPU usage after startup
latency stabilization

CodeCache is full warning
performance degradation
higher CPU
less optimized hot path

JIT is not free.

What is latency SLO?
What heap size?
What allocation rate?
What CPU limit?
What pause tolerance?
What throughput requirement?

-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

-Xlog:gc*,safepoint:file=/app/logs/gc.log:time,level,tags:filecount=5,filesize=20M

-Xlog:gc*:stdout:time,level,tags

stdout integrates with platform logging
file logs can be easier to download/analyze but require volume/rotation

Are GC logs enabled in performance environments?
Can they be enabled in production during incident?
Where do they go?
Can operators correlate GC pauses with request latency?

request = scheduling reservation signal
limit   = maximum CPU quota before throttling

request latency
GC duration
JIT compilation
thread scheduling
startup time
Kafka polling
timeout behavior

Service has enough memory but high latency.
CPU throttling delays GC and request handling.
Timeouts trigger retries.
Retries increase CPU.
System enters cascading failure.

Is CPU limit set?
Is throttling monitored?
Does JVM see correct processor count?
Are GC/JIT threads too many for CPU quota?
Are timeouts calibrated for throttled environment?

GC threads
JIT compiler threads
ForkJoinPool parallelism
common pool behavior
framework defaults

-XX:ActiveProcessorCount=2

container CPU quota is interpreted unexpectedly
framework defaults create too many threads
performance tests need deterministic CPU model

Is ActiveProcessorCount set because of measured behavior or cargo-cult tuning?

java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory
unable to create native thread

kernel/container runtime kills process because cgroup memory limit is exceeded

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

No heap dump does not mean no memory problem.
No Java OOME does not mean JVM was not memory-related.

1. Check pod status and exit code
2. Check memory limit and actual memory usage
3. Check JVM heap max
4. Check GC logs if available
5. Check thread count
6. Check direct buffer/native memory indicators
7. Check recent traffic/file streaming/batch workload
8. Check deployment change: flags, image, dependency, config
9. Check whether heap dump exists
10. Check node pressure events

kubectl describe pod <pod>
kubectl top pod <pod>
kubectl logs <pod> --previous

jcmd <pid> VM.native_memory summary
jcmd <pid> GC.heap_info
jcmd <pid> Thread.print

-XX:NativeMemoryTracking=summary

PII
access tokens
secrets
customer data
pricing data
request payloads

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/app/dumps

heap dump can exceed ephemeral storage
heap dump path may not be writable
OOMKilled may prevent dump from being written
heap dump must be protected as sensitive artifact

Is heap dump enabled?
Where does it write?
Is there enough storage?
Is access controlled?
Is retention defined?
Is PII/security handling documented?

-XX:+ExitOnOutOfMemoryError

process exits clearly
Kubernetes can restart
service avoids corrupted/unstable state

restart loop if root cause persists
heap dump may need time/storage before exit
can hide slow memory leak if observability is weak

record reason
emit metrics/logs if possible
capture dump if policy allows
restart cleanly
page if repeated

image pull
container creation
JVM startup
classpath scan
DI/bootstrap
JAX-RS resource/provider registration
configuration fetch
secret access
database pool initialization
Kafka client initialization
cache warmup
readiness probe success

slow image pull
CPU throttling
classloading/JIT
misconfigured probes
remote dependency during startup
schema migration at startup
secret/config service latency

liveness starts too early
JVM still warming/bootstrap
Kubernetes kills container
CrashLoopBackOff begins

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5

classloading
JIT compilation
connection pool establishment
TLS handshake/cache
DNS cache
JAX-RS provider initialization
JSON serializer cache
database prepared statement cache
branch/profile collection

startup warmup endpoint calls
synthetic internal requests
preload serializers/mappers
initialize connection pools lazily or eagerly by policy
use readiness only after minimum warmup
canary before full traffic

startup becomes too slow
warmup hammers dependencies
warmup masks lazy initialization bugs
readiness delayed unnecessarily

Warm only what is needed to make first real traffic safe and predictable.

CPU limit too low
JVM throttled during GC
request latency rises
HTTP client timeouts fire
callers retry
traffic increases
CPU pressure worsens
circuit breakers open
service appears unhealthy

CPU throttling metrics
GC pause metrics
thread pool queue depth
request rate
retry rate
pod restarts
node pressure

unbounded executor
large servlet thread pool
large Kafka concurrency
large DB pool
common ForkJoinPool overuse
blocking calls inside async executor

context switching
memory pressure
queueing
timeout amplification
harder debugging

HTTP worker threads
DB pool size
Kafka consumer concurrency
outbound HTTP pool
executor pools
CPU limit/request
latency SLO

request queueing
slow throughput
false timeout

database overload
more memory per connection
more threads blocked
transaction contention
connection storms during rollout

More request threads than DB connections can create waiting.
More DB connections than CPU can create overload.
More pods can multiply total DB connections.

total_db_connections = pods * max_pool_size

poll loop delay
rebalance
consumer lag
duplicate processing
commit delay
producer timeout
batch memory pressure

consumer concurrency vs CPU
max.poll.interval.ms vs worst-case processing time
producer buffer.memory vs container memory
compression CPU cost
retry settings vs throttled CPU

JAX-RS provider
servlet container
reverse proxy
gateway
HTTP client
cloud SDK
compression layer
logging filter

direct buffer growth
temp file growth
ephemeral storage exhaustion
heap pressure from accidental byte[] buffering
slow client tying up worker thread

max file size
streaming implementation
buffer size
proxy buffering
read/write timeout
client disconnect handling
ephemeral storage limit

request = scheduling hint/reservation
limit   = hard-ish cgroup cap that can trigger OOMKill

pod may be scheduled on crowded node
node pressure may increase eviction risk
performance may vary

small spikes kill the pod
heap dump may fail
GC cannot recover native memory spikes

normal memory usage
peak memory usage
heap max
native headroom
ephemeral spike behavior
node pressure policy

JIT warmup
class loading
cache warmup
connection pool initialization
serializer cache
thread pool startup

memory grows without bound
heap after full GC trends upward
thread count keeps increasing
direct memory grows after traffic stops
metaspace grows after redeploy/reload
cache ignores size/TTL

heap usage after GC
native memory summary
thread dump over time
cache metrics
GC logs
allocation profiling

container memory working set
container memory limit
container CPU usage
container CPU throttling
pod restart count
OOMKilled events
JVM heap used/max
JVM non-heap used
GC pause duration/count
thread count
class count/metaspace
HTTP latency/error rate
DB pool usage
Kafka lag

memory near limit + rising trend + GC pressure
CPU throttling + latency increase + retry increase
restarts + OOMKilled reason
DB pool saturation + request timeout

-XX:MaxRAMPercentage=70
-XX:InitialRAMPercentage=25
-XX:+ExitOnOutOfMemoryError
-Xlog:gc*:stdout:time,level,tags
-Djava.security.egd=file:/dev/urandom
-Duser.timezone=UTC

container memory limit
native memory needs
thread counts
GC behavior
startup time
latency target
file streaming behavior
framework/runtime overhead

[ ] Set -Xmx equal to container memory limit
[ ] Ignore native memory
[ ] Use huge thread pools because requests are slow
[ ] Tune GC without GC logs
[ ] Disable CPU limits or change memory limits without platform review
[ ] Treat OOMKilled as normal restart behavior
[ ] Store heap dumps in ephemeral filesystem without size/security policy
[ ] Use liveness probe as dependency health check
[ ] Mark pod ready before app can serve real traffic
[ ] Run performance tests only outside container
[ ] Assume local Docker behavior equals Kubernetes behavior
[ ] Ignore CPU throttling during timeout incidents

[ ] Are memory request/limit changed?
[ ] Is heap sizing aligned with memory limit?
[ ] Is non-heap headroom considered?
[ ] Are JVM flags visible and documented?
[ ] Is Java version/distribution changed?
[ ] Is GC changed? Is evidence provided?
[ ] Are GC logs enabled in performance/diagnostic environments?
[ ] Is Native Memory Tracking needed or enabled?
[ ] Are CPU request/limit changed?
[ ] Is CPU throttling monitored?
[ ] Are thread pool sizes aligned with CPU/memory?
[ ] Is DB pool size aligned with pod count and DB capacity?
[ ] Is startup/readiness/liveness behavior affected?
[ ] Is warmup behavior explicit?
[ ] Is OOM handling configured?
[ ] Is heap dump path writable, sized, and secured?
[ ] Are file streaming/temp storage implications considered?
[ ] Are canary/performance results attached for tuning changes?

[ ] Which JVM distribution is used?
[ ] Which Java version is used in runtime image?
[ ] Are JVM flags set in Dockerfile, Kubernetes manifest, Helm values, GitOps config, or platform default?
[ ] Is `JAVA_TOOL_OPTIONS` used?
[ ] Is `JDK_JAVA_OPTIONS` used?
[ ] What memory request/limit is configured per environment?
[ ] What CPU request/limit is configured per environment?
[ ] What is actual max heap at runtime?
[ ] Is heap fixed with -Xmx or percentage-based?
[ ] What non-heap headroom is expected?
[ ] Are GC logs enabled anywhere?
[ ] Which GC is used?
[ ] Are CPU throttling metrics available?
[ ] Are OOMKilled events alerted?
[ ] Are heap dumps enabled? Where do they go?
[ ] Is heap dump access controlled due to PII/secrets?
[ ] Is Native Memory Tracking allowed in production?
[ ] What is typical thread count under load?
[ ] What are HTTP worker, DB pool, Kafka consumer, and executor sizes?
[ ] Is startup probe configured?
[ ] Does readiness wait for true service readiness?
[ ] Are performance tests run in container/Kubernetes-like limits?
[ ] Are warmup or canary strategies used?
[ ] Are JVM/container dashboards available?

1. Confirm exit code 137/OOMKilled.
2. Check memory limit and usage trend.
3. Check max heap.
4. Check GC logs for heap pressure.
5. Check thread count and direct/native memory.
6. Check recent rollout/config change.
7. Check traffic spike, file upload, batch job, Kafka replay.
8. Check heap dump availability.
9. Decide: rollback, increase limit, reduce heap, reduce concurrency, fix leak.

1. Check CPU throttling.
2. Check GC pauses.
3. Check request thread saturation.
4. Check DB pool saturation.
5. Check outbound dependency latency.
6. Check retries.
7. Check canary vs baseline pods.
8. Check node pressure.

1. Check previous logs.
2. Check startup probe timing.
3. Check config/secret fetch.
4. Check CPU throttling during startup.
5. Check classpath scan/DI bootstrap.
6. Check remote dependency at startup.
7. Check liveness killing app before startup completes.

Heap is not container memory.
CPU limit is not free.
Thread count is memory and scheduling pressure.
GC tuning without evidence is superstition.
OOMKilled is platform evidence, not an application exception.
Readiness protects users; liveness protects the platform.
Warmup is part of rollout design, not only performance tuning.
Every JVM flag in production should have an owner and a reason.

Why is memory limit this value?
Why is heap this percentage?
How much native headroom do we expect?
What happens on OOM?
How do we get evidence?
What signal tells us CPU throttling is hurting latency?
Can this service survive canary traffic before full rollout?

1. Container memory limit includes heap and non-heap memory.
2. Heap sizing must leave headroom for native memory, thread stacks, metaspace, direct buffers, and JVM internals.
3. OOMKilled is external to JVM and may not produce heap dumps or Java exceptions.
4. CPU throttling can cause latency, GC delay, timeout, retry storm, and Kafka instability.
5. Java 17 GC defaults are good starting points, not a substitute for measurement.
6. Startup, readiness, liveness, warmup, and canary strategy must be aligned.
7. JVM/container tuning must be reviewed with observability evidence and rollback strategy.