Part 004 — Java Runtime Inside Containers

Fokus part ini: memahami bagaimana Java 17+ / JAX-RS / Jakarta RESTful Web Services benar-benar berjalan di dalam container yang dibatasi CPU, memory, filesystem, signal, network, dan orchestrator lifecycle.

Banyak incident Kubernetes pada service Java terlihat seperti masalah aplikasi, padahal akar masalahnya adalah mismatch antara:

• JVM memory model;
• container memory limit;
• Kubernetes resource request/limit;
• CPU quota;
• GC behavior;
• thread pool;
• native memory;
• readiness/liveness probes;
• graceful shutdown;
• traffic draining;
• consumer lifecycle;
• database transaction lifecycle.

Senior engineer harus bisa membaca symptom seperti OOMKilled, latency spike, high GC, CPU throttling, stuck rollout, atau dropped request sebagai interaksi antara JVM dan container runtime.

1. Mental Model: JVM Tidak Berjalan di Mesin “Normal”

Dalam container, JVM tidak hidup di host penuh. JVM hidup di environment yang dibatasi oleh:

• memory cgroup;
• CPU quota/cpuset;
• process namespace;
• filesystem namespace;
• network namespace;
• file descriptor limit;
• Kubernetes lifecycle;
• probes;
• termination signal;
• resource pressure node;
• eviction policy.

Aplikasi Java mungkin merasa “saya hanya process biasa”, tetapi platform melihatnya sebagai container workload yang harus:

• start cepat;
• expose health endpoint;
• siap menerima traffic hanya saat ready;
• berhenti saat SIGTERM;
• tidak melewati memory limit;
• tidak menyerap CPU berlebihan;
• menulis log ke stdout/stderr;
• tidak bergantung pada host filesystem;
• tidak menyimpan state lokal penting.

2. Container Memory Limit vs JVM Memory

Kubernetes memory limit contoh:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1024Mi"

Memory limit ini bukan hanya untuk heap. Semua memory process masuk hitungan container, termasuk:

• Java heap;
• metaspace;
• thread stacks;
• direct buffer;
• code cache;
• GC structures;
• JIT compiler memory;
• native libraries;
• TLS/native crypto;
• mmap files;
• process overhead;
• temporary allocations;
• off-heap cache;
• Netty direct memory;
• compression/native buffers.

Kesalahan klasik:

container memory limit = 1024Mi
-Xmx = 1024m

Ini berbahaya karena tidak menyisakan ruang untuk non-heap memory. Akibatnya container bisa OOMKilled walaupun Java heap belum terlihat penuh dari metrics aplikasi.

3. JVM Container Awareness

JVM modern sudah container-aware. Java 17 biasanya membaca cgroup limit untuk menentukan default heap dan CPU availability.

Namun “container-aware” bukan berarti otomatis optimal.

Masih perlu memahami:

• berapa default heap yang dipilih JVM;
• bagaimana MaxRAMPercentage dihitung;
• apakah memory limit Kubernetes realistis;
• apakah native memory cukup;
• apakah CPU quota membuat GC lambat;
• apakah thread pool terlalu besar;
• apakah metrics yang dilihat adalah heap saja atau total container memory.

Useful commands di container:

java -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|InitialHeapSize|MaxRAMPercentage|InitialRAMPercentage|ActiveProcessorCount"

Di Kubernetes, akses ini tergantung apakah image punya shell/tooling. Untuk distroless, gunakan logs, startup print, atau debug container.

4. Heap Sizing: Xmx vs MaxRAMPercentage

Ada dua pola utama.

4.1 Explicit heap

-Xms512m -Xmx512m

Kelebihan:

• predictable;
• mudah dibandingkan dengan memory limit;
• cocok untuk workload stabil.

Kekurangan:

• harus dituning per environment;
• mudah lupa disesuaikan saat limit berubah;
• bisa terlalu kaku.

4.2 Percentage-based heap

-XX:InitialRAMPercentage=25
-XX:MaxRAMPercentage=70

Kelebihan:

• mengikuti memory limit container;
• lebih fleksibel antar environment;
• cocok untuk Helm values yang berbeda per environment.

Kekurangan:

• perlu paham non-heap headroom;
• default JVM mungkin tidak cocok;
• bisa misleading jika container limit tidak diset.

Rule praktis

Untuk service Java enterprise:

container memory limit = heap + non-heap + native + safety margin

Contoh kasar:

Angka ini bukan formula final. Harus divalidasi dengan metrics production.

5. InitialRAMPercentage

InitialRAMPercentage menentukan initial heap relatif terhadap available memory.

Terlalu kecil:

• heap tumbuh sering;
• startup/warmup bisa lebih noisy;
• GC awal lebih sering;
• latency warmup naik.

Terlalu besar:

• memory langsung tinggi;
• bin packing buruk;
• startup banyak pod sekaligus bisa menekan node;
• risk OOM jika non-heap juga besar.

Contoh:

-XX:InitialRAMPercentage=25
-XX:MaxRAMPercentage=70

Untuk latency-sensitive service, initial heap sering dibuat lebih dekat ke expected steady-state, tetapi trade-off cost dan startup footprint harus dihitung.

6. Memory Request vs Limit untuk JVM

Kubernetes request menentukan scheduling. Limit menentukan hard ceiling.

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "1Gi"

Jika request = limit, pod masuk QoS Guaranteed jika CPU juga request=limit.

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "2Gi"

Jika request < limit, pod biasanya Burstable.

Trade-off:

Untuk JVM service production, selalu butuh request dan biasanya butuh limit. Namun limit harus memberi ruang non-heap.

7. OOMKilled vs Java OutOfMemoryError

Dua hal ini berbeda.

Java OutOfMemoryError

Terjadi di dalam JVM.

Contoh:

java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory

JVM masih bisa menulis log, heap dump, atau exit tergantung konfigurasi.

Kubernetes OOMKilled

Container dibunuh oleh kernel/cgroup karena melewati memory limit.

Gejala:

kubectl describe pod <pod>

Menunjukkan:

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Pada OOMKilled, JVM bisa tidak sempat menulis log OOM.

Recommended JVM flag

-XX:+ExitOnOutOfMemoryError

Untuk Java OOM internal, JVM exit sehingga Kubernetes bisa restart pod. Tanpa ini, service bisa tetap hidup dalam kondisi rusak.

Heap dump:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/app/heapdump.hprof

Tapi hati-hati:

• heap dump besar;
• butuh writable storage;
• bisa memenuhi disk;
• mengandung data sensitif/PII;
• perlu policy akses dan retention.

8. Native Memory dan Non-Heap

Heap hanya sebagian dari total memory.

Komponen penting:

Metaspace

Menyimpan class metadata.

Risk meningkat jika:

• banyak dynamic class generation;
• framework berat;
• classloader leak;
• app server redeploy pattern;
• plugin architecture.

Flag:

-XX:MaxMetaspaceSize=256m

Jangan asal set terlalu kecil. Bisa menyebabkan OOM Metaspace.

Thread stack

Setiap thread punya stack memory.

Flag:

-Xss512k

Jika thread banyak, stack total signifikan.

Contoh:

500 threads x 1Mi stack = ~500Mi virtual/reserved memory

Tidak semua committed penuh, tetapi tetap perlu awareness.

Direct memory

Dipakai oleh NIO, Netty, HTTP client, Kafka client, database driver tertentu.

Flag:

-XX:MaxDirectMemorySize=256m

Jika tidak dikontrol, direct memory bisa mengejutkan.

Code cache

JIT compiled code.

Flag:

-XX:ReservedCodeCacheSize=128m

Biasanya bukan masalah utama, tetapi masuk total memory.

9. Native Memory Tracking

Untuk investigasi memory native:

-XX:NativeMemoryTracking=summary

Lalu:

jcmd <pid> VM.native_memory summary

Di container minimal, jcmd mungkin tidak tersedia karena runtime image hanya JRE atau distroless.

Alternatif:

• aktifkan NMT di staging dengan debug image;
• buat diagnostic build sementara;
• gunakan ephemeral container dengan tools jika memungkinkan;
• expose relevant JVM metrics;
• korelasikan heap metrics vs container memory metrics.

Important distinction:

JVM heap used rendah + container memory tinggi = curiga non-heap/native/direct/thread/mmap

10. CPU Quota Awareness

Kubernetes CPU:

resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "1"

500m berarti 0.5 vCPU request. Limit 1 berarti container dibatasi sampai 1 vCPU.

JVM container-aware akan melihat available processor berdasarkan quota/cgroup. Namun behavior bisa tetap tidak intuitif.

Dampak CPU limit:

• GC thread count;
• JIT compiler behavior;
• ForkJoinPool common pool;
• application thread pool default;
• Netty event loop default;
• parallel stream;
• async executor;
• HTTP server worker;
• Kafka consumer processing throughput.

Cek:

java -XX:+PrintFlagsFinal -version | grep ActiveProcessorCount

Bisa override:

-XX:ActiveProcessorCount=2

Gunakan hanya jika paham konsekuensinya.

11. CPU Throttling

CPU limit di Kubernetes dapat menyebabkan throttling.

Gejala:

• latency p95/p99 naik;
• CPU usage terlihat “tidak 100%” tetapi throttling tinggi;
• GC pause lebih lama;
• request timeout;
• readiness probe timeout;
• Kafka/RabbitMQ consumer lag naik;
• HPA tidak bereaksi sesuai harapan jika metric CPU misleading.

Metrics penting:

• container CPU usage;
• CPU throttled seconds;
• CPU throttled periods;
• request latency;
• GC pause;
• run queue/node pressure.

Failure pattern:

CPU limit terlalu rendah
→ JVM/GC/app threads throttled
→ request lambat
→ readiness timeout
→ pod removed from service
→ traffic ke pod lain naik
→ cascade

Review concern

• Apakah CPU limit benar-benar perlu?
• Apakah CPU request realistis?
• Apakah HPA scaling berdasarkan CPU?
• Apakah service latency-sensitive?
• Apakah CPU throttling sudah dimonitor?

Beberapa platform memilih tidak memberi CPU limit untuk latency-sensitive Java services, tetapi tetap memberi CPU request. Ini harus mengikuti policy internal platform.

12. GC Behavior dalam Container

Java 17 default umumnya memakai G1 GC untuk server-class machine.

GC dipengaruhi oleh:

• heap size;
• allocation rate;
• CPU quota;
• number of GC threads;
• object lifetime;
• memory pressure;
• traffic pattern;
• startup warmup;
• container throttling.

Flags umum:

-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

Namun jangan tuning GC terlalu awal tanpa data.

Yang harus dimonitor:

• heap used/committed/max;
• GC pause duration;
• GC frequency;
• allocation rate;
• old gen occupancy;
• promotion failure;
• humongous allocation;
• container memory working set;
• CPU throttling.

Failure mode

13. Startup Warmup

Java service tidak langsung optimal saat process started.

Startup phases:

1. JVM start;
2. class loading;
3. framework initialization;
4. dependency injection;
5. config load;
6. DB/Kafka/RabbitMQ/Redis client initialization;
7. JIT warmup;
8. connection pool warmup;
9. cache warmup;
10. readiness true.

Kubernetes tidak tahu app benar-benar siap kecuali readiness probe benar.

Kesalahan:

Container started != Application ready != Safe to receive production traffic

Gunakan:

• startupProbe untuk slow start;
• readinessProbe untuk traffic eligibility;
• liveness hanya untuk stuck/dead condition;
• warmup endpoint jika perlu;
• initial delay/period/timeout realistis.

14. Thread Pool dalam Container

Java backend punya banyak pool:

• HTTP server worker thread;
• async executor;
• scheduler;
• DB connection pool;
• Kafka consumer thread;
• RabbitMQ listener thread;
• Redis client event loop;
• HTTP client dispatcher;
• Netty event loop;
• ForkJoinPool;
• GC threads;
• JIT compiler threads.

Masalah umum: default thread count diasumsikan host besar, padahal container hanya 1 vCPU.

Contoh failure:

CPU limit 1 core
HTTP worker 200 threads
DB pool 100 connections
Kafka consumers 20 threads
GC/JIT threads aktif
→ context switching tinggi
→ latency naik
→ timeout cascade

Review checklist:

• Berapa CPU request/limit?
• Berapa HTTP worker thread?
• Berapa DB connection pool max?
• Berapa Kafka/RabbitMQ concurrency?
• Apakah thread pool bounded?
• Apakah queue bounded?
• Apa rejection policy?
• Apakah pool size berubah per environment?

15. Database Pool dalam Container

PostgreSQL connection pool perlu disesuaikan dengan replica count.

Formula kasar:

max total DB connections = pod replicas x max pool size per pod + admin/maintenance connections

Jika:

20 pods x 50 connections = 1000 DB connections

Itu bisa menghancurkan database jika capacity tidak cukup.

Container/Kubernetes memperbesar risiko karena autoscaling bisa menambah pod.

Review concern:

• DB pool max per pod;
• HPA max replicas;
• database max_connections;
• PgBouncer/proxy usage;
• connection timeout;
• idle timeout;
• readiness dependency anti-pattern;
• graceful shutdown menutup pool.

16. Kafka/RabbitMQ Consumer Runtime

Consumer di container punya lifecycle khusus.

Saat SIGTERM:

• stop menerima message baru;
• selesaikan message in-flight;
• commit offset/ack message;
• close consumer/channel;
• exit sebelum grace period habis.

Failure jika tidak graceful:

• duplicate processing;
• lost ack;
• message redelivery storm;
• offset commit terlambat;
• long processing dipaksa SIGKILL;
• rebalance berulang;
• deployment menyebabkan consumer lag spike.

Review concern:

• consumer concurrency;
• max poll records;
• processing timeout;
• ack mode;
• idempotency;
• DLQ strategy;
• terminationGracePeriodSeconds;
• preStop hook;
• readiness false before shutdown;
• HPA/KEDA behavior.

17. Redis Client Runtime

Redis sering dipakai untuk:

• cache;
• distributed lock;
• rate limiting;
• session-like storage;
• idempotency key;
• temporary state.

Dalam container, perhatian utama:

• connection pool size;
• timeout;
• retry;
• TLS truststore;
• DNS resolution;
• failover behavior;
• memory pressure jika local cache juga aktif;
• startup dependency;
• readiness anti-pattern.

Jangan membuat liveness probe gagal hanya karena Redis sementara unreachable. Itu bisa membuat restart storm.

18. File Descriptor Limit

Banyak koneksi berarti banyak file descriptor.

FD dipakai untuk:

• sockets HTTP;
• DB connections;
• Kafka/RabbitMQ/Redis sockets;
• log files;
• TLS files;
• DNS;
• temporary files;
• jar files;
• native libraries.

Gejala FD exhaustion:

java.io.IOException: Too many open files

Dampak:

• tidak bisa menerima koneksi baru;
• gagal connect DB/broker;
• gagal buka file;
• health endpoint gagal;
• cascading failure.

Debug:

ls /proc/1/fd | wc -l
cat /proc/1/limits

Pada distroless, command ini tidak tersedia langsung. Gunakan debug container atau metrics.

Review concern:

• connection leak;
• HTTP client response tidak ditutup;
• DB connection leak;
• file stream tidak ditutup;
• limit node/runtime;
• load test FD count.

19. Entropy Source

Beberapa operasi crypto/random bisa terpengaruh entropy source, terutama pada environment tertentu.

Flag lama yang sering terlihat:

-Djava.security.egd=file:/dev/urandom

Di Java modern, problem ini lebih jarang, tetapi masih bisa muncul dalam legacy runtime atau image tertentu.

Area terkait:

• TLS handshake;
• token generation;
• key generation;
• secure random initialization;
• startup latency.

Review:

• apakah flag ini masih diperlukan?
• apakah ada startup delay terkait SecureRandom?
• apakah base image menyediakan device random dengan benar?

20. Timezone di JVM Container

Default terbaik untuk backend production biasanya UTC.

Risiko timezone:

• log timestamp beda antar service;
• schedule job salah;
• SLA window salah;
• date-only field salah interpretasi;
• billing/order timestamp bug;
• audit trail membingungkan.

JVM flags:

-Duser.timezone=UTC

Atau environment:

env:
  - name: TZ
    value: UTC

Prinsip:

• log dan persistence timestamp sebaiknya UTC;
• timezone user/bisnis ditangani eksplisit di domain layer;
• jangan bergantung pada timezone node;
• pastikan serialization/deserialization konsisten.

21. Signal Handling: PID 1 dan SIGTERM

Di container, process utama biasanya PID 1. Kubernetes menghentikan pod dengan mengirim SIGTERM, lalu menunggu terminationGracePeriodSeconds, lalu SIGKILL jika belum exit.

Lifecycle:

kubectl delete pod / rollout update
→ Kubernetes marks pod terminating
→ Endpoint removed after readiness/termination handling
→ SIGTERM sent to process
→ app should stop accepting new work
→ app finishes in-flight work
→ app exits 0
→ container stopped

Jika Java tidak menerima SIGTERM:

• wrapper shell tidak memakai exec;
• process bukan PID 1;
• signal tidak diteruskan;
• app tidak register shutdown hook;
• server tidak graceful.

Dockerfile concern:

ENTRYPOINT ["java", "-jar", "/app/app.jar"]

Lebih aman daripada:

CMD java -jar /app/app.jar

Jika pakai script:

exec java ${JAVA_OPTS:-} -jar /app/app.jar

22. Graceful Shutdown untuk JAX-RS Service

JAX-RS service bisa berjalan di:

• embedded Jetty;
• embedded Tomcat;
• Undertow;
• Netty-based runtime;
• Open Liberty;
• WildFly;
• Payara;
• Quarkus;
• Helidon;
• custom main.

Graceful shutdown berarti:

1. readiness menjadi false;
2. service berhenti menerima request baru;
3. existing request diberi waktu selesai;
4. background worker berhenti ambil kerja baru;
5. consumer commit/ack;
6. DB transaction selesai/rollback aman;
7. connection pool ditutup;
8. telemetry flush;
9. process exit sebelum SIGKILL.

Kubernetes setting terkait:

terminationGracePeriodSeconds: 60
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

PreStop sleep adalah blunt tool. Bisa membantu memberi waktu endpoint removal menyebar, tetapi bukan pengganti graceful shutdown aplikasi.

23. Readiness Removal dan Traffic Draining

Saat pod akan dihentikan, traffic harus berhenti masuk sebelum process mati.

Flow ideal:

pod terminating
→ readiness false / endpoint removed
→ ingress/service stop routing new traffic
→ existing connections drain
→ app shutdown

Masalah:

• endpoint removal tidak instan;
• ingress/load balancer punya cache;
• keep-alive connection masih aktif;
• client retry bisa memperparah;
• timeout chain tidak selaras;
• preStop terlalu pendek;
• termination grace terlalu pendek.

Review concern:

• readiness endpoint behavior saat shutdown;
• server graceful shutdown timeout;
• ingress upstream keepalive;
• load balancer deregistration delay;
• HTTP client retry policy;
• idempotency request;
• long-running endpoint.

24. Liveness vs Readiness Mistake

Liveness menjawab:

Apakah process ini rusak total dan perlu direstart?

Readiness menjawab:

Apakah pod ini boleh menerima traffic sekarang?

Kesalahan fatal:

• liveness mengecek database;
• liveness mengecek Kafka;
• liveness mengecek Redis;
• liveness timeout terlalu agresif;
• readiness terlalu cepat true;
• startup lambat tetapi tidak ada startupProbe.

Failure cascade:

Database lambat sementara
→ liveness gagal
→ semua pod restart
→ connection storm ke database
→ outage makin parah

Untuk Java/JAX-RS:

• liveness: process/event loop/server masih hidup;
• readiness: app siap melayani request;
• dependency check harus hati-hati dan context-specific;
• startupProbe untuk slow boot/warmup.

25. Container Filesystem dan Temporary Files

Container filesystem ephemeral. Data hilang saat container restart.

Java app bisa menulis temporary files untuk:

• multipart upload;
• PDF/report generation;
• compression;
• native library extraction;
• heap dump;
• TLS material;
• cache lokal.

Jika root filesystem read-only:

securityContext:
  readOnlyRootFilesystem: true

Maka perlu writable mount:

volumes:
  - name: tmp
    emptyDir: {}
containers:
  - name: app
    volumeMounts:
      - name: tmp
        mountPath: /tmp

Review:

• app menulis ke mana?
• ukuran temp bisa berapa?
• apakah ada cleanup?
• apakah temp mengandung PII?
• apakah heap dump boleh disimpan?
• apakah ephemeral storage limit diset?

26. Ephemeral Storage

Kubernetes juga dapat membatasi ephemeral storage.

resources:
  requests:
    ephemeral-storage: "512Mi"
  limits:
    ephemeral-storage: "1Gi"

Jika app menulis banyak log/file/temp, pod bisa dievict.

Gejala:

• pod evicted;
• node disk pressure;
• log volume besar;
• temp file leak;
• heap dump memenuhi storage.

Java concern:

• file logging;
• temp upload;
• report generation;
• local cache;
• heap dump;
• GC logs;
• large stack trace logs.

27. Logging dan stdout/stderr

Container runtime menangkap stdout/stderr. Java logging harus diarahkan ke console.

Concern:

• async logging queue bounded atau tidak;
• log burst saat error;
• PII/sensitive data;
• JSON structured logging;
• correlation ID;
• request ID;
• pod metadata enrichment;
• multiline stack trace handling;
• log sampling;
• logging cost.

Failure mode:

dependency outage
→ app log error per request
→ log volume naik 100x
→ logging cost naik
→ node disk pressure/log pipeline pressure
→ observability degraded saat incident

28. JVM Metrics yang Wajib Ada

Minimal metrics untuk Java container:

• heap used/committed/max;
• non-heap used;
• metaspace used;
• direct buffer usage jika tersedia;
• thread count;
• GC count/duration;
• CPU usage;
• process/container memory;
• request latency;
• request rate;
• error rate;
• DB pool active/idle/pending;
• HTTP client metrics;
• Kafka/RabbitMQ consumer lag/ack failure;
• Redis latency/error;
• JVM uptime;
• readiness/liveness result;
• container restart count.

Yang sering hilang:

container memory != JVM heap memory

Harus punya keduanya.

29. Startup, Readiness, dan HPA Interaction

HPA dapat menambah pod saat load naik. Pod baru tidak langsung efektif jika startup/warmup lama.

Scaling latency terdiri dari:

metric delay
+ HPA decision delay
+ scheduler delay
+ image pull time
+ container start time
+ JVM startup
+ app warmup
+ readiness delay
+ traffic distribution delay

Jika Java service startup 90 detik, HPA tidak akan menyelesaikan spike 10 detik.

Review concern:

• image size;
• startup time p50/p95;
• readiness warmup;
• min replicas;
• HPA target;
• traffic burst pattern;
• queue buffering;
• upstream timeout;
• load shedding.

30. Cloud SDK Credential Resolution dalam JVM Container

Java service di Kubernetes sering memakai AWS/Azure SDK.

Container runtime concern:

• credential provider chain;
• projected service account token;
• IRSA/Azure Workload Identity;
• env var credential accidentally overriding workload identity;
• token file mount;
• DNS to STS/AAD endpoints;
• HTTP proxy;
• TLS truststore;
• retry/backoff;
• clock skew.

Failure mode:

App works locally with static env credentials
→ deployed to Kubernetes expecting workload identity
→ old env var still set
→ SDK uses wrong credential source
→ access denied or wrong account/tenant

Checklist:

• log credential provider type safely, not secret;
• avoid static credentials in env;
• verify service account annotation/federation;
• verify SDK version supports workload identity;
• verify network path to identity endpoint;
• verify time sync.

31. Runtime Interaction with PostgreSQL, Kafka, RabbitMQ, Redis, Camunda, NGINX

PostgreSQL

• pool size must align with replicas;
• connection timeout matters during rollout;
• transaction must handle SIGTERM;
• TLS truststore may be needed;
• DNS failover behavior matters.

Kafka

• consumer shutdown must commit/leave group cleanly;
• CPU throttling can increase lag;
• readiness should consider whether consumer is assigned/healthy if service role requires it;
• rebalance during rollout can spike latency.

RabbitMQ

• ack/nack behavior must be graceful;
• prefetch count affects in-flight work during shutdown;
• connection/channel recovery must be bounded.

Redis

• connection pool and timeout must be bounded;
• Redis outage should not always kill liveness;
• local cache memory must be included in container memory thinking.

Camunda-like workflow workers

• job lock duration and pod termination must align;
• worker shutdown must stop fetching jobs;
• idempotency matters;
• long-running tasks need careful termination grace.

NGINX/Ingress

• upstream timeout must exceed app expected response time where valid;
• keepalive/draining behavior affects shutdown;
• X-Forwarded headers must be interpreted correctly;
• request body size/timeouts can fail before reaching Java.

32. Failure Mode Catalog

32.1 OOMKilled

Signal:

Exit code 137, Reason OOMKilled

Likely causes:

• heap too large for container;
• native/direct memory leak;
• thread count too high;
• log/temp/heap dump storage issue misread as memory;
• local cache unbounded;
• traffic spike allocation rate.

Debug:

• compare heap metrics vs container memory;
• check restart count;
• check previous logs;
• check GC metrics;
• check direct buffer/thread/metaspace if available;
• inspect memory limit and JVM flags.

32.2 Java heap OOM

Signal:

OutOfMemoryError: Java heap space

Likely causes:

• heap too small;
• memory leak;
• unbounded collection/cache;
• large request body;
• batch load too large;
• inefficient serialization.

Debug:

• heap dump if allowed;
• allocation profiling in staging;
• metrics old gen;
• traffic/request size correlation.

32.3 CPU throttling

Signal:

• high throttled seconds;
• latency spike;
• GC pause increases;
• request timeout;
• readiness timeout.

Likely causes:

• CPU limit too low;
• thread pool too large;
• allocation/GC pressure;
• synchronous blocking calls;
• noisy neighbor/node pressure.

32.4 Slow startup

Signal:

• startupProbe failure;
• CrashLoopBackOff;
• readiness never true;
• rollout stuck.

Likely causes:

• DB/broker dependency blocking startup;
• migration at startup;
• huge classpath;
• DNS/secret/config delay;
• image pull slow;
• JIT/warmup.

32.5 Graceful shutdown failure

Signal:

• 5xx during rollout;
• duplicate message processing;
• interrupted transaction;
• SIGKILL after grace period;
• long termination.

Likely causes:

• no shutdown hook;
• shell entrypoint no exec;
• grace period too short;
• readiness not false before shutdown;
• consumer not closing cleanly;
• ingress draining mismatch.

33. Debugging Workflow: JVM Container Issue

Gunakan urutan ini:

Step 1 — Identify symptom

kubectl get pod -n <namespace>
kubectl describe pod <pod> -n <namespace>

Cari:

• restart count;
• last state;
• exit code;
• OOMKilled;
• events;
• probe failure;
• image pull issue.

Step 2 — Check logs

kubectl logs <pod> -n <namespace> --previous
kubectl logs <pod> -n <namespace>

Cari:

• OutOfMemoryError;
• startup error;
• config missing;
• permission denied;
• bind port failure;
• DB/broker timeout;
• shutdown log.

Step 3 — Compare JVM metrics and container metrics

Pertanyaan:

• heap tinggi atau container memory tinggi?
• CPU usage tinggi atau throttling tinggi?
• GC pause naik?
• thread count naik?
• DB pool exhausted?
• consumer lag naik?

Step 4 — Check manifest

kubectl get deploy <deploy> -n <namespace> -o yaml

Cari:

• resources;
• env/JAVA_OPTS;
• probes;
• terminationGracePeriodSeconds;
• securityContext;
• volume mounts;
• config/secret references.

Step 5 — Check rollout/change

kubectl rollout history deploy/<deploy> -n <namespace>
kubectl rollout status deploy/<deploy> -n <namespace>

Cari:

• image tag berubah;
• config berubah;
• resource berubah;
• probe berubah;
• secret berubah;
• base image berubah.

34. Production-Safe Debugging Principles

Jangan langsung:

• exec sembarangan ke production pod;
• restart semua pod;
• scale down tanpa memahami traffic;
• menaikkan memory/CPU tanpa root cause;
• disable liveness/readiness tanpa mitigasi;
• dump heap production tanpa privacy review;
• log secret/env;
• menjalankan command destructive;
• mengubah manifest manual jika GitOps aktif.

Lakukan:

• baca events/logs/metrics dulu;
• pahami blast radius;
• cek recent deployment/config change;
• gunakan rollback jika jelas regression;
• gunakan canary jika perlu eksperimen;
• koordinasi dengan SRE/platform/security bila menyentuh cluster policy;
• dokumentasikan temuan untuk runbook.

35. JVM Option Baseline

Baseline contoh, bukan standard universal:

-XX:InitialRAMPercentage=25
-XX:MaxRAMPercentage=70
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Djava.security.egd=file:/dev/urandom

Opsional tergantung observability:

-Xlog:gc*:stdout:time,level,tags

Heap dump hanya jika storage/privacy siap:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/app/heapdump.hprof

Native memory tracking untuk investigasi:

-XX:NativeMemoryTracking=summary

Jangan copy-paste flag tanpa memahami:

• Java version;
• memory limit;
• app framework;
• GC target;
• observability pipeline;
• security/privacy policy.

36. Kubernetes Manifest Alignment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote-order-service
spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: registry.example.com/quote-order-service:1.2.3
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: JAVA_OPTS
              value: >-
                -XX:InitialRAMPercentage=25
                -XX:MaxRAMPercentage=70
                -XX:+ExitOnOutOfMemoryError
                -Duser.timezone=UTC
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "1"
              memory: "1Gi"
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            periodSeconds: 5
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

Review notes:

• memory limit harus cukup untuk heap + non-heap;
• CPU limit bisa menyebabkan throttling;
• startupProbe melindungi slow startup;
• readiness bukan liveness;
• termination grace harus sesuai shutdown behavior;
• endpoint names harus sesuai aplikasi;
• /health/* hanya contoh, bukan standard universal.

37. Internal Verification Checklist

JVM configuration

• Java version production apa?
• Apakah service memakai Java 17+?
• Apakah JAVA_OPTS/JDK_JAVA_OPTIONS/JAVA_TOOL_OPTIONS dipakai?
• Di mana JVM options dikonfigurasi: Dockerfile, Helm values, ConfigMap, pipeline, atau platform default?
• Apakah MaxRAMPercentage dipakai?
• Apakah Xmx eksplisit dipakai?
• Apakah ExitOnOutOfMemoryError aktif?
• Apakah timezone diset?

Memory

• Berapa memory request/limit per service?
• Berapa heap max aktual?
• Apakah container memory sering mendekati limit?
• Apakah pernah terjadi OOMKilled?
• Apakah heap metrics tersedia?
• Apakah non-heap/metaspace/direct memory metrics tersedia?
• Apakah heap dump diizinkan?
• Apakah heap dump mengandung data sensitif?

CPU

• Berapa CPU request/limit?
• Apakah CPU throttling dimonitor?
• Apakah latency spike berkorelasi dengan throttling?
• Apakah thread pool disesuaikan dengan CPU?
• Apakah HPA memakai CPU metric?

Thread dan connection pool

• Berapa HTTP worker thread?
• Berapa DB pool max?
• Berapa Kafka/RabbitMQ consumer concurrency?
• Berapa Redis connection pool?
• Apakah pool bounded?
• Apakah queue bounded?
• Apakah rejection policy jelas?

Lifecycle dan shutdown

• Apakah Docker entrypoint memakai exec form?
• Apakah app menerima SIGTERM?
• Apakah graceful shutdown diaktifkan di framework/server?
• Apakah consumer stop/ack/commit aman?
• Apakah DB transaction ditutup benar?
• Apakah telemetry flush sebelum exit?
• Berapa terminationGracePeriodSeconds?
• Apakah pernah ada 5xx saat rollout?

Probes

• Apakah startupProbe ada untuk slow startup?
• Apakah readiness endpoint benar-benar mewakili traffic readiness?
• Apakah liveness tidak mengecek dependency eksternal secara agresif?
• Apakah probe timeout realistis?
• Apakah probe pernah menyebabkan restart storm?

Observability

• Apakah metrics JVM tersedia di dashboard?
• Apakah container memory dan heap memory sama-sama terlihat?
• Apakah GC pause terlihat?
• Apakah restart count alert ada?
• Apakah OOMKilled alert ada?
• Apakah CPU throttling alert ada?
• Apakah shutdown logs bisa dibaca?

CSG/team-specific verification

• Apakah ada standard JVM options internal?
• Apakah ada base Helm chart untuk Java service?
• Apakah resource sizing sudah distandardisasi?
• Apakah platform team punya policy CPU limit?
• Apakah SRE punya runbook OOMKilled/CPU throttling?
• Apakah security mengizinkan heap dump?
• Apakah observability stack memakai Prometheus/Grafana, CloudWatch, Azure Monitor, atau lainnya?
• Apakah service Quote & Order punya workload REST, consumer, worker, atau scheduler yang berbeda?

38. PR Review Checklist

Gunakan checklist ini saat review perubahan JVM/container runtime:

[ ] Memory request dan limit eksplisit
[ ] Heap sizing masuk akal terhadap memory limit
[ ] Non-heap/native/direct/thread headroom dipertimbangkan
[ ] CPU request realistis
[ ] CPU limit policy dipahami
[ ] CPU throttling metric tersedia
[ ] JVM options tidak hardcoded sembarangan di Dockerfile
[ ] JAVA_OPTS/JAVA_TOOL_OPTIONS dikelola per environment
[ ] ExitOnOutOfMemoryError dipertimbangkan
[ ] Heap dump policy jelas dan aman
[ ] Timezone konsisten
[ ] Entrypoint meneruskan SIGTERM dengan benar
[ ] Graceful shutdown aplikasi aktif
[ ] terminationGracePeriodSeconds cukup
[ ] Readiness/liveness/startup probe benar
[ ] Thread pool sesuai CPU/container limit
[ ] DB pool sesuai replica count dan DB capacity
[ ] Kafka/RabbitMQ shutdown aman
[ ] Redis timeout/retry bounded
[ ] Logs ke stdout/stderr
[ ] JVM/container metrics tersedia
[ ] OOMKilled/CPU throttling/restart alert tersedia

39. Key Takeaways

1. Container memory limit mencakup seluruh process memory, bukan hanya heap.
2. -Xmx sama dengan memory limit adalah konfigurasi berbahaya.
3. JVM container-aware membantu, tetapi tidak menggantikan sizing dan observability.
4. OOMKilled berbeda dari Java OutOfMemoryError.
5. CPU limit bisa menyebabkan throttling dan latency spike walau CPU usage terlihat tidak ekstrem.
6. Thread pool dan DB pool harus dihitung berdasarkan CPU, replica count, dan downstream capacity.
7. Graceful shutdown adalah bagian dari correctness, bukan nice-to-have.
8. Liveness tidak boleh menjadi dependency health check agresif.
9. Startup/warmup Java harus dikaitkan dengan startupProbe, readiness, HPA, dan rollout strategy.
10. Senior engineer harus membaca JVM metrics dan Kubernetes metrics bersama-sama.

40. Latihan Praktis

Ambil satu service Java/JAX-RS yang berjalan di Kubernetes.

Kumpulkan:

Service name:
Image:
Java version:
JVM options:
Memory request:
Memory limit:
CPU request:
CPU limit:
Max heap:
Replica count:
HTTP worker thread:
DB pool max:
Kafka/RabbitMQ concurrency:
Readiness endpoint:
Liveness endpoint:
Startup probe:
Termination grace period:
Recent restart count:
Recent OOMKilled event:
CPU throttling metric:
GC pause metric:

Analisis:

1. Apakah heap sizing aman terhadap memory limit?
2. Apakah CPU throttling mungkin terjadi?
3. Apakah pool size cocok dengan replica count?
4. Apakah graceful shutdown cukup untuk REST request dan async consumer?
5. Apakah probe bisa menyebabkan outage?
6. Apakah observability cukup untuk membedakan heap OOM vs container OOM?
7. Apa 3 risiko production terbesar dari runtime config saat ini?

41. Transisi ke Part 005

Setelah memahami Dockerfile dan JVM runtime, langkah berikutnya adalah melihat image sebagai artifact engineering:

• layer;
• size;
• reproducibility;
• tag;
• digest;
• promotion;
• vulnerability scanning;
• SBOM;
• SCA;
• license scanning;
• signing;
• provenance;
• registry governance.

Itulah fokus Part 005.