Part 018 — Compose Development Workflows: Inner Loop, Hot Reload, Databases, Queues

Target pembelajaran: setelah part ini, kita mampu mendesain workflow development berbasis Compose yang cepat, repeatable, aman, dan cukup mirip production tanpa membuat developer tersiksa oleh rebuild lambat, volume kacau, atau dependency lokal yang rapuh.

Part 017 membahas startup order dan readiness. Part ini membahas inner loop: siklus edit, run, observe, debug, test, reset. Compose yang baik bukan hanya menyalakan banyak container; ia harus mempercepat feedback loop engineer.

1. Mental Model: Compose sebagai Developer Platform Mini

Compose development stack adalah platform kecil di laptop.

Tujuan utama:

meminimalkan waktu dari perubahan kode ke feedback yang benar

Namun ada trade-off:

Top engineer memilih mode berdasarkan tujuan: coding cepat, debugging, integration test, demo, atau release validation.

2. Inner Loop vs Outer Loop

2.1 Inner Loop

Inner loop adalah aktivitas menit-ke-menit:

edit code -> service update -> run request/test -> inspect logs -> fix

Kriteria inner loop bagus:

• startup cepat;
• hot reload atau sync efektif;
• logs mudah dibaca;
• dependency tersedia otomatis;
• data bisa di-reset;
• debug port mudah dipakai;
• environment tidak tergantung setup manual panjang.

2.2 Outer Loop

Outer loop adalah validasi lebih berat:

build production-like image -> run integration tests -> scan -> push -> deploy

Jangan mencampur semua kebutuhan outer loop ke inner loop. Jika dev stack terlalu production-like, feedback menjadi lambat. Jika terlalu dev-only, bug environment parity muncul terlambat.

3. Tiga Mode Development Compose

3.1 Mode A — Bind Mount Source

services:
  api:
    build:
      context: .
      target: dev
    volumes:
      - ./src:/workspace/src
    command: ["./dev-server"]

Cocok untuk:

• runtime mendukung hot reload;
• source tree tidak terlalu besar;
• filesystem host cepat;
• dependency compiled tidak dibagikan dari host.

Risiko:

• file ownership mismatch;
• performa buruk di macOS/Windows untuk tree besar;
• dependency host masuk ke container;
• build artifact host berbeda arsitektur dengan container.

3.2 Mode B — Compose Watch

services:
  api:
    build:
      context: .
      target: dev
    command: ["./dev-server"]
    develop:
      watch:
        - action: sync
          path: ./src
          target: /workspace/src
          initial_sync: true
        - action: rebuild
          path: ./package-lock.json

Cocok untuk:

• granular sync;
• menghindari bind mount tree besar;
• dev container yang punya hot reload;
• project multi-platform.

Compose Watch tidak mengganti bind mount sepenuhnya. Ia adalah companion untuk development in containers, terutama ketika kita ingin mengabaikan file besar atau non-portable.

3.3 Mode C — Rebuild and Recreate

docker compose up --build api

Cocok untuk:

• perubahan dependency;
• perubahan Dockerfile;
• compiled artifact yang harus masuk image;
• validasi production-like.

Risiko:

• feedback lambat jika Dockerfile tidak cache-friendly;
• developer cenderung menghindari rebuild sehingga bug terlambat.

4. Compose Watch Deep Dive

Compose Watch menggunakan develop.watch rules.

services:
  api:
    build:
      context: .
      target: dev
    develop:
      watch:
        - action: sync
          path: ./src
          target: /workspace/src
          initial_sync: true
          ignore:
            - build/
            - target/
            - node_modules/
        - action: sync+restart
          path: ./config/application.yaml
          target: /workspace/config/application.yaml
        - action: rebuild
          path: ./Dockerfile

4.1 Actions

Catatan: ketersediaan action tertentu bergantung versi Docker Compose. Untuk baseline modern, gunakan Compose V2 terbaru yang mendukung Compose Develop Specification.

4.2 Prasyarat Image

Compose Watch membutuhkan image yang dapat menerima file sync.

Checklist:

• container punya tool dasar seperti stat, mkdir, dan rmdir;
• USER container punya write permission ke target path;
• file awal di-copy dengan ownership benar;
• target path konsisten dengan working directory aplikasi.

Contoh Dockerfile dev:

FROM eclipse-temurin:21-jdk AS dev

RUN useradd -ms /bin/sh -u 1001 app
WORKDIR /workspace

COPY --chown=app:app gradlew settings.gradle build.gradle ./
COPY --chown=app:app gradle ./gradle
RUN ./gradlew dependencies || true

COPY --chown=app:app . .
USER app

CMD ["./gradlew", "bootRun"]

Fokusnya bukan Java, tetapi prinsipnya umum: file yang akan di-sync harus writable oleh user runtime.

5. Dependency Cache Design

Dev workflow lambat sering bukan karena Docker, tetapi karena dependency cache salah.

5.1 Named Volume untuk Dependency Cache

Contoh Maven/Gradle/NPM-style cache:

services:
  api:
    build:
      context: .
      target: dev
    volumes:
      - gradle-cache:/home/app/.gradle
      - ./src:/workspace/src

volumes:
  gradle-cache: {}

Benefit:

• dependency tidak di-download ulang setiap container recreate;
• cache dikelola Docker;
• tidak mencampur cache host dengan container user/arch.

Risiko:

• cache bisa stale;
• cache bisa besar;
• debugging dependency conflict perlu reset.

Reset:

docker volume rm <project>_gradle-cache

Atau:

docker compose down -v

Jika semua volume dihapus, database dev juga hilang. Pisahkan cache volume dan data volume dengan nama jelas.

6. Project Layout yang Maintainable

Struktur sederhana:

my-service/
  compose.yaml
  compose.dev.yaml
  compose.test.yaml
  Dockerfile
  .dockerignore
  .env.example
  src/
  config/
  scripts/
    wait-for-ready.sh
    seed-dev-data.sh
    reset-local.sh

Guideline:

• compose.yaml: baseline shared.
• compose.dev.yaml: dev-only overrides.
• compose.test.yaml: integration test behavior.
• .env.example: dokumentasi environment variable.
• .env: local private, tidak commit.
• scripts/: operasi repeatable, bukan instruksi wiki manual.

Jalankan dev:

docker compose -f compose.yaml -f compose.dev.yaml up --build

Jalankan test:

docker compose -f compose.yaml -f compose.test.yaml up --build --abort-on-container-exit --exit-code-from test

7. Environment Variable Strategy

Compose environment harus jelas membedakan:

1. interpolation variable untuk Compose file;
2. container environment variable untuk aplikasi;
3. secret untuk data sensitif;
4. developer override untuk local machine.

7.1 .env.example

APP_PORT=8080
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_PASSWORD=app
REDIS_PORT=6379

7.2 Compose Usage

services:
  api:
    build: .
    ports:
      - "127.0.0.1:${APP_PORT:-8080}:8080"
    environment:
      DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}

7.3 env_file

services:
  api:
    env_file:
      - path: ./default.env
        required: true
      - path: ./local.env
        required: false

Guideline:

• commit .env.example, bukan .env berisi secret;
• gunakan default aman;
• jangan menyimpan production secret di Compose dev file;
• dokumentasikan precedence agar developer tidak bingung kenapa value berubah.

8. Local Database Design

Database lokal harus punya dua mode:

8.1 Persistent Mode

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
    volumes:
      - postgres-data:/var/lib/postgresql/data
    ports:
      - "127.0.0.1:5432:5432"

volumes:
  postgres-data: {}

Cocok untuk coding harian.

Benefit:

• data bertahan;
• startup lebih cepat;
• developer dapat inspect manual.

Risiko:

• schema/data drift;
• bug hanya muncul di clean DB;
• data lama membuat test menipu.

8.2 Ephemeral Mode

services:
  postgres:
    image: postgres:16-alpine
    tmpfs:
      - /var/lib/postgresql/data

Atau cukup hapus volume tiap test:

docker compose down -v

Cocok untuk integration test dan clean-room validation.

8.3 Reset Script

#!/usr/bin/env bash
set -euo pipefail

docker compose down -v --remove-orphans
docker compose up --build

Reset harus menjadi command, bukan ritual manual.

9. Migrations and Seed Data in Dev

Compose dev stack ideal:

services:
  api:
    build: .
    depends_on:
      migrate:
        condition: service_completed_successfully

  migrate:
    build: .
    command: ["./bin/migrate", "up"]
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"

  seed:
    build: .
    command: ["./bin/seed-dev-data"]
    depends_on:
      migrate:
        condition: service_completed_successfully
    restart: "no"
    profiles:
      - seed

Jalankan dengan seed:

docker compose --profile seed up --build

Guideline:

• migration harus deterministic;
• seed dev harus idempotent;
• seed data jangan menyerupai production data sensitif;
• gunakan profile agar seed tidak selalu berjalan.

10. Queue and Broker Dependencies

Contoh Redis:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "127.0.0.1:6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  worker:
    build: .
    command: ["./bin/worker"]
    depends_on:
      redis:
        condition: service_healthy

Queue workflow perlu memperhatikan:

• idempotency consumer;
• retry policy;
• poison message;
• dead-letter behavior;
• local visibility tooling;
• reset queue state.

Jika menggunakan RabbitMQ/Kafka/NATS, jangan hanya menjalankan broker. Tambahkan observability lokal seperti management UI jika membantu, tetapi letakkan di profile agar stack ringan.

11. External Service Emulators

Untuk development, sering perlu emulator:

Compose profile:

services:
  mailpit:
    image: axllent/mailpit:latest
    ports:
      - "127.0.0.1:8025:8025"
    profiles:
      - tools

  minio:
    image: minio/minio:latest
    command: ["server", "/data", "--console-address", ":9001"]
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    volumes:
      - minio-data:/data
    ports:
      - "127.0.0.1:9000:9000"
      - "127.0.0.1:9001:9001"
    profiles:
      - tools

volumes:
  minio-data: {}

Jalankan tools:

docker compose --profile tools up

Guideline:

• bind ke 127.0.0.1, bukan semua interface;
• credentials dev harus jelas dan tidak mirip production;
• data emulator bisa di-reset;
• jangan memasukkan emulator berat ke default path jika tidak selalu dipakai.

12. Debugging from IDE

Compose dev stack sering butuh debug port.

services:
  api:
    build:
      context: .
      target: dev
    ports:
      - "127.0.0.1:8080:8080"
      - "127.0.0.1:5005:5005"
    environment:
      JAVA_TOOL_OPTIONS: >-
        -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

Prinsip umum:

• debug port bind ke localhost;
• jangan aktifkan debug mode di production file;
• letakkan di compose.dev.yaml;
• dokumentasikan attach configuration di repository.

13. Logs and Developer Observability

Minimal command:

docker compose logs -f --tail=100

Service spesifik:

docker compose logs -f api postgres

Good local logging:

• log ke stdout/stderr;
• structured log jika memungkinkan;
• request id/correlation id;
• startup configuration summary tanpa secret;
• healthcheck failure jelas;
• migration/seed logs eksplisit.

Bad local logging:

• log hanya ke file di container;
• secret tercetak;
• stack trace hilang;
• service silent saat gagal connect dependency;
• semua service memakai nama log sama.

14. Profiles for Optional Complexity

Profiles mencegah default stack menjadi terlalu berat.

services:
  api:
    build: .

  postgres:
    image: postgres:16-alpine

  redis:
    image: redis:7-alpine

  opensearch:
    image: opensearchproject/opensearch:latest
    profiles:
      - search

  mailpit:
    image: axllent/mailpit:latest
    profiles:
      - tools

  prometheus:
    image: prom/prometheus:latest
    profiles:
      - observability

Commands:

docker compose up

docker compose --profile tools up

docker compose --profile tools --profile observability up

Guideline:

• default profile harus cukup untuk mayoritas coding;
• optional profile untuk service berat;
• test profile untuk CI behavior;
• observability profile untuk debugging kompleks.

15. Port Publishing Discipline

Bad:

ports:
  - "8080:8080"

Ini bind ke semua interface jika tidak dibatasi.

Better untuk local dev:

ports:
  - "127.0.0.1:8080:8080"

Guideline:

• publish hanya service yang perlu diakses dari host;
• service-to-service traffic pakai Docker network internal;
• hindari port collision dengan variable;
• dokumentasikan port map di README;
• jangan publish database ke network publik.

16. File Ownership and User Strategy

Masalah umum:

permission denied: cannot write target/classes

Penyebab:

• container berjalan sebagai UID berbeda;
• bind mount membawa ownership host;
• file di image milik root;
• Compose Watch target tidak writable.

Pattern:

RUN useradd -ms /bin/sh -u 1001 app
WORKDIR /workspace
COPY --chown=app:app . .
USER app

Compose:

services:
  api:
    user: "1001:1001"

Namun di beberapa local workflow, memaksa UID/GID sama dengan host bisa lebih nyaman. Jangan gunakan root sebagai solusi default untuk semua permission problem.

17. Hot Reload Patterns by Runtime

Top-level principle:

source change -> cheapest correct update action

Do not rebuild image for every source change if hot reload is correct. Do rebuild when dependency graph or runtime artifact changes.

18. Compose Commands for Daily Work

Start:

docker compose up --build

Start with watch:

docker compose up --watch

Detached:

docker compose up -d

Logs:

docker compose logs -f --tail=100 api

Execute command:

docker compose exec api sh

Run one-shot:

docker compose run --rm api ./bin/migrate status

Restart one service:

docker compose restart api

Rebuild one service:

docker compose up --build api

Clean containers but keep volumes:

docker compose down --remove-orphans

Clean everything including volumes:

docker compose down -v --remove-orphans

19. Development Stack Anti-Patterns

19.1 “Works on My Laptop” Compose

Symptoms:

• depends on local absolute path;
• .env missing from repo and undocumented;
• hidden manual database setup;
• image requires private credential at build without documented secret flow;
• bind mount masks files built into image.

Fix:

• commit .env.example;
• use relative paths;
• encode setup as scripts/jobs;
• make reset reproducible;
• test from clean clone.

19.2 Production Secret in Dev Compose

Never put real secret in Compose file.

Bad:

environment:
  STRIPE_SECRET_KEY: sk_live_...

Better:

env_file:
  - path: ./local.env
    required: false

And keep local.env ignored by git.

19.3 Default Stack Too Heavy

If docker compose up starts 20 services, developers stop using it.

Fix:

• profiles;
• lightweight mocks;
• documented minimal path;
• separate integration stack.

19.4 Unbounded Persistent Drift

If local DB volume lives for months, it stops representing a real setup.

Fix:

• reset script;
• seed versioning;
• migration check;
• periodic clean-room validation.

20. Example: Production-Like but Developer-Friendly Stack

name: sample-app

services:
  api:
    build:
      context: .
      target: dev
    command: ["./dev-server"]
    environment:
      APP_ENV: development
      DATABASE_URL: postgres://app:app@postgres:5432/app
      REDIS_URL: redis://redis:6379
      MAIL_HOST: mailpit
      MAIL_PORT: 1025
    ports:
      - "127.0.0.1:${APP_PORT:-8080}:8080"
      - "127.0.0.1:${DEBUG_PORT:-5005}:5005"
    depends_on:
      migrate:
        condition: service_completed_successfully
      redis:
        condition: service_healthy
    develop:
      watch:
        - action: sync
          path: ./src
          target: /workspace/src
          initial_sync: true
          ignore:
            - build/
            - target/
        - action: sync+restart
          path: ./config/application-dev.yaml
          target: /workspace/config/application-dev.yaml
        - action: rebuild
          path: ./Dockerfile
    volumes:
      - dependency-cache:/home/app/.cache

  migrate:
    build:
      context: .
      target: dev
    command: ["./bin/migrate", "up"]
    environment:
      DATABASE_URL: postgres://app:app@postgres:5432/app
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
    ports:
      - "127.0.0.1:${POSTGRES_PORT:-5432}:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 20
      start_period: 10s

  redis:
    image: redis:7-alpine
    ports:
      - "127.0.0.1:${REDIS_PORT:-6379}:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  mailpit:
    image: axllent/mailpit:latest
    ports:
      - "127.0.0.1:${MAILPIT_UI_PORT:-8025}:8025"
    profiles:
      - tools

volumes:
  postgres-data: {}
  dependency-cache: {}

Properties:

• API tidak start sebelum migration sukses.
• DB data persistent.
• Dependency cache persistent.
• Mailpit optional via profile.
• Ports bind ke localhost.
• Source sync via Compose Watch.
• Rebuild hanya untuk Dockerfile changes.

21. Clean-Room Validation

Minimal seminggu sekali atau sebelum merge besar, jalankan dari kondisi bersih:

docker compose down -v --remove-orphans
docker compose build --no-cache
docker compose up

Tujuan:

• memastikan onboarding developer baru tidak rusak;
• mendeteksi dependency implicit;
• mendeteksi seed/migration tidak idempotent;
• mendeteksi Dockerfile hanya bekerja karena cache lama;
• mendeteksi environment variable tidak terdokumentasi.

22. Developer Experience Checklist

Stack Compose yang baik harus bisa menjawab:

1. Bagaimana developer start dari clean clone?
2. Command apa untuk start default stack?
3. Command apa untuk reset semua state?
4. Command apa untuk menjalankan migration?
5. Command apa untuk seed data?
6. Service apa yang optional?
7. Port apa saja yang dibuka ke host?
8. Di mana log dilihat?
9. Bagaimana attach debugger?
10. Bagaimana menjalankan integration test?
11. Bagaimana memperbarui dependency?
12. Bagaimana membersihkan volume/cache?
13. Secret dev disimpan di mana?
14. Apa bedanya dev stack dan test stack?
15. Apa yang tidak boleh dianggap production-ready?

Jika jawaban tersebar di Slack lama, Compose workflow belum production-grade sebagai internal developer platform.

23. Practice Lab

Lab 1 — Build a Fast Inner Loop

Buat service api dengan:

• dev Dockerfile target;
• source sync atau bind mount;
• dependency cache volume;
• debug port localhost;
• logs ke stdout.

Ukur:

clean start time
first build time
code change feedback time
dependency change feedback time

Lab 2 — Add Optional Tooling Profiles

Tambahkan:

• mail UI profile;
• object storage profile;
• observability profile.

Pastikan default docker compose up tetap ringan.

Lab 3 — Reset and Clean-Room Test

Buat script:

./scripts/local-reset.sh
./scripts/local-test.sh

Kriteria sukses:

• bisa jalan dari clean clone;
• tidak butuh instruksi manual;
• failure memberi error jelas;
• tidak bergantung pada state lama.

24. Review Rubric

25. Ringkasan

Compose development workflow yang baik memiliki invariant berikut:

Default stack harus cepat.
Optional complexity harus pakai profiles.
State harus bisa di-reset.
Dependency cache harus mempercepat, bukan menyembunyikan bug.
Source update harus memakai action termurah yang tetap benar.
Ports dev harus bind ke localhost.
Secrets tidak boleh masuk Compose file.
Clean-room validation wajib untuk menjaga onboarding dan reproducibility.

Part berikutnya membahas Compose untuk testing: integration test, contract test, E2E, fixture, ephemeral stack, parallel isolation, CI behavior, dan cara menjadikan Compose sebagai test harness yang reliable.