unit
component
mock/fake
state machine
negative path
time/concurrency
property-based
mutation
fuzzing
contract
integration
E2E

Can we write a test?

Can thousands of tests remain useful over years of codebase evolution?

developers
reviewers
release managers
SRE/platform engineers
security/compliance teams

PR feedback cepat
failure actionable
false negative rendah
false positive rendah
release confidence tinggi
maintenance cost terkendali

flaky tests
slow tests
unclear failures
missing ownership
environment contention
data collision
duplicate coverage
obsolete tests
hidden dependencies
non-deterministic ordering
CI runner saturation

confidence
speed
diagnostic precision
cost
maintainability
determinism
scope realism

Cheapest valid evidence wins.

scope
speed
owner
risk
stability
dependency
execution frequency
blocking policy

scope:
  - unit
  - component
  - contract
  - integration
  - e2e
  - performance
  - formal

speed:
  - fast      # < 1s/test or very small suite
  - medium    # seconds
  - slow      # minutes

stability:
  - deterministic
  - async-controlled
  - environment-sensitive
  - flaky-quarantined

blocking:
  - pr-blocking
  - merge-blocking
  - release-blocking
  - non-blocking
  - nightly-only

owner:
  - team name

@Tag("scope:integration")
@Tag("owner:case-platform")
@Tag("risk:critical")
@Tag("speed:medium")
class CaseRepositoryIT {
}

suites:
  - name: case-platform-unit
    scope: unit
    owner: case-platform
    command: mvn -pl case-domain test
    blocking: pr
    target_runtime: 2m

  - name: case-platform-integration
    scope: integration
    owner: case-platform
    command: mvn -pl case-service verify -Pintegration
    dependencies:
      - postgresql
      - kafka
    blocking: merge
    target_runtime: 8m

  - name: case-submission-e2e
    scope: e2e
    owner: case-platform
    environment: staging
    blocking: release
    target_runtime: 3m
    flake_budget: 0.5%

new engineers can understand suite topology
CI can route tests intelligently
ownership is visible
runtime budget is explicit
obsolete suites can be retired

case-platform/
  case-domain/
    src/main/java/...
    src/test/java/...              # unit/property tests
  case-application/
    src/main/java/...
    src/test/java/...              # component tests
  case-adapters-postgres/
    src/main/java/...
    src/test/java/...              # repository unit tests if any
    src/integrationTest/java/...   # PostgreSQL integration tests
  case-adapters-kafka/
    src/integrationTest/java/...   # Kafka integration tests
  case-service/
    src/test/java/...              # controller/component tests
    src/integrationTest/java/...   # full service integration
  case-contract-tests/
    src/test/java/...
  case-e2e-tests/
    src/test/java/...
  case-performance-tests/
    src/jmh/java/...

what test belongs where
what dependencies are allowed
what command runs it
what runtime to expect

src/test/java

*Test      -> unit/component fast tests
*IT        -> integration tests
*E2ETest   -> end-to-end tests
*ContractTest -> contract tests
*PropertyTest -> property-based tests
*Benchmark -> JMH benchmark class

CaseTransitionPolicyTest
CaseLifecyclePropertyTest
CaseRepositoryIT
CaseApiContractTest
CaseSubmissionE2ETest
CaseTransitionBenchmark

IDE discovery
Maven Surefire/Failsafe filtering
human navigation
CI partitioning

maven-surefire-plugin  -> unit tests
maven-failsafe-plugin  -> integration tests

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${maven.surefire.version}</version>
      <configuration>
        <includes>
          <include>**/*Test.java</include>
          <include>**/*PropertyTest.java</include>
        </includes>
        <excludedGroups>slow,integration,e2e</excludedGroups>
      </configuration>
    </plugin>

    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-failsafe-plugin</artifactId>
      <version>${maven.failsafe.version}</version>
      <configuration>
        <includes>
          <include>**/*IT.java</include>
          <include>**/*E2ETest.java</include>
        </includes>
      </configuration>
      <executions>
        <execution>
          <goals>
            <goal>integration-test</goal>
            <goal>verify</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

module boundaries
tags
profiles
CI workflow selection

scope:unit
scope:integration
scope:e2e
owner:case-platform
risk:critical
feature:case-lifecycle
requires:postgres
requires:kafka
stability:quarantined

important
new
bob
random
slowmaybe

scope:*     exactly one required
owner:*     exactly one required for non-unit suites
risk:*      optional but recommended
requires:*  for external dependency needs
stability:* for quarantine/flaky management

mvn test -Dgroups='scope:unit | scope:component'
mvn verify -Dgroups='scope:integration & requires:postgres'
mvn verify -Dgroups='scope:e2e & risk:critical'

tags are routing metadata.

1. Local fast tests
2. PR required tests
3. Merge/main tests
4. Release candidate tests
5. Nightly/weekly heavy tests
6. Production synthetic checks

developer feedback in seconds/minutes

unit
component
small property tests
small contract tests

Docker
network
shared environment
long sleeps

block obvious regressions before merge

unit
component
contract
selected integration for changed modules
static checks

validate integrated mainline

full integration suite
API smoke E2E
migration checks

validate deployable artifact and environment config

critical E2E journeys
selected UI smoke
backward compatibility checks

catch expensive/rare problems without blocking every PR

broader E2E
mutation testing
fuzzing
long-running property tests
soak tests
performance regression checks

not every test belongs to every pipeline.

local core tests:       < 2 minutes
PR required pipeline:   < 10 minutes
mainline integration:   < 20 minutes
release E2E smoke:      < 15 minutes
nightly heavy:          < 2 hours

runtime by module
runtime by test class
runtime trend
slowest tests
queue time
setup time
container startup time

changed module -> run module tests
changed API schema -> run contract tests
changed migration -> run DB integration tests
changed shared library -> run dependent modules
changed workflow definition -> run lifecycle tests + E2E smoke

build graph dependency analysis
historical failure mapping
coverage mapping
service ownership mapping
risk-based selection

full mainline tests
nightly full suite
periodic random selection

same code + same test + inconsistent result

1. detect flake
2. classify cause
3. assign owner
4. quarantine if blocking signal is harmed
5. fix or delete within SLA
6. track recurrence

rerun until green
permanent quarantine
unknown owner
ignored red builds

PR required suite: < 0.1% flaky failure rate
mainline suite: < 0.5%
nightly exploratory: < 2% but must be classified
release blocking: effectively zero tolerated known flakes

test: CaseEscalationJourneyE2ETest
owner: case-platform
quarantined_at: 2026-07-02
reason: intermittent timeout waiting for projection
suspected_cause: async projection lag or weak wait condition
tracking_ticket: CASE-12345
expires_at: 2026-07-16
blocking_removed_from: release

still run quarantined tests in non-blocking lane
publish failures separately
notify owner
expire quarantine automatically

escalate or delete test

PRODUCT_BUG
TEST_BUG
ENVIRONMENT_FAILURE
FLAKY_TEST
INFRASTRUCTURE_FAILURE
DATA_COLLISION
CONTRACT_DRIFT
PERFORMANCE_REGRESSION
UNKNOWN

100 failures this month
70 environment failures
20 flaky tests
8 product bugs
2 contract drifts

test method owner
suite owner
module owner
platform CI owner
quality architecture owner

unit tests -> owning module team
integration tests -> owning service/team
contract tests -> provider + consumer ownership
E2E journeys -> business capability owner
CI runners/build infra -> platform team

code annotations or metadata
CODEOWNERS
test inventory
CI dashboard
failure notification routing

readable names
small tests
clear arrangement
no hidden global state
no sleeps
no random without seed capture
semantic assertions
helper methods with clear intent
minimal mocking
controlled fixture builders

TestHelper.doEverything();

var caseId = fixtures.caseReadyForAssignment()
    .withPriority(HIGH)
    .ownedBy("team-a")
    .create();

builder for domain objects
fixture factory for persistence state
scenario factory for workflows
golden samples for contracts
corpus for fuzzing
seeded generators for property tests
runId for E2E/integration isolation

one giant shared SQL fixture
one static JSON reused everywhere
manual staging data
hidden dependency on execution order

unit data: generated in test
integration data: inserted via repository/API/migration-aware fixture
E2E data: created via public/test fixture API with runId
contract data: stored as versioned samples
fuzz corpus: curated with minimized failures

in-process fake environment
Testcontainers environment
local compose environment
PR preview environment
shared staging
production synthetic environment

who owns it?
what data isolation exists?
what dependencies are real?
how often is it reset?
what observability exists?
what health gate exists?
what tests can run there?

tests are independent
fixtures are isolated
ports are not hardcoded
temporary directories are unique
containers are not mutated globally
database schemas/tenants/data are isolated
external accounts are not shared unsafely

tests mutate global feature flags
tests clear shared tables
tests reuse same username/account
one test changes JVM global timezone/default locale
static mutable state leaks across tests

System properties
static fields
default timezone
default locale
security manager/policies
logging configuration
shared ExecutorService
shared Clock
singletons
random seeds
MDC context

1. avoid mutable global state
2. restore global state after test
3. isolate tests that must mutate global state
4. mark non-parallel-safe tests explicitly

@Test
@ResourceLock("default-time-zone")
void formatsDateInJakartaTimezone() {
    var previous = TimeZone.getDefault();
    try {
        TimeZone.setDefault(TimeZone.getTimeZone("Asia/Jakarta"));
        // test
    } finally {
        TimeZone.setDefault(previous);
    }
}

fail then pass on rerun
pass/fail pattern across same commit
failure frequency per test
failure category
failure environment
failure duration

TestResult(testId, commitSha, environment, status, duration, failureHash, timestamp)

same testId + same commitSha + both pass and fail => flaky candidate

exception type
top stack frame
assertion message category
failure category

clear assertion message
seed on property failure
minimal reproduction input

container logs
database state snapshot if safe
migration version
Kafka topic offsets
application logs by test run id

screenshots/videos/traces
request/response summary
correlation ID
observability links
last known business state

benchmark parameters
hardware/runner info
JVM flags
GC logs/JFR/flamegraphs
baseline comparison

assertEquals(3, result.size());
assertEquals("A", result.get(0).getStatus());

assertThat(result)
    .hasExactlyOneOpenCaseFor(customerId)
    .hasNoDuplicateCaseReferences()
    .containsAuditAction("CASE_ACCEPTED");

failures are readable
intent is clear
implementation details hidden
assertion reuse improves consistency

Expected case CASE-123 to be ACCEPTED
Actual status: SUBMITTED
Audit trail: CASE_CREATED, CASE_SUBMITTED
Last transition error: risk-score-timeout

global TestUtils class with 300 methods
helper that creates hidden database state
helper that catches exceptions silently
helper that sleeps/retries internally
helper that makes network calls without naming it

helpers live near domain/module
helper names reveal scope and side effects
no hidden sleeps
no hidden random without seed
no catch-and-ignore
helper APIs evolve with tests

DomainFixture
RepositoryFixture
ApiFixture
E2EJourneyFixture

CommonUtil
TestMagic
BaseTestEverything

class BaseIntegrationTest {
    // starts containers
    // creates users
    // resets database
    // creates HTTP clients
    // mocks auth
    // configures Kafka
    // contains 80 helper methods
}

hidden setup
slow tests by default
hard to understand dependency
subclass coupling
difficult parallelization

compose extensions/fixtures explicitly

@ExtendWith(PostgresTestExtension.class)
@ExtendWith(KafkaTestExtension.class)
class CaseOutboxPublisherIT {
}

class CaseRepositoryIT {
    private final PostgresFixture postgres = PostgresFixture.shared();
    private CaseRepository repository;
}

container per test?
container per class?
container per suite?
reusable containers locally?
unique database/schema per test?
network per suite?

prefer deterministic startup
avoid relying on manually shared containers
capture logs
use health checks
use unique schemas/databases where possible

provider verifies it satisfies published contract

consumer verifies it uses provider contract correctly

new schema must read old messages
old consumers must tolerate new compatible messages where required

contracts stored/versioned centrally or per provider
contract changes reviewed by consumers
CI verifies provider before publish
breaking changes require migration path

which field changed
which consumer affected
whether change is backward/forward compatible

small deterministic property suite for PR
larger generated suite for main/nightly
long fuzz campaign for scheduled runs
failure corpus regression suite for PR

always capture seed
minimize failing input
promote discovered bug input into regression corpus

src/test/resources/corpus/case-parser/
  malformed-date-001.json
  nested-array-depth-attack.json
  duplicate-field-id.json

PR: targeted mutation for changed critical modules if affordable
main: selected mutation on core domain
nightly/weekly: broader mutation report
release: review mutation trend for critical areas

mutation score must be 100% everywhere

critical domain modules require threshold
surviving mutants must be triaged
equivalent mutants documented/excluded
trend should not regress without review

microbenchmarks -> JMH module/profile
macrobenchmarks -> deployed workload harness
load tests -> staging/performance environment
regression checks -> controlled CI runners

JDK version
JVM flags
hardware/runner type
container limits
GC configuration
dataset size
warmup
measurement duration
baseline version

total runtime
runtime by suite/module
test count by type
failure rate
flake rate
rerun rate
quarantine count
quarantine age
slowest tests
coverage trend where useful
mutation score for critical modules
contract break count
mean time to fix broken test

line coverage as sole quality metric
test count as productivity metric
mutation score without equivalent mutant review
flake rate without owner accountability

this line executed

assertion was meaningful
edge cases covered
invariant was checked
concurrency was safe
performance did not regress

Are critical invariants tested?
Are state transitions covered?
Are failure modes tested?
Are contracts verified?
Are compatibility paths tested?
Are performance-sensitive paths benchmarked?

mutation testing
property testing
contract testing
review of risk matrix

business criticality
compliance impact
change frequency
historical defect density
complexity
external dependency
concurrency/asynchrony
blast radius
observability quality

1. What behavior does this test prove?
2. Is this the cheapest valid layer?
3. Is the assertion semantic enough?
4. Is setup minimal and explicit?
5. Is data isolated?
6. Is time/randomness controlled?
7. Is failure diagnostic?
8. Is test parallel-safe?
9. Does it have an owner if non-unit?
10. Is it likely to become flaky?
11. Does it duplicate existing evidence?
12. What is the retirement condition?

suites:
  - id: case-domain-fast
    command: mvn -pl case-domain test
    scope:
      - unit
      - property-small
    owner: case-platform
    blocking: pr
    target_runtime: 90s

  - id: case-service-contract
    command: mvn -pl case-contract-tests test
    scope:
      - contract
    owner: case-platform
    blocking: pr
    target_runtime: 120s

  - id: case-service-integration
    command: mvn -pl case-service verify -Pintegration
    scope:
      - integration
    requires:
      - postgres
      - kafka
    owner: case-platform
    blocking: main
    target_runtime: 8m

  - id: case-critical-e2e
    command: mvn -pl case-e2e-tests verify -Pe2e-critical
    scope:
      - e2e
    owner: case-platform
    blocking: release
    target_runtime: 10m

  - id: case-domain-mutation
    command: mvn -pl case-domain org.pitest:pitest-maven:mutationCoverage
    scope:
      - mutation
    owner: case-platform
    blocking: non-blocking
    schedule: nightly

Suite: case-service-integration
Test: CaseOutboxPublisherIT.publishesPendingOutboxRowsExactlyOnce
Commit: abc123
Owner: case-platform
Category: PRODUCT_BUG candidate
Duration: 22s
Environment: CI runner linux-x64 / JDK 21 / PostgreSQL 16 container
Correlation ID: it-20260702-9f3a
Failure:
  Expected exactly one Kafka event with key CASE-123
  Found two events
Evidence:
  outbox rows: one marked PUBLISHED
  Kafka events: two CaseAccepted events
  application logs: duplicate publish after retry timeout
Suggested triage:
  inspect transaction boundary around publish confirmation

expected 1 but was 2

Daily:
  triage broken blocking suites
  assign flaky tests
  monitor CI runtime

Weekly:
  review quarantine list
  review slowest tests
  review new flaky candidates
  review suite ownership gaps

Monthly:
  review test inventory
  retire obsolete tests
  review coverage/evidence by risk area
  review mutation/performance trends for critical modules

Before major release:
  run full integration/E2E/performance suites
  review known flakes
  review contract compatibility
  review production synthetic readiness

every test in src/test/java with no classification.

classify scope, split modules/profiles, define tags.

90% coverage, weak assertions, many bugs escape.

mutation testing, invariant review, risk-based test design.

100 quarantined tests, nobody cares.

expiry, owner, delete-or-fix policy.

PR waits 60 minutes for tests unrelated to change.

layered pipeline, test impact analysis, mainline/nightly full coverage.

E2E fails randomly because staging is shared and dirty.

runId isolation, health gates, fixture ownership, preview env where possible.

1. inventory existing tests
2. classify by runtime/scope/owner
3. identify top 20 slowest tests
4. identify top flaky tests
5. split fast PR suite from slow suite
6. add quarantine with expiry
7. add diagnostics for integration/E2E failures
8. move obvious E2E-overcoverage down the pyramid
9. add contract/integration tests where E2E is carrying too much
10. create monthly suite health review

A test suite is an evidence pipeline.

What claim about the system does this test support?
At what cost?
With what false-positive/false-negative risk?
Who owns the claim?
When should this claim be checked?

Do we have enough tests?

Do we have the right evidence at the right layer with the right feedback time?

A large test suite without architecture becomes a legacy system.

Tags, naming, modules, and CI profiles are not cosmetic. They are execution architecture.

Flaky tests are trust leaks. Permanent quarantine is dead code.

The goal is not more tests. The goal is cheaper, faster, more trustworthy evidence.