This part does not assume CSG uses Pact, Spring Cloud Contract, REST Assured,
OpenAPI diff tools, AsyncAPI tooling, k6, Gatling, JMeter, JMH, GitHub Actions,
Jenkins, GitLab CI, Azure DevOps, Buildkite, or any specific internal quality gate.

Treat all tooling, CI stages, performance baselines, contract ownership, and
release gates as internal verification items.

breaking API change
breaking event schema change
hidden performance regression
slow query under production cardinality
retry storm under dependency failure
flaky CI hiding real failures
benchmark result interpreted incorrectly
load test that proves nothing useful

What risk does this test reduce?
At which boundary does the risk exist?
Who is protected by this test?
What failure should block merge/release?
What failure should trigger investigation but not block?

Unit test:
  Does this function/class behave correctly?

Integration test:
  Do real components work together?

Contract test:
  Did we preserve producer/consumer agreement?

Performance test:
  Does this survive realistic load and data shape?

Benchmark:
  Is this implementation faster/slower under controlled micro conditions?

CI gate:
  Should this change be allowed to merge or deploy?

many tests
unclear risk coverage
slow pipeline
flaky failures
ignored failures
no compatibility gate
no performance baseline
no ownership of failures

small number of high-signal tests
clear boundary coverage
fast feedback for common changes
slow tests isolated into proper stages
compatibility checked before release
performance regressions visible
flaky tests aggressively triaged

JAX-RS API is consumed by UI, other services, partners, batch jobs, or workflow engines.
Kafka events are consumed by downstream services, analytics, reconciliation jobs, or audit systems.
gRPC/protobuf APIs may be consumed by generated clients.

Provider changes field type.
Consumer deploys later.
Production breaks after release.
The team that caused the break may not see it in its own tests.

Breaking change is detected before merge or before release.

path exists
method exists
required request fields unchanged
required response fields unchanged
status codes documented
error shape stable
headers documented
content type stable
pagination shape stable
nullable/optional semantics stable

- GET /quotes/{id} returns 200 with field "quoteId"
+ GET /quotes/{id} returns 200 with field "id"

// Before
public record QuoteResponse(String quoteId, String status) {}

// After
public record QuoteResponse(String id, String status) {}

removed path
removed method
removed request field accepted by old clients
new required request field
removed response field used by clients
changed field type
changed enum values without compatibility policy
changed status code semantics
changed auth requirement
changed pagination structure
changed error envelope
changed content type

{
  "request": {
    "method": "GET",
    "path": "/quotes/Q-123"
  },
  "response": {
    "status": 200,
    "body": {
      "quoteId": "Q-123",
      "status": "DRAFT"
    }
  }
}

consumer expresses what it actually needs
provider can verify before release
less reliance on tribal knowledge

too many brittle contracts
contracts coupled to incidental fields
stale consumer contracts
no ownership when contract fails

Is this contract protecting a real dependency, or just duplicating implementation detail?

topic/event name
schema compatibility
required fields
optional fields
field type
semantic meaning
event key
ordering assumption
idempotency key
event version
deprecation policy
replay compatibility

{
  "eventId": "evt-123",
  "eventType": "QuoteSubmitted",
  "eventVersion": 2,
  "occurredAt": "2026-07-10T10:15:30Z",
  "tenantId": "tenant-a",
  "quoteId": "Q-123",
  "submittedBy": "user-456"
}

remove quoteId
change quoteId from string to object
rename occurredAt to timestamp
change event key from quoteId to tenantId
change event meaning without version bump
emit duplicate semantic events without consumer policy
remove old enum value consumer still handles

producer compatibility:
  can producer still emit schema-valid event?

consumer compatibility:
  can consumer still process old and new acceptable events?

               Consumer v1   Consumer v2   Consumer v3
Provider v1       OK            OK?           ?
Provider v2       OK?           OK            OK?
Provider v3       BREAK?        OK?           OK

blue-green deployment
canary deployment
multi-region rollout
customer-specific rollout
tenant-specific rollout
on-prem delayed upgrade
partner integration delayed upgrade

If provider and consumer cannot be deployed atomically, compatibility is not optional.

Does the service meet latency target under expected load?
Where does it saturate?
What is the bottleneck?
How does it fail under overload?
Does it recover after overload?
Does it respect downstream limits?

Under 150 RPS with p95 payload size and production-like DB cardinality,
GET /quotes/{id} should remain below 200ms p95 and error rate below 0.1%.

Let's run 1000 users and see what happens.

p50 latency
p95 latency
p99 latency
error rate
throughput
CPU utilization
memory utilization
GC pause
DB connection pool usage
DB query latency
Kafka publish latency
consumer lag
Redis latency
HTTP downstream latency
queue depth
thread pool saturation

GET /quotes/{id}
  p95 latency < 200ms
  p99 latency < 500ms
  5xx rate < 0.1%
  timeout rate < 0.05%

POST /quotes/{id}/submit
  p95 latency < 800ms
  5xx rate < 0.2%
  duplicate submission prevented
  event publish succeeds or outbox records pending event

Latency target tanpa payload shape dan data cardinality tidak lengkap.

number of tenants
number of quotes per tenant
number of quote lines
catalog size
pricing rule complexity
order history size
index selectivity
payload size distribution
large customer edge cases
archived vs active data ratio

Quote A:
  1 line item
  no discount
  one-time charge only

Quote B:
  2,000 line items
  nested bundles
  discounts
  tax rules
  effective dates
  multi-currency

serialization strategy
mapping strategy
regex/parser
pricing algorithm hot path
cache key generation
hashing/signature operation

real endpoint capacity
DB performance
network latency
gateway behavior
GC behavior under full service load

warmup configured
measurement iterations configured
forks configured
blackhole used when needed
inputs realistic enough
allocation measured
GC considered
CPU frequency noise considered
result compared with confidence

Compare MapStruct mapper vs manual mapper for 10,000 DTO conversions.
Compare JSON serialization config A vs B for representative payload.
Compare pricing rule evaluation strategy for 100/1,000/10,000 line items.

Benchmark one endpoint by calling localhost once.

Stage 1: format/lint/static checks
Stage 2: unit tests
Stage 3: API/event contract checks
Stage 4: integration tests with containers
Stage 5: migration validation
Stage 6: security/SCA/SBOM/image scan
Stage 7: performance smoke or nightly performance test
Stage 8: packaging and artifact publishing

Fast feedback for common mistakes.
Deep feedback for expensive risks.
Release feedback for production compatibility.

real failures ignored because "CI is flaky"
engineers rerun instead of diagnose
pipeline cost increases
merge discipline collapses
production bugs escape

time-dependent assertion
random port conflict
shared database state
shared Kafka topic/consumer group
non-isolated Redis key
async wait with fixed sleep
race condition
ordering assumption
external dependency
clock/timezone dependency
resource leak between tests
parallel test interference
container startup race

1. Is the failure deterministic locally?
2. Does it only fail under parallel execution?
3. Does it depend on time/order/randomness?
4. Does it share external state?
5. Does it assume async work completed too early?
6. Does it depend on slow CI hardware?
7. Does it leak resources from prior tests?
8. Does it depend on timezone/locale/environment variable?

service.submitQuote("Q-123");
Thread.sleep(500);
assertThat(repository.findStatus("Q-123")).isEqualTo("SUBMITTED");

service.submitQuote("Q-123");

await()
    .atMost(Duration.ofSeconds(5))
    .untilAsserted(() ->
        assertThat(repository.findStatus("Q-123")).isEqualTo("SUBMITTED")
    );

Even await-based tests must have clear timeout and isolated state.

unique tenant ID per test
unique resource ID per test
unique Kafka topic or consumer group per test suite
unique Redis key prefix per test
transaction rollback only if it matches runtime behavior
clean DB schema or clean fixtures per suite
no dependency on test execution order
no wall-clock assumption without clock abstraction

use deterministic migration baseline
clean schema intentionally
avoid hidden shared rows
avoid sequence assumptions unless controlled

avoid shared consumer group across tests
wait for assignment before producing if needed
assert eventually, not immediately
control topic retention if possible

prefix keys by test run
avoid global FLUSHALL in shared environment
assert TTL behavior with bounded tolerance

Contract Testing
[ ] Does the PR update API/event contracts when behavior changes?
[ ] Is the change backward-compatible?
[ ] Is there an explicit breaking-change decision if not compatible?
[ ] Are error shapes and status codes still compatible?
[ ] Are generated clients/servers updated if used?
[ ] Are OpenAPI/AsyncAPI/protobuf artifacts validated in CI?

Performance
[ ] Could this change affect latency, throughput, memory, DB load, or Kafka lag?
[ ] Is there benchmark/load evidence for performance-critical change?
[ ] Are payload size and data cardinality considered?
[ ] Are slow queries or new indexes reviewed?
[ ] Does the change add blocking calls or unbounded concurrency?

CI
[ ] Does the test run in the right stage?
[ ] Is the test deterministic?
[ ] Is the test isolated from shared state?
[ ] Does the test fail for the right reason?
[ ] Is a flaky failure being hidden by retries?

Flaky Diagnosis
[ ] Are sleeps replaced with bounded awaits?
[ ] Are Kafka/Redis/DB resources isolated?
[ ] Is time controlled with clock abstraction where needed?
[ ] Are parallel test interactions controlled?

Contract governance
[ ] Is OpenAPI generated from code, written manually, or source of truth?
[ ] Is AsyncAPI used for Kafka/event documentation?
[ ] Is protobuf/gRPC used anywhere?
[ ] Is there an API linting rule set?
[ ] Is contract diff enforced in CI?
[ ] Is there consumer-driven contract testing?
[ ] Is there a contract broker or central contract repository?
[ ] Are generated clients used internally?
[ ] Who approves breaking API/event changes?

Performance/testing
[ ] What performance test tool is standard?
[ ] Are there endpoint-level latency targets?
[ ] Are there service-level SLOs/SLA/error budgets?
[ ] Is there production-like test data?
[ ] Are nightly load/stress/soak tests run?
[ ] Are performance regressions gated or only reported?
[ ] Is benchmark evidence expected for hot path changes?

CI
[ ] What CI system is used?
[ ] Which tests run on PR vs merge vs nightly vs release?
[ ] Are integration tests container-based?
[ ] Is Docker available in CI?
[ ] Are test results tracked over time?
[ ] Are flaky tests tagged/quarantined/owned?
[ ] Is rerun policy documented?

Release
[ ] What tests block release?
[ ] Are contract compatibility checks part of release gate?
[ ] Are migration checks part of release gate?
[ ] Is smoke performance run before or after deployment?

A test that no one trusts is operational debt.
A contract that is not enforced is documentation, not governance.
A benchmark without warmup is suspicious.
A load test without production-like data shape is weak evidence.
A flaky test should be treated like a production reliability issue.
A CI gate should be tied to a risk, not to tradition.
A compatibility matrix is more useful than a generic claim of backward compatibility.

What failure would this test catch before production, and is that failure worth the cost of the test?

right tests
right boundary
right signal
right stage
right ownership