Series MapLesson 42 / 112
Focus mode active/Press Alt+Shift+R to toggle/Esc to exit
Build CoreOrdered learning track

Feature Flags and Kill Switch

Feature Flags Kill Switch and Dark Launch

Kontrol perilaku runtime untuk safe rollout, progressive enablement, tenant-specific behavior, dan emergency disablement

5 min read888 words
PrevNext
Lesson 42112 lesson track22–61 Build Core
#feature-flags#kill-switch#dark-launch#rollout+5 more

Part 042 — Feature Flags, Kill Switch, and Dark Launch

Fokus part ini: memahami feature flag sebagai mekanisme runtime control, bukan sekadar if statement. Kita akan membahas flag taxonomy, kill switch, dark launch, tenant targeting, safe defaults, observability, lifecycle cleanup, dan failure mode pada JAX-RS enterprise service.

Feature flag adalah control plane kecil di dalam application behavior.

Ia memungkinkan sistem menjawab:

Should this code path be active for this request, tenant, user, environment, or integration right now?

Dalam sistem enterprise seperti CPQ/quote/order management, feature flag dapat mengontrol:

- endpoint baru
- pricing rule baru
- catalog behavior baru
- integration route baru
- validation rule baru
- Kafka event publication baru
- downstream client baru
- tenant-specific enablement
- rollback behavior
- emergency disablement

Senior warning:

Feature flags reduce release risk only when their lifecycle, defaults, ownership, and observability are disciplined.

Tanpa discipline, feature flags menjadi hidden complexity.


1. Deployment vs Release

Feature flag memisahkan deployment dari release.

Deployment = code exists in runtime.
Release    = behavior is enabled for users/tenants/traffic.

Without feature flags:

merge code
  -> deploy
  -> behavior immediately active

With feature flags:

merge code
  -> deploy inactive or partially active
  -> enable gradually
  -> observe
  -> expand or disable

This is important because production risk is not binary.

Risk can be exposed gradually.

2. Feature Flag Is Not Just Boolean

Simple boolean flag:

if (flags.enabled("new-pricing")) {
    return newPricing.calculate(input);
}
return oldPricing.calculate(input);

Real enterprise flag can depend on:

- environment
- tenant
- market/region
- user role
- product type
- request type
- percentage rollout
- account/customer
- contract version
- API version
- operation type
- time window

So the real question is:

What is the evaluation context?

Example:

public record FlagContext(
    String environment,
    String tenantId,
    String userId,
    String region,
    String operation,
    String apiVersion
) {}

3. Flag Taxonomy

Not all flags are the same.

Useful taxonomy:

release flag
  enables unfinished/new functionality safely

experiment flag
  compares behavior variants

ops flag
  controls runtime operational behavior

permission flag
  enables capability for customer/tenant/user

kill switch
  disables dangerous behavior quickly

migration flag
  switches between old/new implementation during migration

integration flag
  routes calls to new downstream/provider

schema/contract flag
  controls compatibility behavior during schema/API transition

Each type has different lifecycle.

Release flag should be removed after rollout.
Ops flag may be permanent.
Kill switch may be permanent but must be tested.
Migration flag should be temporary.
Permission flag may be productized.

4. Kill Switch

A kill switch disables behavior quickly during incident or elevated risk.

Examples:

- disable event publication
- disable expensive recalculation
- disable third-party integration
- disable non-critical validation
- disable async enrichment
- disable preview endpoint
- disable batch job
- disable optional notification

A kill switch should be:

- fast to change
- safe by default
- observable
- documented
- tested
- owned
- scoped carefully

Bad kill switch:

flag exists but nobody knows it
flag change needs full redeploy
flag disables too much
flag silently corrupts state
flag has never been tested

Good kill switch:

Disables non-critical external enrichment for tenant X while preserving core quote creation.

5. Dark Launch

Dark launch means code path runs without visible user impact.

Examples:

- calculate new pricing in shadow mode but return old result
- call new downstream but ignore response
- publish event to shadow topic
- validate new rule but do not reject request
- compute new catalog eligibility but do not expose it

Purpose:

- measure correctness
- measure performance
- compare old vs new result
- detect integration failure
- warm cache
- build confidence before release

Dark launch risk:

- it still consumes resources
- it may mutate state accidentally
- it may call downstream with real side effects
- it may leak data
- it may generate confusing logs/events

Senior rule:

Dark launch must be side-effect controlled.

6. Shadow Mode Pattern

For deterministic computation:

QuotePrice oldPrice = oldPricing.calculate(request);

if (flags.enabled("pricing.shadow", context)) {
    QuotePrice newPrice = newPricing.calculate(request);
    comparisonRecorder.record(request.quoteId(), oldPrice, newPrice);
}

return oldPrice;

Important safeguards:

- new path must not mutate canonical state
- comparison result must be sampled or bounded
- failure in shadow path must not fail user path unless intended
- latency overhead must be measured
- sensitive data must be protected

Bad shadow mode:

New pricing writes final price while old pricing response is returned.

That is not shadow mode. That is inconsistent state mutation.


7. Flag Evaluation Location

Where should flag be evaluated?

Possible locations:

- JAX-RS resource method
- application service
- domain service
- repository/infrastructure adapter
- HTTP client wrapper
- Kafka publisher
- scheduler/job
- UI/client
- API gateway

Rule of thumb:

Evaluate the flag at the layer that owns the behavioral decision.

Examples:

Endpoint availability -> resource/API layer
Pricing algorithm -> application/domain service
Downstream provider switch -> integration adapter
Event publication -> event publisher boundary
Batch job enablement -> scheduler/job boundary

Avoid scattering the same flag everywhere.


8. Feature Flag in JAX-RS Resource

Acceptable for API surface control:

@POST
@Path("/quotes/preview")
public Response preview(QuotePreviewRequest request, @Context SecurityContext security) {
    FlagContext context = flagContextFactory.from(security);

    if (!flags.enabled("quote.preview.endpoint", context)) {
        return Response.status(404).build();
    }

    return Response.ok(service.preview(request)).build();
}

Design decision:

Return 404, 403, or 501?

Depends on governance:

404 hides endpoint existence.
403 says endpoint exists but not allowed.
501 says not implemented/supported.
409/422 may apply when business capability unavailable.

Verify internal API standard.


9. Feature Flag in Application Service

Better for business behavior:

public QuoteResult createQuote(CreateQuoteCommand command) {
    FlagContext context = flagContextFactory.from(command);

    PricingResult price = flags.enabled("pricing.v2", context)
        ? pricingV2.calculate(command)
        : pricingV1.calculate(command);

    return quoteRepository.save(command, price);
}

Important:

- both paths must honor same invariant
- both paths must emit compatible response
- both paths must handle validation consistently
- both paths must be observable

If old and new paths write different schema, you need migration compatibility.


10. Feature Flag in Integration Adapter

Example: route to new downstream pricing provider.

public PricingResponse price(PricingRequest request) {
    FlagContext context = flagContextFactory.from(request);

    if (flags.enabled("pricing.provider.v2", context)) {
        return pricingProviderV2.price(request);
    }

    return pricingProviderV1.price(request);
}

Check:

- timeout parity
- retry policy parity
- error mapping parity
- response shape compatibility
- audit/log consistency
- idempotency behavior
- downstream side effects

Switching provider is not just changing URL.


11. Flag Evaluation Context

A flag decision must be explainable.

Evaluation context may include:

environment
service name
service version
tenant ID
user ID
role/permission
region
market
operation
resource type
contract version
API version
request header
percentage bucket

But avoid high-cardinality observability explosion.

Do not blindly add these as metric labels:

tenant ID
user ID
customer ID
quote ID
order ID

Use them carefully in logs/traces with redaction and sampling.


12. Tenant-Specific Feature Flags

Enterprise systems often enable capability by tenant/customer.

Example:

Tenant A uses pricing v1.
Tenant B uses pricing v2.
Tenant C is in shadow mode.

This supports controlled rollout but introduces complexity.

Risks:

- tenant behavior drift
- hard-to-reproduce bugs
- inconsistent support runbooks
- hidden compatibility matrix
- stale tenant exceptions
- cross-tenant leakage in flag context

Senior rule:

Tenant-specific flag must have ownership, expiry, and observability.

13. Flag Defaults

Defaults are critical.

Bad:

boolean enabled = flagClient.getBoolean("new-order-flow", true);

If flag service is down, new behavior activates.

Better depends on risk:

For release flag: default disabled.
For kill switch: default may be safe-disabled or safe-enabled depending semantics.
For critical dependency disablement: default should preserve safest core behavior.

Explicit example:

boolean enabled = flags.booleanFlag(
    "pricing.v2",
    context,
    Default.DISABLED_ON_ERROR
);

Make fallback behavior visible in logs/metrics.


14. Flag Service Dependency Failure

Feature flag systems can fail.

Failure modes:

flag service unavailable
network timeout
stale cache
invalid flag payload
permission denied
SDK initialization failure
flag deleted unexpectedly
clock skew affects scheduled rule

App must define behavior:

- use cached value
- use default value
- fail startup
- fail request
- disable feature
- enable safety mode

For most release flags:

flag failure -> old stable behavior

For security-sensitive flags:

flag failure -> fail closed

15. Flag Caching

Flag evaluation can be:

- local static config
- in-memory cache
- streaming SDK update
- polling SDK update
- remote evaluation per request
- environment file/config map

Trade-offs:

Remote per request:
  + fresh
  - latency dependency
  - availability risk

Local cache:
  + fast
  + resilient
  - stale value risk

Streaming update:
  + near real-time
  - SDK/network complexity

Senior question:

How stale can a flag value safely be?

For kill switch, staleness matters.


16. Percentage Rollout

Percentage rollout needs stable bucketing.

Bad:

if (Math.random() < 0.1) enable();

Problem:

same user/tenant may flip between paths per request
hard to debug
state becomes inconsistent

Better:

hash(tenantId + flagName) % 100 < rolloutPercentage

For stateful business workflows, prefer tenant/customer-level bucketing over request-level bucketing.

Reason:

A quote/order process should not randomly switch behavior halfway through lifecycle.

17. Feature Flags and API Compatibility

Feature flags do not excuse breaking API contracts.

Risk:

flag enabled for some tenants
  -> response shape changes
  -> generated client breaks
  -> OpenAPI no longer describes behavior

Safe pattern:

- additive response fields only
- old fields remain valid
- versioned endpoint if contract changes
- explicit media type/version negotiation
- documentation of conditional behavior

If behavior differs by tenant, contract must still be understandable.


18. Feature Flags and Database Migration

Flagged code often depends on schema changes.

Safe migration flow:

1. expand schema backward-compatibly
2. deploy code that can read/write old and new if needed
3. enable flag for shadow/small scope
4. verify metrics and data correctness
5. increase rollout
6. migrate/backfill data
7. remove old code path
8. contract schema if safe

Bad:

flag off but old code cannot run because schema was changed incompatibly

Feature flag is not rollback if database migration is not backward-compatible.


19. Feature Flags and Event Publication

Flags may control event behavior:

- publish new event
- publish old and new event
- publish to shadow topic
- change payload field population
- enable CDC/outbox publisher

Risks:

- consumer does not expect new event
- event versioning unclear
- duplicate publication
- replay behavior differs by flag state
- flag value at replay time differs from original processing time

Important rule:

Event replay should not accidentally use today's flag state if it must reproduce historical behavior.

Store decision if needed:

- flag snapshot
- event version
- processing mode
- saga state

20. Feature Flags and Auditability

For business-critical behavior, flag decision may need audit trail.

Examples:

- pricing algorithm selected
- catalog rule version selected
- discount eligibility rule selected
- tax calculation mode selected
- tenant-specific behavior selected

Audit record may include:

- flag name
- evaluated value
- rule/version
- tenant
- operation
- timestamp
- service version

Do not include sensitive flag context unnecessarily.


21. Observability for Flags

You need to know which path ran.

Useful telemetry:

- count by flag variant
- error rate by old/new path
- latency by old/new path
- fallback/default usage
- flag evaluation failure
- kill switch state change
- shadow mismatch count

Avoid high cardinality labels.

Good metric dimensions:

flag_name
variant
environment
service

Risky dimensions:

tenant_id
user_id
quote_id
order_id

For tenant-level debug, use logs/traces with sampling and access control.


22. Shadow Comparison Metrics

For dark launch/shadow mode, compare outcomes.

Examples:

old price vs new price
old validation result vs new validation result
old catalog eligibility vs new catalog eligibility
old downstream response category vs new downstream response category

Do not only compare success/failure.

Compare:

- exact result when deterministic
- tolerance window for monetary calculation
- category/classification
- latency
- downstream error type
- data completeness

For money:

Use BigDecimal comparison with explicit scale/rounding policy.

23. Kill Switch Runbook

Every real kill switch should have a runbook.

Runbook should answer:

- what does it disable?
- what remains functional?
- who can toggle it?
- how fast does it propagate?
- how do we verify it worked?
- what customer impact occurs?
- how do we re-enable safely?
- what metrics should be watched?

Example:

Kill switch: external-pricing-enrichment.enabled=false
Impact: quote can still be created using cached/default enrichment policy.
Risk: price enrichment may be less precise for affected tenants.
Verification: downstream call rate drops to zero, quote creation success recovers.

24. Flag Lifecycle and Cleanup

Temporary flags must be removed.

Lifecycle:

proposal
  -> implementation
  -> rollout
  -> full enablement
  -> stabilization
  -> cleanup
  -> deletion

If not cleaned:

- dead code remains
- tests multiply
- behavior matrix grows
- onboarding becomes harder
- production debugging becomes ambiguous

Flag definition should include:

- owner
- purpose
- type
- default
- creation date
- expected removal date
- cleanup issue/link

25. Flag Naming

Good names are boring and explicit.

Bad:

newFlow
v2
enableThing
fastMode

Better:

quote.pricing.algorithm.v2.enabled
order.submission.async-validation.enabled
catalog.eligibility.shadow.enabled
integration.crm.v2.route.enabled
notification.email.kill-switch.disabled

Naming should communicate:

- domain/component
- behavior
- variant/version
- enabled/disabled semantics

Avoid inverse confusion:

disableNewPricing=false

Prefer positive names:

pricing.v2.enabled=true

For kill switch, be extra clear:

pricing.external-enrichment.disabled=true

26. Testing Feature Flags

Tests must cover:

- flag enabled
- flag disabled
- flag service failure
- default behavior
- tenant-specific behavior
- rollout bucket stability
- kill switch behavior
- shadow mode failure isolation

Example unit test matrix:

pricing.v2 disabled -> old pricing called
pricing.v2 enabled  -> new pricing called
flag failure        -> old pricing called + metric emitted

For integration tests:

- use fake flag provider
- make flag context explicit
- avoid tests depending on external flag service
- assert telemetry if critical

27. Feature Flags and Local Development

Local dev should make flags visible.

Options:

- local YAML/properties flag file
- fake flag service
- environment variable override
- dev UI
- test fixture flag provider

Local setup should answer:

How do I enable pricing.v2 locally?
How do I simulate flag service down?
How do I test tenant-specific flag?
How do I inspect effective flag values safely?

Avoid:

local dev always enables everything

That hides production behavior.


28. Feature Flags and Security

Do not use feature flags as the only authorization mechanism.

Bad:

if feature flag enabled -> allow user action

Better:

if feature enabled for tenant
  and user authorized
  and operation allowed
  then proceed

Feature flag controls availability. Authorization controls permission.

They are different.

Security-sensitive flags must fail closed.

Example:

If entitlement flag cannot be evaluated, do not grant capability by default.

29. Feature Flags and Performance

Flag evaluation must not become request bottleneck.

Check:

- per-request remote calls
- cache hit rate
- SDK initialization
- lock contention
- context construction overhead
- JSON parsing per request
- high-cardinality telemetry

If flag evaluation sits on hot path:

- cache locally
- precompute context
- avoid blocking remote call
- bound timeout tightly
- default safely on error

30. Failure Modes

Common failure modes:

flag default unsafe
flag service unavailable
stale flag value
flag deleted while code still depends on it
wrong tenant context
wrong environment config
percentage rollout unstable
kill switch too slow
shadow path mutates state
flagged response breaks contract
flagged event breaks consumer
flag remains forever
flag value not observable

Systemic causes:

- no ownership
- no cleanup policy
- no evaluation context standard
- no observability
- no compatibility discipline
- no runbook
- no test matrix

31. Debugging Playbook

When behavior differs unexpectedly:

1. Identify request/tenant/user/environment.
2. Identify relevant flag names.
3. Check evaluated flag value, not only configured value.
4. Check evaluation context.
5. Check flag default/fallback used or not.
6. Check flag service/cache health.
7. Check recent flag changes.
8. Check service version.
9. Check whether DB/event/API contract is compatible.
10. Check telemetry by old/new path.

Important distinction:

configured flag value != evaluated flag value

Evaluation may include targeting rules, environment, tenant, percentage bucket, and defaults.


32. PR Review Checklist

When reviewing feature flag changes:

[ ] Is the flag type clear: release, ops, kill switch, migration, permission, experiment?
[ ] Is default behavior safe?
[ ] Is failure behavior defined?
[ ] Is evaluation context explicit?
[ ] Is tenant targeting safe if used?
[ ] Is authorization still enforced separately?
[ ] Is contract compatibility preserved?
[ ] Is DB migration compatible with flag rollback?
[ ] Is event publication compatible?
[ ] Is telemetry added for old/new path?
[ ] Is kill switch runbook documented if applicable?
[ ] Are tests covering enabled/disabled/failure/default cases?
[ ] Is owner and cleanup date documented?
[ ] Is stale flag cleanup planned?

33. Internal Verification Checklist

Untuk codebase/internal platform CSG, verifikasi:

[ ] Apakah ada feature flag platform/library internal.
[ ] Apakah memakai LaunchDarkly, Unleash, ConfigCat, FF4J, Togglz, custom config, atau lainnya.
[ ] Apakah flag dievaluasi lokal, remote, polling, streaming, atau config file.
[ ] Apakah ada standard flag naming.
[ ] Apakah ada standard default-on-error.
[ ] Apakah ada kill switch untuk downstream integration kritikal.
[ ] Apakah ada runbook untuk kill switch.
[ ] Apakah tenant-specific flag didukung.
[ ] Apakah flag context memasukkan tenant/user/region/market.
[ ] Apakah flag decision muncul di logs/traces/metrics secara aman.
[ ] Apakah ada cleanup policy untuk temporary flags.
[ ] Apakah flag perubahan diaudit.
[ ] Apakah flag dapat diubah tanpa redeploy.
[ ] Apakah flag propagation delay diketahui.
[ ] Apakah flag service dependency punya timeout/cache.
[ ] Apakah feature flag digunakan untuk migration database/API/event.
[ ] Apakah rollback strategy bergantung pada flag.

34. Senior Mental Model

Feature flag adalah runtime decision boundary.

Untuk setiap flag, senior engineer harus bisa menjawab:

What behavior does this change?
Who owns it?
What is the safe default?
What happens if flag evaluation fails?
Who/tenant/environment is affected?
Does it preserve API, DB, and event compatibility?
How do we observe old vs new behavior?
How do we disable it during incident?
When will we delete it?

Final rule:

Feature flags are powerful because they add runtime control.
They are dangerous because they add runtime ambiguity.

Good engineering turns that ambiguity into explicit context, telemetry, ownership, and cleanup.

Lesson Recap

You just completed lesson 42 in build core. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.