Feature Flags and Kill Switch
Feature Flags Kill Switch and Dark Launch
Kontrol perilaku runtime untuk safe rollout, progressive enablement, tenant-specific behavior, dan emergency disablement
Part 042 — Feature Flags, Kill Switch, and Dark Launch
Fokus part ini: memahami feature flag sebagai mekanisme runtime control, bukan sekadar
ifstatement. Kita akan membahas flag taxonomy, kill switch, dark launch, tenant targeting, safe defaults, observability, lifecycle cleanup, dan failure mode pada JAX-RS enterprise service.
Feature flag adalah control plane kecil di dalam application behavior.
Ia memungkinkan sistem menjawab:
Should this code path be active for this request, tenant, user, environment, or integration right now?
Dalam sistem enterprise seperti CPQ/quote/order management, feature flag dapat mengontrol:
- endpoint baru
- pricing rule baru
- catalog behavior baru
- integration route baru
- validation rule baru
- Kafka event publication baru
- downstream client baru
- tenant-specific enablement
- rollback behavior
- emergency disablement
Senior warning:
Feature flags reduce release risk only when their lifecycle, defaults, ownership, and observability are disciplined.
Tanpa discipline, feature flags menjadi hidden complexity.
1. Deployment vs Release
Feature flag memisahkan deployment dari release.
Deployment = code exists in runtime.
Release = behavior is enabled for users/tenants/traffic.
Without feature flags:
merge code
-> deploy
-> behavior immediately active
With feature flags:
merge code
-> deploy inactive or partially active
-> enable gradually
-> observe
-> expand or disable
This is important because production risk is not binary.
Risk can be exposed gradually.
2. Feature Flag Is Not Just Boolean
Simple boolean flag:
if (flags.enabled("new-pricing")) {
return newPricing.calculate(input);
}
return oldPricing.calculate(input);
Real enterprise flag can depend on:
- environment
- tenant
- market/region
- user role
- product type
- request type
- percentage rollout
- account/customer
- contract version
- API version
- operation type
- time window
So the real question is:
What is the evaluation context?
Example:
public record FlagContext(
String environment,
String tenantId,
String userId,
String region,
String operation,
String apiVersion
) {}
3. Flag Taxonomy
Not all flags are the same.
Useful taxonomy:
release flag
enables unfinished/new functionality safely
experiment flag
compares behavior variants
ops flag
controls runtime operational behavior
permission flag
enables capability for customer/tenant/user
kill switch
disables dangerous behavior quickly
migration flag
switches between old/new implementation during migration
integration flag
routes calls to new downstream/provider
schema/contract flag
controls compatibility behavior during schema/API transition
Each type has different lifecycle.
Release flag should be removed after rollout.
Ops flag may be permanent.
Kill switch may be permanent but must be tested.
Migration flag should be temporary.
Permission flag may be productized.
4. Kill Switch
A kill switch disables behavior quickly during incident or elevated risk.
Examples:
- disable event publication
- disable expensive recalculation
- disable third-party integration
- disable non-critical validation
- disable async enrichment
- disable preview endpoint
- disable batch job
- disable optional notification
A kill switch should be:
- fast to change
- safe by default
- observable
- documented
- tested
- owned
- scoped carefully
Bad kill switch:
flag exists but nobody knows it
flag change needs full redeploy
flag disables too much
flag silently corrupts state
flag has never been tested
Good kill switch:
Disables non-critical external enrichment for tenant X while preserving core quote creation.
5. Dark Launch
Dark launch means code path runs without visible user impact.
Examples:
- calculate new pricing in shadow mode but return old result
- call new downstream but ignore response
- publish event to shadow topic
- validate new rule but do not reject request
- compute new catalog eligibility but do not expose it
Purpose:
- measure correctness
- measure performance
- compare old vs new result
- detect integration failure
- warm cache
- build confidence before release
Dark launch risk:
- it still consumes resources
- it may mutate state accidentally
- it may call downstream with real side effects
- it may leak data
- it may generate confusing logs/events
Senior rule:
Dark launch must be side-effect controlled.
6. Shadow Mode Pattern
For deterministic computation:
QuotePrice oldPrice = oldPricing.calculate(request);
if (flags.enabled("pricing.shadow", context)) {
QuotePrice newPrice = newPricing.calculate(request);
comparisonRecorder.record(request.quoteId(), oldPrice, newPrice);
}
return oldPrice;
Important safeguards:
- new path must not mutate canonical state
- comparison result must be sampled or bounded
- failure in shadow path must not fail user path unless intended
- latency overhead must be measured
- sensitive data must be protected
Bad shadow mode:
New pricing writes final price while old pricing response is returned.
That is not shadow mode. That is inconsistent state mutation.
7. Flag Evaluation Location
Where should flag be evaluated?
Possible locations:
- JAX-RS resource method
- application service
- domain service
- repository/infrastructure adapter
- HTTP client wrapper
- Kafka publisher
- scheduler/job
- UI/client
- API gateway
Rule of thumb:
Evaluate the flag at the layer that owns the behavioral decision.
Examples:
Endpoint availability -> resource/API layer
Pricing algorithm -> application/domain service
Downstream provider switch -> integration adapter
Event publication -> event publisher boundary
Batch job enablement -> scheduler/job boundary
Avoid scattering the same flag everywhere.
8. Feature Flag in JAX-RS Resource
Acceptable for API surface control:
@POST
@Path("/quotes/preview")
public Response preview(QuotePreviewRequest request, @Context SecurityContext security) {
FlagContext context = flagContextFactory.from(security);
if (!flags.enabled("quote.preview.endpoint", context)) {
return Response.status(404).build();
}
return Response.ok(service.preview(request)).build();
}
Design decision:
Return 404, 403, or 501?
Depends on governance:
404 hides endpoint existence.
403 says endpoint exists but not allowed.
501 says not implemented/supported.
409/422 may apply when business capability unavailable.
Verify internal API standard.
9. Feature Flag in Application Service
Better for business behavior:
public QuoteResult createQuote(CreateQuoteCommand command) {
FlagContext context = flagContextFactory.from(command);
PricingResult price = flags.enabled("pricing.v2", context)
? pricingV2.calculate(command)
: pricingV1.calculate(command);
return quoteRepository.save(command, price);
}
Important:
- both paths must honor same invariant
- both paths must emit compatible response
- both paths must handle validation consistently
- both paths must be observable
If old and new paths write different schema, you need migration compatibility.
10. Feature Flag in Integration Adapter
Example: route to new downstream pricing provider.
public PricingResponse price(PricingRequest request) {
FlagContext context = flagContextFactory.from(request);
if (flags.enabled("pricing.provider.v2", context)) {
return pricingProviderV2.price(request);
}
return pricingProviderV1.price(request);
}
Check:
- timeout parity
- retry policy parity
- error mapping parity
- response shape compatibility
- audit/log consistency
- idempotency behavior
- downstream side effects
Switching provider is not just changing URL.
11. Flag Evaluation Context
A flag decision must be explainable.
Evaluation context may include:
environment
service name
service version
tenant ID
user ID
role/permission
region
market
operation
resource type
contract version
API version
request header
percentage bucket
But avoid high-cardinality observability explosion.
Do not blindly add these as metric labels:
tenant ID
user ID
customer ID
quote ID
order ID
Use them carefully in logs/traces with redaction and sampling.
12. Tenant-Specific Feature Flags
Enterprise systems often enable capability by tenant/customer.
Example:
Tenant A uses pricing v1.
Tenant B uses pricing v2.
Tenant C is in shadow mode.
This supports controlled rollout but introduces complexity.
Risks:
- tenant behavior drift
- hard-to-reproduce bugs
- inconsistent support runbooks
- hidden compatibility matrix
- stale tenant exceptions
- cross-tenant leakage in flag context
Senior rule:
Tenant-specific flag must have ownership, expiry, and observability.
13. Flag Defaults
Defaults are critical.
Bad:
boolean enabled = flagClient.getBoolean("new-order-flow", true);
If flag service is down, new behavior activates.
Better depends on risk:
For release flag: default disabled.
For kill switch: default may be safe-disabled or safe-enabled depending semantics.
For critical dependency disablement: default should preserve safest core behavior.
Explicit example:
boolean enabled = flags.booleanFlag(
"pricing.v2",
context,
Default.DISABLED_ON_ERROR
);
Make fallback behavior visible in logs/metrics.
14. Flag Service Dependency Failure
Feature flag systems can fail.
Failure modes:
flag service unavailable
network timeout
stale cache
invalid flag payload
permission denied
SDK initialization failure
flag deleted unexpectedly
clock skew affects scheduled rule
App must define behavior:
- use cached value
- use default value
- fail startup
- fail request
- disable feature
- enable safety mode
For most release flags:
flag failure -> old stable behavior
For security-sensitive flags:
flag failure -> fail closed
15. Flag Caching
Flag evaluation can be:
- local static config
- in-memory cache
- streaming SDK update
- polling SDK update
- remote evaluation per request
- environment file/config map
Trade-offs:
Remote per request:
+ fresh
- latency dependency
- availability risk
Local cache:
+ fast
+ resilient
- stale value risk
Streaming update:
+ near real-time
- SDK/network complexity
Senior question:
How stale can a flag value safely be?
For kill switch, staleness matters.
16. Percentage Rollout
Percentage rollout needs stable bucketing.
Bad:
if (Math.random() < 0.1) enable();
Problem:
same user/tenant may flip between paths per request
hard to debug
state becomes inconsistent
Better:
hash(tenantId + flagName) % 100 < rolloutPercentage
For stateful business workflows, prefer tenant/customer-level bucketing over request-level bucketing.
Reason:
A quote/order process should not randomly switch behavior halfway through lifecycle.
17. Feature Flags and API Compatibility
Feature flags do not excuse breaking API contracts.
Risk:
flag enabled for some tenants
-> response shape changes
-> generated client breaks
-> OpenAPI no longer describes behavior
Safe pattern:
- additive response fields only
- old fields remain valid
- versioned endpoint if contract changes
- explicit media type/version negotiation
- documentation of conditional behavior
If behavior differs by tenant, contract must still be understandable.
18. Feature Flags and Database Migration
Flagged code often depends on schema changes.
Safe migration flow:
1. expand schema backward-compatibly
2. deploy code that can read/write old and new if needed
3. enable flag for shadow/small scope
4. verify metrics and data correctness
5. increase rollout
6. migrate/backfill data
7. remove old code path
8. contract schema if safe
Bad:
flag off but old code cannot run because schema was changed incompatibly
Feature flag is not rollback if database migration is not backward-compatible.
19. Feature Flags and Event Publication
Flags may control event behavior:
- publish new event
- publish old and new event
- publish to shadow topic
- change payload field population
- enable CDC/outbox publisher
Risks:
- consumer does not expect new event
- event versioning unclear
- duplicate publication
- replay behavior differs by flag state
- flag value at replay time differs from original processing time
Important rule:
Event replay should not accidentally use today's flag state if it must reproduce historical behavior.
Store decision if needed:
- flag snapshot
- event version
- processing mode
- saga state
20. Feature Flags and Auditability
For business-critical behavior, flag decision may need audit trail.
Examples:
- pricing algorithm selected
- catalog rule version selected
- discount eligibility rule selected
- tax calculation mode selected
- tenant-specific behavior selected
Audit record may include:
- flag name
- evaluated value
- rule/version
- tenant
- operation
- timestamp
- service version
Do not include sensitive flag context unnecessarily.
21. Observability for Flags
You need to know which path ran.
Useful telemetry:
- count by flag variant
- error rate by old/new path
- latency by old/new path
- fallback/default usage
- flag evaluation failure
- kill switch state change
- shadow mismatch count
Avoid high cardinality labels.
Good metric dimensions:
flag_name
variant
environment
service
Risky dimensions:
tenant_id
user_id
quote_id
order_id
For tenant-level debug, use logs/traces with sampling and access control.
22. Shadow Comparison Metrics
For dark launch/shadow mode, compare outcomes.
Examples:
old price vs new price
old validation result vs new validation result
old catalog eligibility vs new catalog eligibility
old downstream response category vs new downstream response category
Do not only compare success/failure.
Compare:
- exact result when deterministic
- tolerance window for monetary calculation
- category/classification
- latency
- downstream error type
- data completeness
For money:
Use BigDecimal comparison with explicit scale/rounding policy.
23. Kill Switch Runbook
Every real kill switch should have a runbook.
Runbook should answer:
- what does it disable?
- what remains functional?
- who can toggle it?
- how fast does it propagate?
- how do we verify it worked?
- what customer impact occurs?
- how do we re-enable safely?
- what metrics should be watched?
Example:
Kill switch: external-pricing-enrichment.enabled=false
Impact: quote can still be created using cached/default enrichment policy.
Risk: price enrichment may be less precise for affected tenants.
Verification: downstream call rate drops to zero, quote creation success recovers.
24. Flag Lifecycle and Cleanup
Temporary flags must be removed.
Lifecycle:
proposal
-> implementation
-> rollout
-> full enablement
-> stabilization
-> cleanup
-> deletion
If not cleaned:
- dead code remains
- tests multiply
- behavior matrix grows
- onboarding becomes harder
- production debugging becomes ambiguous
Flag definition should include:
- owner
- purpose
- type
- default
- creation date
- expected removal date
- cleanup issue/link
25. Flag Naming
Good names are boring and explicit.
Bad:
newFlow
v2
enableThing
fastMode
Better:
quote.pricing.algorithm.v2.enabled
order.submission.async-validation.enabled
catalog.eligibility.shadow.enabled
integration.crm.v2.route.enabled
notification.email.kill-switch.disabled
Naming should communicate:
- domain/component
- behavior
- variant/version
- enabled/disabled semantics
Avoid inverse confusion:
disableNewPricing=false
Prefer positive names:
pricing.v2.enabled=true
For kill switch, be extra clear:
pricing.external-enrichment.disabled=true
26. Testing Feature Flags
Tests must cover:
- flag enabled
- flag disabled
- flag service failure
- default behavior
- tenant-specific behavior
- rollout bucket stability
- kill switch behavior
- shadow mode failure isolation
Example unit test matrix:
pricing.v2 disabled -> old pricing called
pricing.v2 enabled -> new pricing called
flag failure -> old pricing called + metric emitted
For integration tests:
- use fake flag provider
- make flag context explicit
- avoid tests depending on external flag service
- assert telemetry if critical
27. Feature Flags and Local Development
Local dev should make flags visible.
Options:
- local YAML/properties flag file
- fake flag service
- environment variable override
- dev UI
- test fixture flag provider
Local setup should answer:
How do I enable pricing.v2 locally?
How do I simulate flag service down?
How do I test tenant-specific flag?
How do I inspect effective flag values safely?
Avoid:
local dev always enables everything
That hides production behavior.
28. Feature Flags and Security
Do not use feature flags as the only authorization mechanism.
Bad:
if feature flag enabled -> allow user action
Better:
if feature enabled for tenant
and user authorized
and operation allowed
then proceed
Feature flag controls availability. Authorization controls permission.
They are different.
Security-sensitive flags must fail closed.
Example:
If entitlement flag cannot be evaluated, do not grant capability by default.
29. Feature Flags and Performance
Flag evaluation must not become request bottleneck.
Check:
- per-request remote calls
- cache hit rate
- SDK initialization
- lock contention
- context construction overhead
- JSON parsing per request
- high-cardinality telemetry
If flag evaluation sits on hot path:
- cache locally
- precompute context
- avoid blocking remote call
- bound timeout tightly
- default safely on error
30. Failure Modes
Common failure modes:
flag default unsafe
flag service unavailable
stale flag value
flag deleted while code still depends on it
wrong tenant context
wrong environment config
percentage rollout unstable
kill switch too slow
shadow path mutates state
flagged response breaks contract
flagged event breaks consumer
flag remains forever
flag value not observable
Systemic causes:
- no ownership
- no cleanup policy
- no evaluation context standard
- no observability
- no compatibility discipline
- no runbook
- no test matrix
31. Debugging Playbook
When behavior differs unexpectedly:
1. Identify request/tenant/user/environment.
2. Identify relevant flag names.
3. Check evaluated flag value, not only configured value.
4. Check evaluation context.
5. Check flag default/fallback used or not.
6. Check flag service/cache health.
7. Check recent flag changes.
8. Check service version.
9. Check whether DB/event/API contract is compatible.
10. Check telemetry by old/new path.
Important distinction:
configured flag value != evaluated flag value
Evaluation may include targeting rules, environment, tenant, percentage bucket, and defaults.
32. PR Review Checklist
When reviewing feature flag changes:
[ ] Is the flag type clear: release, ops, kill switch, migration, permission, experiment?
[ ] Is default behavior safe?
[ ] Is failure behavior defined?
[ ] Is evaluation context explicit?
[ ] Is tenant targeting safe if used?
[ ] Is authorization still enforced separately?
[ ] Is contract compatibility preserved?
[ ] Is DB migration compatible with flag rollback?
[ ] Is event publication compatible?
[ ] Is telemetry added for old/new path?
[ ] Is kill switch runbook documented if applicable?
[ ] Are tests covering enabled/disabled/failure/default cases?
[ ] Is owner and cleanup date documented?
[ ] Is stale flag cleanup planned?
33. Internal Verification Checklist
Untuk codebase/internal platform CSG, verifikasi:
[ ] Apakah ada feature flag platform/library internal.
[ ] Apakah memakai LaunchDarkly, Unleash, ConfigCat, FF4J, Togglz, custom config, atau lainnya.
[ ] Apakah flag dievaluasi lokal, remote, polling, streaming, atau config file.
[ ] Apakah ada standard flag naming.
[ ] Apakah ada standard default-on-error.
[ ] Apakah ada kill switch untuk downstream integration kritikal.
[ ] Apakah ada runbook untuk kill switch.
[ ] Apakah tenant-specific flag didukung.
[ ] Apakah flag context memasukkan tenant/user/region/market.
[ ] Apakah flag decision muncul di logs/traces/metrics secara aman.
[ ] Apakah ada cleanup policy untuk temporary flags.
[ ] Apakah flag perubahan diaudit.
[ ] Apakah flag dapat diubah tanpa redeploy.
[ ] Apakah flag propagation delay diketahui.
[ ] Apakah flag service dependency punya timeout/cache.
[ ] Apakah feature flag digunakan untuk migration database/API/event.
[ ] Apakah rollback strategy bergantung pada flag.
34. Senior Mental Model
Feature flag adalah runtime decision boundary.
Untuk setiap flag, senior engineer harus bisa menjawab:
What behavior does this change?
Who owns it?
What is the safe default?
What happens if flag evaluation fails?
Who/tenant/environment is affected?
Does it preserve API, DB, and event compatibility?
How do we observe old vs new behavior?
How do we disable it during incident?
When will we delete it?
Final rule:
Feature flags are powerful because they add runtime control.
They are dangerous because they add runtime ambiguity.
Good engineering turns that ambiguity into explicit context, telemetry, ownership, and cleanup.
You just completed lesson 42 in build core. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.