Secret Management
Secret Management and Secure Configuration
Menangani secret tanpa bocor ke code, config, log, artifact, pipeline, atau runtime surface
Part 041 — Secret Management and Secure Configuration
Fokus part ini: membedakan configuration biasa dari secret, memahami lifecycle secret dari source sampai runtime, menghindari leak, dan membuat service JAX-RS yang aman terhadap rotasi, logging, debugging, deployment, dan incident.
Secret management bukan sekadar menyimpan password di tempat yang “lebih aman”.
Secret management adalah kontrol lifecycle untuk nilai sensitif yang memberi akses ke sistem, data, tenant, credential, atau trust boundary.
Dalam sistem enterprise seperti CPQ/quote/order management, secret dapat menyentuh:
- database credential
- Kafka credential
- HTTP downstream API key
- OAuth/OIDC client secret
- private key untuk signing
- mTLS private key
- cloud access credential
- storage credential
- encryption key reference
- webhook secret
- integration credential per customer/tenant
Senior rule:
A secret is not secure because it is hidden once.
A secret is secure when its entire lifecycle is controlled.
1. Configuration vs Secret
Tidak semua configuration adalah secret. Tidak semua secret terlihat seperti password.
Configuration biasa:
- port
- base URL
- feature flag default
- timeout value
- retry count
- environment name
- topic name
- pool size
- endpoint path
Secret:
- password
- API token
- private key
- client secret
- signing key
- encryption key material
- bearer token
- basic auth credential
- database credential
- service account credential
- webhook verification secret
Gray area:
- internal hostnames
- tenant-specific routing key
- customer integration endpoint
- key ID
- certificate fingerprint
- username without password
- account ID
Gray area belum tentu secret, tetapi bisa sensitif.
Rule praktis:
If disclosure helps an attacker move, enumerate, impersonate, decrypt, or exfiltrate, treat it as sensitive.
2. Secret Lifecycle
Secret punya lifecycle lengkap:
creation
-> storage
-> distribution
-> injection
-> use
-> rotation
-> revocation
-> audit
-> deletion
Bug umum terjadi karena engineer hanya memikirkan bagian use.
Contoh problem:
Secret safely stored in vault
but copied into CI log
but dumped in exception message
but printed by config debug endpoint
but committed into test fixture
but left in old Kubernetes Secret
but not rotated after incident
Mental model production:
Secret safety = weakest lifecycle step.
3. Common Secret Sources
Secret dapat datang dari banyak sumber.
- environment variable
- properties file
- mounted file
- Kubernetes Secret
- external secret operator
- cloud secret manager
- vault service
- CI/CD secret store
- developer local file
- test fixture
- sealed secret manifest
- service mesh certificate store
Tidak ada satu sumber yang otomatis benar.
Yang harus dinilai:
- siapa bisa membaca secret?
- bagaimana secret sampai ke runtime?
- apakah secret muncul di logs?
- apakah secret muncul di process environment?
- apakah secret bisa dirotasi?
- apakah perubahan butuh redeploy?
- apakah aksesnya diaudit?
- apakah secret scoped per environment?
- apakah secret scoped per tenant/customer?
4. Secret Anti-Patterns
Anti-pattern yang sering muncul:
- hardcoded secret in source code
- secret in Git history
- secret in application.properties committed to repo
- secret in Docker image layer
- secret passed as CLI argument
- secret printed during startup
- secret returned by actuator/debug endpoint
- secret included in exception message
- secret copied into unit test snapshot
- secret stored in shared wiki without access control
- one credential shared by all environments
- one credential shared by all tenants
- secret without owner
- secret without expiry/rotation procedure
CLI argument risk:
java -jar app.jar --db.password=super-secret
Problem:
- visible in process list
- visible in shell history
- visible in orchestrator event/log
- visible in support diagnostic dump
Better:
- file mount with restricted permission
- secret manager lookup with service identity
- short-lived credential
- credential reference rather than raw key material
5. Secure Configuration Mental Model
Secure configuration harus memenuhi beberapa invariant.
1. Secret value must not be part of source code.
2. Secret value must not be part of immutable image layer.
3. Secret value must not be logged.
4. Secret value must not appear in normal exception message.
5. Secret access must be scoped to least privilege.
6. Secret should be rotatable.
7. Secret should be environment-specific.
8. Secret should have an owner.
9. Secret usage should be observable without exposing the value.
10. Secret failure must fail closed.
Fail closed artinya:
If secret cannot be loaded or validated, do not silently run in insecure mode.
Bad default:
String apiKey = env("API_KEY", "dev-key");
Better mental model:
Production startup should fail if mandatory secret is missing.
Local/dev may use explicit local profile with fake credential.
6. Secret Injection into JAX-RS Service
Dalam JAX-RS/Jersey service, secret biasanya dibaca oleh composition layer, bukan resource method langsung.
Bad:
@Path("/orders")
public class OrderResource {
@GET
public Response list() {
String password = System.getenv("DB_PASSWORD");
// use directly
}
}
Problem:
- secret lookup repeated per request
- hard to test
- hard to rotate
- resource method knows infrastructure detail
- error handling may leak value
Better:
public final class DatabaseCredentialProvider {
private final SecretSource secretSource;
public DatabaseCredentialProvider(SecretSource secretSource) {
this.secretSource = secretSource;
}
public DbCredential current() {
return secretSource.getRequired("database/main");
}
}
Resource method should depend on domain/application service, not secret source.
HTTP Resource
-> Application Service
-> Repository / Client
-> Credential Provider / DataSource / HTTP Client
-> Secret Source
7. Secret Source Abstraction
A useful abstraction is not simply Map<String, String>.
It should preserve intent:
public interface SecretSource {
SecretValue getRequired(String name);
Optional<SecretValue> getOptional(String name);
}
A SecretValue should avoid accidental string exposure:
public final class SecretValue {
private final String name;
private final char[] value;
public SecretValue(String name, char[] value) {
this.name = name;
this.value = value;
}
public String name() {
return name;
}
public char[] copyValue() {
return value.clone();
}
@Override
public String toString() {
return "SecretValue{name='" + name + "', value='****'}";
}
}
Reality check:
Java String is common and often unavoidable.
But the design should still prevent accidental logging and toString leakage.
8. Environment Variables: Useful but Risky
Environment variables are common in containers.
Pros:
- simple
- supported everywhere
- easy for local dev
- easy for twelve-factor style apps
Risks:
- visible to process introspection depending runtime
- may appear in diagnostic dumps
- sometimes exposed in orchestrator UI/event
- hard to rotate without restart
- all process code can read all env vars
- easy to accidentally log at startup
Use env vars carefully:
- avoid printing full environment
- avoid env dumps in debug endpoints
- scope variables per service
- use secret references if platform supports it
- prefer mounted files or secret manager for high-value key material
9. Mounted Secrets
Mounted secret file pattern:
/var/run/secrets/app/db-password
/var/run/secrets/app/client-secret
/var/run/secrets/app/private-key.pem
Pros:
- not stored in Java process environment
- can support file permission controls
- works well for certificates/private keys
- can be updated by external secret mechanism
Risks:
- application may read once and never reload
- permissions may be too broad
- file path may be logged
- rotation semantics depend on platform
- stale file mount can occur
Design question:
Does the app need to reload the secret at runtime, or is restart acceptable?
For database credentials, restart may be acceptable if rollout is controlled. For signing keys or external credentials, runtime refresh may be required.
10. Cloud Secret Managers
Common examples:
- AWS Secrets Manager
- AWS Systems Manager Parameter Store secure strings
- Azure Key Vault
- HashiCorp Vault
- Google Secret Manager
Important concerns:
- credential to access the secret manager
- secret manager availability
- latency and caching
- rate limits
- audit logs
- versioning
- rotation integration
- local development fallback
Do not call secret manager on every request unless the design explicitly supports it.
Better:
startup load + cache
periodic refresh
refresh on version change
short TTL cache
explicit reload endpoint/control plane event
Trade-off:
Long cache TTL: lower latency, slower rotation.
Short cache TTL: faster rotation, more dependency on secret manager availability.
11. Credential Provider Chain
Many SDKs use credential provider chains.
Example mental model:
environment variable
-> system property
-> local profile
-> workload identity
-> instance metadata
-> explicit credential
Risk:
The service may work locally for the wrong reason.
The service may pick a broader credential than expected.
The service may accidentally use developer credentials.
Senior checklist:
- know the provider chain
- pin allowed provider in production when possible
- log credential source safely, not credential value
- test missing credential behavior
- test wrong permission behavior
Safe log example:
Cloud credential source selected: workload-identity/service-account order-service-prod
Unsafe log example:
AWS_SECRET_ACCESS_KEY=...
12. Secret Rotation
Rotation means changing a secret without breaking the system.
Rotation can be:
- manual
- scheduled
- event-driven
- incident-driven
- automatic by provider
Rotation strategy depends on credential type.
Database password rotation:
1. create new password
2. allow old and new during overlap if possible
3. update secret store
4. restart/reload app
5. verify new connections use new credential
6. revoke old password
JWT signing key rotation:
1. publish new key
2. sign new tokens with new key
3. accept old and new keys during overlap
4. wait until old tokens expire
5. remove old key
Webhook secret rotation:
1. allow dual validation
2. notify provider/consumer
3. switch signing secret
4. monitor failures
5. remove old secret
Senior rule:
Rotation is not complete until old credential is revoked and monitoring confirms no user.
13. Zero-Downtime Rotation Requirements
To support zero-downtime rotation, the app needs one of these:
- reloadable credential provider
- dual credential acceptance
- rolling restart with overlapping credentials
- connection pool refresh
- token/key set refresh
- retry-safe downstream behavior
Database rotation failure mode:
Secret updated
-> app still has old DataSource connections
-> old password revoked
-> connection pool gradually fails
-> production outage
Mitigation:
- drain old connections
- set max lifetime shorter than rotation overlap
- validate pool after rotation
- use rolling restart
- monitor auth failures
14. Log Redaction
Secret leak often happens through logs.
Redaction must cover:
- application logs
- access logs
- audit logs
- security logs
- exception logs
- HTTP client logs
- SQL logs
- config dump
- startup logs
- test logs
- CI/CD logs
Common sensitive field names:
password
passwd
secret
token
access_token
refresh_token
api_key
apikey
authorization
cookie
set-cookie
client_secret
private_key
credential
signature
But field-name redaction is not enough.
Problem:
{"header":"Authorization","value":"Bearer eyJ..."}
Better redaction model:
- redact by known field name
- redact by known header name
- redact by value pattern
- redact by type wrapper
- redact before serialization
- test redaction with negative cases
15. HTTP Header Redaction
High-risk headers:
Authorization
Cookie
Set-Cookie
Proxy-Authorization
X-API-Key
X-Signature
X-Auth-Token
JAX-RS request logging filter must not dump all headers blindly.
Bad:
headers.forEach((name, values) -> log.info("{}={}", name, values));
Better:
private static final Set<String> SENSITIVE_HEADERS = Set.of(
"authorization",
"cookie",
"set-cookie",
"proxy-authorization",
"x-api-key",
"x-auth-token",
"x-signature"
);
String safeHeaderValue(String name, List<String> values) {
if (SENSITIVE_HEADERS.contains(name.toLowerCase(Locale.ROOT))) {
return "****";
}
return String.join(",", values);
}
Also consider:
- query parameters may contain tokens
- path segments may contain customer IDs
- multipart filenames may contain sensitive data
- error body may echo invalid input
16. Avoid Secret in Exception Messages
Bad:
throw new IllegalStateException("Cannot connect with password " + password);
Still bad:
throw new ConfigException("Invalid secret: " + secretValue);
Better:
throw new ConfigException("Invalid required secret: database/main");
Log useful metadata:
- secret name/reference
- version/key ID
- provider name
- environment
- operation
- failure category
Never log:
- raw value
- token
- private key
- password
- full Authorization header
17. Secure Config Debugging
Engineers often want debug endpoint for config.
This is dangerous but sometimes useful.
Safe config view should expose:
- key exists: true/false
- source: env/vault/k8s/cloud
- version: if safe
- last refreshed time
- validation status
- hash/fingerprint only if safe and intended
It should not expose:
- raw secret value
- full env dump
- full Java system properties
- full mounted file content
- private key material
- bearer token
Example safe output:
{
"database.password": {
"present": true,
"source": "external-secret",
"lastRefresh": "2026-07-10T03:00:00Z",
"redacted": true
}
}
18. Secret and Dependency Injection
Secrets should usually enter the app at infrastructure boundary.
Common injection targets:
- DataSource factory
- Kafka producer/consumer factory
- HTTP client factory
- OAuth client
- signing service
- encryption service
- cloud SDK client builder
Avoid injecting raw secrets broadly.
Bad:
public class OrderService {
private final String dbPassword;
private final String kafkaPassword;
private final String apiSecret;
}
Better:
public class OrderService {
private final OrderRepository repository;
private final PricingClient pricingClient;
private final EventPublisher eventPublisher;
}
The secret is consumed inside the infrastructure component that needs it.
19. Secret Scope and Least Privilege
One service should not have broad secrets “just in case”.
Scope by:
- service
- environment
- tenant/customer
- permission
- operation
- data classification
- time
Bad:
one shared database admin credential for all services
Better:
service-specific database user with minimum privileges
Even better:
separate read/write credential if operationally justified
For tenant-specific integrations:
tenant A credential must not be usable for tenant B integration
20. Secrets in Local Development
Local development needs convenience, but production safety must not be weakened.
Good local pattern:
- local fake credentials
- local-only profile
- `.env.example` without real values
- `.env.local` ignored by Git
- Docker Compose secrets for local infra
- seed scripts without real customer secrets
Bad local pattern:
- production-like credentials shared in Slack
- real customer integration credentials in sample files
- copying prod secret into local file
- committing local secret accidentally
Rule:
Local convenience must never normalize unsafe production behavior.
21. Secrets in Tests
Tests should not need real secrets.
Use:
- fake secret provider
- ephemeral generated values
- test containers with local credentials
- mock cloud secret manager
- contract tests with dummy token
Avoid:
- real API key in test fixture
- real private key committed to repo
- real customer credential in integration test
- test snapshot containing token
Test redaction explicitly:
@Test
void shouldRedactAuthorizationHeader() {
var output = redactor.redactHeader("Authorization", "Bearer abc.def.ghi");
assertThat(output).isEqualTo("****");
}
22. Secrets in CI/CD
CI/CD secret risks:
- echoing env vars
- shell tracing with `set -x`
- dumping Maven settings
- publishing test reports with secrets
- storing secrets in build artifacts
- using broad deploy credential
- leaking secret in failed command
CI/CD checklist:
- secret masking enabled
- no debug echo for secret commands
- least privilege deploy credential
- separate build and deploy identity
- no secret in artifact
- no secret in Docker layer
- no secret in generated docs
Docker layer risk:
ARG DB_PASSWORD
RUN echo "$DB_PASSWORD" > /app/secret.txt
Even if removed later, secret may remain in image history/layer.
23. Secret Rotation and Connection Pools
Connection pools complicate rotation.
Typical database pool lifecycle:
startup
-> create pool
-> open connections lazily/eagerly
-> keep connections alive
-> validate/recycle connections
If password rotates while old connections stay open:
existing connections may work
new connections may fail
or old connections may be killed
Pool settings that matter:
- max lifetime
- idle timeout
- validation timeout
- minimum idle
- connection test query
- leak detection
Senior review question:
Can this pool survive credential rotation without full outage?
24. Secret Rotation and HTTP Clients
For outbound HTTP integrations:
- API key may rotate
- OAuth client secret may rotate
- signing key may rotate
- mTLS certificate may rotate
If client caches headers or token forever, rotation fails.
Better design:
HTTP Client
-> Auth Interceptor
-> Token Provider / Signing Provider
-> Refreshable Secret Source
Do not bake token at client construction unless lifecycle is clear.
Bad:
Client client = ClientBuilder.newClient()
.register(new ApiKeyFilter(secretAtStartup));
Potentially better:
Client client = ClientBuilder.newClient()
.register(new ApiKeyFilter(refreshableSecretProvider));
25. Secret Rotation and Kafka
Kafka credentials may be configured in producer/consumer properties.
Potential issue:
Kafka client initialized at startup
-> SASL credential rotates
-> existing connection fails later
-> client cannot re-authenticate
Senior checklist:
- credential rotation support by auth mechanism
- client restart requirement
- rolling restart procedure
- consumer group rebalance impact
- producer error handling
- auth failure alert
Do not treat Kafka credential rotation as purely configuration change. It can affect consumer group stability and event processing.
26. Certificate and Private Key Handling
Private key material is a high-value secret.
Rules:
- avoid storing private key in source
- avoid printing certificate chain with private key
- restrict file permission
- prefer managed certificate where possible
- rotate before expiry
- monitor expiry
- support truststore/keystore reload if required
mTLS certificate failure modes:
- expired certificate
- wrong SAN
- wrong truststore
- wrong intermediate CA
- old cert not revoked
- key/cert mismatch
- certificate mounted but app did not reload
27. Secret Observability Without Secret Exposure
You need operational visibility without leakage.
Useful metrics:
- secret load success/failure
- secret refresh success/failure
- secret age
- credential auth failure count
- secret manager latency
- secret manager error rate
- certificate days to expiry
Useful logs:
secret database/main loaded from external-secret version v12
secret refresh failed for pricing/client-secret: provider unavailable
certificate order-service-mtls expires in 14 days
Do not log:
secret value
full token
private key
password
raw Authorization header
28. Failure Modes
Common failure modes:
missing secret
wrong secret
expired secret
revoked secret
stale secret cache
secret manager unavailable
permission denied to secret
wrong environment secret
wrong tenant secret
secret leaked to log
secret leaked to artifact
secret not rotated after incident
rotation broke connection pool
rotation broke token validation
Root-cause patterns:
- ambiguous configuration precedence
- no secret ownership
- no rotation runbook
- no validation at startup
- no redaction test
- no environment separation
- overly broad credential
29. Debugging Playbook
When a service fails due to secret/config issue:
1. Identify failing dependency.
2. Identify credential/reference used.
3. Verify secret source and version.
4. Verify service identity permission to read secret.
5. Verify injected runtime value is present without printing it.
6. Verify downstream accepts credential.
7. Check recent rotation/change/deployment.
8. Check old/new credential overlap.
9. Check connection pool/client cache behavior.
10. Check logs for redacted failure category.
Do not “debug” by printing the secret.
Better:
- compare safe hash if approved
- verify key ID
- verify version ID
- verify certificate fingerprint
- use provider audit log
30. PR Review Checklist
When reviewing secret/config changes, check:
[ ] Is this value a secret or sensitive config?
[ ] Is it absent from source code and image layers?
[ ] Is it absent from logs and exception messages?
[ ] Is access least-privilege?
[ ] Is environment separation clear?
[ ] Is tenant separation clear if tenant-specific?
[ ] Is startup validation explicit?
[ ] Is missing/wrong secret behavior fail-closed?
[ ] Is rotation procedure known?
[ ] Does connection pool/client support rotation?
[ ] Are tests using fake/ephemeral secrets?
[ ] Are redaction tests present?
[ ] Is CI/CD masking safe?
[ ] Is ownership documented?
31. Internal Verification Checklist
Untuk codebase/internal platform CSG, verifikasi:
[ ] Secret source utama: env var, Kubernetes Secret, external secret, cloud secret manager, Vault, atau lainnya.
[ ] Ada/tidak standard internal untuk naming secret.
[ ] Ada/tidak standard internal untuk secret rotation.
[ ] Ada/tidak standard internal untuk log redaction.
[ ] Ada/tidak wrapper untuk sensitive values.
[ ] Ada/tidak config debug endpoint dan apakah aman.
[ ] Bagaimana database credential di-inject ke DataSource.
[ ] Bagaimana Kafka credential di-inject ke producer/consumer.
[ ] Bagaimana HTTP downstream credential di-inject ke client.
[ ] Apakah secret scoped per environment.
[ ] Apakah secret scoped per tenant/customer jika relevan.
[ ] Apakah mTLS/private key digunakan.
[ ] Apakah certificate expiry dimonitor.
[ ] Apakah CI/CD menyimpan secret dan masking aktif.
[ ] Apakah Docker image pernah menerima secret saat build.
[ ] Apakah ada secret scanner di repo/pipeline.
[ ] Apakah ada runbook rotasi.
[ ] Apakah ada incident playbook untuk leaked secret.
32. Senior Mental Model
Secret management bukan tugas “DevOps saja”.
Di service enterprise, secret mempengaruhi:
- application startup
- dependency access
- tenant isolation
- integration correctness
- incident blast radius
- audit/compliance
- deployment safety
- debugging safety
Senior engineer harus bisa menjawab:
What secrets does this service need?
Where do they come from?
Who can read them?
How are they rotated?
What breaks if they are wrong?
How do we debug without leaking them?
Final rule:
A production service should be observable enough to diagnose secret failure, but disciplined enough to never reveal the secret itself.
You just completed lesson 41 in build core. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.