Configuration, Secrets, and Runtime Contract
Learn Kubernetes with Cloud Services AWS & Azure - Part 006
Configuration, Secrets, and runtime contract engineering for Kubernetes workloads across AWS EKS and Azure AKS: ConfigMap, Secret, immutable config, reload strategy, secret delivery, external secret stores, and production failure modes.
Configuration, Secrets, and Runtime Contract
Configuration is where application design meets platform reality.
A container image should be immutable, but a production system must still vary by environment, tenant, region, feature flag, credential, endpoint, policy, limit, timeout, and operational mode.
That tension creates one of the most important Kubernetes design boundaries:
Build once. Configure at runtime. Keep secrets confidential. Make changes observable, reversible, and safe.
This part is about that boundary.
We will cover ConfigMap, Secret, environment variables, mounted configuration files, immutable configuration, reload strategy, cloud secret integration, failure modes, and review checklists for EKS and AKS.
1. The Runtime Contract Mental Model
A container image should answer:
What software is this?
Runtime configuration should answer:
How should this software behave in this environment right now?
Secrets should answer:
Which confidential material does this workload need to access protected systems?
The platform should make runtime behavior explicit without baking environment-specific state into images.
Bad pattern:
Build one image for dev, one for staging, one for production.
Each image contains different config files and credentials.
Good pattern:
Build one immutable image.
Inject environment-specific configuration and secrets at deployment/runtime.
Use cloud identity for access to cloud resources.
Audit and rotate sensitive material outside the image.
2. Configuration Taxonomy
Not all configuration is the same.
| Type | Example | Storage | Change Frequency | Risk |
|---|---|---|---|---|
| Static app setting | server port, log format | image/default config | rare | low |
| Environment setting | database host, API endpoint | ConfigMap | occasional | medium |
| Operational tuning | timeout, pool size, rate limit | ConfigMap / dynamic config | frequent | medium/high |
| Feature flag | enable new checkout flow | feature flag service | frequent | high if unsafe |
| Secret | password, token, private key | Secret/external secret store | rotated | high |
| Identity binding | workload-to-cloud permission | IAM / managed identity | rare but critical | very high |
| Policy | allowed region, compliance mode | policy engine / config | occasional | high |
Kubernetes ConfigMap and Secret are not a complete configuration platform. They are primitives.
A mature platform often combines:
- image defaults;
- Kubernetes ConfigMaps;
- Kubernetes Secrets;
- external secret managers;
- workload identity;
- feature flag service;
- policy-as-code;
- GitOps-controlled environment overlays;
- runtime dynamic config for high-risk toggles.
3. ConfigMap Pattern
A ConfigMap stores non-confidential configuration data.
Use it for:
- environment-specific endpoints;
- non-secret feature switches;
- logging level defaults;
- application config files;
- tuning values;
- static allowlists/blocklists if not confidential;
- operational mode flags.
Do not use it for:
- passwords;
- access tokens;
- private keys;
- database credentials;
- OAuth client secrets;
- signing secrets;
- anything confidential.
3.1 ConfigMap as Environment Variables
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-config
labels:
app.kubernetes.io/name: order-api
data:
LOG_LEVEL: "INFO"
ORDER_TIMEOUT_MS: "3000"
PAYMENT_ENDPOINT: "https://payment.internal.example.com"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: order-api
template:
metadata:
labels:
app.kubernetes.io/name: order-api
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
envFrom:
- configMapRef:
name: order-api-config
This is simple, but it has an important property:
Environment variables are captured when the container starts.
If the ConfigMap changes later, already-running containers do not magically update their environment variables.
That is usually good for deterministic rollouts.
3.2 ConfigMap as Explicit Environment Variables
Prefer explicit mapping for important values.
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
template:
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: order-api-config
key: LOG_LEVEL
- name: ORDER_TIMEOUT_MS
valueFrom:
configMapKeyRef:
name: order-api-config
key: ORDER_TIMEOUT_MS
Explicit mapping is more verbose but safer for review.
It avoids accidentally injecting every key into the process environment.
3.3 ConfigMap as Mounted File
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-files
data:
application.yaml: |
server:
port: 8080
order:
timeoutMs: 3000
maxItems: 100
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
template:
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
volumeMounts:
- name: config
mountPath: /app/config/application.yaml
subPath: application.yaml
readOnly: true
volumes:
- name: config
configMap:
name: order-api-files
Mounted files are useful when the application already expects structured configuration files.
Be careful with subPath: updates to the ConfigMap may not propagate to files mounted with subPath in the same way as directory mounts. In production, prefer explicit rollout after config changes rather than relying on implicit file refresh.
4. Config Change Strategy
The hardest question is not “how do I mount config?”
It is:
What should happen when config changes?
There are three broad strategies.
4.1 Restart-on-Change
Config changes create a new Pod template or trigger a rollout.
Pattern:
Typical implementation:
spec:
template:
metadata:
annotations:
checksum/config: "sha256-of-rendered-config"
When the checksum changes, the Pod template changes, causing a rollout.
This is a strong default for most production systems.
Pros:
- deterministic;
- auditable;
- rollback-friendly;
- aligns app lifecycle with config lifecycle;
- avoids hidden runtime mutation.
Cons:
- slower than live reload;
- may restart many Pods;
- requires safe rollout strategy.
4.2 Live Reload
The application watches mounted files or receives a signal and reloads config without restarting.
Pros:
- useful for low-risk dynamic tuning;
- avoids full restart;
- can support fast operational response.
Cons:
- harder to test;
- app may enter mixed internal state;
- not every setting is safe to reload;
- observability must show active config version;
- rollback semantics are weaker.
Only use live reload when the application explicitly supports it and the config keys are classified by reload safety.
Example classification:
| Setting | Live Reload Safe? | Why |
|---|---|---|
| log level | yes | low semantic risk |
| request timeout | maybe | existing requests may differ from new requests |
| database host | usually no | connection pool and transaction semantics |
| encryption key | no | cryptographic boundary |
| feature flag | depends | should use flag platform with guardrails |
4.3 Dynamic Configuration Service
A separate configuration or feature flag platform controls dynamic runtime behavior.
Use it when:
- changes are frequent;
- rollout must be targeted;
- business toggles need audit;
- kill switches are required;
- experiments are used;
- config needs gradual exposure.
Kubernetes ConfigMap is not a replacement for a full feature flag control plane.
5. Immutable ConfigMaps
Kubernetes supports marking ConfigMaps and Secrets as immutable.
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-config-v20260703
immutable: true
data:
ORDER_TIMEOUT_MS: "3000"
LOG_LEVEL: "INFO"
Immutable config has a powerful operational property:
Once created, the object cannot be changed; a new version must be created instead.
This reduces accidental mutation and improves auditability.
Pattern:
order-api-config-v20260703-001
order-api-config-v20260703-002
order-api-config-v20260704-001
Then Deployment references the intended version.
Pros:
- safer GitOps semantics;
- easier rollback;
- avoids accidental in-place mutation;
- makes config version explicit.
Cons:
- requires cleanup policy;
- object names change;
- templates must manage version references.
For regulated or high-risk environments, immutable config is often superior to in-place config mutation.
6. Kubernetes Secret Pattern
A Kubernetes Secret stores confidential data for Pods to consume.
But the name “Secret” can mislead engineers.
By default, Secret values are base64-encoded, not magically secure. Cluster configuration determines encryption at rest. RBAC determines who can read them. Workload design determines how broadly they are exposed.
Use Kubernetes Secrets for:
- small confidential values needed by Pods;
- TLS certificates when managed carefully;
- credentials synced from external secret stores;
- short-lived tokens when supported by rotation workflow;
- bootstrap credentials when unavoidable.
Avoid long-lived static credentials where workload identity can be used instead.
6.1 Secret as Environment Variables
apiVersion: v1
kind: Secret
metadata:
name: order-api-secret
type: Opaque
stringData:
DATABASE_PASSWORD: "replace-me-through-secret-pipeline"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
template:
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
env:
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: order-api-secret
key: DATABASE_PASSWORD
This is common but has trade-offs.
Environment variable secrets can leak through:
- process dumps;
- debug endpoints;
- accidental logging;
- child process environments;
- misconfigured diagnostics.
For highly sensitive material, mounted files or direct external secret retrieval with workload identity may be safer.
6.2 Secret as Mounted File
apiVersion: v1
kind: Secret
metadata:
name: order-api-tls
type: kubernetes.io/tls
stringData:
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
template:
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
volumeMounts:
- name: tls
mountPath: /app/tls
readOnly: true
volumes:
- name: tls
secret:
secretName: order-api-tls
Mounted secrets can be better for:
- certificate files;
- private keys;
- credentials consumed by libraries expecting file paths;
- rotation through mounted volume update, if application supports reload.
Still, mounted secrets are visible inside the container filesystem. Container compromise means secret compromise.
7. Secret Delivery Architecture
There are four common models.
7.1 Model A: Native Kubernetes Secret Only
Secrets are created directly as Kubernetes Secret objects.
Use when:
- simple cluster;
- low rotation complexity;
- secrets are generated by deployment pipeline;
- encryption at rest and RBAC are correctly configured.
Risks:
- secret sprawl;
- weak rotation;
- broad RBAC exposure;
- accidental Git leakage if not handled properly.
7.2 Model B: External Secret Sync
A controller syncs cloud secret values into Kubernetes Secrets.
Examples:
- AWS Secrets Manager → Kubernetes Secret;
- Azure Key Vault → Kubernetes Secret;
- HashiCorp Vault → Kubernetes Secret.
Pros:
- Kubernetes-native consumption;
- apps do not need cloud secret SDK;
- rotation can be centralized.
Cons:
- synced secret still exists in Kubernetes;
- controller permissions are sensitive;
- sync delay and failure modes matter;
- must audit both external store and Kubernetes access.
7.3 Model C: Secret Store CSI Mount
Secrets are mounted into Pods as volumes from external providers.
Pros:
- avoids storing long-lived secret value as a normal Kubernetes Secret when configured that way;
- good for file-based secrets;
- integrates well with cloud identity.
Cons:
- mount failures block Pod startup;
- application reload behavior must be defined;
- CSI/provider availability becomes part of startup path;
- operational debugging is more complex.
7.4 Model D: Direct App Retrieval
The application calls cloud secret manager directly using workload identity.
Pros:
- no secret object required in Kubernetes;
- application can implement caching/refresh logic;
- cloud audit trail is direct.
Cons:
- app now depends on cloud SDK and secret manager availability;
- local development/testing becomes more complex;
- every app may reinvent caching/retry/security logic;
- migration across clouds is harder.
No model is universally best. The correct answer depends on sensitivity, rotation needs, app maturity, platform ownership, and audit requirements.
8. EKS Secret and Identity Patterns
On AWS EKS, prefer identity-based access over static cloud credentials.
Common tools and patterns:
- IAM Roles for Service Accounts (IRSA);
- EKS Pod Identity;
- AWS Secrets Manager;
- AWS Systems Manager Parameter Store;
- Secrets Store CSI Driver with AWS provider;
- External Secrets Operator with AWS provider;
- envelope encryption for Kubernetes Secrets using AWS KMS, where configured.
8.1 EKS Pattern: Pod Identity + Secrets Manager
Design rule:
Do not put AWS access keys in Kubernetes Secrets for normal workloads.
Use workload identity.
8.2 EKS IAM Boundary Questions
Ask:
- which service account can access which AWS secret?
- is access namespace-scoped?
- can a compromised Pod read unrelated secrets?
- are IAM policies resource-scoped, not wildcarded?
- is CloudTrail/audit enabled for secret reads?
- how is rotation handled?
- what happens if Secrets Manager is temporarily unavailable?
9. AKS Secret and Identity Patterns
On Azure AKS, prefer managed identity / workload identity over static credentials.
Common tools and patterns:
- Microsoft Entra Workload ID;
- managed identities;
- Azure Key Vault;
- Secrets Store CSI Driver with Azure Key Vault provider;
- External Secrets Operator with Azure provider;
- Kubernetes Secret encryption at rest through platform-managed configuration;
- Azure RBAC / Kubernetes RBAC integration.
9.1 AKS Pattern: Workload Identity + Key Vault
Design rule:
Do not store Azure client secrets in Kubernetes Secrets for normal workloads.
Use workload identity or managed identity.
9.2 AKS Key Vault Boundary Questions
Ask:
- is the workload identity bound only to the intended service account?
- does Key Vault policy/RBAC grant least privilege?
- are secrets, keys, and certificates separated by access need?
- is secret version pinning required?
- how are rotations propagated?
- what alerts exist for failed secret access?
10. Configuration Layering
Production systems need layering, but layering can become chaos.
A clean model:
Example:
| Layer | Owner | Example |
|---|---|---|
| Image defaults | app team | default port, safe internal defaults |
| Base manifest | app/platform | probes, resources, labels |
| Environment overlay | platform/app | dev/stage/prod endpoint |
| Cluster overlay | platform | storage class, ingress class, node pool selector |
| Runtime config | app/platform | timeout, feature toggle default |
| Secret store | security/platform | credentials, certificates |
Rules:
- Lower layers should be safe defaults.
- Higher layers should override intentionally.
- Secrets should not appear in non-secret layers.
- The final rendered config should be reviewable.
- A running Pod should expose config version, not raw secret values.
11. Environment Variable vs File Mount
| Criterion | Env Var | Mounted File |
|---|---|---|
| Simple scalar values | excellent | acceptable |
| Structured config | weak | excellent |
| Secret leakage risk | higher | lower but still present |
| App reload possibility | requires restart | possible if app watches file |
| Compatibility with libraries | common | common for certs/config files |
| Review clarity | explicit env is clear | file content can be clearer |
| Dynamic updates | not for existing process | possible for directory mounts |
Default guidance:
- use env vars for small non-secret scalar config;
- use mounted files for structured config;
- use mounted files or external retrieval for sensitive files/certificates;
- prefer restart-on-change unless live reload is explicitly designed.
12. Missing Config and Startup Failure
Configuration failure should be loud.
Bad app behavior:
Missing PAYMENT_ENDPOINT.
Defaulting to http://localhost:8080.
Application starts successfully.
Production traffic fails later.
Good app behavior:
Missing required PAYMENT_ENDPOINT.
Startup validation failed.
Exit non-zero.
Pod enters CrashLoopBackOff.
Rollout stops before receiving traffic.
At first, CrashLoopBackOff feels bad. But for invalid config, failing early is correct.
A production app should validate:
- required config exists;
- values have valid format;
- numeric values are within bounds;
- URLs use expected scheme;
- mutually exclusive settings are not both enabled;
- secrets are readable;
- dangerous defaults are forbidden in production;
- config version is logged without exposing secrets.
13. Config Compatibility During Rollout
Config changes are deployment changes.
Treat them with the same rigor as code.
13.1 Safe Config Change
A safe config change is:
- backward-compatible with currently running app version;
- forward-compatible with next app version if both coexist;
- reversible;
- observable;
- tested in staging or canary;
- bounded in blast radius.
13.2 Unsafe Config Change
Unsafe examples:
- change database endpoint and deploy all Pods at once;
- reduce timeout below realistic latency;
- enable feature requiring schema not yet deployed;
- rotate credential without overlapping validity;
- change Kafka topic name while old Pods still publish to old topic;
- change encryption key without keyring/migration plan.
13.3 Config and App Version Matrix
Use a compatibility matrix for high-risk changes.
| App Version | Config v1 | Config v2 | Notes |
|---|---|---|---|
1.8 | yes | no | old app cannot parse new setting |
1.9 | yes | yes | bridge version |
2.0 | no | yes | final version |
Rollout sequence:
- deploy bridge app version
1.9compatible with both configs; - switch config from v1 to v2;
- deploy final app version
2.0; - remove old config after safety window.
This is the same principle as safe database migrations.
14. Secret Rotation Strategy
Secret rotation must be designed before the incident.
A rotation plan needs:
- source of truth;
- propagation path;
- compatibility window;
- application reload/restart behavior;
- verification step;
- rollback or emergency revoke;
- audit trail;
- owner and runbook.
14.1 Rotation Timeline
The critical detail is overlap.
If dependency revokes old credential before all Pods use the new credential, you cause outage.
If old credential remains valid forever, rotation did not reduce risk.
14.2 Rotation Patterns
| Pattern | Use When | Risk |
|---|---|---|
| restart all Pods after secret update | app reads secret at startup | rollout must be safe |
| live reload mounted secret | app supports reload | mixed state and reload bugs |
| dual credential window | dependency supports two credentials | operational sequencing |
| short-lived tokens | identity provider supports refresh | clock/token/cache issues |
| direct cloud identity | cloud service supports IAM/managed identity | permission boundary complexity |
Best long-term direction:
replace static secrets with workload identity wherever possible.
15. RBAC and Secret Blast Radius
RBAC for Secrets must be restrictive.
Important rule:
Permission to read Secrets in a namespace is often equivalent to permission to impersonate many workloads in that namespace.
Why?
Secrets may contain:
- database credentials;
- service tokens;
- TLS keys;
- image pull credentials;
- cloud credentials if badly designed;
- application signing keys.
Avoid broad roles like:
resources: ["secrets"]
verbs: ["get", "list", "watch"]
Especially avoid list/watch unless necessary. Reading all secrets is much more dangerous than reading one named secret.
Prefer:
- least privilege;
- namespace isolation;
- named resource access when possible;
- separate service accounts per workload;
- external secret store access scoped per workload;
- audit logs for secret reads;
- admission policy preventing risky secret patterns.
16. GitOps and Secret Handling
Never commit raw secrets to Git.
GitOps options include:
- references to external secret names only;
- sealed/encrypted secret manifests;
- SOPS-encrypted files;
- External Secrets Operator resources;
- CSI SecretProviderClass resources;
- pipeline-created Secrets;
- cloud-native secret stores as source of truth.
16.1 GitOps-Friendly External Secret
Example shape, intentionally generic:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: order-api-secret
spec:
refreshInterval: 1h
secretStoreRef:
name: platform-secret-store
kind: ClusterSecretStore
target:
name: order-api-secret
creationPolicy: Owner
data:
- secretKey: DATABASE_PASSWORD
remoteRef:
key: prod/order-api/database
property: password
This manifest does not contain the secret value. It contains the reference and sync policy.
Review questions:
- who can change the remote secret reference?
- who can read the resulting Kubernetes Secret?
- who can read the external secret store value?
- how is rotation tested?
- what happens when sync fails?
17. Naming and Versioning
Config and secret names should communicate ownership and lifecycle.
Bad names:
config
secret
app-secret
prod-config
Better names:
order-api-runtime-config
order-api-database-credentials
ledger-reconciler-schedule-config
payment-gateway-client-certificate
For immutable/versioned config:
order-api-runtime-config-v20260703-001
order-api-runtime-config-v20260710-001
For Secrets, avoid putting secret values or sensitive meaning in names. The name itself is metadata and may appear in logs/events.
18. Observability for Configuration
You should be able to answer:
- which config version is a Pod running?
- which secret version is it using, if safe to expose as version metadata?
- when did config last change?
- who changed it?
- did the rollout happen?
- did the app reload successfully?
- are some Pods on old config and some on new config?
- did secret sync fail?
Expose non-sensitive metadata:
config_version="order-api-runtime-config-v20260703-001"
secret_version="db-password-version-42" # only if not sensitive
feature_profile="prod-stable"
Never expose raw secret values.
18.1 Useful Events and Signals
Track:
- Pod startup validation failure;
- ConfigMap/Secret missing key;
- failed secret mount;
- external secret sync error;
- CSI mount error;
- application config reload success/failure;
- rollout stuck after config change;
- authentication failures after rotation.
19. Failure Mode Catalog
| Failure | Symptom | Cause | Prevention |
|---|---|---|---|
| Missing ConfigMap | Pod stuck or app crash | wrong name/namespace | CI validation, admission checks |
| Missing key | container creation error or app startup failure | manifest/config mismatch | schema validation, startup validation |
| Bad value format | app crash or degraded behavior | weak config validation | typed config validation |
| In-place config mutation | unpredictable Pod behavior | manual edit | immutable config/GitOps |
| Secret expired | auth failures | no rotation monitoring | expiry alerts and rotation runbook |
| Secret not propagated | old value still used | app reads only at startup | restart-on-change or reload strategy |
| RBAC too broad | lateral movement risk | shared secret access | least privilege per workload |
| Secret committed to Git | credential leak | weak process | scanning, encrypted secrets, external stores |
| ConfigMap used for secret | sensitive data exposed | misunderstanding | policy and review |
| Cloud secret unavailable | Pod startup failure | external dependency outage | caching, retry, readiness, fallback strategy |
| Wrong environment config | prod points to staging or vice versa | overlay mistake | environment validation and naming |
20. Configuration Schema and Validation
Kubernetes will not validate your application-specific config semantics.
It can validate object shape, not business correctness.
Your platform should add validation at multiple layers:
Validation examples:
- YAML schema for config files;
- JSON Schema for structured config;
- Helm values schema;
- Kustomize build validation;
- policy-as-code for forbidden secret patterns;
- admission control requiring config checksum annotation;
- application startup validation with clear failure messages.
A mature app treats config as typed input, not as random strings.
21. Production Patterns
21.1 Restart-on-Config-Change Deployment
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-runtime-config-v20260703-001
immutable: true
data:
ORDER_TIMEOUT_MS: "3000"
PAYMENT_ENDPOINT: "https://payment.internal.example.com"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: order-api
template:
metadata:
labels:
app.kubernetes.io/name: order-api
annotations:
platform.example.com/config-version: "order-api-runtime-config-v20260703-001"
checksum/config: "sha256:example"
spec:
containers:
- name: app
image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
envFrom:
- configMapRef:
name: order-api-runtime-config-v20260703-001
21.2 Secret Mount with Explicit File Paths
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-client
spec:
template:
spec:
containers:
- name: app
image: registry.example.com/platform/payment-client:2026.07.03-1a2b3c4
env:
- name: PAYMENT_CERT_PATH
value: /app/secrets/tls.crt
- name: PAYMENT_KEY_PATH
value: /app/secrets/tls.key
volumeMounts:
- name: payment-client-cert
mountPath: /app/secrets
readOnly: true
volumes:
- name: payment-client-cert
secret:
secretName: payment-client-certificate
defaultMode: 0400
21.3 Workload Identity Instead of Static Cloud Secret
Bad:
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-static-credentials
key: accessKeyId
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-static-credentials
key: secretAccessKey
Better:
serviceAccountName: order-api
Then bind the service account to the appropriate AWS IAM role or Azure federated identity through the cloud-specific workload identity mechanism.
The manifest becomes simpler because identity is no longer a secret string inside the Pod.
22. Platform Guardrails
A production platform should enforce common rules.
Potential policies:
- ConfigMaps cannot contain keys matching
PASSWORD,TOKEN,SECRET,PRIVATE_KEY; - Secrets must not be mounted into all containers by default;
- Pods must not reference missing ConfigMaps/Secrets;
- production ConfigMaps must be immutable or checksum-triggered;
- Secrets must be sourced from approved external stores;
- service accounts must be workload-specific;
- default service account token automount should be disabled unless required;
- containers must not log environment variables;
- secret volumes must be read-only;
- dangerous debug endpoints disabled in production.
Guardrails prevent repeated human review mistakes.
23. Review Checklist
23.1 ConfigMap Review
- Is the data non-confidential?
- Are keys explicitly named and documented?
- Are values typed/validated by the app?
- Is there a config version?
- Is the change rollout-triggered?
- Is live reload intentionally supported or avoided?
- Is rollback clear?
- Is the config environment-specific but not image-specific?
23.2 Secret Review
- Why is this a Secret instead of workload identity?
- Where is the source of truth?
- Is it encrypted at rest?
- Who can read it in Kubernetes?
- Who can read it in the external store?
- How is it rotated?
- What happens to existing Pods during rotation?
- Is the secret exposed as env var, file, CSI mount, or direct retrieval?
- Are secret values excluded from logs, metrics, traces, and crash dumps?
23.3 Runtime Contract Review
- Does the app fail fast on missing/invalid config?
- Does the app expose config version safely?
- Does readiness reflect config-dependent startup?
- Does rollback include config rollback?
- Are old and new config versions compatible during rollout?
- Are cloud permissions least-privilege?
24. Exercises
Exercise 1: Classify Configuration
Classify these values as image default, ConfigMap, Secret, workload identity, feature flag, or policy:
- HTTP server port;
- database password;
- database hostname;
- AWS S3 bucket name;
- permission to read the S3 bucket;
- percentage rollout for new checkout flow;
- TLS private key;
- request timeout;
- production compliance mode;
- Azure Key Vault URI.
Explain why.
Exercise 2: Design a Rotation Plan
Design a rotation plan for a database password used by order-api.
Include:
- source of truth;
- old/new overlap window;
- Kubernetes propagation;
- app restart/reload;
- validation;
- rollback;
- alerts;
- audit evidence.
Exercise 3: Config Compatibility Matrix
Create a matrix for an app version change where:
- old app reads
PAYMENT_URL; - new app reads
PAYMENT_ENDPOINT; - both must run during rolling update.
Design the bridge release.
Exercise 4: Secret Blast Radius
Given a namespace with five apps sharing one Secret, redesign it.
Expected direction:
- separate secrets per workload;
- separate service accounts;
- least privilege external store policies;
- remove unused keys;
- audit access;
- rotate after separation.
25. Part Summary
Configuration and secrets are not deployment afterthoughts. They are part of the workload runtime contract.
Use this mental model:
- image = immutable software artifact;
- ConfigMap = non-confidential runtime configuration;
- Secret = confidential runtime material, but not a complete secret management system;
- workload identity = preferred way to access cloud services;
- external secret store = source of truth for sensitive material;
- rollout/reload strategy = what makes config changes safe;
- validation/observability = what makes config failures detectable.
The key production question is not:
Where do I put this value?
It is:
What is the lifecycle, sensitivity, owner, reload behavior, blast radius, and rollback model of this value?
Once you answer that, the Kubernetes objects become implementation details.
References
- Kubernetes Documentation: Configuration
- Kubernetes Documentation: ConfigMaps
- Kubernetes Documentation: Secrets
- Kubernetes Documentation: Good practices for Kubernetes Secrets
- Kubernetes Documentation: Managing Secrets
- Kubernetes Documentation: Resource Management for Pods and Containers
- AWS Documentation: AWS Secrets and Configuration Provider for the Kubernetes Secrets Store CSI Driver
- AWS Documentation: EKS Pod Identity
- Azure Documentation: Use the Azure Key Vault provider for Secrets Store CSI Driver in AKS
- Azure Documentation: Microsoft Entra Workload ID with AKS
You just completed lesson 06 in start here. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.