Start HereOrdered learning track

Configuration, Secrets, and Runtime Contract

Learn Kubernetes with Cloud Services AWS & Azure - Part 006

Configuration, Secrets, and runtime contract engineering for Kubernetes workloads across AWS EKS and Azure AKS: ConfigMap, Secret, immutable config, reload strategy, secret delivery, external secret stores, and production failure modes.

21 min read4029 words
PrevNext
Lesson 0640 lesson track0108 Start Here
#kubernetes#configmap#secret#configuration+6 more

Configuration, Secrets, and Runtime Contract

Configuration is where application design meets platform reality.

A container image should be immutable, but a production system must still vary by environment, tenant, region, feature flag, credential, endpoint, policy, limit, timeout, and operational mode.

That tension creates one of the most important Kubernetes design boundaries:

Build once. Configure at runtime. Keep secrets confidential. Make changes observable, reversible, and safe.

This part is about that boundary.

We will cover ConfigMap, Secret, environment variables, mounted configuration files, immutable configuration, reload strategy, cloud secret integration, failure modes, and review checklists for EKS and AKS.


1. The Runtime Contract Mental Model

A container image should answer:

What software is this?

Runtime configuration should answer:

How should this software behave in this environment right now?

Secrets should answer:

Which confidential material does this workload need to access protected systems?

The platform should make runtime behavior explicit without baking environment-specific state into images.

Bad pattern:

Build one image for dev, one for staging, one for production.
Each image contains different config files and credentials.

Good pattern:

Build one immutable image.
Inject environment-specific configuration and secrets at deployment/runtime.
Use cloud identity for access to cloud resources.
Audit and rotate sensitive material outside the image.

2. Configuration Taxonomy

Not all configuration is the same.

TypeExampleStorageChange FrequencyRisk
Static app settingserver port, log formatimage/default configrarelow
Environment settingdatabase host, API endpointConfigMapoccasionalmedium
Operational tuningtimeout, pool size, rate limitConfigMap / dynamic configfrequentmedium/high
Feature flagenable new checkout flowfeature flag servicefrequenthigh if unsafe
Secretpassword, token, private keySecret/external secret storerotatedhigh
Identity bindingworkload-to-cloud permissionIAM / managed identityrare but criticalvery high
Policyallowed region, compliance modepolicy engine / configoccasionalhigh

Kubernetes ConfigMap and Secret are not a complete configuration platform. They are primitives.

A mature platform often combines:

  • image defaults;
  • Kubernetes ConfigMaps;
  • Kubernetes Secrets;
  • external secret managers;
  • workload identity;
  • feature flag service;
  • policy-as-code;
  • GitOps-controlled environment overlays;
  • runtime dynamic config for high-risk toggles.

3. ConfigMap Pattern

A ConfigMap stores non-confidential configuration data.

Use it for:

  • environment-specific endpoints;
  • non-secret feature switches;
  • logging level defaults;
  • application config files;
  • tuning values;
  • static allowlists/blocklists if not confidential;
  • operational mode flags.

Do not use it for:

  • passwords;
  • access tokens;
  • private keys;
  • database credentials;
  • OAuth client secrets;
  • signing secrets;
  • anything confidential.

3.1 ConfigMap as Environment Variables

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-api-config
  labels:
    app.kubernetes.io/name: order-api
data:
  LOG_LEVEL: "INFO"
  ORDER_TIMEOUT_MS: "3000"
  PAYMENT_ENDPOINT: "https://payment.internal.example.com"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: order-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order-api
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          envFrom:
            - configMapRef:
                name: order-api-config

This is simple, but it has an important property:

Environment variables are captured when the container starts.

If the ConfigMap changes later, already-running containers do not magically update their environment variables.

That is usually good for deterministic rollouts.

3.2 ConfigMap as Explicit Environment Variables

Prefer explicit mapping for important values.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          env:
            - name: LOG_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: order-api-config
                  key: LOG_LEVEL
            - name: ORDER_TIMEOUT_MS
              valueFrom:
                configMapKeyRef:
                  name: order-api-config
                  key: ORDER_TIMEOUT_MS

Explicit mapping is more verbose but safer for review.

It avoids accidentally injecting every key into the process environment.

3.3 ConfigMap as Mounted File

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-api-files
data:
  application.yaml: |
    server:
      port: 8080
    order:
      timeoutMs: 3000
      maxItems: 100
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          volumeMounts:
            - name: config
              mountPath: /app/config/application.yaml
              subPath: application.yaml
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: order-api-files

Mounted files are useful when the application already expects structured configuration files.

Be careful with subPath: updates to the ConfigMap may not propagate to files mounted with subPath in the same way as directory mounts. In production, prefer explicit rollout after config changes rather than relying on implicit file refresh.


4. Config Change Strategy

The hardest question is not “how do I mount config?”

It is:

What should happen when config changes?

There are three broad strategies.

4.1 Restart-on-Change

Config changes create a new Pod template or trigger a rollout.

Pattern:

Typical implementation:

spec:
  template:
    metadata:
      annotations:
        checksum/config: "sha256-of-rendered-config"

When the checksum changes, the Pod template changes, causing a rollout.

This is a strong default for most production systems.

Pros:

  • deterministic;
  • auditable;
  • rollback-friendly;
  • aligns app lifecycle with config lifecycle;
  • avoids hidden runtime mutation.

Cons:

  • slower than live reload;
  • may restart many Pods;
  • requires safe rollout strategy.

4.2 Live Reload

The application watches mounted files or receives a signal and reloads config without restarting.

Pros:

  • useful for low-risk dynamic tuning;
  • avoids full restart;
  • can support fast operational response.

Cons:

  • harder to test;
  • app may enter mixed internal state;
  • not every setting is safe to reload;
  • observability must show active config version;
  • rollback semantics are weaker.

Only use live reload when the application explicitly supports it and the config keys are classified by reload safety.

Example classification:

SettingLive Reload Safe?Why
log levelyeslow semantic risk
request timeoutmaybeexisting requests may differ from new requests
database hostusually noconnection pool and transaction semantics
encryption keynocryptographic boundary
feature flagdependsshould use flag platform with guardrails

4.3 Dynamic Configuration Service

A separate configuration or feature flag platform controls dynamic runtime behavior.

Use it when:

  • changes are frequent;
  • rollout must be targeted;
  • business toggles need audit;
  • kill switches are required;
  • experiments are used;
  • config needs gradual exposure.

Kubernetes ConfigMap is not a replacement for a full feature flag control plane.


5. Immutable ConfigMaps

Kubernetes supports marking ConfigMaps and Secrets as immutable.

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-api-config-v20260703
immutable: true
data:
  ORDER_TIMEOUT_MS: "3000"
  LOG_LEVEL: "INFO"

Immutable config has a powerful operational property:

Once created, the object cannot be changed; a new version must be created instead.

This reduces accidental mutation and improves auditability.

Pattern:

order-api-config-v20260703-001
order-api-config-v20260703-002
order-api-config-v20260704-001

Then Deployment references the intended version.

Pros:

  • safer GitOps semantics;
  • easier rollback;
  • avoids accidental in-place mutation;
  • makes config version explicit.

Cons:

  • requires cleanup policy;
  • object names change;
  • templates must manage version references.

For regulated or high-risk environments, immutable config is often superior to in-place config mutation.


6. Kubernetes Secret Pattern

A Kubernetes Secret stores confidential data for Pods to consume.

But the name “Secret” can mislead engineers.

By default, Secret values are base64-encoded, not magically secure. Cluster configuration determines encryption at rest. RBAC determines who can read them. Workload design determines how broadly they are exposed.

Use Kubernetes Secrets for:

  • small confidential values needed by Pods;
  • TLS certificates when managed carefully;
  • credentials synced from external secret stores;
  • short-lived tokens when supported by rotation workflow;
  • bootstrap credentials when unavoidable.

Avoid long-lived static credentials where workload identity can be used instead.

6.1 Secret as Environment Variables

apiVersion: v1
kind: Secret
metadata:
  name: order-api-secret
type: Opaque
stringData:
  DATABASE_PASSWORD: "replace-me-through-secret-pipeline"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          env:
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: order-api-secret
                  key: DATABASE_PASSWORD

This is common but has trade-offs.

Environment variable secrets can leak through:

  • process dumps;
  • debug endpoints;
  • accidental logging;
  • child process environments;
  • misconfigured diagnostics.

For highly sensitive material, mounted files or direct external secret retrieval with workload identity may be safer.

6.2 Secret as Mounted File

apiVersion: v1
kind: Secret
metadata:
  name: order-api-tls
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          volumeMounts:
            - name: tls
              mountPath: /app/tls
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: order-api-tls

Mounted secrets can be better for:

  • certificate files;
  • private keys;
  • credentials consumed by libraries expecting file paths;
  • rotation through mounted volume update, if application supports reload.

Still, mounted secrets are visible inside the container filesystem. Container compromise means secret compromise.


7. Secret Delivery Architecture

There are four common models.

7.1 Model A: Native Kubernetes Secret Only

Secrets are created directly as Kubernetes Secret objects.

Use when:

  • simple cluster;
  • low rotation complexity;
  • secrets are generated by deployment pipeline;
  • encryption at rest and RBAC are correctly configured.

Risks:

  • secret sprawl;
  • weak rotation;
  • broad RBAC exposure;
  • accidental Git leakage if not handled properly.

7.2 Model B: External Secret Sync

A controller syncs cloud secret values into Kubernetes Secrets.

Examples:

  • AWS Secrets Manager → Kubernetes Secret;
  • Azure Key Vault → Kubernetes Secret;
  • HashiCorp Vault → Kubernetes Secret.

Pros:

  • Kubernetes-native consumption;
  • apps do not need cloud secret SDK;
  • rotation can be centralized.

Cons:

  • synced secret still exists in Kubernetes;
  • controller permissions are sensitive;
  • sync delay and failure modes matter;
  • must audit both external store and Kubernetes access.

7.3 Model C: Secret Store CSI Mount

Secrets are mounted into Pods as volumes from external providers.

Pros:

  • avoids storing long-lived secret value as a normal Kubernetes Secret when configured that way;
  • good for file-based secrets;
  • integrates well with cloud identity.

Cons:

  • mount failures block Pod startup;
  • application reload behavior must be defined;
  • CSI/provider availability becomes part of startup path;
  • operational debugging is more complex.

7.4 Model D: Direct App Retrieval

The application calls cloud secret manager directly using workload identity.

Pros:

  • no secret object required in Kubernetes;
  • application can implement caching/refresh logic;
  • cloud audit trail is direct.

Cons:

  • app now depends on cloud SDK and secret manager availability;
  • local development/testing becomes more complex;
  • every app may reinvent caching/retry/security logic;
  • migration across clouds is harder.

No model is universally best. The correct answer depends on sensitivity, rotation needs, app maturity, platform ownership, and audit requirements.


8. EKS Secret and Identity Patterns

On AWS EKS, prefer identity-based access over static cloud credentials.

Common tools and patterns:

  • IAM Roles for Service Accounts (IRSA);
  • EKS Pod Identity;
  • AWS Secrets Manager;
  • AWS Systems Manager Parameter Store;
  • Secrets Store CSI Driver with AWS provider;
  • External Secrets Operator with AWS provider;
  • envelope encryption for Kubernetes Secrets using AWS KMS, where configured.

8.1 EKS Pattern: Pod Identity + Secrets Manager

Design rule:

Do not put AWS access keys in Kubernetes Secrets for normal workloads.

Use workload identity.

8.2 EKS IAM Boundary Questions

Ask:

  • which service account can access which AWS secret?
  • is access namespace-scoped?
  • can a compromised Pod read unrelated secrets?
  • are IAM policies resource-scoped, not wildcarded?
  • is CloudTrail/audit enabled for secret reads?
  • how is rotation handled?
  • what happens if Secrets Manager is temporarily unavailable?

9. AKS Secret and Identity Patterns

On Azure AKS, prefer managed identity / workload identity over static credentials.

Common tools and patterns:

  • Microsoft Entra Workload ID;
  • managed identities;
  • Azure Key Vault;
  • Secrets Store CSI Driver with Azure Key Vault provider;
  • External Secrets Operator with Azure provider;
  • Kubernetes Secret encryption at rest through platform-managed configuration;
  • Azure RBAC / Kubernetes RBAC integration.

9.1 AKS Pattern: Workload Identity + Key Vault

Design rule:

Do not store Azure client secrets in Kubernetes Secrets for normal workloads.

Use workload identity or managed identity.

9.2 AKS Key Vault Boundary Questions

Ask:

  • is the workload identity bound only to the intended service account?
  • does Key Vault policy/RBAC grant least privilege?
  • are secrets, keys, and certificates separated by access need?
  • is secret version pinning required?
  • how are rotations propagated?
  • what alerts exist for failed secret access?

10. Configuration Layering

Production systems need layering, but layering can become chaos.

A clean model:

Example:

LayerOwnerExample
Image defaultsapp teamdefault port, safe internal defaults
Base manifestapp/platformprobes, resources, labels
Environment overlayplatform/appdev/stage/prod endpoint
Cluster overlayplatformstorage class, ingress class, node pool selector
Runtime configapp/platformtimeout, feature toggle default
Secret storesecurity/platformcredentials, certificates

Rules:

  1. Lower layers should be safe defaults.
  2. Higher layers should override intentionally.
  3. Secrets should not appear in non-secret layers.
  4. The final rendered config should be reviewable.
  5. A running Pod should expose config version, not raw secret values.

11. Environment Variable vs File Mount

CriterionEnv VarMounted File
Simple scalar valuesexcellentacceptable
Structured configweakexcellent
Secret leakage riskhigherlower but still present
App reload possibilityrequires restartpossible if app watches file
Compatibility with librariescommoncommon for certs/config files
Review clarityexplicit env is clearfile content can be clearer
Dynamic updatesnot for existing processpossible for directory mounts

Default guidance:

  • use env vars for small non-secret scalar config;
  • use mounted files for structured config;
  • use mounted files or external retrieval for sensitive files/certificates;
  • prefer restart-on-change unless live reload is explicitly designed.

12. Missing Config and Startup Failure

Configuration failure should be loud.

Bad app behavior:

Missing PAYMENT_ENDPOINT.
Defaulting to http://localhost:8080.
Application starts successfully.
Production traffic fails later.

Good app behavior:

Missing required PAYMENT_ENDPOINT.
Startup validation failed.
Exit non-zero.
Pod enters CrashLoopBackOff.
Rollout stops before receiving traffic.

At first, CrashLoopBackOff feels bad. But for invalid config, failing early is correct.

A production app should validate:

  • required config exists;
  • values have valid format;
  • numeric values are within bounds;
  • URLs use expected scheme;
  • mutually exclusive settings are not both enabled;
  • secrets are readable;
  • dangerous defaults are forbidden in production;
  • config version is logged without exposing secrets.

13. Config Compatibility During Rollout

Config changes are deployment changes.

Treat them with the same rigor as code.

13.1 Safe Config Change

A safe config change is:

  • backward-compatible with currently running app version;
  • forward-compatible with next app version if both coexist;
  • reversible;
  • observable;
  • tested in staging or canary;
  • bounded in blast radius.

13.2 Unsafe Config Change

Unsafe examples:

  • change database endpoint and deploy all Pods at once;
  • reduce timeout below realistic latency;
  • enable feature requiring schema not yet deployed;
  • rotate credential without overlapping validity;
  • change Kafka topic name while old Pods still publish to old topic;
  • change encryption key without keyring/migration plan.

13.3 Config and App Version Matrix

Use a compatibility matrix for high-risk changes.

App VersionConfig v1Config v2Notes
1.8yesnoold app cannot parse new setting
1.9yesyesbridge version
2.0noyesfinal version

Rollout sequence:

  1. deploy bridge app version 1.9 compatible with both configs;
  2. switch config from v1 to v2;
  3. deploy final app version 2.0;
  4. remove old config after safety window.

This is the same principle as safe database migrations.


14. Secret Rotation Strategy

Secret rotation must be designed before the incident.

A rotation plan needs:

  • source of truth;
  • propagation path;
  • compatibility window;
  • application reload/restart behavior;
  • verification step;
  • rollback or emergency revoke;
  • audit trail;
  • owner and runbook.

14.1 Rotation Timeline

The critical detail is overlap.

If dependency revokes old credential before all Pods use the new credential, you cause outage.

If old credential remains valid forever, rotation did not reduce risk.

14.2 Rotation Patterns

PatternUse WhenRisk
restart all Pods after secret updateapp reads secret at startuprollout must be safe
live reload mounted secretapp supports reloadmixed state and reload bugs
dual credential windowdependency supports two credentialsoperational sequencing
short-lived tokensidentity provider supports refreshclock/token/cache issues
direct cloud identitycloud service supports IAM/managed identitypermission boundary complexity

Best long-term direction:

replace static secrets with workload identity wherever possible.


15. RBAC and Secret Blast Radius

RBAC for Secrets must be restrictive.

Important rule:

Permission to read Secrets in a namespace is often equivalent to permission to impersonate many workloads in that namespace.

Why?

Secrets may contain:

  • database credentials;
  • service tokens;
  • TLS keys;
  • image pull credentials;
  • cloud credentials if badly designed;
  • application signing keys.

Avoid broad roles like:

resources: ["secrets"]
verbs: ["get", "list", "watch"]

Especially avoid list/watch unless necessary. Reading all secrets is much more dangerous than reading one named secret.

Prefer:

  • least privilege;
  • namespace isolation;
  • named resource access when possible;
  • separate service accounts per workload;
  • external secret store access scoped per workload;
  • audit logs for secret reads;
  • admission policy preventing risky secret patterns.

16. GitOps and Secret Handling

Never commit raw secrets to Git.

GitOps options include:

  • references to external secret names only;
  • sealed/encrypted secret manifests;
  • SOPS-encrypted files;
  • External Secrets Operator resources;
  • CSI SecretProviderClass resources;
  • pipeline-created Secrets;
  • cloud-native secret stores as source of truth.

16.1 GitOps-Friendly External Secret

Example shape, intentionally generic:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: order-api-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: platform-secret-store
    kind: ClusterSecretStore
  target:
    name: order-api-secret
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_PASSWORD
      remoteRef:
        key: prod/order-api/database
        property: password

This manifest does not contain the secret value. It contains the reference and sync policy.

Review questions:

  • who can change the remote secret reference?
  • who can read the resulting Kubernetes Secret?
  • who can read the external secret store value?
  • how is rotation tested?
  • what happens when sync fails?

17. Naming and Versioning

Config and secret names should communicate ownership and lifecycle.

Bad names:

config
secret
app-secret
prod-config

Better names:

order-api-runtime-config
order-api-database-credentials
ledger-reconciler-schedule-config
payment-gateway-client-certificate

For immutable/versioned config:

order-api-runtime-config-v20260703-001
order-api-runtime-config-v20260710-001

For Secrets, avoid putting secret values or sensitive meaning in names. The name itself is metadata and may appear in logs/events.


18. Observability for Configuration

You should be able to answer:

  • which config version is a Pod running?
  • which secret version is it using, if safe to expose as version metadata?
  • when did config last change?
  • who changed it?
  • did the rollout happen?
  • did the app reload successfully?
  • are some Pods on old config and some on new config?
  • did secret sync fail?

Expose non-sensitive metadata:

config_version="order-api-runtime-config-v20260703-001"
secret_version="db-password-version-42"   # only if not sensitive
feature_profile="prod-stable"

Never expose raw secret values.

18.1 Useful Events and Signals

Track:

  • Pod startup validation failure;
  • ConfigMap/Secret missing key;
  • failed secret mount;
  • external secret sync error;
  • CSI mount error;
  • application config reload success/failure;
  • rollout stuck after config change;
  • authentication failures after rotation.

19. Failure Mode Catalog

FailureSymptomCausePrevention
Missing ConfigMapPod stuck or app crashwrong name/namespaceCI validation, admission checks
Missing keycontainer creation error or app startup failuremanifest/config mismatchschema validation, startup validation
Bad value formatapp crash or degraded behaviorweak config validationtyped config validation
In-place config mutationunpredictable Pod behaviormanual editimmutable config/GitOps
Secret expiredauth failuresno rotation monitoringexpiry alerts and rotation runbook
Secret not propagatedold value still usedapp reads only at startuprestart-on-change or reload strategy
RBAC too broadlateral movement riskshared secret accessleast privilege per workload
Secret committed to Gitcredential leakweak processscanning, encrypted secrets, external stores
ConfigMap used for secretsensitive data exposedmisunderstandingpolicy and review
Cloud secret unavailablePod startup failureexternal dependency outagecaching, retry, readiness, fallback strategy
Wrong environment configprod points to staging or vice versaoverlay mistakeenvironment validation and naming

20. Configuration Schema and Validation

Kubernetes will not validate your application-specific config semantics.

It can validate object shape, not business correctness.

Your platform should add validation at multiple layers:

Validation examples:

  • YAML schema for config files;
  • JSON Schema for structured config;
  • Helm values schema;
  • Kustomize build validation;
  • policy-as-code for forbidden secret patterns;
  • admission control requiring config checksum annotation;
  • application startup validation with clear failure messages.

A mature app treats config as typed input, not as random strings.


21. Production Patterns

21.1 Restart-on-Config-Change Deployment

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-api-runtime-config-v20260703-001
immutable: true
data:
  ORDER_TIMEOUT_MS: "3000"
  PAYMENT_ENDPOINT: "https://payment.internal.example.com"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: order-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order-api
      annotations:
        platform.example.com/config-version: "order-api-runtime-config-v20260703-001"
        checksum/config: "sha256:example"
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/order-api:2026.07.03-1a2b3c4
          envFrom:
            - configMapRef:
                name: order-api-runtime-config-v20260703-001

21.2 Secret Mount with Explicit File Paths

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-client
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/platform/payment-client:2026.07.03-1a2b3c4
          env:
            - name: PAYMENT_CERT_PATH
              value: /app/secrets/tls.crt
            - name: PAYMENT_KEY_PATH
              value: /app/secrets/tls.key
          volumeMounts:
            - name: payment-client-cert
              mountPath: /app/secrets
              readOnly: true
      volumes:
        - name: payment-client-cert
          secret:
            secretName: payment-client-certificate
            defaultMode: 0400

21.3 Workload Identity Instead of Static Cloud Secret

Bad:

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-static-credentials
        key: accessKeyId
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-static-credentials
        key: secretAccessKey

Better:

serviceAccountName: order-api

Then bind the service account to the appropriate AWS IAM role or Azure federated identity through the cloud-specific workload identity mechanism.

The manifest becomes simpler because identity is no longer a secret string inside the Pod.


22. Platform Guardrails

A production platform should enforce common rules.

Potential policies:

  • ConfigMaps cannot contain keys matching PASSWORD, TOKEN, SECRET, PRIVATE_KEY;
  • Secrets must not be mounted into all containers by default;
  • Pods must not reference missing ConfigMaps/Secrets;
  • production ConfigMaps must be immutable or checksum-triggered;
  • Secrets must be sourced from approved external stores;
  • service accounts must be workload-specific;
  • default service account token automount should be disabled unless required;
  • containers must not log environment variables;
  • secret volumes must be read-only;
  • dangerous debug endpoints disabled in production.

Guardrails prevent repeated human review mistakes.


23. Review Checklist

23.1 ConfigMap Review

  • Is the data non-confidential?
  • Are keys explicitly named and documented?
  • Are values typed/validated by the app?
  • Is there a config version?
  • Is the change rollout-triggered?
  • Is live reload intentionally supported or avoided?
  • Is rollback clear?
  • Is the config environment-specific but not image-specific?

23.2 Secret Review

  • Why is this a Secret instead of workload identity?
  • Where is the source of truth?
  • Is it encrypted at rest?
  • Who can read it in Kubernetes?
  • Who can read it in the external store?
  • How is it rotated?
  • What happens to existing Pods during rotation?
  • Is the secret exposed as env var, file, CSI mount, or direct retrieval?
  • Are secret values excluded from logs, metrics, traces, and crash dumps?

23.3 Runtime Contract Review

  • Does the app fail fast on missing/invalid config?
  • Does the app expose config version safely?
  • Does readiness reflect config-dependent startup?
  • Does rollback include config rollback?
  • Are old and new config versions compatible during rollout?
  • Are cloud permissions least-privilege?

24. Exercises

Exercise 1: Classify Configuration

Classify these values as image default, ConfigMap, Secret, workload identity, feature flag, or policy:

  1. HTTP server port;
  2. database password;
  3. database hostname;
  4. AWS S3 bucket name;
  5. permission to read the S3 bucket;
  6. percentage rollout for new checkout flow;
  7. TLS private key;
  8. request timeout;
  9. production compliance mode;
  10. Azure Key Vault URI.

Explain why.

Exercise 2: Design a Rotation Plan

Design a rotation plan for a database password used by order-api.

Include:

  • source of truth;
  • old/new overlap window;
  • Kubernetes propagation;
  • app restart/reload;
  • validation;
  • rollback;
  • alerts;
  • audit evidence.

Exercise 3: Config Compatibility Matrix

Create a matrix for an app version change where:

  • old app reads PAYMENT_URL;
  • new app reads PAYMENT_ENDPOINT;
  • both must run during rolling update.

Design the bridge release.

Exercise 4: Secret Blast Radius

Given a namespace with five apps sharing one Secret, redesign it.

Expected direction:

  • separate secrets per workload;
  • separate service accounts;
  • least privilege external store policies;
  • remove unused keys;
  • audit access;
  • rotate after separation.

25. Part Summary

Configuration and secrets are not deployment afterthoughts. They are part of the workload runtime contract.

Use this mental model:

  • image = immutable software artifact;
  • ConfigMap = non-confidential runtime configuration;
  • Secret = confidential runtime material, but not a complete secret management system;
  • workload identity = preferred way to access cloud services;
  • external secret store = source of truth for sensitive material;
  • rollout/reload strategy = what makes config changes safe;
  • validation/observability = what makes config failures detectable.

The key production question is not:

Where do I put this value?

It is:

What is the lifecycle, sensitivity, owner, reload behavior, blast radius, and rollback model of this value?

Once you answer that, the Kubernetes objects become implementation details.


References

Lesson Recap

You just completed lesson 06 in start here. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.