Focus mode active/Press Alt+Shift+R to toggle/Esc to exit
Build CoreOrdered learning track

Kubernetes Identity and Permission Boundary

ServiceAccount and RBAC Operations

Operasi ServiceAccount dan RBAC untuk backend workloads: ServiceAccount, Role, ClusterRole, RoleBinding, least privilege, token projection, automountServiceAccountToken, kubectl auth can-i, permission debugging, and production-safe access review.

20 min read3941 words
PrevNext
Lesson 3398 lesson track19–53 Build Core
#kubernetes#serviceaccount#rbac#authorization+4 more

Part 033 — ServiceAccount and RBAC Operations

In Kubernetes, not every pod should be treated as the same actor.

Every backend workload that talks to the Kubernetes API, consumes projected tokens, integrates with cloud identity, reads mounted secrets, or is governed by policy needs a clear identity boundary.

The core operational invariant:

A Pod runs as a ServiceAccount.
A ServiceAccount receives permissions through RBAC.
RBAC determines what Kubernetes API actions that workload can perform.

This sounds simple, but production failures around ServiceAccount and RBAC are often misdiagnosed as:

  • application bugs;
  • missing config;
  • bad Secret;
  • cloud IAM problem;
  • network failure;
  • GitOps drift;
  • platform outage.

Part ini membahas ServiceAccount dan RBAC dari sudut pandang senior backend engineer: apa yang harus dipahami, apa yang aman diinvestigasi, bagaimana membaca permission failure, dan kapan harus eskalasi ke platform/SRE/security.

Untuk CSG atau environment enterprise apa pun, detail namespace, ServiceAccount naming, RoleBinding pattern, admission policy, workload identity convention, break-glass policy, dan audit mechanism harus diverifikasi internal.


1. Why ServiceAccount and RBAC Matter Operationally

ServiceAccount and RBAC matter because they control what a workload can do inside Kubernetes.

For backend engineers, this becomes important when a service:

  • reads Kubernetes API objects;
  • discovers pods/services dynamically;
  • runs leader election through ConfigMap or Lease objects;
  • creates Jobs or CronJobs;
  • reads Secrets or ConfigMaps;
  • watches custom resources;
  • integrates with service mesh sidecars;
  • uses external secrets controllers;
  • uses workload identity mapping;
  • is blocked by admission/security policy;
  • runs operational automation inside the cluster.

Most Java/JAX-RS backend services should not need broad Kubernetes API access.

A typical REST API service should usually only need:

ServiceAccount identity for pod runtime
+ cloud workload identity if it accesses cloud services
+ no direct Kubernetes API permissions unless explicitly required

A service that requires broad Kubernetes permissions should trigger architectural review.


2. The Core Objects

ServiceAccount and RBAC are composed from a small set of Kubernetes objects.

ObjectScopePurposeBackend engineer concern
ServiceAccountNamespaceIdentity used by podsWhich identity does my workload run as?
RoleNamespaceNamespaced permission setWhat can this workload do inside one namespace?
ClusterRoleClusterCluster-scoped or reusable permission setIs this too broad?
RoleBindingNamespaceGrants Role/ClusterRole to subject in a namespaceWho receives permissions?
ClusterRoleBindingClusterGrants ClusterRole cluster-wideHigh-risk; usually platform/security-owned
Token projectionPodProvides token to workloadIs token mounted? Is it needed?
automountServiceAccountTokenPod/SAControls token auto-mountingShould this workload receive a token at all?

Operational shorthand:

ServiceAccount = identity
Role/ClusterRole = permission definition
RoleBinding/ClusterRoleBinding = permission assignment

3. Mental Model: Pod Identity to API Authorization

When a pod calls the Kubernetes API, the request flows through authentication and authorization.

sequenceDiagram autonumber participant App as Java/JAX-RS App in Pod participant Token as Projected ServiceAccount Token participant API as Kubernetes API Server participant AuthN as Authentication participant RBAC as RBAC Authorization participant Obj as Kubernetes Object App->>Token: Reads token if mounted App->>API: Sends API request with bearer token API->>AuthN: Validate token and ServiceAccount identity AuthN-->>API: system:serviceaccount:namespace:name API->>RBAC: Authorize verb/resource/scope alt Allowed RBAC-->>API: allow API->>Obj: Perform operation API-->>App: 2xx response else Denied RBAC-->>API: deny API-->>App: 403 Forbidden end

Important distinction:

401 Unauthorized = identity/authentication problem
403 Forbidden    = identity authenticated, but not authorized

A backend engineer should avoid treating every 403 as "Kubernetes is down".

Most 403 errors mean:

the workload identity exists, but does not have permission for that action

4. Backend Engineer Responsibility

Backend engineers usually own:

  • understanding which ServiceAccount the workload uses;
  • knowing whether the application actually needs Kubernetes API access;
  • ensuring the app does not require unnecessary broad permissions;
  • reviewing Helm/Kustomize manifests for ServiceAccount usage;
  • validating that RBAC aligns with runtime behavior;
  • checking permission errors in application logs;
  • providing clear evidence when asking platform/security for permission changes;
  • ensuring code handles authorization failures explicitly;
  • avoiding direct reads of Kubernetes Secrets from application code unless intentionally designed;
  • ensuring cloud identity integration uses the correct workload identity mapping.

A backend engineer should be able to answer:

What identity does my workload run as?
What Kubernetes API permissions does it need?
Why does it need those permissions?
What breaks if those permissions are removed?
What is the smallest permission set that works?

5. Platform/SRE Responsibility

Platform/SRE usually owns:

  • cluster-wide RBAC conventions;
  • namespace bootstrap policies;
  • service account provisioning patterns;
  • admission control policies;
  • GitOps reconciliation of RBAC objects;
  • break-glass access mechanism;
  • audit log integration;
  • cluster role templates;
  • controller/operator permissions;
  • default ServiceAccount hardening;
  • cluster upgrade impact on RBAC APIs;
  • platform documentation for safe access.

Backend engineers should not invent their own cluster-wide permission model in isolation.

If a service needs a ClusterRoleBinding, that should be treated as a serious platform/security discussion.


6. Security Team Responsibility

Security usually owns or co-owns:

  • least privilege standard;
  • production access review;
  • privileged workload approval;
  • secret access policy;
  • audit evidence policy;
  • break-glass procedure;
  • exception process;
  • compliance mapping;
  • sensitive namespace restrictions;
  • service account token policies;
  • cloud IAM mapping standards;
  • workload identity trust boundaries.

Operational rule:

Permission expansion is a security change, not just a deployment fix.

During incidents, permission changes may be required, but they should still be traceable, time-bounded, and reviewed after mitigation.


7. Application Workload Responsibility

The workload itself should:

  • fail with clear error messages when permission is denied;
  • avoid logging raw tokens;
  • avoid dumping environment variables that may include credentials;
  • use official client libraries responsibly;
  • retry authorization failures carefully, not infinitely;
  • expose health endpoints that do not require Kubernetes API access unless intended;
  • not depend on broad Kubernetes list/watch permissions for core request handling unless justified;
  • not use admin kubeconfig inside containers;
  • avoid embedding cluster credentials in ConfigMaps or Secrets.

A Java/JAX-RS service should not require cluster-admin-like access to serve normal HTTP requests.


8. ServiceAccount Operations

A pod uses a ServiceAccount through the pod spec:

spec:
  serviceAccountName: quote-order-api

If omitted, Kubernetes uses the namespace default ServiceAccount.

That is often dangerous because it makes workload identity implicit.

Production preference:

Each production workload should use an explicit ServiceAccount.

Good pattern:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: quote-order-api
  namespace: quote-prod
  labels:
    app.kubernetes.io/name: quote-order-api
    app.kubernetes.io/part-of: quote-order
    app.kubernetes.io/managed-by: gitops

Risky pattern:

spec:
  # no serviceAccountName

Why risky:

  • identity is implicit;
  • audit becomes unclear;
  • future default ServiceAccount changes may affect the app;
  • workload identity mapping may not work;
  • RBAC review becomes ambiguous.

9. Default ServiceAccount Risk

Every namespace has a default ServiceAccount.

For production backend services, relying on it is usually a smell.

The default ServiceAccount may have no permission, or it may accidentally inherit permission through old bindings.

Common failure modes:

FailureSymptomLikely cause
App cannot access Kubernetes API403 ForbiddenDefault SA has no RoleBinding
App unexpectedly can read objectsSecurity concernDefault SA has broad binding
Cloud SDK cannot assume roleAccess deniedWorkload identity annotation missing
Secret integration failsMounted secret unavailableExternal controller uses different SA
Audit unclearHard to traceMultiple workloads use same default SA

Operational recommendation:

Use one explicit ServiceAccount per deployable workload, unless there is an approved shared pattern.

10. automountServiceAccountToken

By default, Kubernetes may mount a ServiceAccount token into pods.

For workloads that do not call the Kubernetes API and do not need projected token-based identity, consider disabling token auto-mounting.

Example:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: quote-order-api
  namespace: quote-prod
automountServiceAccountToken: false

Or at pod level:

spec:
  serviceAccountName: quote-order-api
  automountServiceAccountToken: false

Operational trade-off:

SettingBenefitRisk
Token mountedEnables Kubernetes API access and some identity integrationsToken exposure if pod compromised
Token not mountedReduces attack surfaceBreaks workloads needing API/cloud token projection

Backend engineer checklist:

Does the application actually call Kubernetes API?
Does the application need workload identity token projection?
Does any sidecar require the token?
Does external secret / service mesh / platform injection depend on it?

Do not disable token mounting blindly. Validate sidecar and workload identity behavior first.


11. Token Projection Awareness

Modern Kubernetes uses projected, bounded ServiceAccount tokens.

Important properties:

  • token can have expiration;
  • token can have audience;
  • token can be projected into pod volume;
  • cloud identity systems can trust projected token;
  • application/client library should handle refresh if needed;
  • token should not be copied or logged.

Example projected token volume pattern:

volumes:
  - name: kube-api-token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            expirationSeconds: 3600
            audience: api

Backend concern:

If the app or SDK reads a projected token, it must tolerate token rotation/refresh.

Failure mode:

FailureSymptomExplanation
Token expired401 or cloud access deniedApp cached old token
Wrong audienceToken rejectedToken minted for different audience
Missing projectionCredential not foundVolume not mounted or token disabled
Permission denied reading fileApp startup failureFile path/user permission mismatch

12. Role vs ClusterRole

A Role is namespace-scoped.

A ClusterRole is cluster-scoped or reusable across namespaces.

Use Role when permission is only needed inside one namespace.

Example Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: quote-order-api-reader
  namespace: quote-prod
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

A ClusterRole can grant access to cluster-scoped resources or be reused in multiple namespaces.

Example ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

Operational caution:

ClusterRole is not automatically dangerous.
ClusterRoleBinding is where cluster-wide assignment usually becomes dangerous.

A ClusterRole bound with a namespaced RoleBinding grants permission only in that namespace.


13. RoleBinding vs ClusterRoleBinding

A RoleBinding grants permissions within a namespace.

A ClusterRoleBinding grants permissions cluster-wide.

Example RoleBinding to a Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: quote-order-api-reader
  namespace: quote-prod
subjects:
  - kind: ServiceAccount
    name: quote-order-api
    namespace: quote-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: quote-order-api-reader

Example RoleBinding to a ClusterRole, still namespaced:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: quote-order-api-pod-reader
  namespace: quote-prod
subjects:
  - kind: ServiceAccount
    name: quote-order-api
    namespace: quote-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: namespace-pod-reader

ClusterRoleBinding example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: quote-order-api-cluster-reader
subjects:
  - kind: ServiceAccount
    name: quote-order-api
    namespace: quote-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-reader

For backend services, ClusterRoleBinding should be rare and explicitly justified.


14. Verbs and Resources

RBAC grants actions through verbs.

Common verbs:

VerbMeaningRisk
getRead one objectLow to medium, depends on object
listRead many objectsMedium, can expose broad data
watchSubscribe to changesMedium, long-running access
createCreate objectsMedium to high
updateReplace objectHigh
patchPartial updateHigh
deleteDelete objectHigh
deletecollectionDelete many objectsVery high
impersonateAct as another user/serviceVery high
escalateCreate roles beyond own permissionVery high
bindBind roles to subjectsVery high

Backend service workloads should usually not have:

*
update on secrets
patch on deployments
delete on pods
create rolebindings
impersonate
escalate
bind
cluster-admin

If they do, require architecture/security review.


15. Least Privilege Model

Least privilege does not mean "give no permissions".

It means:

Grant only the verbs, resources, namespaces, and API groups required for the workload's legitimate runtime behavior.

A practical least privilege review asks:

  1. What exact Kubernetes API call does the app make?
  2. What resource does it target?
  3. Does it need namespace scope or cluster scope?
  4. Does it need list/watch or only get?
  5. Does it need write permission?
  6. Is this runtime behavior or deployment-time behavior?
  7. Could this be moved to CI/CD, operator, or platform automation?
  8. What happens if the permission is abused?
  9. How is access audited?
  10. How is permission revoked?

Operational smell:

Permission added because "the app failed" but no one can explain which API call requires it.

16. Backend Workload Patterns

Different backend workloads have different identity needs.

JAX-RS API service

Usually needs:

  • explicit ServiceAccount;
  • no Kubernetes API write permission;
  • possibly cloud workload identity;
  • secret/config consumption;
  • observability sidecar compatibility.

Usually should not need:

  • list all pods;
  • patch deployments;
  • read all secrets;
  • create jobs;
  • cluster-wide permissions.

Kafka consumer service

Usually needs:

  • explicit ServiceAccount;
  • cloud identity if broker uses IAM/cloud auth;
  • secret/config consumption;
  • no Kubernetes API permissions unless using leader election or dynamic discovery.

RabbitMQ consumer service

Usually needs:

  • explicit ServiceAccount;
  • broker credential access through secret mechanism;
  • no Kubernetes API permission for normal consumption.

Camunda worker

Usually needs:

  • explicit ServiceAccount;
  • Camunda endpoint credentials;
  • optional cloud identity;
  • no Kubernetes API permission unless orchestrating jobs.

Migration job

May need:

  • explicit ServiceAccount;
  • DB credential access;
  • possibly no Kubernetes API permission;
  • strict TTL and audit.

Operator/controller

May need:

  • list/watch/update custom resources;
  • create/update child resources;
  • leader election permission;
  • carefully scoped Role/ClusterRole.

Operators are different from normal backend services.


17. Leader Election RBAC

Some Java services, schedulers, or workers use Kubernetes leader election to ensure only one active instance runs a task.

Leader election often uses:

  • Lease objects in coordination.k8s.io;
  • ConfigMaps in older patterns;
  • Endpoints in older patterns.

Example Role for Lease-based leader election:

rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

Operational concern:

Leader election permission is a write permission. It should be scoped to the namespace and preferably to a known resource name where possible.

Failure symptoms:

  • scheduler does not start;
  • multiple instances think they are not leader;
  • logs show 403 on Lease update;
  • task never runs;
  • duplicate task execution if fallback logic is wrong.

18. Secret Access Through Kubernetes API

Reading Secrets through Kubernetes API is highly sensitive.

A workload that can get/list/watch secrets in a namespace can potentially access credentials for other services in that namespace.

Dangerous rule:

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

Better patterns:

  • mount only the specific Secret the pod needs;
  • use external secret projection;
  • use cloud workload identity instead of static credentials;
  • avoid app-level Secret API reads;
  • separate sensitive workloads into namespaces where needed;
  • limit who can read Secret objects.

Backend engineer rule:

Do not request secret read RBAC unless the application genuinely acts as a secret controller or operator.

19. Kubernetes API Access from Java

Java services may access the Kubernetes API using clients such as:

  • Fabric8 Kubernetes Client;
  • official Kubernetes Java client;
  • Spring Cloud Kubernetes;
  • custom HTTP client using in-cluster token;
  • platform SDKs or operators.

Operational checklist:

Which client library is used?
What API objects does it access?
Does it list/watch objects continuously?
Does it handle 401/403/429/5xx properly?
Does it reconnect safely?
Does it cache stale results?
Does it log too much object data?
Does it require token refresh support?

Failure modes:

SymptomPossible cause
Startup fails with 403Missing RoleBinding
App hangs during startupKubernetes API call timeout
CPU/memory highLarge list/watch cache
Stale routing/configWatch disconnected silently
Excess API requestsBad polling loop
API throttlingToo many replicas list/watch same objects

20. Safe Investigation Commands

Before checking permissions, confirm context and namespace.

kubectl config current-context
kubectl config view --minify --output 'jsonpath={..namespace}'

Find pod ServiceAccount:

kubectl -n <namespace> get pod <pod-name> \
  -o jsonpath='{.spec.serviceAccountName}{"\n"}'

Find Deployment ServiceAccount:

kubectl -n <namespace> get deploy <deployment-name> \
  -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'

List ServiceAccounts:

kubectl -n <namespace> get serviceaccounts

Describe ServiceAccount:

kubectl -n <namespace> describe serviceaccount <serviceaccount-name>

Check RoleBindings:

kubectl -n <namespace> get rolebinding
kubectl -n <namespace> describe rolebinding <rolebinding-name>

Check Roles:

kubectl -n <namespace> get role
kubectl -n <namespace> describe role <role-name>

Check if a ServiceAccount can perform an action:

kubectl auth can-i get configmaps \
  --as=system:serviceaccount:<namespace>:<serviceaccount-name> \
  -n <namespace>

Check cluster-scoped permission carefully:

kubectl auth can-i list nodes \
  --as=system:serviceaccount:<namespace>:<serviceaccount-name>

Production safety:

Use auth can-i before changing RBAC.
Use describe/get before patch/apply.
Do not create emergency broad bindings without approval.

21. RBAC Denied Debugging Flow

Use this flow when an application logs Kubernetes API permission denied.

flowchart TD A[App logs 401 or 403] --> B{Is it Kubernetes API?} B -->|No| C[Check cloud IAM / app auth / dependency auth] B -->|Yes| D[Identify pod and namespace] D --> E[Find ServiceAccount] E --> F[Identify verb/resource/apiGroup from error] F --> G[Run kubectl auth can-i as ServiceAccount] G --> H{Allowed?} H -->|No| I[Check Role/RoleBinding or ClusterRoleBinding] H -->|Yes| J[Check token mount / audience / client config] I --> K{Permission justified?} K -->|No| L[Fix app/config to avoid unnecessary API call] K -->|Yes| M[Request least-privilege RBAC change] J --> N[Check token projection, expiry, API URL, namespace] M --> O[Apply through GitOps/pipeline if required] O --> P[Verify logs, events, and audit]

Do not start with "add cluster-admin".

Start with the exact denied action.


22. Reading the Error Message

Kubernetes authorization errors are usually explicit.

Example:

configmaps is forbidden: User "system:serviceaccount:quote-prod:quote-order-api" cannot list resource "configmaps" in API group "" in the namespace "quote-prod"

Extract:

subject     = system:serviceaccount:quote-prod:quote-order-api
verb        = list
resource    = configmaps
apiGroup    = core API group ""
namespace   = quote-prod

This maps directly to an RBAC rule:

rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["list"]

But before adding the rule, ask:

Why does this service need to list ConfigMaps?
Can it get one named ConfigMap instead?
Is this startup-time behavior or request-time behavior?
Could this be provided as mounted config instead?

23. 401 vs 403 vs Connection Failure

RBAC only explains some failures.

SymptomMeaningLikely area
401 UnauthorizedIdentity not acceptedtoken, audience, expiry, API endpoint
403 ForbiddenIdentity accepted, action deniedRBAC
connection refusedAPI endpoint or network problemcluster/network
timeoutnetwork/API server/proxy issueplatform/network
x509 errortrust/certificate issueTLS/CA/truststore
no such file tokentoken not mountedpod spec/SA token config

Backend debugging should classify the failure before proposing fixes.


24. Common RBAC Failure Modes

Failure modeSymptomDebug signalSafe mitigation
Missing RoleBinding403auth can-i = noAdd minimal RoleBinding through approved path
Wrong ServiceAccount403 or cloud deniedPod SA differs from expectedFix serviceAccountName
Default SA used accidentallyUnclear audit / deniedPod spec lacks SAAdd explicit SA
Wrong namespace403 or not foundBinding exists in another namespaceBind in correct namespace or fix app namespace
Missing API group403Rule uses wrong apiGroupCorrect API group
Missing verb403Rule has get but app listsAdd exact required verb if justified
ClusterRoleBinding too broadSecurity findingBroad subject accessReplace with RoleBinding where possible
Token not mountedCredential file missingPod spec disables automountRe-enable only if needed
Token audience mismatch401Token rejectedFix projected token audience
GitOps overwrites manual fixPermission disappearsArgo/Flux syncUpdate source repo

25. Dangerous Permissions to Review Carefully

Certain permissions are high risk.

Secrets

get/list/watch secrets

Risk:

  • credential exposure;
  • cross-service compromise;
  • compliance violation.

Pods exec/attach/portforward

create pods/exec
create pods/attach
create pods/portforward

Risk:

  • interactive access to runtime;
  • data exfiltration;
  • bypassing deployment controls.

Workload mutation

patch/update deployments
patch/update statefulsets
create/delete pods

Risk:

  • workload disruption;
  • unauthorized deployment change;
  • GitOps conflict.

RBAC mutation

create/update rolebindings
bind
escalate
impersonate

Risk:

  • privilege escalation;
  • audit/control bypass.

Cluster-scoped reads

list nodes
list namespaces
list all pods

Risk:

  • inventory exposure;
  • broader blast radius.

26. RBAC and GitOps

In GitOps-managed environments, RBAC changes should usually come from Git.

Manual RBAC change risk:

kubectl apply emergency RoleBinding
→ incident mitigated temporarily
→ GitOps reconciliation removes it
→ failure returns

Operational rule:

If RBAC is GitOps-managed, fix the source of truth, not only the live object.

Check:

kubectl -n <namespace> get rolebinding <name> -o yaml

Look for labels/annotations such as:

app.kubernetes.io/managed-by: argocd
argocd.argoproj.io/instance: quote-order

Do not assume exact internal labels. Verify internal GitOps conventions.


27. RBAC and Helm/Kustomize

RBAC may be generated by Helm or Kustomize.

Helm concerns:

  • serviceAccount.create true/false;
  • serviceAccount.name override;
  • chart creates broad Role;
  • chart creates ClusterRole by default;
  • environment-specific values change permissions;
  • release upgrade changes identity.

Kustomize concerns:

  • overlay changes ServiceAccount name;
  • patch adds RoleBinding in only one environment;
  • generated names break binding references;
  • base and overlay drift;
  • namespace transformer affects subjects.

Review rendered output, not only templates.

helm template ...
kustomize build ...

Then compare against live objects and GitOps source.


28. RBAC and Admission Policy

Admission controllers may reject RBAC changes or pod specs.

Examples:

  • deny automount ServiceAccount token;
  • deny use of default ServiceAccount;
  • deny ClusterRoleBinding from app namespace;
  • deny access to Secrets;
  • require labels/annotations;
  • require approved ServiceAccount name;
  • block privileged permissions.

Symptoms:

Error from server (Forbidden): admission webhook denied the request

This is not always RBAC authorization. It may be policy admission.

Debug path:

  1. Read the admission error message.
  2. Identify policy name if available.
  3. Check GitOps/pipeline logs.
  4. Ask platform/security for policy documentation.
  5. Adjust manifest according to standard.
  6. Use exception process only when justified.

29. Observability and Audit Signals

RBAC issues should leave evidence.

Useful signals:

  • application logs with 401/403;
  • Kubernetes API server audit logs;
  • admission webhook logs;
  • GitOps sync logs;
  • controller/operator logs;
  • deployment events;
  • security dashboard;
  • cloud audit logs if workload identity is involved;
  • incident timeline.

Backend engineer should capture:

timestamp
namespace
pod
ServiceAccount
operation attempted
resource/apiGroup/verb
error text
recent deployment/config changes
expected behavior
impact

Do not capture raw tokens or secret values.


30. Production-Safe Mitigation Options

When RBAC blocks production behavior, possible mitigations include:

Option 1: Rollback

Use when:

  • new deployment introduced unnecessary API call;
  • wrong ServiceAccount was deployed;
  • Helm/Kustomize rendered incorrect binding;
  • permission change is not approved during incident.

Option 2: Fix ServiceAccount reference

Use when:

  • RoleBinding exists for correct SA;
  • pod is using default or wrong SA;
  • rollout mistake changed serviceAccountName.

Option 3: Add minimal RBAC through approved path

Use when:

  • permission is genuinely required;
  • action is known precisely;
  • owner approves;
  • security/platform accepts scope;
  • change is applied through GitOps/pipeline where required.

Option 4: Disable unnecessary Kubernetes API integration

Use when:

  • app library auto-discovers Kubernetes features not needed;
  • framework tries to read ConfigMaps/Secrets dynamically;
  • feature flag can avoid API access.

Option 5: Escalate

Use when:

  • ClusterRoleBinding needed;
  • admission policy involved;
  • cloud IAM/workload identity involved;
  • audit/break-glass required;
  • security exception needed.

31. Java/JAX-RS Specific Concerns

Java frameworks sometimes add Kubernetes behavior indirectly.

Examples:

  • config reload through Kubernetes ConfigMap watch;
  • service discovery through Kubernetes API;
  • leader election;
  • actuator/management integration;
  • cloud SDK token reading;
  • service mesh sidecar injection;
  • OpenTelemetry auto-instrumentation sidecar/init container.

Operational risk:

The application team may not realize a library introduced Kubernetes API access.

Review dependencies and config flags.

Questions:

Does the app fail if Kubernetes API is unavailable?
Does request handling call Kubernetes API?
Can startup proceed without ConfigMap watch?
Does the app need list/watch or only mounted config?

Request path should not depend on Kubernetes API unless explicitly designed.


32. Dependency Impact

RBAC problems can indirectly break dependency access.

DependencyRBAC linkage
PostgreSQLSecret/config access may fail; migration Job may use wrong SA
KafkaCredentials may not mount; IAM auth may require workload identity
RabbitMQSecret/config access may fail; consumer startup blocked
RedisSecret/config access may fail; TLS cert secret unavailable
CamundaWorker credentials/config may fail; job worker cannot start
NGINX/IngressController permissions are platform-owned; app should not mutate ingress at runtime
External SecretsController RBAC or workload SA mapping may affect secret sync

Do not conclude "database is down" before checking whether the workload can even read its credential/config.


33. EKS/AKS/On-Prem/Hybrid Awareness

EKS

ServiceAccount may be tied to AWS IAM through IRSA.

RBAC controls Kubernetes API access. AWS IAM controls AWS API access.

These are separate authorization planes.

AKS

ServiceAccount may be tied to Azure Workload Identity.

RBAC controls Kubernetes API access. Azure RBAC/Entra ID controls Azure API/resource access.

On-prem/hybrid

ServiceAccount may integrate with:

  • internal identity platform;
  • service mesh identity;
  • corporate PKI;
  • private registry auth;
  • custom admission policies.

Internal verification is mandatory.


34. PR Review Checklist

When reviewing Kubernetes RBAC changes, check:

  • Is the ServiceAccount explicit?
  • Is default ServiceAccount avoided?
  • Is automountServiceAccountToken intentionally set?
  • Does the workload need Kubernetes API access?
  • Are verbs minimal?
  • Are resources minimal?
  • Is namespace scope sufficient?
  • Is ClusterRoleBinding avoided unless justified?
  • Is Secret read permission avoided?
  • Is list/watch justified?
  • Are bind, escalate, and impersonate absent?
  • Is the RBAC generated correctly by Helm/Kustomize?
  • Is GitOps source updated?
  • Is security/platform approval required?
  • Is audit trail preserved?
  • Is rollback path clear?

35. Incident Checklist

During incident, capture:

  • Impacted service/workload.
  • Namespace.
  • Pod name.
  • ServiceAccount name.
  • Exact error message.
  • Verb/resource/apiGroup/namespace.
  • kubectl auth can-i result.
  • Relevant Role/RoleBinding.
  • Recent deployment/config/RBAC change.
  • GitOps sync status.
  • Whether cloud IAM is involved.
  • Whether rollback is safer than permission expansion.
  • Escalation owner.
  • Evidence captured without tokens/secrets.

36. Internal Verification Checklist

For CSG/team-specific validation, verify:

  • ServiceAccount naming convention.
  • Whether every workload has explicit ServiceAccount.
  • Whether default ServiceAccount usage is allowed.
  • automountServiceAccountToken standard.
  • RBAC templates for backend services.
  • RBAC templates for Jobs/CronJobs.
  • RBAC templates for operators/controllers.
  • Whether RBAC is managed by Helm, Kustomize, or GitOps.
  • How to request RBAC changes.
  • Who approves RoleBinding changes.
  • Who approves ClusterRoleBinding changes.
  • Whether Secret read RBAC is forbidden or exception-based.
  • Whether admission policies enforce ServiceAccount/RBAC standards.
  • How API server audit logs are accessed.
  • How to inspect RBAC safely in production.
  • Whether kubectl exec / port-forward / debug are allowed.
  • Whether workload identity depends on ServiceAccount annotation.
  • Whether break-glass RBAC exists and how it is audited.
  • Incident escalation path for RBAC denied.
  • Security review process for permission expansion.

37. Practical Review Questions

Ask these before approving a ServiceAccount/RBAC change:

  1. What is the exact runtime use case?
  2. Which code path triggers the Kubernetes API call?
  3. Is this request-time or startup-time behavior?
  4. Can mounted config/secret replace API access?
  5. Can permission be namespaced?
  6. Can permission be narrowed to get instead of list/watch?
  7. Is write permission genuinely required?
  8. What is the blast radius if this pod is compromised?
  9. What audit evidence will exist?
  10. How do we rollback safely?

38. Senior Engineer Mental Model

A senior backend engineer should not think:

The app needs permission, add RBAC.

They should think:

The app attempted verb X on resource Y as ServiceAccount Z.
Is that behavior necessary?
Is the permission scoped minimally?
Is the change owned by app, platform, or security?
Will GitOps preserve it?
What is the incident-safe mitigation?

RBAC is not just YAML.

It is a production safety boundary.


39. Key Takeaways

  • ServiceAccount is the pod's Kubernetes identity.
  • RBAC controls Kubernetes API authorization, not cloud API authorization.
  • Explicit ServiceAccount is preferable for production workloads.
  • Default ServiceAccount usage should be questioned.
  • automountServiceAccountToken should be intentional.
  • Least privilege requires knowing exact verbs/resources/API groups.
  • Secret read RBAC is high-risk.
  • ClusterRoleBinding is usually platform/security-sensitive.
  • GitOps-managed RBAC must be fixed in source control.
  • Authorization debugging starts with the exact error message.
  • Permission expansion is a security change, not just an operational tweak.

Next part: Workload Identity Operations on EKS and AKS.

Lesson Recap

You just completed lesson 33 in build core. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.