Part 049 — Container Registry and Artifact Governance

A container registry is not only a place to store images.

In enterprise systems, the registry is part of the production control plane for artifact identity, promotion, traceability, security, rollback, and disaster recovery.

For Java/JAX-RS services, the registry answers several operational questions:

• Which exact image is running in production?
• Which source commit produced that image?
• Which base image and dependencies are inside it?
• Has it been scanned?
• Was it promoted through the expected environments?
• Can the cluster still pull it during a registry incident?
• Can an old version still be rolled back safely?

If the registry is weakly governed, Kubernetes deployments become difficult to trust.

CSG/internal note: this part does not assume CSG uses Docker Hub, Amazon ECR, Azure Container Registry, Harbor, Artifactory, Nexus, private registry mirrors, specific image promotion rules, immutable tags, admission policies, or registry replication. Verify all registry and artifact governance details internally through CI/CD pipeline, GitOps/IaC repository, registry configuration, platform/SRE/DevOps/security teams, and incident/runbook documentation.

1. Core Mental Model

A container image is a deployable artifact.

A registry is the artifact distribution and governance layer.

source code
   ↓
CI build
   ↓
container image
   ↓
scan / sign / attest
   ↓
registry push
   ↓
promotion to environment
   ↓
Kubernetes image pull
   ↓
running pod

The registry sits between build and runtime. That makes it a control point for:

• artifact identity,
• supply-chain security,
• environment promotion,
• rollback availability,
• vulnerability visibility,
• deployment auditability,
• incident recovery.

A mature registry strategy treats images as immutable release artifacts, not disposable build outputs.

2. Registry Types

Common registry options include:

• Docker Hub,
• Amazon Elastic Container Registry,
• Azure Container Registry,
• Google Artifact Registry,
• Harbor,
• JFrog Artifactory,
• Sonatype Nexus,
• internal enterprise registry,
• registry cache or mirror.

For enterprise Java/JAX-RS systems, the exact product matters less than the governance properties:

• authentication,
• authorization,
• immutable artifact identity,
• vulnerability scanning,
• retention,
• replication,
• audit logging,
• availability,
• integration with Kubernetes clusters,
• integration with CI/CD and GitOps.

A registry without governance is just a shared folder with HTTP APIs.

3. Image Name, Tag, and Digest

A Kubernetes container image reference usually looks like this:

image: registry.example.com/quote/order-service:1.24.7

This has three important parts:

registry.example.com       registry host
quote/order-service        repository path
1.24.7                     tag

A digest reference looks like this:

image: registry.example.com/quote/order-service@sha256:abc123...

The difference is critical.

Example:

image: registry.example.com/quote/order-service:1.24.7@sha256:abc123...

In production, a tag should not be trusted unless immutability is enforced by policy.

4. Mutable Tag Failure

Mutable tags are a common source of deployment ambiguity.

Bad pattern:

image: registry.example.com/quote/order-service:latest

Why this is dangerous:

• latest does not mean newest in a meaningful release sense.
• The same tag can point to different image content over time.
• Rollback becomes ambiguous.
• Incident investigation cannot prove which binary ran.
• Different nodes may pull different content depending on cache and pull timing.

More subtle bad pattern:

image: registry.example.com/quote/order-service:dev
image: registry.example.com/quote/order-service:staging
image: registry.example.com/quote/order-service:production

Environment tags often hide promotion history. They are pointers, not release artifacts.

Better pattern:

image: registry.example.com/quote/order-service:1.24.7-20260711-8f3a91c

Even better when supported:

image: registry.example.com/quote/order-service:1.24.7-20260711-8f3a91c@sha256:...

5. Image Tag Strategy

A useful tag strategy should answer:

• Which service is this?
• Which version is this?
• Which commit built it?
• Which build produced it?
• Is it suitable for rollback?
• Can it be traced from pod to source?

Possible tag components:

<semantic-version>-<build-date>-<git-sha>
<release-version>-<ci-build-number>-<git-sha>
<branch>-<git-sha>-<timestamp>

Examples:

1.12.3-20260711-a1b2c3d
1.12.3-build.8421-a1b2c3d
main-a1b2c3d-20260711T101530Z

For production, prefer tags that are:

• unique,
• immutable,
• traceable,
• promoted rather than rebuilt per environment,
• compatible with rollback and audit.

Avoid using only:

• latest,
• dev,
• test,
• prod,
• branch name without commit,
• timestamp without source reference.

6. Image Promotion Model

There are two broad deployment models.

Rebuild per environment

source → build dev image
source → build test image
source → build prod image

This is risky because production may not run the exact artifact that was tested.

Build once, promote same image

source
  ↓
build image once
  ↓
scan/sign/attest
  ↓
promote same digest dev → test → staging → prod

This is stronger because each environment runs the same artifact identity.

Configuration should vary by environment. The image should not.

Same image digest + environment-specific config = predictable promotion
Different image per environment + hidden build variance = weak traceability

For Java/JAX-RS systems, environment-specific differences should usually live in:

• ConfigMap,
• Secret,
• Helm values,
• Kustomize overlay,
• external config service,
• cloud parameter service,
• GitOps environment overlay.

They should not require rebuilding the Java image.

7. Registry Authentication

Kubernetes pulls private images using registry credentials.

Common mechanisms:

• node-level cloud integration,
• Kubernetes imagePullSecrets,
• workload identity integration,
• managed identity integration,
• registry-specific IAM/RBAC,
• private registry credentials stored as Kubernetes Secret.

Example Kubernetes reference:

spec:
  imagePullSecrets:
    - name: registry-pull-secret

But in cloud-managed Kubernetes, explicit imagePullSecrets may not always be necessary when nodes or kubelet have managed access to the registry.

Common failure modes:

• wrong registry credential,
• expired token,
• missing imagePullSecret,
• secret in wrong namespace,
• node identity lacks registry pull permission,
• private endpoint or firewall blocks registry,
• DNS cannot resolve registry endpoint,
• registry repository path is wrong,
• tag/digest does not exist.

8. ImagePullBackOff and ErrImagePull

ImagePullBackOff means Kubernetes tried to pull the image, failed, and is now backing off before retrying.

Typical debug flow:

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Look for messages such as:

Failed to pull image
pull access denied
repository does not exist
manifest unknown
unauthorized: authentication required
x509: certificate signed by unknown authority
i/o timeout
no such host

Map symptoms to cause:

Do not restart pods blindly. The pull will keep failing until the underlying registry, auth, network, or reference issue is fixed.

9. Image Pull Policy

Kubernetes supports image pull policies:

imagePullPolicy: Always
imagePullPolicy: IfNotPresent
imagePullPolicy: Never

Mental model:

For production, the safer pattern is not to rely on pull policy to compensate for weak tags.

Use immutable image identity instead.

Correct image identity > clever imagePullPolicy

If tags are immutable and unique, IfNotPresent is less dangerous because the tag points to one artifact. If tags are mutable, node cache behavior can create confusing inconsistency.

10. Registry Availability and Runtime Risk

A registry outage may not immediately break running pods.

Running containers usually continue running after image pull has completed.

But registry unavailability can break:

• new deployments,
• rollbacks to images not cached on nodes,
• pod rescheduling after node failure,
• autoscaling to new nodes,
• disaster recovery,
• cluster rebuild,
• blue-green/canary rollout,
• Job/CronJob start,
• node image refresh events.

This is why registry availability affects production resilience even if current pods look healthy.

Important question:

Can we recover the service if all pods must be recreated right now?

If the answer depends on a single registry endpoint that is down or unreachable, the deployment is not operationally resilient.

11. Registry Replication and Locality

Registry replication may be used for:

• multi-region resilience,
• lower pull latency,
• disaster recovery,
• compliance locality,
• disconnected or air-gapped environments,
• on-prem cluster support.

But replication introduces governance questions:

• Which registry is the source of truth?
• Are tags immutable across replicas?
• Is digest preserved?
• What is the replication delay?
• What happens if promotion occurs before replication completes?
• How is vulnerability metadata replicated?
• How is access controlled per region/environment?

For hybrid systems, registry locality matters because image pulls may cross:

• corporate firewall,
• private network link,
• VPN,
• Direct Connect,
• ExpressRoute,
• proxy,
• private endpoint,
• registry mirror.

Image pull latency can become a startup and autoscaling problem.

12. Retention Policy

Registry retention saves cost and reduces clutter, but aggressive deletion can break rollback.

Retention policy must account for:

• production rollback window,
• hotfix rollback window,
• compliance evidence retention,
• audit requirements,
• disaster recovery,
• long-lived customer environments,
• release support lifecycle,
• air-gapped sync delay,
• image scan history.

Dangerous pattern:

Delete all images older than 30 days without checking whether they are still deployed.

Better pattern:

Keep all images currently deployed anywhere.
Keep all images in active release support window.
Keep images referenced by release records.
Expire only unreferenced build artifacts after policy threshold.

For Kubernetes, retention should be reconciled with actual cluster state:

kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'

In GitOps environments, also check manifests and release tags, not only live pods.

13. Vulnerability Scanning in Registry

Registry scanning may detect:

• OS package CVEs,
• language dependency CVEs,
• vulnerable base images,
• outdated libraries,
• malware signatures depending on tooling,
• license issues depending on scanner.

Scanning must be interpreted carefully.

Important questions:

• Is the vulnerable package actually present in the final runtime layer?
• Is it reachable by the application?
• Is it exploitable in this deployment context?
• Is a patched base image available?
• Is the vulnerability in JDK/JRE, OS layer, native library, or app dependency?
• Does the scanner support Java dependency visibility inside fat jars?
• What is the SLA for critical/high CVEs?

A scan finding is not automatically an incident. But ignoring scan findings without triage is supply-chain negligence.

14. SBOM and Provenance

A Software Bill of Materials describes what is inside an artifact.

For containerized Java systems, SBOM should ideally cover:

• base image,
• OS packages,
• JDK/JRE distribution,
• Maven dependencies,
• application artifact,
• native libraries,
• transitive dependencies,
• build metadata.

Provenance answers:

Who built this image?
From which source commit?
Using which build pipeline?
With which dependencies?
Was it signed?
Was it promoted through approved gates?

Operational value:

• faster CVE impact analysis,
• stronger audit trail,
• safer rollback,
• compliance evidence,
• incident traceability.

Without SBOM/provenance, production investigation often becomes archaeology.

15. Image Signing and Admission

Image signing creates a cryptographic trust signal for images.

Admission control can enforce rules such as:

• image must come from approved registry,
• image must be signed,
• image must not use latest,
• image must use immutable digest,
• image must pass vulnerability threshold,
• image must not run privileged,
• image must have approved labels/annotations,
• image repository must match namespace/team ownership.

The point is not bureaucracy. The point is preventing untrusted artifacts from becoming running workloads.

Failure mode:

Emergency hotfix bypasses signing and admission controls.
The workload runs but is not traceable or compliant.
Later incident cannot prove artifact origin.

If break-glass exists, it must be audited, time-bound, and reviewed after use.

16. Registry Access Control

Registry access should follow least privilege.

Separate permissions:

• push images,
• pull images,
• delete images,
• mutate tags,
• manage repositories,
• configure scanning,
• configure replication,
• manage lifecycle policy,
• read scan results,
• administer registry.

Typical roles:

Do not give broad registry admin access to every pipeline token.

17. Registry and GitOps Relationship

In GitOps, the Git repository usually stores the desired image reference.

Example:

image:
  repository: registry.example.com/quote/order-service
  tag: 1.24.7-20260711-a1b2c3d
  digest: sha256:...

Promotion may update:

• Helm values,
• Kustomize image override,
• Argo CD Application parameter,
• Flux image automation policy,
• environment-specific manifest.

A clean GitOps flow should make it clear:

• which image was approved,
• which environment was updated,
• who approved it,
• when sync happened,
• whether the running pod matches Git state,
• whether rollback means Git revert or controller rollback.

If the registry says one thing and GitOps says another, the live cluster becomes difficult to reason about.

18. Java/JAX-RS Impact

For Java/JAX-RS services, registry governance affects:

• deployment reproducibility,
• rollback safety,
• vulnerability response,
• base JDK update rollout,
• JVM flag changes baked into image,
• startup time during autoscaling,
• incident traceability,
• compliance evidence.

Example production incident:

A pod crashes after deployment.
Team rolls back to previous tag.
The previous tag was mutable and now points to a different image.
Rollback does not restore previous behavior.

Corrective control:

• immutable tags,
• digest pinning,
• release record,
• signed image,
• retained previous artifact,
• GitOps rollback.

19. PostgreSQL, Kafka, RabbitMQ, Redis, Camunda, and NGINX Impact

Registry governance is not only for Java API services.

It also affects images for:

• database migration jobs,
• Kafka consumers,
• RabbitMQ workers,
• Redis-dependent services,
• Camunda workers or engine-adjacent services,
• NGINX ingress/custom proxy images,
• sidecars,
• init containers,
• observability agents,
• security agents.

High-risk examples:

• migration Job image tag changed after approval,
• init container uses latest,
• NGINX custom image lacks CVE patching,
• sidecar image is pulled from public registry without approval,
• consumer rollback uses deleted image,
• air-gapped environment lacks replicated dependency image.

Every image in a pod is part of the production artifact surface.

20. EKS, AKS, On-Prem, and Hybrid Concerns

EKS

Verify:

• ECR repository policy,
• node IAM permission to pull images,
• cross-account ECR access,
• private ECR endpoint if used,
• registry lifecycle policy,
• ECR scanning configuration,
• replication if multi-region,
• image pull behavior through VPC endpoint/NAT.

AKS

Verify:

• ACR integration,
• kubelet managed identity permission,
• private ACR endpoint if used,
• Azure RBAC/ACR pull role,
• retention policy,
• vulnerability scanning integration,
• replication/geo-replication if used,
• private DNS for registry endpoint.

On-prem

Verify:

• internal registry availability,
• registry certificate trust,
• image mirror process,
• air-gapped sync process,
• retention and backup,
• registry disaster recovery.

Hybrid

Verify:

• pull path across network boundaries,
• firewall/proxy requirements,
• registry locality,
• replication delay,
• DNS and TLS trust,
• what happens during private connectivity outage.

21. Common Failure Modes

22. Production Debugging Workflow

When a pod cannot pull an image:

kubectl get pod <pod> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Then check:

1. Is the image reference spelled correctly?
2. Does the tag/digest exist in the registry?
3. Is the registry reachable from the node/cluster?
4. Does Kubernetes have credentials to pull?
5. Is the secret in the correct namespace?
6. Does node identity have pull permission?
7. Is private DNS resolving correctly?
8. Is TLS certificate trusted?
9. Is the image blocked by policy/admission?
10. Did retention delete the image?

For cloud-managed clusters, correlate Kubernetes events with:

• registry audit logs,
• IAM/RBAC decision logs,
• private endpoint metrics,
• DNS logs if available,
• firewall/proxy logs,
• CI/CD artifact publish logs.

23. Registry Review Checklist

Use this checklist in PR/release review.

Image identity

• Image tag is unique.
• Production image does not use latest.
• Digest is recorded or pinned where required.
• Image can be traced to commit/build.
• Release record includes image reference.

Promotion

• Image is built once and promoted.
• Environment-specific config is outside image.
• Promotion is auditable.
• Rollback artifact exists.
• GitOps state references approved image.

Security

• Image scan passed or exception documented.
• SBOM is generated if required.
• Image signing/provenance exists if required.
• Registry access follows least privilege.
• Build secrets are not baked into image.

Availability

• Registry is reachable from cluster.
• Private endpoint/DNS is configured if used.
• Replication is understood if multi-region/hybrid.
• Retention does not delete active rollback images.
• Registry outage runbook exists.

Kubernetes integration

• imagePullSecret or node identity is configured.
• Secret exists in correct namespace if needed.
• Image pull policy is intentional.
• Admission policy is compatible with release flow.
• ImagePullBackOff troubleshooting path is documented.

24. Internal Verification Checklist

Verify these in the CSG/team environment:

• Which registry or registries are used: ECR, ACR, Docker Hub, Harbor, Artifactory, Nexus, or other.
• Whether image tags are immutable.
• Whether production deployments pin digest.
• Whether images are built once and promoted.
• How image tag maps to Git commit and CI build.
• Where SBOM is generated and stored.
• Which vulnerability scanner is used.
• CVE severity threshold and exception workflow.
• Whether images are signed.
• Whether admission policy enforces trusted registry/signature/digest.
• Which identity pulls images in EKS/AKS/on-prem.
• Whether imagePullSecrets are used.
• Registry network path from cluster.
• Private endpoint or proxy requirement.
• Registry retention policy.
• Rollback image retention window.
• Registry replication or mirror strategy.
• Air-gapped or hybrid registry sync process if relevant.
• Registry outage runbook.
• ImagePullBackOff incident history.
• Ownership between backend, platform, DevOps, and security teams.

25. Key Takeaways

• A registry is a production artifact governance system, not just image storage.
• Tags are names; digests are content identity.
• Mutable tags weaken rollback, audit, and incident analysis.
• Build once, promote the same image digest.
• Registry auth, network, DNS, TLS, and retention all affect production recovery.
• Every container in a pod matters: app, init, sidecar, proxy, agent, and migration job.
• ImagePullBackOff is usually an artifact identity, permission, registry, DNS, network, or TLS problem.
• Strong registry governance improves deployment confidence, security, rollback, and compliance evidence.