external client
  -> DNS
  -> cloud/on-prem load balancer
  -> NGINX Ingress Controller Service
  -> NGINX Ingress Controller Pod
  -> generated NGINX configuration
  -> Kubernetes Service
  -> EndpointSlice
  -> Pod
  -> Java/JAX-RS endpoint

Ingress resource
IngressClass
Ingress Controller
Service
EndpointSlice
Pod readiness
TLS Secret
ConfigMap
annotation
load balancer integration
network policy

host example.company.com
path /api/orders
backend service quote-order-api:8080
TLS secret example-company-tls

watch Ingress resources
watch Service resources
watch EndpointSlice/Endpoints
watch Secret resources
watch ConfigMap resources
translate desired state into NGINX runtime config
reload/update NGINX safely
serve traffic
emit logs/metrics/events

Ingress = desired routing state
Ingress Controller = runtime implementation
NGINX = data plane that handles actual request

Pod
  -> container JVM
  -> listens on port 8080/8443
  -> exposes /health, /ready, /api/*

pod recreated
pod IP changes
readiness false
container port mismatch
network policy blocks traffic
application listens on localhost only

kubectl get pod -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
containerPort matches service targetPort
readinessProbe matches actual Java endpoint

apiVersion: v1
kind: Service
metadata:
  name: quote-order-api
spec:
  selector:
    app: quote-order-api
  ports:
    - name: http
      port: 80
      targetPort: 8080

Ingress backend servicePort 80
  -> Service port 80
  -> targetPort 8080
  -> Pod container port 8080

Ingress points to wrong Service name
Ingress uses wrong Service port
Service selector does not match Pods
Service targetPort wrong
Pods not ready, so EndpointSlice empty
application listens on different port

kubectl get svc quote-order-api -n quote
kubectl describe svc quote-order-api -n quote
kubectl get endpointslice -n quote -l kubernetes.io/service-name=quote-order-api
kubectl get endpoints quote-order-api -n quote

Service exists
but no ready endpoints
=> Ingress may return 503

label selector mismatch
readiness probe failing
container crash loop
port name mismatch
namespace mismatch
rollout not complete

kubectl get endpointslice -n <namespace>
kubectl describe endpointslice <name> -n <namespace>
kubectl get pod -n <namespace> --show-labels
kubectl describe deploy <deployment> -n <namespace>

spec:
  type: ClusterIP

NGINX Ingress Controller Pod
  -> ClusterIP Service
  -> Java/JAX-RS Pods

external load balancer
  -> nodeIP:nodePort
  -> ingress controller Service
  -> ingress controller Pod

node exposure lebih luas
firewall rule harus presisi
source IP preservation bisa tricky
port range operational overhead

DNS
  -> AWS NLB / Azure Load Balancer / cloud LB
  -> Service type LoadBalancer
  -> NGINX Ingress Controller Pods

Is ingress controller exposed using LoadBalancer, NodePort, or hostNetwork?
Is the cloud load balancer public or internal?
Is TLS terminated at cloud LB, NGINX, or both?
Is source IP preserved?
Is proxy protocol enabled?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: quote-order-api
  namespace: quote
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - quote.example.company.com
      secretName: quote-example-company-tls
  rules:
    - host: quote.example.company.com
      http:
        paths:
          - path: /api/quotes
            pathType: Prefix
            backend:
              service:
                name: quote-order-api
                port:
                  number: 80

For HTTPS request with Host quote.example.company.com
and path beginning with /api/quotes
route to Service quote-order-api port 80
using TLS material from Secret quote-example-company-tls
handled by controller for IngressClass nginx

Kubernetes itself becomes a reverse proxy
Ingress independently opens public endpoint
all controllers will automatically use this rule
path behavior is identical across controllers

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/ingress-nginx

spec:
  ingressClassName: nginx

one cluster can have multiple ingress controllers
public and internal ingress can be separate
NGINX and cloud-native ingress can coexist
migration between controllers becomes possible
wrong class can make ingress ignored

Ingress exists but no ADDRESS assigned
controller ignores resource
wrong load balancer receives traffic
internal route accidentally exposed publicly

kubectl get ingressclass
kubectl describe ingressclass nginx
kubectl get ingress -A -o wide
kubectl describe ingress <name> -n <namespace>

What IngressClass names exist?
Which one is default?
Which class is public vs internal?
Are application teams allowed to choose class freely?
Is class enforced by policy?

control plane: watches Kubernetes API and generates config
 data plane: NGINX process handles traffic

routing stale
new Ingress not applied
TLS Secret update not reflected
Service endpoint change not picked up
config reload failed
controller pod restarted
admission webhook blocks bad config

Annotation names, defaults, generated config, health behavior, canary behavior, snippets, and NGINX Plus features can differ.

Which ingress controller implementation is installed?
What image repository and tag are used?
What Helm chart is used?
Is it community ingress-nginx or F5 NGINX Ingress Controller?
Is NGINX Plus enabled?
Which annotation prefix is valid?
What controller version is deployed in each environment?

controller process
NGINX master process
NGINX worker processes
mounted service account token
mounted ConfigMap/Secret data or generated config
logs to stdout/stderr
metrics endpoint
health endpoint

kubectl get pod -n ingress-nginx
kubectl describe pod <controller-pod> -n ingress-nginx
kubectl logs <controller-pod> -n ingress-nginx
kubectl exec -n ingress-nginx <controller-pod> -- nginx -T

controller cannot watch API due to RBAC issue
controller cannot read TLS Secret across namespace
controller cannot render config due to invalid annotation
NGINX reload fails
controller pod OOMKilled
controller pod CPU throttled
controller has stale generated config

validate Ingress resources before Kubernetes accepts them
reject invalid annotations or dangerous config
prevent broken generated NGINX config

kubectl apply fails
GitOps sync fails
Argo CD shows admission webhook denial
Helm upgrade fails
Ingress not created

Error from server: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request

The failure happened before runtime traffic path.
The controller is refusing a configuration shape.
Do not debug Java service first.
Debug Ingress manifest/annotation/class first.

Is validating webhook enabled?
Which annotations are denied?
Are snippets allowed?
Does CI run the same validation before merge?
Are webhook failures visible in GitOps dashboard?

proxy timeout defaults
body size defaults
log format
real IP behavior
TLS defaults
server tokens
proxy buffers
gzip
allow/deny snippet usage

ConfigMap = platform-level defaults
Annotation = per-Ingress override
Helm values = deployment-time controller configuration

global ConfigMap says timeout 30s
one team adds annotation timeout 600s
another layer has cloud LB idle timeout 60s
result: confusing 504/499 behavior

kubectl get configmap -n <ingress-namespace>
kubectl describe configmap <controller-configmap> -n <ingress-namespace>
helm get values <release> -n <ingress-namespace>

Which ConfigMap controls global NGINX behavior?
Who owns it?
Are changes reviewed by platform/SRE?
Which settings may application teams override?
Are defaults different across dev/staging/prod?

rewrite behavior
timeout behavior
body size behavior
rate limiting
CORS
auth
TLS redirect
backend protocol
proxy buffering
canary
whitelist/allowlist
custom snippets

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"

Annotations are not portable across controllers.
Annotation semantics can change across controller versions.
Some annotations can bypass platform defaults.
Snippet annotations can become code execution at proxy configuration layer.

List all annotations on the Ingress.
Confirm the annotation prefix matches the installed controller.
Confirm the annotation is supported in the deployed version.
Confirm whether it overrides platform defaults.
Confirm whether it changes security, timeout, buffering, or auth behavior.
Confirm whether it needs platform approval.

spec:
  tls:
    - hosts:
        - quote.example.company.com
      secretName: quote-example-company-tls

tls.crt
tls.key

Ingress declares host -> TLS Secret binding
Controller reads Secret
NGINX serves certificate for matching SNI/Host

Secret missing
Secret in wrong namespace
certificate expired
private key mismatch
certificate SAN does not include host
intermediate certificate missing
controller lacks permission to read Secret
TLS terminated before NGINX so Secret unused

kubectl get secret quote-example-company-tls -n quote
kubectl describe ingress quote-order-api -n quote
openssl s_client -connect quote.example.company.com:443 -servername quote.example.company.com

Is TLS terminated at NGINX or upstream cloud LB?
Is cert-manager used?
Is certificate source ACM, Azure Key Vault, internal CA, or Kubernetes Secret?
Is certificate rotation automated?
Is expiry alert configured?

backend:
  service:
    name: quote-order-api
    port:
      number: 80

ports:
  - name: http
    port: 80
    targetPort: 8080

Ingress backend port: 8080
Service port: 80
targetPort: 8080

kubectl describe ingress <ingress> -n <namespace>
kubectl describe svc <service> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>

404 from ingress controller
unexpected default page
request reaches NGINX but no host/path match
wrong Host header
missing Ingress rule
IngressClass mismatch

curl -vk https://host.example.com/path
curl -vk -H 'Host: host.example.com' https://lb-address/path
kubectl describe ingress -A | grep -A20 host.example.com
kubectl logs -n <ingress-ns> <controller-pod>

cloud LB health check against ingress controller
Kubernetes readiness probe for ingress controller pod
Kubernetes readiness probe for Java backend pod
NGINX passive upstream failure behavior
application-level /health or /ready endpoint

AWS NLB health checks ingress controller node/pod
Kubernetes checks ingress controller readiness
NGINX proxies to Service
Service only includes ready Java pods
Java readiness depends on DB/Kafka/config/cache

What does cloud LB health check?
What does ingress controller readiness check?
What does Java service readiness check?
Are readiness probes too shallow or too deep?
Are dependency failures reflected in readiness or only in request errors?

cloud load balancer
  -> ingress controller Service
  -> selected controller pods

easier HPA
fewer pods
standard rolling update

pod placement matters
anti-affinity matters
PDB matters
node failure can reduce ingress capacity

one ingress pod per node
external LB sends traffic to nodes

predictable node-level presence
useful with hostNetwork/hostPort patterns

more pods
node exposure
rollout across many nodes
resource usage per node

Is controller deployed as Deployment or DaemonSet?
How many replicas?
Is anti-affinity configured?
Is PDB configured?
Does HPA exist?
What happens during node drain?

number of replicas
CPU limit/request
memory limit/request
worker_processes
worker_connections
TLS workload
logging volume
buffering/temp file usage
connection count
long-lived WebSocket/SSE connections
cloud LB distribution

single NGINX pod overloaded
cloud LB uneven distribution
node-level connection tracking pressure
CPU throttling
log output bottleneck
Prometheus scraping overhead
TLS handshake spike
long-lived connection exhaustion

Ingress controller HPA metric: CPU, memory, custom metric, or none?
Replica count per environment?
PDB minimum available?
Resource requests/limits?
Worker connection setting?
Dashboard for active connections and 5xx?

X-Forwarded-Proto: https
X-Forwarded-Host: quote.example.company.com
X-Forwarded-Port: 443
Forwarded: proto=https;host=quote.example.company.com

wrong redirect URL
wrong absolute link generation
wrong OpenAPI server URL
wrong callback URL
wrong cookie Secure behavior

public: /quote-order/api/v1/quotes
backend: /api/v1/quotes

NGINX 404
Java 404
wrong JAX-RS resource matched
redirect loses prefix
OpenAPI docs incorrect

audit
fraud detection
rate limiting
tenant policy
security investigation

Does Java framework trust forwarded headers?
Which proxy IP ranges are trusted?
Is external client allowed to inject X-Forwarded-*?
Does NGINX overwrite or append these headers?
Is client IP logged consistently in NGINX and Java logs?

Route 53 -> NLB -> NGINX Ingress Controller -> Service -> Pod
Route 53 -> ALB -> NGINX Ingress Controller -> Service -> Pod
Route 53 -> ALB directly to Service/Pod via AWS LB Controller

NLB vs ALB
proxy protocol
source IP preservation
security group
target type instance vs IP
ACM vs Kubernetes Secret

Azure DNS -> Azure Load Balancer -> NGINX Ingress -> Service -> Pod
Azure Front Door -> Application Gateway -> NGINX Ingress -> Service -> Pod
Azure Application Gateway Ingress Controller instead of NGINX

Front Door/Application Gateway position
TLS termination point
NSG rules
Private Link/Private Endpoint
certificate store
source IP headers

enterprise DNS -> firewall -> L4/L7 load balancer -> NGINX Ingress -> Service -> Pod
enterprise DNS -> corporate reverse proxy -> Kubernetes ingress -> Java service

DMZ boundary
internal CA
proxy chaining
firewall allowlist
split-horizon DNS
hybrid connectivity

1. Does DNS resolve to expected load balancer?
2. Does load balancer route to ingress controller?
3. Is the Ingress assigned to the correct IngressClass?
4. Does host/path match the Ingress rule?
5. Does the backend Service exist?
6. Does the Service have ready endpoints?
7. Does the backend protocol/port match?
8. Does Java/JAX-RS service receive the request?
9. Do NGINX and application logs share a correlation ID?

kubectl get ingress -A -o wide
kubectl describe ingress <name> -n <namespace>
kubectl get ingressclass
kubectl get svc -A
kubectl describe svc <service> -n <namespace>
kubectl get endpointslice -n <namespace>
kubectl get pod -n <namespace> -o wide
kubectl logs -n <ingress-ns> <controller-pod>
kubectl exec -n <ingress-ns> <controller-pod> -- nginx -T
curl -vk https://host/path
curl -vk -H 'Host: host.example.com' https://lb-address/path

Which host was requested?
Which path was requested?
Which ingress/service/upstream handled it?
What was the response status?
Was it NGINX-generated or backend-generated?
How long did the total request take?
How long did upstream take?
Which upstream address was used?
Was the request terminated by client, proxy, or backend?

request_id
traceparent
remote_addr
real_client_ip
host
request_method
request_uri
status
request_time
upstream_response_time
upstream_status
upstream_addr
bytes_sent
request_length
user_agent

public vs internal class
TLS termination point
default backend exposure
admin path exposure
snippet annotation allowance
CORS annotation usage
auth annotation usage
source IP trust
X-Forwarded-* trust
body size limits
rate limits
WAF integration
log redaction
namespace isolation

Application teams should not be able to bypass platform security defaults through arbitrary annotations without review.

controller replica count
CPU/memory limits
worker_connections
TLS handshake rate
upstream keepalive
body buffering
response buffering
large uploads/downloads
WebSocket/SSE long-lived connections
log volume
cloud LB behavior

NGINX can accept more connections than Java can process.
NGINX can hide backend slowness until queues build up.
NGINX buffering can protect backend or increase latency/disk use.
Timeouts can fail requests before Java completes work.

IngressClass is correct.
Host is correct.
Path and pathType are intentional.
Backend Service name and port are correct.
TLS Secret is correct and in same namespace.
Annotations are supported by installed controller.
Annotations do not bypass platform defaults unexpectedly.
Timeout/body size/rewrite behavior is reviewed.
Auth/CORS/rate limit behavior is reviewed.
Public/internal exposure is correct.
Service has ready endpoints.
Java/JAX-RS base path and forwarded headers are compatible.
Observability fields remain available.
Rollback plan exists.

Identify ingress controller implementation:
  - community ingress-nginx?
  - F5 NGINX Ingress Controller?
  - NGINX Plus?
  - cloud-native controller instead?

Identify exposure path:
  - public load balancer?
  - internal load balancer?
  - on-prem L4/L7 appliance?
  - API gateway before ingress?

Identify controller ownership:
  - application team?
  - platform team?
  - SRE/DevOps?
  - network/security team?

Inspect Kubernetes resources:
  - IngressClass
  - Ingress
  - Service
  - EndpointSlice
  - Secret
  - ConfigMap
  - Deployment/DaemonSet
  - HPA
  - PDB
  - NetworkPolicy

Inspect GitOps/CI/CD:
  - Helm chart
  - Kustomize overlay
  - rendered manifests
  - policy validation
  - admission webhook
  - rollout strategy
  - rollback process

Inspect observability:
  - access log format
  - controller metrics
  - cloud LB metrics
  - error dashboards
  - 499/502/503/504 alerts
  - trace correlation

Ingress resource declares desired HTTP/S routing.
IngressClass selects the controller implementation.
Controller watches Kubernetes resources.
Controller generates and reloads NGINX configuration.
NGINX handles actual request traffic.
Service and EndpointSlice determine available backend pods.
Java/JAX-RS service sees the result of all proxy decisions before it.

DNS?
cloud/on-prem load balancer?
IngressClass?
Ingress host/path?
NGINX generated config?
Service?
EndpointSlice?
Pod readiness?
Java/JAX-RS route?
Downstream dependency?

Who owns the ingress layer?
What are the platform defaults?
What can application teams override?
How are risky annotations governed?
How is TLS managed?
How are source IP and forwarded headers trusted?
How are ingress failures observed?
How is rollback performed?