Kubernetes Service Deep Dive
Service discovery, ClusterIP, NodePort, LoadBalancer, headless service, ExternalName, selector, EndpointSlice, session affinity, port, targetPort, named port, Service without selector, and production troubleshooting for Java/JAX-RS backend systems.
Kubernetes Service Deep Dive
1. Core Mental Model
A Kubernetes Service is a stable network abstraction in front of unstable pods.
Pods are not stable deployment targets:
Pod IP changes when pod is recreated.
Pod can move to another node.
Pod can become not ready.
Pod can be scaled up or down.
Pod can be replaced during rollout.
Clients should not call pod IPs directly.
Clients should call a stable Service identity:
http://quote-service.quote.svc.cluster.local:8080
The Service then routes traffic to currently selected pod endpoints.
The useful mental model:
Service = stable virtual address + selection rule + port mapping + endpoint set
For a Java/JAX-RS backend service, the Service is the stable contract inside Kubernetes:
Other service calls quote-service Service
Kubernetes resolves Service DNS
Service routes to ready pod endpoint
Pod receives request on container port
JAX-RS resource method handles request
Response flows back through the same network path
A Service does not run your application.
A Service does not create pods.
A Service does not know business logic.
A Service connects a stable network identity to a dynamic set of backend endpoints.
2. Why Kubernetes Service Exists
Without a Service, clients would need to discover pods themselves.
That would be fragile:
quote-service pod-1: 10.244.1.21
quote-service pod-2: 10.244.2.18
quote-service pod-3: 10.244.3.44
During rollout:
pod-1 terminating
pod-4 created
pod-4 not ready yet
pod-2 still ready
pod-3 crashing
A client should not need to know all of this.
The Service hides pod churn:
Client calls quote-service
Service routes only to eligible endpoints
EndpointSlice updates as pods change readiness
The Service exists to provide:
Stable name
Stable virtual IP where applicable
Load distribution across ready endpoints
Port abstraction
Selector-based endpoint discovery
DNS-based service discovery
Integration point for ingress and load balancers
Operational debugging boundary
For enterprise Java systems, this matters because service-to-service calls are common:
quote-api -> catalog-service
quote-api -> pricing-service
quote-api -> order-service
quote-api -> eligibility-service
quote-api -> workflow-service
If every service needed to track pod IPs manually, deployment would be unsafe.
3. Service Object Anatomy
A basic Service:
apiVersion: v1
kind: Service
metadata:
name: quote-service
namespace: quote
labels:
app.kubernetes.io/name: quote-service
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: quote-service
ports:
- name: http
port: 8080
targetPort: http
protocol: TCP
Key fields:
metadata.name
Stable service name inside the namespace.
metadata.namespace
Namespace scope for DNS and object lookup.
spec.type
Exposure model: ClusterIP, NodePort, LoadBalancer, ExternalName.
spec.selector
Label query used to select backing pods.
spec.ports[].port
Port exposed by the Service.
spec.ports[].targetPort
Port on the selected pod/container endpoint.
spec.ports[].name
Named port used by humans, probes, Ingress, Gateway, and targetPort references.
spec.ports[].protocol
Usually TCP. UDP and SCTP exist for specific use cases.
The selector is the link between Service and pods.
Example pod template:
metadata:
labels:
app.kubernetes.io/name: quote-service
spec:
containers:
- name: app
ports:
- name: http
containerPort: 8080
If the Service selector and pod labels do not match, the Service has no endpoints.
That is one of the most common Kubernetes routing failures.
4. Service Type Overview
Kubernetes supports several Service types.
Use them intentionally.
ClusterIP
Stable internal virtual IP inside the cluster.
NodePort
Exposes service on a static port on every node.
LoadBalancer
Requests an external load balancer from cloud/provider integration.
Headless Service
No virtual IP; DNS resolves directly to backing endpoints.
ExternalName
DNS alias to an external hostname.
A simplified decision model:
Internal service-to-service call?
Use ClusterIP.
Expose to ingress controller inside cluster?
Usually ClusterIP behind Ingress/Gateway.
Expose directly through cloud load balancer?
Use LoadBalancer only when intentionally needed.
Need stable DNS per pod for StatefulSet?
Use Headless Service.
Need Kubernetes DNS alias to external service?
Use ExternalName carefully.
For most Java/JAX-RS microservices, the default should be:
Deployment + ClusterIP Service + Ingress/Gateway/API Gateway if external exposure is needed
Do not expose every application as LoadBalancer by default.
That usually increases cost, attack surface, and operational complexity.
5. ClusterIP Service
ClusterIP is the default Service type.
spec:
type: ClusterIP
It creates an internal virtual IP reachable inside the cluster.
Example:
Service name: quote-service
Namespace: quote
ClusterIP: 10.96.42.17
Port: 8080
Clients can call:
http://quote-service:8080
http://quote-service.quote:8080
http://quote-service.quote.svc.cluster.local:8080
The exact search behavior depends on pod DNS configuration, but the fully qualified form is:
<service-name>.<namespace>.svc.cluster.local
ClusterIP is suitable for:
Internal REST APIs
Internal gRPC services
Internal admin services where access is controlled
Backend services behind ingress
Service-to-service communication
For Java/JAX-RS:
JAX-RS app binds to 0.0.0.0:8080 inside pod
Service exposes port 8080
Other workloads call Service DNS
A common mistake:
Java app binds to 127.0.0.1 only
Service targetPort points to 8080
Traffic reaches pod network namespace
No process is listening on pod IP:8080
Client sees connection refused or timeout
Inside containers, backend servers should usually bind to:
0.0.0.0
not only:
localhost
127.0.0.1
6. NodePort Service
NodePort exposes a Service on a static port on each node.
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: http
nodePort: 30080
Traffic can reach:
<node-ip>:30080
Then Kubernetes routes it to a backend pod endpoint.
NodePort is often used as a lower-level building block for cloud load balancers.
But direct NodePort exposure is usually not preferred for enterprise application exposure.
Reasons:
Harder firewall management
Node IP dependency
Large exposed node surface
Port range constraints
Less clean TLS/routing model
Operationally awkward in autoscaled clusters
Use NodePort deliberately.
For most Java/JAX-RS services:
Do not expose REST API directly through NodePort unless platform architecture requires it.
Prefer Ingress/Gateway/API Gateway/LoadBalancer-managed edge.
Internal verification matters because some on-prem or legacy Kubernetes environments may rely on NodePort behind external load balancers.
7. LoadBalancer Service
LoadBalancer asks the infrastructure provider to provision an external or internal load balancer.
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: http
In cloud environments, this may create resources such as:
AWS Network Load Balancer
AWS Classic Load Balancer in older setups
Azure Load Balancer
Provider-specific forwarding rules
Public or private IP
Cloud target pools or target groups
A LoadBalancer Service is powerful but expensive and broad.
Use it for:
Ingress controller exposure
Gateway controller exposure
Specific infrastructure services
Explicitly approved direct service exposure
Avoid this anti-pattern:
Every microservice gets its own public LoadBalancer.
Why this is bad:
Cost grows quickly
TLS management fragments
Ingress policy fragments
Security groups/NSGs become harder to audit
DNS management becomes noisy
External attack surface expands
Traffic governance becomes inconsistent
Common production pattern:
External cloud load balancer
-> ingress controller / gateway controller
-> ClusterIP Service
-> Java pods
LoadBalancer Service belongs in the platform/edge discussion, not as a default choice in every backend PR.
8. Headless Service
A headless Service has no ClusterIP.
spec:
clusterIP: None
Instead of resolving to a virtual Service IP, DNS returns pod endpoint addresses directly.
This is useful for workloads that need stable network identity per pod.
Common examples:
StatefulSet members
Databases
Message brokers
Consensus systems
Peer discovery
Cluster membership protocols
Example StatefulSet DNS pattern:
postgres-0.postgres-headless.db.svc.cluster.local
postgres-1.postgres-headless.db.svc.cluster.local
rabbitmq-0.rabbitmq-headless.messaging.svc.cluster.local
For stateless Java/JAX-RS REST APIs, headless Service is usually not needed.
For stateful dependencies:
RabbitMQ cluster nodes may need stable peer DNS.
Kafka brokers may require careful advertised listener configuration.
Redis cluster/sentinel may need predictable endpoint behavior.
PostgreSQL HA setups may use operator-managed Services.
Do not use headless Service just because it seems more direct.
You give up the normal Service virtual IP abstraction and push more endpoint behavior to clients.
9. ExternalName Service
ExternalName maps a Kubernetes Service name to an external DNS name.
apiVersion: v1
kind: Service
metadata:
name: external-pricing-api
namespace: quote
spec:
type: ExternalName
externalName: pricing.example.internal
A pod resolving:
external-pricing-api.quote.svc.cluster.local
gets a CNAME-like alias to:
pricing.example.internal
ExternalName can be useful for:
Migration compatibility
Temporary abstraction over external service DNS
Keeping app config Kubernetes-native
But it has caveats:
No Kubernetes endpoint health
No load balancing by Kubernetes
No port mapping in the same sense as normal Services
TLS hostname mismatch risk
DNS-only abstraction
Harder observability at Kubernetes layer
For Java clients using TLS, hostname verification matters.
If app calls:
https://external-pricing-api.quote.svc.cluster.local
but the certificate is issued for:
pricing.example.internal
TLS verification may fail.
Prefer clear external endpoint configuration unless there is a strong reason to use ExternalName.
10. Selector and Label Matching
The Service selector determines which pods receive traffic.
Example:
spec:
selector:
app.kubernetes.io/name: quote-service
app.kubernetes.io/component: api
Only pods with both labels match:
metadata:
labels:
app.kubernetes.io/name: quote-service
app.kubernetes.io/component: api
Common selector failure:
Deployment label:
app: quote-api
Service selector:
app: quote-service
Result:
Service exists
ClusterIP exists
DNS resolves
No endpoints
Traffic fails
This failure is deceptive because many objects look healthy:
kubectl get service quote-service
Looks fine
kubectl get deployment quote-service
Looks fine
kubectl get pods
Pods running
kubectl get endpointslice
Empty or no matching endpoints
Always debug Service routing by checking selectors and endpoints, not only Service existence.
11. Endpoint and EndpointSlice
A Service routes to endpoints.
Historically Kubernetes had Endpoints objects.
Modern Kubernetes uses EndpointSlice for scalable endpoint tracking.
The flow:
Deployment creates pods
Pods get labels
Service selector matches pod labels
EndpointSlice controller creates EndpointSlices
Only ready endpoints are generally used for traffic
kube-proxy/dataplane programs routing rules
Example observation:
kubectl get endpointslice -n quote -l kubernetes.io/service-name=quote-service
A healthy EndpointSlice should show backend addresses and ports.
If Service DNS resolves but there is no endpoint, request routing fails.
Common reasons for missing endpoints:
Selector mismatch
Pods not ready
Pods in wrong namespace
Pods have missing labels
Service points to wrong port name
Readiness probe failing
Deployment has zero replicas
Rollout stuck
EndpointSlice is the bridge between stable Service identity and dynamic pod instances.
Senior engineers should inspect EndpointSlice early during service routing incidents.
12. Port, TargetPort, and Named Port
Service port configuration has three concepts:
port
The port exposed by the Service.
targetPort
The port on the pod/container endpoint.
containerPort
The port declared in the pod container spec.
Example:
containers:
- name: app
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
spec:
ports:
- name: http
port: 80
targetPort: http
Meaning:
Client calls Service on port 80.
Service routes to pod named port http.
Named port resolves to containerPort 8080.
Java app must listen on 8080.
Named target ports are recommended because they decouple Service from numeric container port.
Example benefit:
Container port changes from 8080 to 8081.
Pod named port remains http.
Service targetPort remains http.
Ingress remains stable.
But named ports can also fail.
Failure case:
Service targetPort: http
Pod port name: web
Result:
Endpoint port resolution fails or routes incorrectly depending on configuration.
Traffic does not reach expected listener.
Review rule:
Service port, targetPort, containerPort, named port, app bind port, readiness probe port, and ingress backend port must form one consistent chain.
13. Service Discovery and DNS Names
Kubernetes creates DNS records for Services.
Within the same namespace:
http://quote-service:8080
Across namespaces:
http://quote-service.quote:8080
Fully qualified:
http://quote-service.quote.svc.cluster.local:8080
Use short names carefully.
If an app in namespace order calls:
http://quote-service:8080
Kubernetes first searches in the caller namespace.
That may resolve to:
quote-service.order.svc.cluster.local
not:
quote-service.quote.svc.cluster.local
If multiple namespaces contain similarly named services, short names can cause wrong-target calls.
For enterprise systems, prefer explicit namespace in cross-namespace calls:
quote-service.quote.svc.cluster.local
or at least:
quote-service.quote
This is especially important in environments with:
Multiple tenants
Multiple product domains
Preview environments
Blue/green namespaces
Shared platform services
14. Service Without Selector
A Service can exist without a selector.
apiVersion: v1
kind: Service
metadata:
name: external-database
spec:
ports:
- name: postgres
port: 5432
targetPort: 5432
Then endpoints are managed separately.
Use cases:
Represent external database inside Kubernetes DNS
Represent manually managed external endpoints
Migration bridge
Hybrid service abstraction
But this pattern requires extra governance.
Risks:
Endpoint drift
Manual endpoint update failure
No pod readiness integration
Misleading Kubernetes abstraction
Harder ownership
Security review ambiguity
If used, document clearly:
Who owns the external endpoint?
How health is checked?
How DNS and endpoint changes are tracked?
How TLS hostname verification works?
What happens during failover?
For PostgreSQL, Kafka, RabbitMQ, Redis, or external enterprise APIs, a selectorless Service may be useful but should not hide critical operational facts.
15. Session Affinity
Kubernetes Service supports client IP based session affinity:
spec:
sessionAffinity: ClientIP
This attempts to route requests from the same client IP to the same backend pod for a period.
Use it carefully.
It is not a replacement for proper stateless service design.
For Java/JAX-RS APIs, prefer statelessness:
No in-memory session dependency
No local cache correctness dependency
No pod-local workflow state
No pod-local transaction continuation
Session affinity can create problems:
Uneven load distribution
Hot pods
Sticky failure during pod degradation
Unexpected behavior behind proxies
Reduced rollout smoothness
If a service needs affinity, ask why.
Sometimes the real issue is:
Missing distributed cache
Stateful workflow hidden in REST layer
Improper client retry behavior
Legacy session design
Non-idempotent API behavior
For enterprise systems, affinity should be an explicit architecture decision, not a default.
16. Service and Readiness
A pod should only receive Service traffic after it is ready.
Readiness is the application-level gate.
Flow:
Pod starts
Readiness probe fails initially
Pod not included as ready endpoint
Application warms up
Readiness probe passes
EndpointSlice marks endpoint ready
Service can route traffic
If readiness is too optimistic:
Pod receives traffic before caches initialized
Database pool not ready
Kafka metadata client still initializing
JAX-RS server accepts traffic but dependencies fail
Rollout causes 5xx spike
If readiness is too strict:
Pod never becomes ready
Service has no endpoints
Rollout stalls
Traffic drains away from healthy but dependency-degraded pods
Do not confuse readiness with dependency perfection.
A good readiness endpoint answers:
Can this pod safely receive normal traffic right now?
not:
Is every downstream dependency globally perfect?
The Service depends on readiness to protect traffic.
Bad readiness design becomes a Service routing incident.
17. Service and Ingress/Gateway Relationship
Ingress and Gateway usually route traffic to Services.
Pattern:
Client
-> DNS
-> Load Balancer
-> Ingress Controller / Gateway
-> Kubernetes Service
-> EndpointSlice
-> Pod
Ingress does not normally route directly to arbitrary pods.
It routes to Service backends.
This means Service correctness is required even when external traffic appears to be an ingress problem.
A 503 from ingress may be caused by:
Service has no endpoints
Service port mismatch
TargetPort mismatch
Pods not ready
Backend protocol mismatch
NetworkPolicy blocking ingress controller to pod
During ingress incidents, always inspect the Service layer.
18. Java/JAX-RS Backend Implications
18.1 Bind Address
Inside Kubernetes, the Java server must listen on the pod network interface.
Good:
0.0.0.0:8080
Dangerous:
127.0.0.1:8080
localhost:8080
If the server binds only to localhost, the process may look healthy from inside itself but fail from Service traffic.
18.2 Port Consistency
For JAX-RS service, verify the chain:
Application server port
Container port
Readiness probe port
Liveness probe port
Service targetPort
Service port
Ingress/Gateway backend port
A mismatch at any layer can cause confusing symptoms.
18.3 Thread Pool and Connection Pool
Service load distribution does not protect a pod from internal saturation.
If a pod receives traffic but its HTTP worker pool or database pool is exhausted:
Service still routes traffic
Pod is still ready unless readiness detects degradation
Latency increases
Client timeouts increase
Retries amplify traffic
Service-level routing is not application-level backpressure.
18.4 Client DNS and Connection Reuse
Java HTTP clients often reuse connections.
A client may resolve Service DNS once, open connections, and keep them alive.
Usually this is fine because Service IP is stable for ClusterIP.
But for headless Services or external DNS, DNS caching behavior matters.
Check JVM DNS cache settings if using direct endpoint discovery or external DNS names.
18.5 Timeouts
Service routing can succeed while the application still fails due to timeout mismatch.
Example:
Client timeout: 2s
Ingress timeout: 30s
Service routing: fine
JAX-RS server timeout: 60s
Database timeout: 45s
This causes premature client failure and wasted backend work.
Service design must be reviewed with application timeout chains.
19. PostgreSQL, Kafka, RabbitMQ, Redis, Camunda, and NGINX Implications
PostgreSQL
For PostgreSQL, avoid hiding operational reality behind a vague Service.
Ask:
Is PostgreSQL inside Kubernetes or external managed service?
Is Service selector-based or selectorless?
Is there a primary/replica routing model?
Does failover change endpoint behavior?
Does DNS TTL matter?
Is TLS hostname verification aligned?
A simple Service name can hide failover and connection pooling complexity.
Kafka
Kafka is sensitive to advertised listeners.
A Service can expose a broker, but the broker may advertise an address clients cannot reach.
Check:
Bootstrap Service
Broker-specific Services
Headless Service for broker identity
Advertised listeners
TLS/SASL config
NetworkPolicy egress
Client DNS behavior
Kafka connectivity bugs are often not just Service bugs.
They are address advertisement bugs.
RabbitMQ
RabbitMQ may use Services for:
Client AMQP access
Management UI
Cluster peer discovery
Headless Service for StatefulSet nodes
Check whether clients connect to a stable Service and whether cluster nodes use stable identities.
Redis
Redis standalone, Sentinel, and Redis Cluster have different endpoint models.
A single Service may be insufficient for cluster topology awareness.
Check:
Standalone vs Sentinel vs Cluster
Headless Service usage
Client library topology support
Failover endpoint behavior
Camunda-like Workloads
Camunda-style systems may include:
REST API
Job workers
External task workers
Database dependency
Message broker dependency
Each component may need different Service exposure and readiness semantics.
Do not assume one Service pattern fits all Camunda-adjacent workloads.
NGINX / Ingress
NGINX Ingress routes to Services.
Check:
Ingress backend service name
Backend service port
Endpoint availability
Protocol expectations
Timeouts
Headers
Path rewrite
Many NGINX 502/503 issues are actually Service/Endpoint issues.
20. EKS, AKS, On-Prem, and Hybrid Considerations
EKS
In EKS, Service behavior interacts with AWS networking.
Check:
VPC CNI behavior
Pod IPs from VPC subnets
AWS Load Balancer Controller
NLB/ALB integration
Security group rules
Target type: instance vs IP
Internal vs internet-facing load balancer
Route 53 DNS
ACM certificate path
A LoadBalancer Service may provision an AWS load balancer depending on annotations and controller setup.
Internal verification is required because AWS behavior depends on cluster add-ons, controller versions, annotations, and platform standards.
AKS
In AKS, Service behavior interacts with Azure networking.
Check:
Azure CNI or kubenet
Azure Load Balancer
Internal vs public load balancer
NSG rules
UDR route behavior
Application Gateway / AGIC if used
Private DNS zone
Azure DNS
A LoadBalancer Service may create an Azure Load Balancer frontend.
Review whether the service is supposed to be public or internal.
On-Prem
On-prem clusters may not have automatic cloud LoadBalancer provisioning.
Options may include:
MetalLB
External hardware/software load balancer
NodePort behind enterprise LB
Manual DNS
Ingress controller with static VIP
Do not assume type: LoadBalancer behaves the same on-prem as in EKS/AKS.
Hybrid
Hybrid environments create cross-boundary service discovery issues.
Questions:
Can pods resolve on-prem DNS?
Can on-prem systems resolve Kubernetes-exposed DNS?
Is traffic routed through VPN, Direct Connect, ExpressRoute, proxy, or firewall?
Are Service IP ranges routable outside cluster?
Is access through ingress/load balancer instead?
ClusterIP is usually not directly reachable from outside the cluster.
Hybrid access should be designed explicitly.
21. Common Failure Modes
21.1 Service Has No Endpoints
Symptoms:
Ingress returns 503
Client connection refused or timeout
kubectl get service looks fine
kubectl get pods shows pods running
Likely causes:
Selector mismatch
Pods not ready
Wrong namespace
Deployment scaled to zero
Readiness probe failing
Label changed during refactor
Debug:
kubectl get svc -n quote quote-service -o yaml
kubectl get pods -n quote --show-labels
kubectl get endpointslice -n quote -l kubernetes.io/service-name=quote-service
21.2 Wrong TargetPort
Symptoms:
DNS resolves
Service has endpoints
Connection refused
Ingress 502
Likely causes:
Service targetPort points to wrong number
Named port mismatch
App listens on different port
Management port used instead of API port
Debug:
kubectl get svc -n quote quote-service -o yaml
kubectl get pod -n quote <pod> -o yaml
kubectl exec -n quote <pod> -- ss -lntp
21.3 Pod Not Listening on Pod IP
Symptoms:
App works with localhost inside pod
Service traffic fails
Likely cause:
Java app bound to 127.0.0.1 instead of 0.0.0.0
Debug:
kubectl exec -n quote <pod> -- ss -lntp
Look for:
127.0.0.1:8080
instead of:
0.0.0.0:8080
21.4 Short DNS Name Resolves Wrong Service
Symptoms:
Request reaches unexpected service
Only fails in certain namespace
Works locally or in another namespace
Likely cause:
Short service name resolves inside caller namespace first
Fix:
Use namespace-qualified service DNS for cross-namespace calls.
21.5 LoadBalancer Exposed Publicly by Mistake
Symptoms:
Unexpected public IP
External scanners hit service
Security review fails
Unexpected cloud cost
Likely cause:
Service type LoadBalancer without internal annotation or platform-approved exposure
Fix:
Use ClusterIP behind approved ingress/gateway unless direct LB is required.
22. Debugging Workflow
When Service routing fails, debug in layers.
Step 1: Confirm Service Exists
kubectl get svc -n quote quote-service
kubectl describe svc -n quote quote-service
Check:
Type
ClusterIP
Ports
Selector
Events
Step 2: Check Pod Labels
kubectl get pods -n quote --show-labels
Verify labels match Service selector exactly.
Step 3: Check EndpointSlice
kubectl get endpointslice -n quote -l kubernetes.io/service-name=quote-service -o wide
If empty, routing cannot work.
Step 4: Check Readiness
kubectl get pods -n quote
kubectl describe pod -n quote <pod>
Look for readiness probe failures.
Step 5: Check Port Chain
kubectl get svc -n quote quote-service -o yaml
kubectl get deploy -n quote quote-service -o yaml
Verify:
Service port
Service targetPort
Pod named port
Container port
Application bind port
Step 6: Test from Inside Cluster
Use a temporary debug pod if allowed:
kubectl run tmp-curl -n quote --rm -it --image=curlimages/curl -- sh
Then:
curl -v http://quote-service.quote.svc.cluster.local:8080/health/ready
Step 7: Check Application Listener
kubectl exec -n quote <pod> -- ss -lntp
Verify the app listens on expected port and address.
Step 8: Check NetworkPolicy
If endpoints exist and app listens but traffic times out, inspect NetworkPolicy.
kubectl get networkpolicy -n quote
Step 9: Check Ingress/Gateway
If internal Service works but external traffic fails, move upward to Ingress/Gateway.
23. Observability Concerns
Service-level observability should answer:
Does Service have ready endpoints?
Is traffic reaching pods?
Which backend pods receive traffic?
Are errors concentrated on specific pods?
Is latency caused by routing, app, dependency, or client retry?
Useful signals:
Endpoint count
Ready endpoint count
Pod readiness state
Ingress/backend 5xx by Service
Request rate by Service
Latency by Service
Connection errors
DNS resolution errors
NetworkPolicy deny events if available
For Java/JAX-RS apps, correlate:
HTTP server metrics
JAX-RS request latency
Thread pool saturation
Connection pool saturation
GC pause
Pod readiness transitions
Deployment rollout events
A Service does not emit rich application metrics by itself.
You need application, ingress, and platform metrics together.
24. Security Concerns
Services can unintentionally expand access.
Review:
Is this Service internal only?
Is it exposed through LoadBalancer?
Is ingress restricted?
Does NetworkPolicy limit callers?
Does namespace boundary matter?
Is the Service name discoverable by workloads that should not call it?
Is TLS required at ingress or service mesh layer?
A ClusterIP Service is not public, but it is not automatically private from all pods.
Without NetworkPolicy or service mesh policy, many clusters allow broad east-west connectivity.
For sensitive APIs:
Use authentication and authorization at application layer.
Use NetworkPolicy for network-level restriction where supported.
Use TLS/mTLS where required by platform standard.
Avoid relying on Service obscurity.
Do not treat Service DNS name as a security boundary.
25. Performance Concerns
Service routing adds an abstraction layer, but most performance issues are not caused by the Service itself.
Common real causes:
Pod CPU throttling
JVM GC pause
Thread pool exhaustion
Connection pool exhaustion
Client retry storm
Ingress timeout mismatch
DNS caching issue
NetworkPolicy/dataplane overhead in extreme cases
Cross-zone traffic
Overloaded backend pod
Still, Service design can affect performance:
Session affinity can create hot pods.
Headless Service can shift load balancing to clients.
LoadBalancer cross-zone routing can add latency/cost.
Wrong readiness can send traffic to cold pods.
Too few replicas reduces endpoint diversity.
For Java systems, always analyze Service performance with:
request rate
latency percentile
pod CPU/memory
GC
HTTP worker threads
DB/client pool
retry rate
endpoint distribution
26. Cost Concerns
ClusterIP Services are low-cost abstractions.
LoadBalancer Services can create real cloud cost.
Cost review questions:
Does this Service create a cloud load balancer?
Is it public or internal?
Is one load balancer shared through ingress/gateway enough?
Does cross-zone load balancing increase data transfer cost?
Does exposing this directly require extra WAF/TLS/DNS resources?
Is traffic better aggregated at API gateway/ingress?
Avoid turning every microservice into separately exposed infrastructure.
A Service PR can be a cost-impacting PR when type: LoadBalancer is involved.
27. Correctness Concerns
Service correctness depends on stable naming and exact selection.
Critical invariants:
Service selector matches intended pods only.
Service does not accidentally select old/new incompatible pods.
Service targetPort resolves to the correct application listener.
Readiness accurately controls endpoint eligibility.
Cross-namespace clients call the intended namespace.
ExternalName does not break TLS hostname verification.
Headless Service is used only when clients can handle endpoint-level behavior.
A subtle but dangerous failure:
Two Deployments share the same label.
One Service selector matches both.
Traffic goes to incompatible pods.
Some requests fail randomly.
Label governance is correctness governance.
28. PR Review Checklist
When reviewing a Service manifest, ask:
Is Service type appropriate?
Is ClusterIP enough?
Is LoadBalancer intentionally approved?
Is NodePort required by platform architecture?
Is headless Service justified?
Is ExternalName safe for TLS and observability?
Does selector match only intended pods?
Do pod labels match Service selector?
Is targetPort correct?
Are named ports used consistently?
Does readiness gate traffic correctly?
Does ingress/gateway backend point to the right Service and port?
Is NetworkPolicy needed for caller restriction?
Is cross-namespace DNS explicit?
Does this Service affect cloud cost?
Does this Service expose sensitive API surface?
Is there a dashboard or alert for endpoint health?
For Java/JAX-RS specifically:
Does application bind to 0.0.0.0?
Does Service target the API port, not management-only port?
Are readiness/liveness endpoints on the correct port?
Are timeouts consistent with ingress and clients?
Does the app expose any admin endpoints through the same Service?
29. Internal Verification Checklist
Verify in the CSG/team environment before making assumptions:
Which Service types are allowed by platform policy?
Are direct LoadBalancer Services allowed for application teams?
Is ingress/gateway the standard external exposure mechanism?
Which labels are mandatory for Services and pods?
Is there a namespace naming convention?
Are cross-namespace calls allowed?
Is NetworkPolicy enforced by the CNI?
Are Services generated by Helm, Kustomize, operator, or raw manifests?
Are Service names stable across environments?
Are ports standardized for API, management, metrics, and debug endpoints?
Are LoadBalancer annotations standardized for EKS/AKS/on-prem?
Are internal/public exposure rules documented?
Is EndpointSlice used in dashboards or runbooks?
Are service routing incidents documented?
Is there a standard curl/debug pod image?
Who owns DNS, ingress, and load balancer configuration?
How are service-to-service calls authenticated?
How are sensitive internal services restricted?
Do not infer CSG topology from generic Kubernetes knowledge.
Use this part as the review model, then validate actual implementation with platform/SRE/DevOps/backend teams.
30. Senior Engineer Summary
A Kubernetes Service is simple in YAML but critical in production.
The core invariant:
Stable Service identity must route only to correct, ready, compatible pod endpoints.
Most Service incidents come from broken assumptions:
Selector mismatch
Wrong targetPort
Pods not ready
Wrong namespace DNS
Unexpected LoadBalancer exposure
Headless Service used without client awareness
Ingress blamed while Service has no endpoints
For enterprise Java/JAX-RS systems, Service review is not just networking trivia.
It affects:
API reachability
Rollout safety
Zero-downtime deployment
Security boundary design
Cloud cost
Observability
Incident debugging
Cross-service correctness
A senior engineer should be able to trace:
Client DNS name
-> Service
-> EndpointSlice
-> Pod IP
-> Container port
-> Java listener
-> JAX-RS endpoint
and identify where the chain is broken.
You just completed lesson 17 in build core. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.
Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.