Series MapLesson 52 / 60
Focus mode active/Press Alt+Shift+R to toggle/Esc to exit
Final StretchOrdered learning track

On-Prem and Hybrid Deployment

Hybrid deployment untuk enterprise systems: on-prem, cloud, corporate firewall, private DNS, internal CA, proxy, air-gapped deployment, cloud control plane dependency, SDK from on-prem, private access, dan hybrid observability.

17 min read3227 words
PrevNext
Lesson 5260 lesson track51–60 Final Stretch
#aws#azure#on-prem#hybrid+11 more

Part 052 — On-Prem and Hybrid Deployment

Target pembaca: Senior Java/JAX-RS backend engineer yang perlu memahami constraint enterprise ketika aplikasi tidak hanya berjalan di public cloud, tetapi juga terhubung ke on-prem, corporate network, private DNS, firewall, proxy, internal CA, atau customer-controlled environment.

Hybrid deployment adalah salah satu area paling sering diremehkan oleh cloud-native engineer.

Di cloud murni, banyak asumsi terasa natural:

DNS works
outbound internet exists
cloud metadata endpoint exists
managed identity works
certificate chain is public
container registry reachable
logs can be pushed
control plane is available

Di enterprise hybrid/on-prem environment, asumsi itu bisa salah.

Aplikasi Java/JAX-RS yang sempurna di EKS/AKS bisa gagal total ketika berjalan di environment yang memiliki:

corporate firewall
forward proxy
no direct internet
private DNS only
internal certificate authority
restricted outbound allowlist
air-gapped deployment
customer-managed Kubernetes
on-prem PostgreSQL/Kafka/RabbitMQ/Redis
limited cloud control plane access

Part ini membangun mental model agar backend engineer bisa memahami hybrid deployment secara production-oriented.


1. Konsep inti

On-prem system adalah workload yang berjalan di data center atau environment milik enterprise/customer, bukan di public cloud provider.

Hybrid system adalah sistem yang menggabungkan on-prem dan cloud.

on-prem only  = semua komponen di data center/customer environment
cloud only    = semua komponen di AWS/Azure
hybrid        = sebagian di on-prem, sebagian di cloud
cross-cloud   = lebih dari satu cloud provider

Hybrid bukan sekadar network connection. Hybrid berarti runtime, identity, DNS, secrets, observability, deployment, dan operational ownership berada di boundary berbeda.


2. Kenapa konsep ini ada

Hybrid deployment umum di enterprise karena:

  1. data residency,
  2. regulatory constraint,
  3. customer-controlled environment,
  4. legacy system integration,
  5. telco/BSS/OSS integration,
  6. low-latency connection ke on-prem system,
  7. existing middleware,
  8. security policy,
  9. phased cloud migration,
  10. contractual deployment model.

Untuk Quote & Order style system, hybrid bisa muncul ketika:

quote/order service runs in cloud
catalog source remains on-prem
billing integration remains on-prem
provisioning system remains on-prem
identity provider is corporate
Kafka/RabbitMQ bridge crosses network boundary
customer document storage is cloud
workflow engine is cloud but calls on-prem adapters

3. Hybrid topology patterns

Pattern A — Cloud app calls on-prem system

EKS/AKS Java service
  ↓
cloud VPC/VNet route
  ↓
VPN / Direct Connect / ExpressRoute
  ↓
corporate firewall
  ↓
on-prem API / database / broker

Typical failure:

  • firewall rule missing,
  • DNS not resolvable from pod,
  • route not advertised,
  • TLS certificate not trusted,
  • proxy required but not configured,
  • latency exceeds timeout.

Pattern B — On-prem app calls cloud service

on-prem Java service
  ↓
corporate DNS/proxy/firewall
  ↓
private/public cloud endpoint
  ↓
API Gateway/APIM/Load Balancer
  ↓
EKS/AKS service

Typical failure:

  • private DNS zone not forwarded,
  • cloud private endpoint not reachable,
  • outbound firewall blocks endpoint,
  • source IP not allowed,
  • mTLS/JWT validation fails,
  • DNS resolves public IP instead of private IP.

Pattern C — Split deployment

frontend/api in cloud
workflow adapter on-prem
messaging bridge between both
shared identity provider
central observability

Typical failure:

  • event ordering issue,
  • duplicate delivery across bridge,
  • partial outage hidden by async queue,
  • on-prem adapter cannot reach cloud config/secret service,
  • trace correlation breaks at boundary.

Pattern D — Customer-hosted product deployment

customer Kubernetes/on-prem VM
  ↓
customer PostgreSQL/Kafka/Redis
  ↓
optional cloud licensing/update/telemetry endpoint

Typical failure:

  • no internet,
  • old Kubernetes version,
  • custom ingress controller,
  • private registry requirement,
  • internal CA,
  • restricted logging/export,
  • no managed cloud identity.

4. Cloud control plane dependency

Cloud-native systems often rely on cloud control plane APIs:

AWS STS
AWS Secrets Manager
AWS Systems Manager Parameter Store
AWS KMS
Amazon ECR
Azure Entra ID token endpoint
Azure Key Vault
Azure App Configuration
Azure Container Registry
Azure Monitor

In hybrid/on-prem deployment, these may be unavailable, blocked, proxied, or too slow.

Important distinction:

control plane = create/configure/manage resource
runtime data plane = serve actual business traffic/data

A Java service that fetches secret from cloud at startup has a runtime dependency on cloud control plane availability.

If that service is deployed on-prem or in restricted network, startup can fail before serving any request.


5. AWS hybrid implementation awareness

AWS hybrid patterns often involve:

Site-to-Site VPN
Direct Connect
Transit Gateway
Virtual Private Gateway
Customer Gateway
Route 53 Resolver inbound/outbound endpoints
VPC Endpoint / PrivateLink
IAM Roles Anywhere awareness
Systems Manager hybrid activations awareness
ECR private registry access
CloudWatch agent / OpenTelemetry collector

Key AWS concerns:

  1. VPC route table must know on-prem CIDR.
  2. On-prem network must know VPC CIDR.
  3. Security Group must allow traffic from on-prem CIDR or specific source.
  4. NACL and firewall must not silently block ephemeral ports.
  5. Route 53 private hosted zone is not automatically available to on-prem unless resolver forwarding exists.
  6. VPC endpoints are regional and must be reachable from the right network path.
  7. IAM role assumption from on-prem needs a deliberate identity pattern.
  8. ECR image pull from on-prem may require internet/proxy/private connectivity strategy.

6. Azure hybrid implementation awareness

Azure hybrid patterns often involve:

VPN Gateway
ExpressRoute
Virtual WAN
Hub-spoke VNet
Azure Firewall
Private Endpoint
Private Link
Private DNS Zone
Azure DNS Private Resolver
Application Gateway / APIM private endpoint
Azure Arc awareness
Azure Monitor Agent / OpenTelemetry collector
ACR private endpoint
Key Vault private endpoint

Key Azure concerns:

  1. VNet route table/UDR must send traffic correctly.
  2. On-prem route advertisement must include Azure address space.
  3. NSG and Azure Firewall can both affect flow.
  4. Private DNS Zone must be linked/resolved from required networks.
  5. Private Endpoint creates IPs inside VNet but DNS must resolve to those IPs.
  6. Managed Identity is not automatically available outside Azure compute.
  7. Service principal/federated credential strategy may be needed for on-prem workloads.
  8. ACR/Key Vault/App Configuration private endpoint can break clients that still resolve public names.

7. Corporate firewall

Corporate firewall is often the real boundary.

Cloud engineers may see:

VPN connected
routes propagated
endpoint reachable from network test VM

But application still fails because corporate firewall policies are more specific.

Firewall rules may depend on:

  • source subnet,
  • destination IP,
  • destination FQDN,
  • protocol,
  • port,
  • TLS SNI,
  • certificate issuer,
  • application identity,
  • user/device posture,
  • time window,
  • inspection policy.

For Java services, failure can appear as:

ConnectTimeoutException
SSLHandshakeException
Connection reset
No route to host
HTTP 403 from proxy
HTTP 407 proxy authentication required

Do not collapse all of these into “network issue”. Each implies a different layer.


8. Private DNS in hybrid

Hybrid DNS is a frequent root cause.

Cloud private DNS is not automatically visible to on-prem.

On-prem DNS is not automatically visible to cloud.

You need resolver/forwarder design.

cloud workload resolves on-prem name
  ↓
cloud DNS resolver
  ↓
forwarding rule
  ↓
on-prem DNS

on-prem workload resolves cloud private endpoint
  ↓
on-prem DNS
  ↓
forwarding rule
  ↓
cloud private DNS resolver

Failure modes:

  • resolves public IP instead of private IP,
  • resolves stale IP,
  • no conditional forwarder,
  • split-horizon mismatch,
  • CoreDNS cannot resolve corporate domain,
  • private endpoint record missing,
  • search suffix changes result,
  • DNS TTL too long.

Debug from actual runtime location, not from laptop.

nslookup dependency.internal
nslookup dependency.internal <specific-dns-server>
dig dependency.internal
kubectl exec -it debug-pod -- nslookup dependency.internal

9. Internal CA and TLS trust

On-prem and enterprise environments often use internal certificate authorities.

Java does not automatically trust every corporate/internal CA unless truststore is configured.

Common Java failure:

javax.net.ssl.SSLHandshakeException:
PKIX path building failed

This usually means the server certificate chain is not trusted by the JVM truststore.

Production-safe options:

  1. include approved internal CA in container image or mounted truststore,
  2. configure JVM truststore explicitly,
  3. use platform-approved certificate bundle,
  4. automate certificate rotation,
  5. monitor certificate expiry.

Dangerous anti-patterns:

disable certificate validation
trust all certificates
ignore hostname verification
pin expired certificate
manually patch truststore in running pod

In hybrid systems, TLS trust chain is an architecture dependency.


10. Forward proxy and NO_PROXY

Enterprise networks often require outbound traffic through a proxy.

Java, AWS SDK, Azure SDK, PostgreSQL clients, Kafka clients, and HTTP clients can behave differently with proxy settings.

Common environment variables:

HTTP_PROXY
HTTPS_PROXY
NO_PROXY

Common JVM properties:

-Dhttp.proxyHost
-Dhttp.proxyPort
-Dhttps.proxyHost
-Dhttps.proxyPort
-Dhttp.nonProxyHosts

Critical issue:

cloud private endpoints and cluster-internal services should usually bypass proxy

Wrong NO_PROXY can send internal service calls to corporate proxy, causing:

  • timeout,
  • 403/407,
  • TLS interception,
  • broken service discovery,
  • failed AWS/Azure SDK call,
  • high latency.

Checklist:

localhost
127.0.0.1
*.svc
*.svc.cluster.local
cluster domain
VPC/VNet CIDR
private endpoint domains
metadata endpoint if applicable
on-prem internal domains

Use the exact internal standard; do not copy this blindly.


11. Air-gapped deployment

Air-gapped or restricted environments cannot reach public internet directly.

Implications:

  • container images must be mirrored,
  • OS packages must be preloaded,
  • Maven dependencies must be vendored or proxied,
  • Helm charts must be mirrored,
  • cloud SDK endpoints may be unreachable,
  • license checks may fail,
  • telemetry export may be blocked,
  • certificate chain may be internal,
  • time synchronization may use internal NTP,
  • vulnerability scanning must be local or offline.

For Java/JAX-RS services, avoid startup logic that assumes internet access.

Anti-pattern:

service starts
  ↓
fetch remote config from public endpoint
  ↓
fetch secret from cloud service
  ↓
call license server
  ↓
start HTTP listener

In restricted environments, service may never start.

Better pattern:

load local config/secret projection
validate required dependencies with bounded timeout
start listener with readiness=false
become ready only after required dependencies pass

12. Cloud SDK from on-prem

Using AWS SDK or Azure SDK from on-prem is possible, but runtime assumptions change.

AWS SDK from on-prem

Potential credential sources:

  • static credentials from approved secret store,
  • web identity/federation if available,
  • IAM Roles Anywhere awareness,
  • STS AssumeRole through corporate identity pattern,
  • environment/profile-based credentials for tooling only.

Concerns:

  • STS endpoint reachability,
  • region endpoint reachability,
  • proxy configuration,
  • request signing through proxy,
  • clock skew,
  • credential rotation,
  • audit evidence.

Azure SDK from on-prem

Potential credential sources:

  • service principal,
  • workload identity federation if supported by identity provider,
  • certificate credential,
  • managed identity only if running on Azure-managed compute or supported environment,
  • Azure Arc-related patterns if applicable.

Concerns:

  • Entra ID token endpoint reachability,
  • proxy configuration,
  • private endpoint DNS,
  • client secret/certificate rotation,
  • tenant/subscription scope,
  • Conditional Access impact.

Do not assume DefaultAzureCredential or AWS default credential chain will magically work in on-prem production.


13. On-prem to cloud private access

Private access from on-prem to cloud usually requires:

private network connectivity
route advertisement
firewall rules
private DNS resolution
private endpoint / private link / VPC endpoint strategy
TLS trust
identity/authentication
monitoring

Example Azure:

on-prem client
  ↓
ExpressRoute/VPN
  ↓
Azure VNet
  ↓
Private Endpoint IP
  ↓
Azure service

Example AWS:

on-prem client
  ↓
Direct Connect/VPN
  ↓
VPC route
  ↓
VPC Endpoint / PrivateLink endpoint
  ↓
AWS service or private service

The hard part is usually DNS and firewall, not the endpoint itself.


14. Cloud to on-prem private access

Cloud-to-on-prem access must account for:

  • return routing,
  • firewall rules,
  • NAT behavior,
  • source IP expectations,
  • TLS server name,
  • on-prem load balancer health,
  • corporate DNS,
  • maintenance windows,
  • latency/jitter,
  • ownership escalation.

Example:

EKS pod calls on-prem billing API
  ↓
CoreDNS resolves billing.corp.internal
  ↓
Route 53 Resolver forwards to corporate DNS
  ↓
VPC route sends packet to Transit Gateway/VPN
  ↓
corporate firewall permits source subnet
  ↓
on-prem load balancer routes to billing service

Failure at any layer may look like “billing API timeout”.


15. Hybrid observability

Hybrid observability is hard because telemetry crosses boundaries.

Questions:

  1. Where do logs go?
  2. Is telemetry allowed to leave on-prem/customer environment?
  3. Are traces sampled consistently?
  4. Is correlation ID preserved across proxy/firewall/gateway?
  5. Are cloud and on-prem clocks synchronized?
  6. Are network devices logs available to app team?
  7. Is PII redacted before export?
  8. Who owns dashboard for cross-boundary flow?

Minimum useful labels:

runtime.location = cloud | on-prem | customer
cloud.provider = aws | azure | none
network.zone = internal | dmz | private-cloud | public-cloud
k8s.cluster.name
service.name
service.version
dependency.location
dependency.type

Without these labels, hybrid incident triage becomes guesswork.


16. Deployment packaging for hybrid

Cloud-native deployment often assumes CI/CD can reach cluster and registry.

Hybrid deployment may require:

  • offline artifact bundle,
  • private container registry,
  • Helm chart package,
  • Terraform plan handed to customer/platform team,
  • signed images,
  • SBOM,
  • vulnerability report,
  • config template,
  • secret injection instruction,
  • migration script,
  • rollback package,
  • compatibility matrix.

For Java service:

application image
config schema
required env vars
required ports
health endpoints
required DNS names
required outbound endpoints
required certificates
required secrets
required database migrations
required broker topics/queues

This must be explicit. Hidden runtime dependency is the enemy of hybrid deployment.


17. Runtime configuration in hybrid

Config strategy must adapt to environment.

Cloud runtime may use:

AWS AppConfig
SSM Parameter Store
Azure App Configuration
Key Vault
Secrets Manager
Kubernetes Secret/ConfigMap projection

On-prem runtime may use:

Kubernetes Secret/ConfigMap
sealed secret
external secret operator connected to local vault
mounted file
customer secret manager
environment variables

Design principle:

application code should not require a specific cloud config service unless the deployment model guarantees it

Use abstraction carefully, but do not hide failure semantics.


18. Identity in hybrid

Hybrid identity can involve:

  • corporate IdP,
  • Azure Entra ID,
  • AWS IAM federation,
  • service principal,
  • certificate-based identity,
  • Kubernetes ServiceAccount,
  • customer LDAP/OIDC,
  • mTLS client certificate.

Important distinction:

human identity       = engineer/operator/user access
workload identity    = service runtime access
deployment identity  = CI/CD or GitOps access
integration identity = API-to-API trust

Do not reuse one identity for all four.

For production review, check:

  • who can deploy,
  • who can read secrets,
  • what identity service uses at runtime,
  • how tokens are rotated,
  • how access is revoked,
  • how audit log maps action to principal.

19. Impact ke Java/JAX-RS backend

Hybrid constraints affect Java service design directly.

Startup

Avoid unbounded startup dependency calls.

Use:

  • bounded timeout,
  • clear readiness state,
  • explicit error logs,
  • dependency health classification,
  • retry with backoff.

HTTP clients

Configure:

  • proxy,
  • timeout,
  • connection pool,
  • TLS truststore,
  • retry policy,
  • DNS refresh behavior,
  • circuit breaker.

File/object access

Cloud object storage may not be reachable from on-prem without private endpoint/proxy/public allowlist.

Presigned URL/SAS strategy must account for whether the client can reach the storage endpoint.

Messaging

Kafka/RabbitMQ across hybrid boundary requires explicit handling of:

  • latency,
  • duplicate delivery,
  • TLS trust,
  • SASL/auth,
  • broker advertised listener,
  • firewall ports,
  • consumer lag.

Database

JDBC across WAN/hybrid link is risky.

If Java service in cloud calls on-prem PostgreSQL directly, review latency, connection pool, failover, firewall idle timeout, transaction duration, and backup ownership.


20. Impact ke PostgreSQL, Kafka, RabbitMQ, Redis, Camunda, NGINX

PostgreSQL

  • Direct cross-boundary JDBC should be scrutinized.
  • Connection pool must handle firewall idle timeout.
  • Long transaction over WAN is dangerous.
  • Backup/restore ownership must be clear.

Kafka

  • Advertised listeners often break hybrid clients.
  • Broker ports must be allowed explicitly.
  • TLS/SASL config must match environment.
  • High latency affects producer ack and consumer lag.

RabbitMQ

  • Management UI access may be blocked.
  • AMQP port and TLS port must be known.
  • Heartbeat timeout must account for network jitter.
  • Federation/shovel across network boundary must be monitored.

Redis

  • Cross-boundary Redis calls are usually bad for latency-sensitive paths.
  • Firewall idle timeout can break long-lived connections.
  • Redis should not be used as a WAN coordination primitive.

Camunda

  • External task workers across hybrid boundary need timeout and idempotency.
  • Job executor placement must be deliberate.
  • Workflow integration adapters need clear retry semantics.

NGINX

  • Acts as reverse proxy boundary in many hybrid setups.
  • Must preserve headers and correlation ID.
  • TLS termination and upstream trust must be explicit.
  • Timeout values must match backend behavior.

21. Impact ke EKS/AKS/on-prem Kubernetes

Managed Kubernetes and on-prem Kubernetes differ.

EKS/AKS provide cloud integrations:

managed control plane
cloud load balancer integration
cloud identity integration
cloud storage CSI
cloud registry auth
cloud monitoring integration

On-prem Kubernetes may require:

custom ingress
MetalLB or appliance load balancer
local storage provisioner
manual registry secret
local DNS integration
custom certificate management
no managed identity
manual node lifecycle

A Helm chart that works in EKS/AKS may fail on-prem because annotations, ingress class, storage class, Service type LoadBalancer, and identity assumptions differ.

Use environment overlays.

base manifests
  ├─ overlays/aws-eks
  ├─ overlays/azure-aks
  └─ overlays/on-prem

22. Failure mode

Common hybrid failure modes:

  1. Pod cannot resolve corporate DNS name.
  2. On-prem client resolves cloud endpoint to public IP instead of private IP.
  3. Firewall allows test VM but blocks Kubernetes node subnet.
  4. Proxy intercepts TLS and Java truststore rejects certificate.
  5. NO_PROXY missing cluster/internal domains.
  6. AWS/Azure SDK cannot reach token/STS endpoint.
  7. Container image pull fails from private registry.
  8. Cloud secret/config service unreachable from on-prem.
  9. Kafka advertised listener points to unreachable hostname.
  10. PostgreSQL connection pool dies due to firewall idle timeout.
  11. RabbitMQ heartbeat timeout due to network jitter.
  12. Redis latency spike causes application thread exhaustion.
  13. Observability exporter blocked by firewall.
  14. Time sync issue causes token/signature failure.
  15. On-prem maintenance window breaks cloud workflow unexpectedly.

23. Detection signals

Track signals by boundary:

DNS resolution success/failure
connect timeout count
TLS handshake failure count
proxy 403/407 count
firewall deny logs
VPN/ExpressRoute/Direct Connect tunnel status
route propagation status
packet loss/latency/jitter
SDK credential failure
secret/config retrieval latency
registry pull failure
broker consumer lag
JDBC pool reconnects
trace gaps at boundary

Application logs should differentiate:

DNS failure
TCP connect failure
TLS failure
HTTP proxy failure
authentication failure
authorization failure
dependency timeout

Generic “dependency unavailable” is not enough.


24. Debugging flow

Use this flow for cloud ↔ on-prem issue:

1. identify source runtime location
2. identify destination endpoint and expected IP
3. resolve DNS from source runtime
4. verify route from source subnet
5. verify firewall/security group/NSG rules
6. verify proxy requirement and NO_PROXY
7. verify TLS certificate chain and SNI
8. verify application identity/token
9. verify dependency health from its side
10. verify logs on both network boundaries
11. correlate with trace/correlation ID
12. decide whether issue is DNS, route, firewall, TLS, identity, or app

For Kubernetes, always test from inside a pod in the same namespace/node class if possible.


25. Trade-off

DecisionBenefitCost/Risk
Direct cloud-to-on-prem API callsimple integrationlatency/firewall coupling
Async messaging bridgedecouples availabilityreplay/ordering complexity
Private connectivityreduces public exposureDNS/routing complexity
Public endpoint + strict autheasier reachabilityexposure and allowlist risk
Internal CAenterprise controltruststore maintenance
Proxy-based egresscentralized controlSDK/TLS/performance issues
Air-gapped packagestrong isolationoperational overhead
Cloud config servicecentralized configruntime dependency on cloud control plane

26. Correctness concern

Hybrid failures can create correctness bugs, not just downtime.

Examples:

  • timeout causes retry and duplicate order submission,
  • message bridge redelivers old event,
  • workflow task completes after timeout and is retried elsewhere,
  • stale DNS sends write to old endpoint,
  • partial connectivity makes read succeed but write fail,
  • on-prem system processes request but cloud service records failure.

Design mutating operations with:

idempotency key
external reference ID
deduplication store
retry classification
compensating action
observable state transition

27. Networking concern

Hybrid networking review must include:

  • CIDR overlap,
  • routing domain,
  • VPN/Direct Connect/ExpressRoute path,
  • firewall zones,
  • NAT behavior,
  • DNS forwarding,
  • private endpoint resolution,
  • source IP preservation,
  • asymmetric routing,
  • MTU,
  • proxy,
  • TLS inspection.

If diagram does not show DNS and firewall, it is incomplete.


28. Identity/security concern

Hybrid security review:

  • no shared static credential across environments,
  • no broad service principal/IAM user reused by all services,
  • no disabled TLS validation,
  • no public endpoint opened without approval,
  • no secret copied manually to on-prem without lifecycle,
  • no unmanaged certificate expiry,
  • no undocumented firewall bypass.

Ask security team how workload identity should be represented across on-prem/cloud boundary.


29. Performance concern

Hybrid performance is shaped by network reality.

Measure:

  • round-trip latency,
  • packet loss,
  • jitter,
  • TLS handshake time,
  • DNS lookup time,
  • proxy latency,
  • firewall idle timeout,
  • database query latency,
  • broker publish latency,
  • consumer lag,
  • trace span duration across boundary.

Avoid chatty synchronous calls across hybrid boundary.

Bad:

for each quote line:
  call on-prem pricing API

Better:

batch request
cache reference data
async integration
local projection

30. Cost concern

Hybrid cost includes hidden operational cost:

  • private connectivity circuit,
  • VPN appliance,
  • firewall throughput,
  • data transfer,
  • proxy infrastructure,
  • duplicate observability stack,
  • support effort,
  • manual deployment effort,
  • DR test effort,
  • certificate/secret management effort.

Cost review should include human operations, not only cloud bill.


31. Privacy/compliance concern

Hybrid often exists because privacy/compliance matters.

Check:

  • where PII is processed,
  • where PII is logged,
  • whether telemetry leaves customer/on-prem boundary,
  • whether object files are stored in approved location,
  • whether backup leaves region/customer environment,
  • whether support engineers can access customer data,
  • whether audit logs are retained,
  • whether encryption keys are customer-managed.

32. Observability concern

Hybrid observability should support joint triage.

Need:

  • app logs,
  • platform logs,
  • firewall logs,
  • DNS resolver logs,
  • VPN/ExpressRoute/Direct Connect status,
  • ingress/proxy logs,
  • distributed traces,
  • synthetic checks from both sides,
  • dashboard with dependency boundary.

If app team cannot see firewall deny logs, define escalation path and required evidence.


33. PR review checklist

Deployment assumptions

  • Does the service assume public internet?
  • Does it require cloud metadata endpoint?
  • Does it fetch cloud secret/config at startup?
  • Does it require external license/telemetry endpoint?
  • Is there offline/air-gapped mode?

Network

  • Source and destination CIDR known?
  • DNS resolution path known?
  • Firewall rules documented?
  • Proxy/NO_PROXY configured?
  • Private endpoint path validated?
  • TLS trust chain documented?

Identity/security

  • Runtime identity defined?
  • Deployment identity defined?
  • Secret lifecycle defined?
  • Certificate rotation defined?
  • Audit evidence available?

Application behavior

  • Timeouts bounded?
  • Retries safe?
  • Idempotency available for mutating calls?
  • Dependency health reflected in readiness?
  • Error classification specific enough?

Operations

  • On-prem/cloud ownership clear?
  • Escalation path clear?
  • Dashboard covers both sides?
  • Runbook includes DNS/firewall/proxy/TLS steps?
  • Deployment package supports target environment?

34. Internal verification checklist

Verifikasi ke platform/SRE/security/backend team:

  • Apakah Quote & Order memiliki on-prem deployment atau customer-hosted deployment?
  • Komponen mana yang cloud, on-prem, atau hybrid?
  • Apakah ada AWS Direct Connect, Azure ExpressRoute, VPN, Transit Gateway, Virtual WAN, atau hub-spoke network?
  • Apakah ada corporate firewall/proxy yang wajib dilewati?
  • Domain DNS internal apa yang harus di-resolve dari EKS/AKS?
  • Apakah on-prem bisa resolve private endpoint AWS/Azure?
  • Apakah ada internal CA yang harus masuk JVM truststore?
  • Apakah container image harus dimirror ke private registry?
  • Apakah deployment bisa air-gapped?
  • Apakah AWS/Azure SDK dipakai dari on-prem runtime?
  • Credential pattern apa yang disetujui untuk on-prem runtime?
  • Apakah cloud config/secret service boleh menjadi startup dependency?
  • Apakah PostgreSQL/Kafka/RabbitMQ/Redis/Camunda berada lintas boundary?
  • Apakah observability data boleh keluar dari on-prem/customer environment?
  • Apakah firewall deny logs bisa diakses saat incident?
  • Apakah runbook hybrid sudah pernah diuji?
  • Incident apa yang pernah terjadi terkait DNS, proxy, firewall, certificate, atau private connectivity?

35. Senior engineer mental model

Untuk hybrid deployment, jangan mulai dari “aplikasi jalan di mana”.

Mulai dari boundary.

Where does the code run?
Where is the data?
Where is identity issued?
Where is DNS resolved?
Where is traffic inspected?
Where are secrets stored?
Where are logs allowed to go?
Who owns failure at each boundary?

Hybrid architecture yang baik bukan yang paling cloud-native, tetapi yang paling eksplisit terhadap boundary dan failure semantics.


36. Ringkasan

Hybrid/on-prem deployment memaksa backend engineer memahami hal di luar kode Java:

DNS
routing
firewall
proxy
TLS trust
identity
secret/config delivery
observability
artifact packaging
operational ownership

Jika area ini tidak eksplisit, aplikasi akan terlihat benar di development tetapi gagal di customer/enterprise production environment.

Untuk senior engineer, kontribusi paling bernilai adalah membuat asumsi runtime terlihat, terukur, dan dapat diuji sebelum deployment menyentuh production.

Lesson Recap

You just completed lesson 52 in final stretch. Use the series map if you want to review the broader track, or continue directly into the next lesson while the context is still warm.

Continue The Track

Keep the momentum while the lesson is still fresh. Move backward for review or continue forward into the next concept.