Cheatsheet Kafka Part 042 — Kafka On-Prem and Hybrid Deployment

1. Core idea

Kafka on-prem and hybrid deployment is where Kafka stops being just an application dependency and becomes infrastructure with explicit operational ownership.

In cloud-managed Kafka, many broker concerns are abstracted.

In on-prem or hybrid Kafka, the organization must own more of the stack:

• server hardware or VM sizing,
• operating system tuning,
• disk layout,
• filesystem behavior,
• network routing,
• firewall rules,
• DNS,
• TLS certificates,
• security principals,
• broker configuration,
• upgrades,
• patching,
• monitoring,
• backups,
• cluster replication,
• disaster recovery,
• incident response.

For a Java/JAX-RS backend engineer, the goal is not to become the Kafka platform operator.

The goal is to understand enough of the runtime to reason about latency, outages, connectivity failures, certificate failures, disk pressure, broker degradation, cross-network event flow, replay feasibility, data consistency risk, and operational ownership boundaries.

2. Deployment models

2.1 Fully on-prem Kafka

Kafka brokers run inside a company data center.

Applications may also run on-prem, or cloud services may connect into the on-prem cluster.

Common motivations:

• regulatory or data residency constraints,
• legacy integration,
• existing data center investment,
• air-gapped or restricted environments,
• tight coupling to on-prem databases or systems,
• internal platform standardization.

Operational burden is high because the organization owns the full stack.

2.2 Self-managed Kafka on private cloud

Kafka runs on private cloud or virtualization platforms.

This still behaves like self-managed Kafka.

The team owns broker lifecycle, storage performance, networking, monitoring, upgrades, access control, and incident handling.

2.3 Cloud application to on-prem Kafka

Java/JAX-RS services run in cloud, while Kafka is on-prem.

This introduces hybrid connectivity concerns:

• VPN,
• ExpressRoute/Direct Connect equivalent,
• firewall rules,
• DNS resolution,
• private routing,
• TLS termination,
• cross-network latency,
• packet loss,
• connection stability,
• maintenance windows.

2.4 On-prem application to cloud Kafka

Legacy or data-center applications produce/consume from cloud Kafka.

Common concerns:

• outbound firewall restrictions,
• proxy limitations,
• certificate truststore,
• cloud private endpoint reachability,
• egress cost,
• latency,
• retry behavior during connectivity incidents.

2.5 Cross-cluster hybrid Kafka

Separate Kafka clusters exist across on-prem and cloud, with replication between them.

Possible tools:

• MirrorMaker 2,
• Cluster Linking if available,
• Confluent Replicator,
• custom replication,
• CDC-based integration,
• event gateway patterns.

This model creates a second layer of semantics:

• source cluster offset,
• target cluster offset,
• topic name mapping,
• schema replication,
• consumer failover,
• duplicate events,
• delayed replication,
• conflict risk,
• operational ownership.

3. Why on-prem/hybrid Kafka is different

Kafka is sensitive to storage and network behavior.

On-prem/hybrid adds constraints that may not exist in managed cloud deployments:

• hardware variability,
• manual patching,
• fixed capacity procurement,
• slower expansion,
• strict firewall change process,
• certificate governance,
• network segmentation,
• shared storage contention,
• limited observability tooling,
• organizational silos,
• air-gapped release process,
• maintenance window constraints.

Kafka's programming model may look identical to developers:

bootstrap.servers=kafka01.internal:9093,kafka02.internal:9093,kafka03.internal:9093

But operational behavior can be very different.

A timeout may come from:

• broker CPU saturation,
• disk queue depth,
• ISR shrink,
• DNS split horizon,
• firewall idle timeout,
• TLS certificate expiry,
• VPN packet loss,
• proxy interference,
• cross-site latency,
• replication lag,
• broker rolling restart,
• maintenance window.

The same Java exception can hide very different root causes.

4. Broker OS tuning

Kafka depends heavily on OS behavior.

Platform/SRE usually owns this, but backend engineers should understand the questions.

4.1 File descriptors

Kafka uses many files and sockets.

Insufficient file descriptor limits can cause broker instability.

Verify:

• open file limits,
• process limits,
• socket limits,
• broker logs for too many open files.

4.2 Page cache

Kafka performance depends heavily on the operating system page cache.

Kafka writes to log segments and relies on sequential disk IO and page cache efficiency.

Memory sizing is not only JVM heap.

Important distinction:

• JVM heap is for broker process internals.
• OS memory/page cache helps Kafka serve reads and writes efficiently.

Oversizing heap can reduce useful page cache.

4.3 Swapping

Swap is dangerous for Kafka latency.

If broker JVM or OS starts swapping, request latency can spike and cluster stability can degrade.

Verify:

• swap disabled or tightly controlled,
• memory pressure alerts,
• broker GC and OS memory metrics.

4.4 Network tuning

Network tuning matters for high-throughput clusters.

Areas to verify:

• TCP buffer sizes,
• connection backlog,
• NIC capacity,
• packet drops,
• MTU consistency,
• firewall timeout behavior,
• cross-site packet loss.

4.5 Time synchronization

Time drift can break TLS validity checks, logs, monitoring correlation, and distributed debugging.

Verify NTP/chrony health.

5. Disk layout and filesystem

Kafka is storage-intensive.

Bad disk design becomes Kafka instability.

5.1 Disk type

Questions:

• SSD or HDD?
• Local disk or network-attached storage?
• Dedicated disk or shared storage?
• What are IOPS and throughput guarantees?
• What is write latency under load?
• What happens during disk failure?

Kafka generally benefits from predictable sequential IO and low-latency disks.

5.2 Separate log directories

Kafka can use multiple log directories.

Design considerations:

• distribute partition logs across disks,
• isolate Kafka logs from OS disk,
• avoid noisy neighbors,
• monitor disk usage per log directory,
• handle disk failure behavior.

5.3 Filesystem

Filesystem choice and mount options can matter.

Platform team should own this, but engineers should verify:

• supported filesystem,
• mount options,
• noatime settings if used,
• disk scheduler,
• monitoring for filesystem errors,
• free space alerts.

5.4 Disk full failure

Disk full can lead to severe broker issues.

Failure path:

1. Retention misconfigured or workload grows.
2. Disk usage rises.
3. Broker cannot write segments.
4. Producer requests fail or stall.
5. Partition availability may degrade.
6. Consumers lag.
7. Incident becomes customer-visible.

Prevention:

• disk usage alerts,
• retention review,
• topic growth dashboards,
• capacity forecasting,
• log segment monitoring,
• emergency cleanup runbook.

6. Network and firewall model

Kafka is not a single-endpoint HTTP service.

Clients connect to bootstrap brokers, fetch metadata, then connect to advertised broker addresses.

Hybrid environments often fail because firewall teams allow bootstrap but not all broker listeners.

6.1 Required network checks

From every producer/consumer network zone, verify:

• bootstrap host resolution,
• every advertised broker hostname,
• every broker listener port,
• TLS certificate hostname,
• reverse route if required,
• firewall idle timeout,
• DNS split-horizon behavior.

6.2 Firewall rules

Firewall rules should be explicit and broker-aware.

Check:

• source subnet,
• destination broker hosts,
• destination listener ports,
• protocol,
• TLS inspection policy,
• idle timeout,
• NAT behavior,
• logging enabled.

6.3 DNS

Hybrid DNS is a frequent source of Kafka breakage.

Common patterns:

• internal DNS for on-prem clients,
• private DNS for cloud clients,
• split-horizon DNS,
• broker hostnames not resolvable from cloud,
• broker certificate SAN missing cloud-visible name.

Debug from the application runtime, not from a laptop.

For Kubernetes:

kubectl exec -it <pod> -- nslookup <broker-host>
kubectl exec -it <pod> -- nc -vz <broker-host> 9093

For TLS:

openssl s_client -connect <broker-host>:9093 -servername <broker-host>

7. TLS and certificate management

On-prem/hybrid Kafka often has stricter TLS and certificate processes.

7.1 Certificate lifecycle

Track:

• broker certificate expiry,
• client certificate expiry,
• CA bundle expiry,
• intermediate CA rotation,
• truststore deployment,
• keystore deployment,
• secret synchronization,
• application restart requirements.

Certificate expiry is a predictable incident.

If it happens unexpectedly, the system lacked operational controls.

7.2 mTLS

With mTLS, clients authenticate using certificates.

Review:

• subject naming convention,
• principal mapping rules,
• certificate issuance process,
• revocation process,
• rotation process,
• environment separation,
• service-level identity.

7.3 Truststore drift

Common failure:

• broker cert rotated,
• platform trust updated,
• application truststore not updated,
• only some pods fail,
• consumers leave groups.

Avoid by:

• central secret management,
• expiry alerts,
• pre-rotation validation,
• rollout coordination,
• canary clients.

8. Security and access control

On-prem/hybrid Kafka security may use:

• TLS,
• mTLS,
• SASL/SCRAM,
• SASL/PLAIN over TLS,
• Kerberos/GSSAPI,
• LDAP-backed identity,
• OAuth/OIDC through enterprise identity,
• ACLs,
• RBAC in vendor distributions,
• network segmentation.

8.1 Least privilege

Access should be scoped by:

• service,
• environment,
• topic,
• consumer group,
• transactional ID,
• schema subject,
• connector.

Avoid:

• one shared producer user,
• one shared consumer user,
• wildcard topic access,
• shared consumer group credentials,
• manual ACLs without audit trail.

8.2 Auditability

For regulated systems, verify:

• who changed ACLs,
• who created topics,
• who changed retention,
• who modified schema compatibility,
• who reset offsets,
• who replayed DLQ,
• who restarted connectors,
• who changed broker configuration.

Auditability is part of production readiness, not a compliance afterthought.

9. Monitoring stack

On-prem Kafka requires an explicit monitoring stack.

Common components:

• JMX exporter,
• Prometheus,
• Grafana,
• Alertmanager,
• ELK/OpenSearch,
• Splunk,
• Datadog,
• vendor console,
• custom scripts,
• Kafka CLI,
• Cruise Control if used.

9.1 Broker metrics

Monitor:

• under-replicated partitions,
• offline partitions,
• ISR shrink/expand rate,
• active controller count,
• request latency,
• request queue time,
• network processor idle percent,
• produce/fetch request rate,
• disk usage,
• log flush metrics,
• controller event queue,
• partition count per broker,
• leader count per broker,
• bytes in/out,
• failed produce/fetch requests.

9.2 Client metrics

Application teams should still emit:

• producer send rate,
• producer error rate,
• producer retry rate,
• producer latency,
• consumer lag,
• processing latency,
• commit latency,
• rebalance count,
• DLQ count,
• deserialization failures,
• idempotency conflict count.

Broker metrics alone are not enough.

9.3 Hybrid metrics

For hybrid deployments, add:

• VPN/ExpressRoute health,
• packet loss,
• cross-site latency,
• firewall drops,
• DNS errors,
• TLS handshake failures,
• replication lag,
• MirrorMaker 2 lag,
• connector lag.

10. Patch management and upgrades

Kafka upgrades are not just binary replacement.

They involve:

• broker version,
• inter-broker protocol version,
• message format version,
• client compatibility,
• rolling restart,
• controller behavior,
• KRaft/ZooKeeper considerations,
• operator version if Kubernetes,
• connector compatibility,
• schema registry compatibility,
• Kafka Streams application compatibility.

10.1 Upgrade planning questions

Ask:

• What version are brokers on?
• What client versions are used by Java services?
• Are any clients too old?
• Are there Kafka Streams apps?
• Are there transactional producers?
• Are there Connect workers?
• Are schemas affected?
• Is rollback possible?
• Has upgrade been tested in lower environments?
• What are success metrics?
• What is the maintenance window?
• What is the customer impact plan?

10.2 Rolling upgrade risk

During rolling broker restart:

• leadership moves,
• clients reconnect,
• producers retry,
• consumers may see fetch latency,
• ISR can shrink,
• under-replicated partitions may appear temporarily,
• poorly tuned clients may fail faster than intended.

Application teams should ensure retry behavior is sane before platform upgrades.

11. Air-gapped deployment

Air-gapped or restricted environments add constraints.

Common challenges:

• no direct internet access,
• restricted container image registry,
• offline package distribution,
• controlled certificate authority,
• manual artifact promotion,
• delayed patching,
• limited external observability,
• restricted vendor support access,
• stricter change approval.

For Kafka ecosystem tools, verify offline availability for:

• broker binaries/images,
• Kafka Connect plugins,
• Debezium connectors,
• Schema Registry,
• ksqlDB,
• monitoring exporters,
• CLI tools,
• security patches.

Air-gapped does not mean static.

It means change control is slower and must be more deliberate.

12. Hybrid event flow patterns

12.1 Cloud-to-on-prem event flow

Example:

1. Java/JAX-RS service runs in cloud.
2. Service writes cloud database.
3. Service publishes event to on-prem Kafka.
4. On-prem consumer integrates with legacy system.

Risks:

• cloud-to-on-prem network outage,
• producer retries exhaust,
• DB commit succeeds but publish fails,
• outbox backlog grows,
• legacy system receives events late,
• reconciliation required.

Correctness pattern:

• use outbox,
• publish asynchronously,
• monitor outbox lag,
• make consumer idempotent,
• define business SLA for propagation delay.

12.2 On-prem-to-cloud event flow

Example:

1. On-prem system writes local database.
2. CDC or producer publishes to on-prem Kafka.
3. Replication sends event to cloud Kafka.
4. Cloud service updates read model.

Risks:

• replication lag,
• duplicate events,
• schema mismatch,
• offset translation,
• cloud consumer sees delayed data,
• stale read model.

Correctness pattern:

• include event ID,
• include event time and source system,
• make projection replay-safe,
• monitor replication lag,
• document staleness SLA.

12.3 Bidirectional event flow

This is much harder.

Risks:

• event loops,
• duplicate propagation,
• conflicting updates,
• split-brain-like business state,
• unclear source of truth,
• compensation complexity.

Avoid active-active bidirectional event flow unless business ownership and conflict resolution are explicitly designed.

13. Cross-cluster replication

13.1 MirrorMaker 2

MirrorMaker 2 can replicate topics across Kafka clusters.

Understand:

• source cluster,
• target cluster,
• topic rename policy,
• offset sync topic,
• checkpoint topic,
• heartbeat topic,
• replication lag,
• consumer group offset translation,
• ACL/config replication limitations,
• operational monitoring.

MirrorMaker 2 is not magic DR.

It replicates records, but your application still must handle:

• duplicate records,
• delayed replication,
• failover,
• offset translation,
• schema registry,
• topic configuration drift,
• consumer state,
• producer routing.

13.2 Cluster Linking

Cluster Linking, where available, can provide broker-level topic replication semantics in specific Kafka distributions.

Verify:

• availability in your Kafka distribution,
• licensing,
• supported topology,
• topic config behavior,
• offset behavior,
• failover model,
• operational tooling,
• limitations.

Do not assume it exists in open-source Apache Kafka unless the platform explicitly supports it.

13.3 Schema replication

Topic replication without schema replication can break consumers.

Verify:

• Schema Registry replication,
• subject naming strategy,
• schema ID portability,
• compatibility settings,
• schema promotion workflow,
• DR restore process.

13.4 ACL and credential replication

Failover is incomplete if applications cannot authenticate or authorize against the target cluster.

Verify:

• principals exist in target,
• ACLs/RBAC exist in target,
• certificates/secrets are valid,
• DNS/bootstrap can switch,
• clients can reach target brokers.

14. Operational responsibility boundary

This is the most important non-technical artifact.

Define ownership explicitly.

Ambiguous ownership creates incidents.

If nobody owns DLQ replay, data is silently lost.

If nobody owns schema compatibility, consumers break.

If nobody owns connector lag, databases accumulate WAL.

If nobody owns certificate rotation, the event backbone fails on a calendar date.

15. Java/JAX-RS backend implications

15.1 Treat Kafka as remote infrastructure

On-prem/hybrid Kafka may have higher and more variable latency than in-cluster Kafka.

Tune:

• request timeout,
• delivery timeout,
• retry backoff,
• metadata max age,
• linger,
• batch size,
• max in-flight requests,
• reconnect backoff.

Do not set aggressive low timeouts without measuring network behavior.

15.2 Use outbox for remote publish

When publishing across network boundaries, direct synchronous publish inside business transaction becomes riskier.

Prefer:

• DB commit includes outbox row,
• publisher sends to Kafka asynchronously,
• publish retry is durable,
• outbox lag is monitored,
• business API does not block on remote Kafka unless explicitly required.

15.3 Consumer idempotency is mandatory

Hybrid replication and failover can create duplicates.

Consumers must handle:

• duplicate event ID,
• replay,
• cross-cluster duplicate,
• delayed old event,
• out-of-order event,
• partially processed event.

15.4 Correlation and traceability

Every event crossing on-prem/cloud boundary should include:

• event ID,
• source system,
• source environment,
• aggregate ID,
• correlation ID,
• causation ID,
• trace ID if available,
• event time,
• publish time,
• schema version,
• partition key.

Without metadata, hybrid incidents become guesswork.

16. Common failure modes

16.1 Broker disk full

Symptom:

• producer timeout,
• broker error logs,
• offline partitions,
• under-replicated partitions,
• consumer lag grows.

Likely causes:

• retention too long,
• unexpected volume spike,
• compacted topic not compacting fast enough,
• replication issue,
• disk alert missing.

Safe response:

• confirm affected broker/log directory,
• identify top topics by disk,
• avoid deleting logs manually without platform approval,
• reduce retention only with business approval,
• add capacity or move partitions,
• document data loss risk.

16.2 Firewall change breaks Kafka

Symptom:

• clients fail after network maintenance,
• bootstrap works but metadata broker fails,
• only some apps affected.

Likely causes:

• broker port not allowed,
• advertised listener route blocked,
• idle timeout changed,
• NAT behavior changed.

Safe response:

• test all broker endpoints from affected runtime,
• compare before/after firewall rules,
• check TLS handshake,
• coordinate with network team.

16.3 Certificate expiry

Symptom:

• SSL handshake failure,
• sudden auth failure,
• all clients fail around same time,
• some clients fail after restart.

Likely causes:

• broker cert expired,
• CA expired,
• truststore outdated,
• client cert expired.

Safe response:

• check certificate chain,
• check truststore in pod/VM,
• verify rotation history,
• roll clients safely,
• add expiry alert.

16.4 MirrorMaker replication lag

Symptom:

• target cluster stale,
• cloud read model behind,
• DR target not ready,
• consumer lag normal on target but source replication delayed.

Likely causes:

• network bandwidth,
• source topic spike,
• replication task failure,
• target throttling,
• partition imbalance.

Safe response:

• inspect replication metrics,
• inspect MirrorMaker task status,
• check network,
• compare source and target offsets,
• avoid failover until lag understood.

16.5 Schema missing in target environment

Symptom:

• consumers fail deserialization after failover,
• records replicated but unreadable,
• schema ID unknown.

Likely causes:

• schema registry not replicated,
• subject mismatch,
• schema ID mismatch,
• environment-specific registry drift.

Safe response:

• verify subject exists,
• verify compatibility,
• promote schema,
• avoid blind consumer restart loops.

16.6 Connector hidden failure

Symptom:

• business data stops syncing,
• Kafka topic stops receiving updates,
• database WAL grows,
• no app deployment occurred.

Likely causes:

• Debezium connector stopped,
• replication slot lag,
• connector task failed,
• DB permission changed,
• network path broken.

Safe response:

• inspect connector status,
• inspect source DB replication slot,
• inspect connector DLQ,
• inspect task logs,
• restart only after cause is known.

17. Capacity planning for on-prem/hybrid

Capacity planning must include more than topic throughput.

17.1 Broker capacity

Track:

• bytes in/sec,
• bytes out/sec,
• request rate,
• partition count,
• leader count,
• disk usage,
• disk throughput,
• disk latency,
• network throughput,
• CPU,
• memory/page cache,
• replication traffic.

17.2 Hybrid link capacity

Track:

• bandwidth,
• latency,
• packet loss,
• firewall throughput,
• VPN tunnel health,
• replication traffic,
• peak batch windows,
• backfill/replay traffic.

A replay can saturate hybrid links and degrade live traffic.

17.3 Retention capacity

Retention must be sized by:

• message volume,
• record size,
• replication factor,
• retention duration,
• compaction behavior,
• burst factor,
• audit requirements,
• replay requirements,
• DR replication.

Formula approximation:

storage_required =
  daily_ingest_bytes
  × retention_days
  × replication_factor
  × overhead_factor

This is only a planning approximation. Validate with real broker metrics.

18. Internal verification checklist

Use this checklist with platform/SRE/network/security/backend/data teams.

Runtime and ownership

• Is Kafka on-prem, private cloud, cloud, or hybrid?
• Who owns broker operations?
• Who owns topics?
• Who owns schemas?
• Who owns ACLs and credentials?
• Who owns Kafka Connect and CDC?
• Who owns replay and data repair?
• Who is on-call for Kafka incidents?

Broker and OS

• What Kafka version is used?
• Is the cluster KRaft or ZooKeeper-based?
• What OS and filesystem are used?
• Are file descriptor limits configured?
• Is swap disabled or controlled?
• Is time synchronization monitored?
• Are broker JVM and page cache sized correctly?
• Are broker logs monitored?

Storage

• What disk type is used?
• Are Kafka logs on dedicated disks?
• Are disk throughput and latency monitored?
• Are disk usage alerts configured?
• Are retention policies reviewed?
• Is there emergency capacity procedure?
• Are disk failure procedures documented?

Network

• Are all advertised broker addresses reachable from all producer/consumer zones?
• Are firewall rules broker-aware?
• Are DNS records valid from cloud and on-prem runtimes?
• Is TLS hostname verification passing?
• Are VPN/ExpressRoute/private links monitored?
• Are firewall idle timeouts known?
• Are packet loss and latency monitored?

Security

• Is TLS/mTLS/SASL/Kerberos/OAuth used?
• Are credentials service-specific?
• Are ACLs least privilege?
• Are certificates monitored before expiry?
• Is certificate rotation rehearsed?
• Are audit logs available for topic/schema/ACL changes?

Hybrid and replication

• Is MirrorMaker 2, Cluster Linking, or another replication mechanism used?
• Are replicated topic names documented?
• Is offset translation understood?
• Is schema replication handled?
• Are ACLs replicated or recreated?
• Are replication lag alerts configured?
• Is failover tested?

Application correctness

• Do Java/JAX-RS services use outbox for DB + publish flows?
• Are consumers idempotent?
• Are replay and duplicate scenarios tested?
• Are retries bounded?
• Are DLQ topics monitored and owned?
• Are correlation IDs propagated?
• Are event timestamps and source metadata present?

Operations

• Are runbooks available for broker down, disk full, lag, DLQ, connector failure, certificate expiry, and replication lag?
• Are dashboards accessible to backend engineers?
• Is safe offset reset procedure documented?
• Is replay approval process documented?
• Are incident notes reviewed by application teams?
• Are maintenance windows communicated?

19. PR review checklist

When reviewing Java/JAX-RS code that interacts with on-prem/hybrid Kafka, ask:

• Does the service assume low-latency broker connectivity?
• Does the producer publish synchronously in request path without justification?
• Is outbox used for DB + Kafka consistency?
• Is outbox lag observable?
• Are producer retries durable and bounded?
• Are consumer handlers idempotent?
• Are duplicate, delayed, and out-of-order events handled?
• Does event metadata include source system and environment?
• Are topic names externalized per environment?
• Are credentials/secrets injected securely?
• Does readiness avoid unsafe liveness dependency on Kafka?
• Are network/auth failures visible in logs and metrics?
• Are hybrid failure modes covered in tests or runbook?
• Does the change require platform/network/security verification?
• Is replay safe if replication or consumer processing fails?

20. Architecture decision checklist

Before approving on-prem/hybrid Kafka architecture, require clear answers.

Business and data

• What business process depends on the event flow?
• What is the source of truth?
• What is the acceptable propagation delay?
• What happens if events are delayed for one hour?
• What happens if events are duplicated?
• What happens if events arrive out of order?
• What reconciliation process exists?

Runtime

• Where are brokers located?
• Where are producers and consumers located?
• What network links are involved?
• What is the expected peak throughput?
• What is the retention requirement?
• What is the replay requirement?

Failure

• What happens if cloud cannot reach on-prem Kafka?
• What happens if on-prem cannot reach cloud Kafka?
• What happens if replication stops?
• What happens if Schema Registry is unavailable?
• What happens if a certificate expires?
• What happens if a firewall rule changes?
• What happens if a broker disk fills?

Operations

• Who receives alerts?
• Who has permission to operate Kafka?
• Who has permission to reset offsets?
• Who has permission to replay?
• Who communicates customer impact?
• Who validates business recovery?

21. Common anti-patterns

Anti-pattern: Hybrid Kafka without outbox

Publishing directly to remote Kafka after DB commit without durable outbox creates data loss risk.

Anti-pattern: Treating replication as consistency

Cross-cluster replication provides data movement, not business consistency.

Consumers still need idempotency, replay safety, and reconciliation.

Anti-pattern: No schema replication plan

Replicated topics are useless if consumers cannot deserialize records.

Anti-pattern: Shared firewall rule for "Kafka"

Kafka requires broker-aware access, not just one bootstrap endpoint.

Anti-pattern: Certificate expiry discovered by outage

Certificate expiry should be monitored and rehearsed.

Anti-pattern: No ownership for DLQ/replay

DLQ without owner is deferred incident handling.

Anti-pattern: Active-active without conflict model

Bidirectional active-active event flow without conflict resolution creates split-brain-like business risk.

Anti-pattern: Platform-only observability

Backend teams need app-level producer, consumer, idempotency, DLQ, and business-lag metrics.

22. Key takeaways

Kafka on-prem and hybrid deployment is primarily about operational clarity.

The core engineering truths:

• Kafka is storage- and network-sensitive.
• Hybrid networking makes Kafka failure modes harder to diagnose.
• Broker reliability does not solve application consistency.
• Outbox, inbox, idempotency, replay safety, and reconciliation remain mandatory.
• Topic replication does not equal DR readiness.
• Schema, ACL, credential, and offset behavior must be part of failover planning.
• Certificate and firewall changes are production events.
• Ownership boundaries must be explicit.

The senior backend engineer posture is:

Do not treat on-prem/hybrid Kafka as an opaque pipe. Treat it as a distributed runtime with explicit semantics, operational boundaries, failure modes, and business recovery obligations.