Cheatsheet Kafka Part 041 — Kafka on Azure

1. Core idea

Running Kafka workloads on Azure is not one thing.

In real enterprise systems, "Kafka on Azure" can mean several different runtime models:

1. Self-managed Apache Kafka on Azure VM
2. Self-managed Apache Kafka on AKS
3. Kafka distributed through an operator on AKS
4. Confluent Cloud on Azure
5. Azure Event Hubs with Kafka-compatible endpoint
6. Hybrid Kafka where Azure applications connect to Kafka hosted elsewhere
7. Kafka clients on AKS consuming from AWS, on-prem, or third-party managed Kafka

These are not equivalent.

They may expose similar client bootstrap semantics, but they differ in broker ownership, operational responsibility, protocol compatibility, networking model, authentication model, observability surface, scaling model, topic and partition limits, transaction support, Kafka Connect support, schema governance options, disaster recovery mechanics, and incident response ownership.

For a Java/JAX-RS backend engineer, the practical question is not only:

Can my service produce and consume Kafka records?

The better question is:

What Kafka semantics are actually available in this Azure deployment, and which operational assumptions are owned by my team versus the platform?

2. Azure deployment options

2.1 Self-managed Kafka on Azure VMs

This is the closest model to traditional bare-metal or cloud VM Kafka.

You provision VM instances, attach disks, configure network, install Kafka, configure broker properties, configure TLS/SASL, operate upgrades, monitor brokers, and manage storage.

Typical characteristics:

• maximum control,
• maximum operational burden,
• direct broker-level tuning,
• direct disk layout control,
• custom security configuration,
• custom monitoring,
• custom backup/replication strategy.

This model is common when the organization already has Kafka operational expertise, managed services cannot satisfy compliance or network constraints, the system requires specific broker configuration, or an existing on-prem Kafka practice is being migrated to Azure.

2.2 Self-managed Kafka on AKS

Kafka can run on AKS, usually through StatefulSets and persistent volumes.

This is operationally harder than running stateless Java services because Kafka brokers are stateful, storage-sensitive, and identity-sensitive.

Key concerns:

• stable broker identity,
• persistent volume performance,
• broker rack awareness,
• StatefulSet lifecycle,
• pod disruption behavior,
• rolling upgrade safety,
• storage class reliability,
• node failure behavior,
• anti-affinity,
• Kubernetes DNS,
• advertised listener configuration.

Running Kafka on AKS is possible, but it is not "just another deployment".

The failure mode is not only pod restart. It can include partition unavailability, ISR shrink, replica reassignment pressure, disk saturation, under-replicated partitions, broker identity mismatch, advertised listener breakage, controller instability, and slow state recovery.

2.3 Kafka operator on AKS

Kafka operators such as Strimzi or Confluent Operator can simplify lifecycle management.

They typically help with broker StatefulSets, listeners, certificates, topic resources, user resources, rolling upgrades, configuration reconciliation, metrics exposure, and entity operator patterns.

However, an operator does not remove the need to understand Kafka. It moves some operational complexity into declarative resources and controller behavior.

Failure questions still remain:

• What happens if the operator is down?
• What changes are reconciled automatically?
• What changes are dangerous?
• How are broker configs promoted across environments?
• How are certificates rotated?
• How are topic configs governed?
• How are user ACLs managed?
• How are upgrades tested?
• How are persistent volumes protected?

2.4 Confluent Cloud on Azure

Managed Kafka services reduce broker operations but do not remove application correctness concerns.

Confluent Cloud on Azure may provide managed Kafka clusters, Schema Registry, connectors, governance features, networking options, security controls, and operational dashboards.

From the application team perspective, the concerns shift toward client configuration, topic ownership, schema compatibility, network peering/private link, service accounts, ACLs/RBAC, quotas, cost, observability integration, incident escalation path, and cloud provider boundary.

Managed Kafka does not solve wrong partition keys, non-idempotent consumers, unsafe retry, bad schema evolution, missing correlation IDs, replay-unsafe handlers, dual-write bugs, stale read models, or inadequate DLQ runbooks.

2.5 Azure Event Hubs Kafka-compatible endpoint

Azure Event Hubs can expose a Kafka-compatible protocol endpoint.

This is useful when Kafka clients need to publish/consume using Kafka protocol while the underlying service is Event Hubs.

But "Kafka-compatible endpoint" is not the same as a full Apache Kafka broker.

Before treating it as Kafka, verify compatibility for:

• client protocol versions,
• producer idempotence,
• transactions,
• consumer group behavior,
• offset behavior,
• retention semantics,
• partition scaling,
• compaction support,
• admin APIs,
• topic creation model,
• Kafka Streams compatibility,
• ksqlDB compatibility,
• Kafka Connect support,
• Schema Registry assumptions,
• authorization model,
• monitoring metrics,
• quotas and throttling.

Do not assume every Kafka feature works just because a Kafka client can connect.

3. Decision model

Use this decision model when evaluating Kafka on Azure.

4. Azure networking mental model

Kafka networking issues are usually not random.

They come from mismatch among bootstrap server, advertised listener, DNS resolution, TLS hostname, routing, firewall/NSG rules, private endpoint configuration, client network location, and broker network location.

A Java service does not connect only to the bootstrap server forever.

The bootstrap server returns broker metadata. The client then connects to broker addresses from metadata.

If those advertised broker addresses are not reachable from the client, the service may bootstrap successfully but fail shortly after.

4.1 VNet

Azure Virtual Network defines private IP address space, subnets, routing, and isolation.

For Kafka clients, verify:

• the AKS subnet,
• broker subnet,
• private endpoint subnet,
• route table,
• DNS zone,
• NSG rules,
• peering relationship,
• firewall inspection path.

4.2 Private endpoint / Private Link

Private endpoint patterns are common for managed services.

Verify:

• DNS resolves to private IP from the pod or VM,
• TLS certificate hostname matches the expected broker endpoint,
• private endpoint is approved,
• route does not hairpin through public internet,
• firewall allows required broker ports,
• security teams understand Kafka's multi-broker connection pattern.

4.3 NSG and firewall

Kafka needs more than a single HTTP-like endpoint mental model.

A client may need to connect to multiple brokers, each on a configured listener port.

Check:

• outbound rules from AKS node subnet,
• inbound rules to broker/private endpoint,
• firewall FQDN rules,
• TLS inspection compatibility,
• ephemeral port behavior,
• cross-region routing.

4.4 DNS

DNS failure often appears as Kafka timeout or metadata errors.

Verify from the application pod:

nslookup <bootstrap-host>
nslookup <advertised-broker-host>

Then verify connectivity:

nc -vz <bootstrap-host> 9092
nc -vz <advertised-broker-host> 9092

For TLS:

openssl s_client -connect <broker-host>:9093 -servername <broker-host>

The important test is not only bootstrap. Test advertised broker hosts too.

5. Authentication and authorization on Azure

Authentication depends on the runtime.

Possible models:

• TLS client certificate,
• SASL/SCRAM,
• SASL/PLAIN over TLS,
• OAuth/OIDC,
• cloud-specific identity,
• Event Hubs connection string,
• Confluent service account,
• enterprise identity integration.

5.1 Managed identity

Managed identity may be relevant for Azure-native services, but Apache Kafka clients typically still need Kafka-compatible authentication parameters unless using a service that maps Azure identity to access control.

Do not assume "we use Azure, so managed identity handles Kafka auth." Verify the exact mechanism.

5.2 SASL

Common client configs may include:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=...

or:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=...

The important questions:

• Where is the secret stored?
• Is it in Kubernetes Secret, external secret manager, or CI/CD variable?
• Who rotates it?
• How does rotation avoid downtime?
• Is it scoped per app, team, or environment?
• Are producer and consumer principals separated?

5.3 TLS and certificates

For TLS/mTLS:

• verify CA bundle,
• verify broker certificate hostname,
• verify keystore/truststore path,
• verify certificate expiry,
• verify rotation procedure,
• verify pod reload behavior,
• verify whether service restart is required.

A common failure mode is certificate rotation without client rollout coordination.

5.4 Authorization

Kafka authorization may be ACL-based, RBAC-based, or service-specific.

Minimum permissions are usually separate for:

• topic read,
• topic write,
• topic describe,
• consumer group read,
• transactional ID,
• connector resources,
• schema registry subject read/write.

A consumer may need topic read and group permission. A producer may need topic write and describe. A transactional producer may need transactional ID permission.

Do not grant wildcard access unless explicitly justified.

6. Java/JAX-RS service implications

For Java/JAX-RS services deployed on Azure, Kafka configuration should be treated as part of runtime contract.

6.1 Configuration surface

Typical environment-specific configuration:

bootstrap.servers=${KAFKA_BOOTSTRAP_SERVERS}
security.protocol=${KAFKA_SECURITY_PROTOCOL}
sasl.mechanism=${KAFKA_SASL_MECHANISM}
schema.registry.url=${SCHEMA_REGISTRY_URL}
client.id=${SERVICE_NAME}-${ENV}
group.id=${SERVICE_NAME}-${CONSUMER_NAME}-${ENV}

Avoid hardcoding broker hostnames, topic names, credentials, schema registry URL, group IDs, or environment suffixes.

6.2 Startup behavior

A service should fail fast for mandatory Kafka configuration errors.

But there is nuance:

• A pure producer API may not need to block startup until Kafka is reachable.
• A critical consumer service may need readiness to reflect Kafka connectivity.
• A service with both HTTP and consumer roles may need separate readiness semantics.
• A service should not enter ready state if it cannot process required event streams.

6.3 Readiness and liveness

Do not make liveness depend on Kafka.

If Kafka is temporarily unavailable and liveness fails, Kubernetes may restart the pod repeatedly and create a restart storm.

Use readiness for dependency availability and liveness for process health.

6.4 Secret rotation

A Java service may need restart or dynamic reload depending on client implementation, keystore/truststore loading, SASL credential provider, deployment pattern, and external secret sync mechanism.

Internal verification is required.

7. PostgreSQL/MyBatis/JDBC implications

Kafka on Azure does not change the core correctness problem:

The database transaction and Kafka publish are still separate systems.

Even if Kafka is managed, Azure-hosted, or Event Hubs-compatible, the Java service must still handle:

• dual-write risk,
• outbox pattern,
• CDC pipeline,
• idempotent consumer,
• inbox pattern,
• replay safety,
• transaction boundary,
• compensation,
• reconciliation.

7.1 Outbox still matters

A service writing PostgreSQL and publishing to Kafka should not rely on "managed Kafka reliability" to solve atomicity.

Managed broker reliability does not make database write + Kafka publish atomic.

Use transactional outbox, CDC outbox, polling publisher, or another explicitly designed consistency pattern.

7.2 CDC on Azure

If PostgreSQL runs as Azure Database for PostgreSQL or on VM/AKS, CDC feasibility depends on:

• logical replication support,
• WAL configuration,
• publication permissions,
• replication slot behavior,
• network path from Debezium/Kafka Connect to PostgreSQL,
• connector deployment location,
• secrets and TLS,
• monitoring replication slot lag,
• database maintenance policies.

Do not assume Debezium works the same in every Azure PostgreSQL deployment mode.

8. Kafka Connect on Azure

Kafka Connect can run as self-managed workers on AKS, on VM, through managed connector offerings, as part of Confluent Cloud, or not at all if using Event Hubs compatibility.

Key questions:

• Where are workers deployed?
• How are connector configs stored?
• How are secrets injected?
• Are internal topics configured correctly?
• How are offsets backed up?
• Is distributed mode used?
• How are tasks scaled?
• How are connector upgrades performed?
• Is DLQ enabled?
• Where are connector logs?
• Who owns restart decisions?

Connector failure modes

Common connector failures:

• invalid credentials,
• network timeout,
• schema evolution mismatch,
• source database permission issue,
• sink throttling,
• poison record,
• offset corruption,
• task imbalance,
• worker memory pressure,
• connector version incompatibility.

A connector should never be treated as fire-and-forget infrastructure.

9. Schema Registry options on Azure

Possible patterns:

• Confluent Schema Registry,
• Apicurio Registry,
• cloud vendor registry if compatible with chosen tooling,
• internal schema registry,
• schema as code without runtime registry,
• Protobuf descriptor governance,
• JSON Schema validation in CI.

Key checks:

• Is registry reachable from producer/consumer pods?
• Is registry access authenticated?
• Are subjects environment-scoped?
• Is compatibility enforced?
• Are schemas promoted via CI/CD?
• Are schema IDs stable across environments?
• Is schema registry included in DR?
• Are schema registry failures visible in dashboards?

A Schema Registry outage can break producers and consumers depending on serializer/deserializer caching and startup behavior.

10. Observability on Azure

Azure-hosted Kafka observability may involve Azure Monitor, Log Analytics, Application Insights, Managed Prometheus, Grafana, Confluent Cloud dashboards, JMX exporter, OpenTelemetry, custom application metrics, and Kafka client metrics.

10.1 Metrics to require

At minimum, track:

• producer send rate,
• producer error rate,
• producer retry rate,
• producer request latency,
• consumer lag,
• consumer processing latency,
• poll loop latency,
• rebalance count/rate,
• DLQ count,
• broker request latency,
• under-replicated partitions,
• offline partitions,
• ISR shrink,
• disk usage,
• network throughput,
• connector task status,
• replication slot lag if CDC is used.

10.2 Logs and tracing

Application logs should include:

• event ID,
• topic,
• partition,
• offset,
• key,
• event type,
• schema version,
• correlation ID,
• causation ID,
• trace ID,
• consumer group,
• retry count,
• DLQ reason.

Do not log sensitive payloads by default.

10.3 End-to-end latency

Measure end-to-end latency separately from broker latency.

Example timeline:

1. HTTP request accepted.
2. DB transaction committed.
3. Outbox row inserted.
4. Outbox published to Kafka.
5. Broker appended record.
6. Consumer fetched record.
7. Consumer committed business state.
8. Projection/read model updated.
9. API/user sees updated state.

If only broker metrics exist, the actual business delay can remain invisible.

11. Azure-specific failure modes

11.1 Event Hubs compatibility assumption

Symptom:

• Kafka client connects, but advanced feature fails.
• Kafka Streams behaves unexpectedly.
• Transaction config does not work.
• Admin operations fail.
• Retention/compaction expectation mismatches.

Likely cause:

• Service is Kafka-compatible, not full Apache Kafka.

Debug approach:

• verify feature support matrix,
• test exact client feature,
• check broker response errors,
• review runtime documentation,
• avoid assuming Apache Kafka semantics.

11.2 Private endpoint DNS mismatch

Symptom:

• works locally through VPN but fails in AKS,
• bootstrap works but broker connection fails,
• TLS hostname verification fails.

Likely cause:

• DNS resolves public endpoint,
• private DNS zone not linked to VNet,
• advertised listener returns unreachable hostname,
• certificate CN/SAN mismatch.

Debug approach:

• run nslookup from pod,
• inspect Kafka metadata,
• test advertised broker addresses,
• validate TLS SNI.

11.3 NSG/firewall partial allow

Symptom:

• client connects to one broker but fails to others,
• intermittent metadata refresh errors,
• consumer lag spikes,
• producer retries increase.

Likely cause:

• only bootstrap endpoint allowed,
• broker endpoints blocked,
• firewall allows one subnet but not another,
• cross-zone route restricted.

Debug approach:

• enumerate all advertised brokers,
• test each host/port from the pod,
• compare firewall/NSG rules.

11.4 Secret rotation outage

Symptom:

• sudden auth failure after deployment or rotation window,
• some pods work, some fail,
• consumers drop from group.

Likely cause:

• rotated server credential but client pods still use old secret,
• external secret not synced,
• truststore not updated,
• rolling restart incomplete.

Debug approach:

• inspect secret version,
• inspect pod environment/mounted secret,
• verify certificate expiry,
• check auth failure logs.

11.5 Connector task silently failed

Symptom:

• data stops flowing,
• no application deployment occurred,
• connector status shows FAILED,
• source database lag increases.

Likely cause:

• connector task failure,
• poison record,
• schema mismatch,
• database permission/network issue.

Debug approach:

• inspect connector status,
• inspect task logs,
• inspect DLQ,
• inspect source lag,
• restart only after root cause is understood.

11.6 Throttling or quota

Symptom:

• producer latency increases,
• request timeout increases,
• throughput plateaus,
• consumer lag grows.

Likely cause:

• managed service quota,
• partition/throughput unit limit,
• network egress limit,
• storage throughput limit.

Debug approach:

• check service quotas,
• compare workload to partition/throughput capacity,
• check throttling metrics,
• review cost/performance limits.

12. Architecture review questions

Runtime

• Is this Apache Kafka or Kafka-compatible?
• Which Kafka version/protocol version is supported?
• Are transactions, idempotent producer, compaction, Kafka Streams, Connect, and AdminClient supported?
• Is the runtime managed, self-managed, or hybrid?
• Who owns upgrades and incident response?

Networking

• Are producers/consumers in AKS, VM, on-prem, AWS, or SaaS?
• Is traffic private?
• How are bootstrap and advertised listeners resolved?
• Are DNS/private zones correctly linked?
• Are NSG/firewall rules broker-aware?
• Is TLS hostname verification enabled?

Security

• What authentication mechanism is used?
• Are credentials per service/principal?
• Are ACLs least-privilege?
• How are secrets rotated?
• Are certificates monitored before expiry?
• Are schema registry and connector permissions separated?

Operations

• Where are metrics and logs?
• Are lag, ISR, broker latency, DLQ, connector, CDC, and client metrics visible?
• Are runbooks attached to alerts?
• Who can reset offsets?
• Who can replay DLQ?
• Who approves topic/schema/ACL changes?

Application correctness

• Does the Java/JAX-RS service use outbox for DB + event consistency?
• Are consumers idempotent?
• Are retries bounded?
• Is DLQ schema standardized?
• Are events replay-safe?
• Is schema compatibility enforced in CI?

13. Java client configuration review checklist

Review these for every Java/JAX-RS service on Azure.

bootstrap.servers=...
client.id=...
group.id=...
security.protocol=...
sasl.mechanism=...
acks=all
enable.idempotence=true
retries=...
delivery.timeout.ms=...
request.timeout.ms=...
max.in.flight.requests.per.connection=...
key.serializer=...
value.serializer=...
key.deserializer=...
value.deserializer=...
schema.registry.url=...
enable.auto.commit=false
max.poll.records=...
max.poll.interval.ms=...
session.timeout.ms=...
heartbeat.interval.ms=...

Questions:

• Are producer configs aligned with ordering and duplicate expectations?
• Are consumer commits manual?
• Is auto commit disabled for business-critical consumers?
• Is schema registry configured per environment?
• Are group IDs stable and environment-scoped?
• Are client IDs useful for metrics?
• Are credentials injected securely?
• Are timeouts appropriate for Azure network latency?

14. Azure observability checklist

Minimum dashboard sections:

Producer

• send rate,
• send latency,
• error rate,
• retry rate,
• record size,
• batch size,
• buffer exhaustion,
• timeout count.

Consumer

• lag by group/topic/partition,
• processing latency,
• poll latency,
• commit latency,
• rebalance rate,
• failed processing count,
• DLQ publish count.

Broker/runtime

• request latency,
• throughput,
• partition count,
• under-replicated partitions,
• offline partitions,
• ISR shrink,
• disk usage,
• network IO,
• controller health.

Connect/CDC

• connector status,
• task status,
• source lag,
• DLQ count,
• connector restart count,
• replication slot lag,
• snapshot progress.

Security

• auth failure rate,
• certificate expiry,
• secret rotation status,
• denied ACL attempts,
• suspicious topic/group access.

15. Common anti-patterns

Anti-pattern: "It uses Kafka client, therefore it is Kafka"

Kafka-compatible endpoint does not guarantee all Kafka semantics.

Verify feature support before depending on transactions, compaction, Streams, Connect, AdminClient, or offset behavior.

Anti-pattern: Public endpoint for internal event backbone

Internal event backbone should normally avoid public exposure unless explicitly approved and controlled.

Prefer private connectivity.

Anti-pattern: One shared credential for all services

This destroys least privilege and auditability.

Use service-specific principals.

Anti-pattern: Treating managed Kafka as correctness guarantee

Managed brokers do not solve dual-write, idempotency, schema compatibility, retry safety, or replay safety.

Anti-pattern: Kubernetes liveness checks depend on Kafka

This causes restart storms during Kafka/network incidents.

Use readiness for dependencies, not liveness.

Anti-pattern: No DLQ/replay ownership

DLQ without owner and runbook is delayed data loss.

Anti-pattern: Schema registry not part of DR

If consumers cannot deserialize after failover, topic replication alone is not enough.

16. Internal verification checklist

Use this with platform, SRE, backend, data, and security teams.

Runtime

• Is the Azure Kafka runtime Apache Kafka, Event Hubs Kafka-compatible endpoint, Confluent Cloud, self-managed VM, or self-managed AKS?
• What version/protocol compatibility is guaranteed?
• Are idempotent producer, transactions, compaction, Kafka Streams, ksqlDB, Kafka Connect, and AdminClient supported?
• Who owns broker/runtime operations?
• What is the escalation path during incident?

Networking

• Where do Java/JAX-RS services run: AKS, VM, on-prem, AWS, or hybrid?
• What bootstrap servers are used per environment?
• Are advertised broker addresses reachable from application pods?
• Is private endpoint/private link used?
• Are DNS zones linked to the right VNets?
• Are NSG/firewall rules broker-aware?
• Is TLS hostname verification enabled and passing?

Security

• What authentication mechanism is used?
• Are credentials/principals service-specific?
• Are ACLs least-privilege?
• How are secrets stored and rotated?
• How are certificates rotated?
• Is Schema Registry secured separately?
• Are connector credentials separated from app credentials?

Application correctness

• Do services writing PostgreSQL and publishing Kafka use outbox or CDC?
• Are consumers idempotent?
• Is auto commit disabled for critical consumers?
• Are retry/DLQ semantics standardized?
• Are events replay-safe?
• Are schemas checked in CI?
• Are event metadata standards enforced?

Operations

• Are consumer lag dashboards available?
• Are producer/consumer metrics exported from Java services?
• Are broker/runtime metrics available in Azure Monitor/Grafana/Confluent dashboards?
• Are connector and CDC metrics visible?
• Are DLQ metrics alerting?
• Are runbooks linked from alerts?
• Is there a safe offset reset and replay procedure?

Cost and capacity

• What are partition, throughput, storage, and retention limits?
• Are quotas monitored?
• Is cross-zone/cross-region traffic cost understood?
• Is private link cost understood?
• Are retention and compaction policies reviewed?

17. PR review checklist

When reviewing code or config that touches Kafka on Azure, ask:

• Does the implementation assume full Apache Kafka while the runtime is Kafka-compatible only?
• Are bootstrap servers and schema registry URLs externalized?
• Are secrets loaded securely?
• Are client IDs and group IDs meaningful?
• Is producer idempotence enabled where needed?
• Are consumer commits manual and after successful processing?
• Does the service handle Kafka unavailability without corrupting DB state?
• Does the service avoid liveness restart storms?
• Does the service emit metrics with topic/group/client labels?
• Are auth/network failures observable?
• Are Azure-specific configs documented?
• Is there a runbook for failure?
• Does the PR include internal verification notes for platform-dependent behavior?

18. Key takeaways

Kafka on Azure must be understood as a runtime choice, not just a connection string.

For senior backend engineers, the key distinctions are:

• Apache Kafka vs Kafka-compatible endpoint,
• managed vs self-managed responsibility,
• private networking vs public endpoint,
• SASL/TLS/identity model,
• Schema Registry and Connect availability,
• operational visibility,
• application correctness independent of broker reliability,
• feature compatibility before architecture dependency.

The strongest engineering posture is:

Treat Azure Kafka deployment as an explicit production contract. Verify the semantics, networking, security, observability, and ownership before relying on it in Java/JAX-RS event-driven systems.