Apache Kafka Reference¶
What this page is
Look-up facts for Apache Kafka 4.x: release and support matrix, Java requirements, default ports, internal topics, KIP status, key configuration defaults, CLI tools, security advisories, hardening checklist, and published benchmark results. Task recipes live in How-to Guides. The reasons behind these facts live in Explanation.
Related Notes
Release and Support Matrix¶
Apache Kafka aims for three minor releases a year (one every four months). Bug-fix releases ship only for releases listed as supported on the downloads page. Dates below come from the official downloads page and release announcements.
| Version | Released | Status (2026-09-25) | Notes |
|---|---|---|---|
| 4.4.0 | Not released | In release process | The 4.4 branch is versioned 4.4.0, but no artifacts are on downloads.apache.org yet. Upgrade notes list share-group DLQ (KIP-1191), broker.id deprecation (KIP-1232), and controller unregistration (KIP-1312) |
| 4.3.1 | 2026-06-25 | Supported (latest) | Bug fix: Kafka Streams RocksDB native memory leak (KAFKA-20616), about 15 fixes |
| 4.3.0 | 2026-05-22 | Archived | 25 KIPs. Log-directory cordoning (KIP-1066), follower fetch from tiered offset (KIP-1023), streams-scala deprecated |
| 4.2.1 | 2026-05-30 | Supported | Fixes a share-group deadlock (KAFKA-20505) and a Streams-protocol migration bug (KAFKA-20254) |
| 4.2.0 | 2026-02-17 | Archived | Queues for Kafka (share groups, KIP-932) production-ready. Streams Rebalance Protocol (KIP-1071) GA for its core feature set. Java 25 support |
| 4.1.2 | 2026-03-17 | Supported | Fixes CVE-2026-33557 and CVE-2026-35554 |
| 4.1.1 | 2025-11-12 | Archived | Fixes the Kafka Streams 4.1.0 memory leak (KAFKA-19748) |
| 4.1.0 | 2025-09-02 | Archived | Share groups preview. Streams Rebalance Protocol early access. ELR on by default for new clusters. Static-to-dynamic controller quorum upgrade |
| 4.0.2 | 2026-03-16 | Archived | Bug-fix release |
| 4.0.1 | 2025-10-13 | Archived | Bug-fix release |
| 4.0.0 | 2025-03-18 | Archived | ZooKeeper removed (KRaft only). KIP-848 consumer protocol GA. ELR preview. Queues early access. Brokers need Java 17 |
| 3.9.2 | 2026-02-21 | Archived | Latest 3.x patch release. Security fixes backported |
| 3.9.1 | 2025-05-21 | Archived | Fixes CVE-2025-27817/27818/27819 |
| 3.9.0 | 2024-11-06 | Archived | Last release line that supports ZooKeeper (the "bridge release" for migration). Tiered storage production-ready. Dynamic KRaft quorums (KIP-853) |
| 3.6.0 | 2023-10-10 | Archived | Tiered storage early access. ZooKeeper-to-KRaft migration production-ready |
| 3.5.0 | 2023-06-15 | Archived | ZooKeeper mode marked deprecated |
| 3.3.1 | 2022-10-03 | Archived | KRaft declared production-ready. 3.3.0 was pulled before its announcement because of a significant bug |
Upgrade floor for 4.x
Brokers can roll to 4.x only from KRaft clusters on 3.3.x or later. ZooKeeper clusters must first migrate to KRaft on a 3.x bridge release (3.9.x recommended). Clients older than 2.1 cannot talk to 4.x brokers because 4.0 removed old protocol API versions (KIP-896).
Java and Platform Requirements¶
| Component | Minimum Java (4.x) | Notes |
|---|---|---|
| Brokers, controllers, Kafka Connect, CLI tools | Java 17 | KIP-1013 dropped Java 11 for the server side in 4.0 |
| Java clients, Kafka Streams | Java 11 | KIP-750 dropped Java 8 in 4.0 |
| Build and test matrix (4.3 branch) | Java 17 and 25 | From the apache/kafka 4.3 README |
| Newest JDK support added | Java 23 (4.0), Java 25 (4.2) | From the release notes |
| Scala | 2.13 only | The binary tarball is kafka_2.13-<version>.tgz |
| Logging | Log4j2 | Log4j 1.x was replaced in 4.0. KafkaLog4jAppender was removed |
Official container images: apache/kafka:<version> (JVM) and apache/kafka-native:<version> (GraalVM native image). Both are published for every supported release, for example apache/kafka:4.3.1.
Default Ports and Listeners¶
| Port | Listener (sample config) | Purpose |
|---|---|---|
| 9092 | PLAINTEXT |
Client and inter-broker traffic in config/server.properties and the Docker quickstart |
| 9093 | CONTROLLER |
KRaft controller quorum (controller.listener.names=CONTROLLER) |
| 8083 | Kafka Connect REST | Default listeners for a Connect worker |
Set by JMX_PORT |
JMX | Remote JMX is off unless you export JMX_PORT before starting the node |
The port numbers are conventions from the sample configs, not protocol requirements. Production clusters often add SSL/SASL_SSL listeners on other ports.
Internal Topics¶
| Topic | Created by | Default partitions | Purpose |
|---|---|---|---|
__cluster_metadata |
KRaft controllers | 1 (Raft log) | Replicated metadata log: topics, partitions, configs, ACLs, feature levels |
__consumer_offsets |
Group coordinator | 50 (offsets.topic.num.partitions) |
Committed offsets and group metadata (compacted) |
__transaction_state |
Transaction coordinator | 50 (transaction.state.log.num.partitions) |
Transactional producer state (compacted) |
__share_group_state |
Share coordinator | 50 | Share-group delivery state (KIP-932). Default RF 3, min ISR 2 |
__remote_log_metadata |
TopicBasedRemoteLogMetadataManager |
50 | Tiered-storage segment metadata. RF 3, min ISR 2 (KIP-1235 in 4.3), infinite retention |
connect-configs, connect-offsets, connect-status |
Kafka Connect (distributed) | Configurable | Connector configs, source offsets, task status (names set by the worker config) |
Small dev clusters
Internal topics default to replication factor 3. On clusters with fewer than 3 brokers, lower offsets.topic.replication.factor, transaction.state.log.replication.factor, and share.coordinator.state.topic.replication.factor (and the matching min.isr settings) before first use. The sample config/server.properties already sets them to 1.
KIP Status Tracker¶
| KIP | Feature | Status |
|---|---|---|
| KIP-500 / KIP-833 | KRaft (ZooKeeper removal) | Early access 2.8, production-ready 3.3, ZooKeeper deprecated 3.5, removed 4.0 |
| KIP-853 | Dynamic KRaft controller quorums (controller.quorum.bootstrap.servers) |
New clusters 3.9. Static-to-dynamic upgrade 4.1 (kraft.version=1) |
| KIP-405 | Tiered storage | Early access 3.6, production-ready 3.9 |
| KIP-848 | Next-generation consumer rebalance protocol | GA 4.0. Consumers opt in with group.protocol=consumer. The client default is still classic in 4.3 |
| KIP-1274 | Deprecate the classic protocol in KafkaConsumer |
Phase 1 in 4.3: the consumer logs a recommendation to switch |
| KIP-890 | Transactions server-side defense | Enabled when the 4.0 upgrade is finalized |
| KIP-966 (part 1) | Eligible Leader Replicas (ELR) | Preview 4.0 (eligible.leader.replicas.version=1). On by default for new clusters from 4.1 |
| KIP-932 | Queues for Kafka (share groups) | Early access 4.0, preview 4.1, production-ready 4.2 (share.version=1) |
| KIP-1191 | Dead-letter queue for share groups | In the 4.4.0 upgrade notes (share.version=2). Not yet released |
| KIP-1071 | Streams Rebalance Protocol (group.protocol=streams) |
Early access 4.1, GA for the core feature set in 4.2. Static membership support in 4.4 |
| KIP-714 | Client metrics and observability (client telemetry push) | Released in 3.7 |
| KIP-1066 | Cordon brokers and log directories (cordoned.log.dirs) |
4.3 |
| KIP-1023 | Follower fetch from the last tiered offset | 4.3 (follower.fetch.last.tiered.offset.enable, default false) |
| KIP-1232 | Deprecate broker.id in favor of node.id |
In the 4.4.0 upgrade notes. Removal is planned for 5.0 |
Feature Flags (kafka-features.sh)¶
| Feature | Levels | Enables |
|---|---|---|
metadata.version |
e.g. 4.3-IV0 |
Finalized cluster metadata format. Set with upgrade --release-version 4.3 |
kraft.version |
0 = static quorum, 1 = dynamic quorum | KIP-853 controller membership changes |
group.version |
1 | New group coordinator and KIP-848 consumer protocol |
transaction.version |
2 | KIP-890 transaction protocol |
eligible.leader.replicas.version |
0 or 1 | ELR tracking (KIP-966) |
share.version |
1 (4.2+), 2 (4.4, DLQ) | Share groups |
streams.version |
1 | Streams Rebalance Protocol |
kafka-features.sh --bootstrap-server <host:port> upgrade --release-version <X.Y> raises every feature to the default for that release.
Key Configuration Defaults¶
Defaults come from the 4.3 source (ProducerConfig, ConsumerConfig, LogConfig, ReplicationConfigs) and the upgrade notes. Tuning guidance for these settings is in How-to Guides.
Broker and Topic¶
| Setting | Default | Notes |
|---|---|---|
process.roles |
none (required) | broker, controller, or broker,controller (combined) |
node.id |
none (required) | Replaces broker.id (deprecated in 4.4, removal planned for 5.0) |
controller.quorum.bootstrap.servers |
none | Replaces controller.quorum.voters for dynamic quorums (KIP-853) |
num.network.threads |
3 | |
num.io.threads |
8 | |
num.replica.fetchers |
1 | |
num.recovery.threads.per.data.dir |
2 | Changed from 1 in 4.0 (KIP-1030) |
socket.send.buffer.bytes / socket.receive.buffer.bytes |
102400 | |
log.segment.bytes / segment.bytes |
1073741824 (1 GiB) | Minimum raised to 1 MiB in 4.0 |
log.retention.hours / retention.ms |
168 h / 604800000 ms (7 days) | |
num.partitions |
1 | Default partitions for auto-created topics |
default.replication.factor |
1 | Set to 3 in production |
min.insync.replicas |
1 | Set to 2 with RF=3. With ELR on, set it at cluster level, not broker level |
unclean.leader.election.enable |
false |
|
replica.lag.time.max.ms |
30000 | A follower that falls behind for longer is removed from the ISR |
auto.create.topics.enable |
true |
Disable in production |
message.timestamp.after.max.ms |
1 hour | Changed from Long.MAX_VALUE in 4.0 |
remote.log.storage.system.enable |
false |
Broker-level tiered storage switch |
remote.storage.enable (topic) |
false |
Per-topic tiered storage switch |
local.retention.ms / local.retention.bytes (topic) |
-2 (inherit retention.*) |
Local-tier retention for tiered topics |
Producer¶
| Setting | Default | Notes |
|---|---|---|
acks |
all |
Default since 3.0 (KIP-679) |
enable.idempotence |
true |
Default since 3.0 |
linger.ms |
5 | Changed from 0 in 4.0 |
batch.size |
16384 | |
compression.type |
none |
gzip, snappy, lz4, zstd |
max.in.flight.requests.per.connection |
5 | Must be 5 or less when idempotence is enabled |
buffer.memory |
33554432 (32 MiB) | |
delivery.timeout.ms |
120000 |
Consumer¶
| Setting | Default | Notes |
|---|---|---|
group.protocol |
classic |
Set consumer for KIP-848 |
isolation.level |
read_uncommitted |
read_committed hides aborted transactional records |
fetch.min.bytes |
1 | |
fetch.max.wait.ms |
500 | |
max.poll.records |
500 | |
max.poll.interval.ms |
300000 | |
auto.offset.reset |
latest |
Share Groups (group-level configs)¶
| Setting | Default | Range (broker min/max) |
|---|---|---|
share.record.lock.duration.ms |
30000 | 15000 to 60000 |
share.delivery.count.limit |
5 | 2 to 10 |
share.partition.max.record.locks |
broker config group.share.partition.max.record.locks |
Added as a group config in 4.3 (KIP-1240) |
CLI Tools¶
| Script | Purpose |
|---|---|
kafka-storage.sh |
random-uuid, format (with --standalone, --initial-controllers, or --no-initial-controllers) |
kafka-server-start.sh / kafka-server-stop.sh |
Start a node, or stop it (controlled shutdown) |
kafka-topics.sh |
Create, list, describe, alter, and delete topics |
kafka-configs.sh |
Dynamic broker, topic, user, and client configs, SCRAM credentials, quotas |
kafka-consumer-groups.sh |
Describe lag and reset offsets for consumer groups |
kafka-share-groups.sh |
Describe share groups. From 4.4 it can also initialize their offsets |
kafka-streams-groups.sh |
Describe streams groups (KIP-1071) |
kafka-acls.sh |
Manage ACLs |
kafka-features.sh |
Describe, upgrade, and downgrade feature levels |
kafka-metadata-quorum.sh |
Quorum status and replication. add-controller, remove-controller |
kafka-cluster.sh |
Cluster ID, unregister broker. api-versions in 4.4 (replaces kafka-broker-api-versions.sh) |
kafka-leader-election.sh |
Preferred or unclean leader election |
kafka-reassign-partitions.sh |
Move partitions between brokers and log directories |
kafka-producer-perf-test.sh / kafka-consumer-perf-test.sh |
Load testing |
kafka-dump-log.sh |
Inspect segment files, including tiered and metadata logs |
connect-distributed.sh / connect-mirror-maker.sh |
Run Kafka Connect and MirrorMaker 2 |
Key JMX Metrics¶
| MBean | Alert when |
|---|---|
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions |
> 0 for a sustained period |
kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount |
> 0 (acks=all produce requests fail) |
kafka.server:type=ReplicaManager,name=IsrShrinksPerSec / IsrExpandsPerSec |
Frequent flapping |
kafka.controller:type=KafkaController,name=ActiveControllerCount |
The sum across controllers is not exactly 1 |
kafka.controller:type=KafkaController,name=OfflinePartitionsCount |
> 0 |
kafka.controller:type=KafkaController,name=PreferredReplicaImbalanceCount |
Grows after broker restarts |
kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce,FetchConsumer,FetchFollower} |
p99 regression |
kafka.server:type=BrokerTopicMetrics,name={BytesInPerSec,BytesOutPerSec} |
Capacity planning |
Deprecated group metrics
The 4.4 upgrade notes deprecate several Yammer kafka.coordinator.group:type=GroupMetadataManager metrics in favor of kafka.server:type=group-coordinator-metrics (KIP-1301). They are due for removal in 5.0.
Security Advisories¶
Source: Apache Kafka CVE list. For CVSS scores, see the NVD entries.
| CVE | Announced | Summary | Affected | Fixed / Action |
|---|---|---|---|---|
| CVE-2026-41115 | 2026-06-02 | CONSUMER_GROUP_DESCRIBE (API 69) checks DESCRIBE on the group, while the docs and KIP-848 said READ. The code is correct. The docs were fixed |
4.0.0 to 4.3.0 | Review group ACLs for least privilege |
| CVE-2026-33558 | 2026-04-17 | At DEBUG level, NetworkClient logs SASL, SCRAM, delegation-token, and config requests in full |
Clients 0.11.0 to 3.9.1, 4.0.0 | 3.9.2, 4.0.1, 4.1.0. Keep NetworkClient at INFO |
| CVE-2026-33557 | 2026-04-17 | The default broker JWT validator (DefaultJwtValidator) accepts OAUTHBEARER tokens without checking signature, issuer, or audience |
4.1.0 to 4.1.1 | 4.1.2, 4.2.0. Or set sasl.oauthbearer.jwt.validator.class=org.apache.kafka.common.security.oauthbearer.BrokerJwtValidator |
| CVE-2026-35554 | 2026-04-07 | Producer buffer-pool race: an expired in-flight batch's buffer can be reused, so records reach the wrong topic | Clients 2.8.0 to 3.9.1, 4.0.0 to 4.0.1, 4.1.0 to 4.1.1 | 3.9.2, 4.0.2, 4.1.2, 4.2.0 |
| CVE-2025-27819 | 2025-06-09 | RCE/DoS through the SASL JAAS JndiLoginModule for principals with AlterConfigs on the cluster |
2.0.0 to 3.3.2 | 3.9.1, 4.0.0 |
| CVE-2025-27818 | 2025-06-09 | RCE through the SASL JAAS LdapLoginModule for AlterConfigs holders or Connect connector authors |
2.3.0 to 3.9.0 | 3.9.1, 4.0.0 (login module disabled by default) |
| CVE-2025-27817 | 2025-06-09 | Client arbitrary file read and SSRF through sasl.oauthbearer.token.endpoint.url / jwks.endpoint.url |
3.1.0 to 3.9.0 | 3.9.1, 4.0.0. Set -Dorg.apache.kafka.sasl.oauthbearer.allowed.urls |
| CVE-2024-56128 | 2024-12-18 | SCRAM does not validate the nonce (RFC 5802), so replay is possible without TLS | 0.10.2.0 to 3.7.1, 3.8.0 | 3.7.2, 3.8.1, 3.9.0. Always use SCRAM over TLS |
| CVE-2024-31141 | 2024-11-18 | Client ConfigProviders (file, directory, env var) can leak disk or env contents when an untrusted party supplies the config |
2.3.0 to 3.7.0 | 3.7.1, 3.8.0. Set -Dorg.apache.kafka.automatic.config.providers=none where appropriate |
| CVE-2024-27309 | 2024-04-12 | ACLs can be enforced incorrectly during a ZooKeeper-to-KRaft migration after an ACL is removed | 3.5.0 to 3.6.1 | 3.6.2. Migrate only on 3.6.2 or later |
Hardening Checklist¶
- Every listener that crosses a trust boundary uses
SSLorSASL_SSL, including broker-to-broker and broker-to-controller. NoPLAINTEXTin production. -
ssl.enabled.protocols=TLSv1.3,TLSv1.2and hostname verification is on (ssl.endpoint.identification.algorithm=https, the default). - An authorizer is configured (
org.apache.kafka.metadata.authorizer.StandardAuthorizer) andallow.everyone.if.no.acl.foundstaysfalse(the default). -
super.usersis limited to the operator and inter-broker principals. - OAUTHBEARER brokers use a real JWKS validator with
sasl.oauthbearer.expected.audienceandexpected.issuerset. 4.4 fails fast at startup without them. - The
org.apache.kafka.sasl.oauthbearer.allowed.urlsJVM property lists only your identity provider's URLs. - On 4.2+, JAAS login modules are restricted with
org.apache.kafka.allowed.login.modules(disallowed.login.modulesis deprecated). - AlterConfigs and Connect REST access are limited to trusted operators (CVE-2025-27818/27819).
-
NetworkClientandkafka.authorizer.loggerrun at INFO, not DEBUG, in production (CVE-2026-33558). - Client quotas (
producer_byte_rate,consumer_byte_rate,request_percentage) are set per principal. - Controllers run in isolated mode on hosts that only the broker network can reach.
-
auto.create.topics.enable=false, andunclean.leader.election.enable=falseunless you accept data loss. - Clients and brokers run a release that fixes the 2026 advisories (3.9.2, 4.0.2, 4.1.2, 4.2.0, or later).
Record Batch Format (Magic v2)¶
Each record batch on disk and on the wire has this header (per the Apache Kafka protocol documentation). The record-level format is covered in Explanation.
baseOffset: int64
batchLength: int32
partitionLeaderEpoch: int32
magic: int8 (= 2)
crc: uint32 (CRC-32C over attributes..end)
attributes: int16 bits 0-2 compression: 0=none 1=gzip 2=snappy 3=lz4 4=zstd
bit 3 timestampType, bit 4 isTransactional, bit 5 isControlBatch
lastOffsetDelta: int32
baseTimestamp: int64
maxTimestamp: int64
producerId: int64 (idempotence / transactions)
producerEpoch: int16
baseSequence: int32
recordsCount: int32
records: [Record] (compressed together as one unit)
Kafka 4.0 removed log.message.format.version and message.format.version. New writes always use magic v2.
Benchmark Results¶
Benchmark caveat
These are published results from different hardware, versions, and vendors. Throughput depends heavily on message size, compression codec, batch size, replication factor, acks, and disks. Treat them as upper bounds and rerun the OpenMessaging Benchmark on your own hardware. For the reasons Kafka is fast, see Explanation.
| Source | Year | Setup | Result |
|---|---|---|---|
| LinkedIn Engineering | 2014 | 3 brokers (Xeon 2.5 GHz 6-core, 6x 7200 RPM SATA, 32 GiB RAM, 1 GbE), Kafka 0.8.2-beta producer, 100-byte records | 2,024,032 records/s (193 MB/s) with 3 producers and 3x asynchronous replication |
| Confluent: Kafka vs Pulsar vs RabbitMQ | 2020 | 3x i3en.2xlarge brokers, OpenMessaging Benchmark, 1 KiB messages, RF=3 |
About 605 MB/s peak stable throughput for Kafka, with lower p99 latency than Pulsar at the same load (vendor benchmark) |
| Dell EMC / Confluent Platform characterization | n/a | 13 concurrent producers on Dell hardware | 18,623,322 records/s aggregate (1,776 MB/s), 83 ms average producer latency (as reported, not re-verified 2026-09) |
| Confluent: Kafka vs Kora latency | 2023 | Confluent Cloud (Kora) vs self-managed Apache Kafka, 1.4 GB/s ingress + 4.2 GB/s egress | Vendor claims up to 10x lower tail latency for Kora |
Practical Sizing Heuristics¶
These are community rules of thumb, not measurements. Validate them against your own workload.
| Cluster size | Brokers | Sustained ingress | Topic and partition budget |
|---|---|---|---|
| Dev | 1 (KRaft combined) | < 50 MB/s | < 100 partitions |
| Small prod | 3 brokers (RF=3) | 100 to 500 MB/s | A few thousand partitions |
| Medium prod | 6 to 12 brokers | 500 MB/s to 2 GB/s | 10k to 50k partitions |
| Large prod | 30+ brokers, tiered storage | 2 GB/s+ | 100k+ partitions |
Sources¶
- Apache Kafka downloads (release dates, supported releases)
- Apache Kafka upgrade notes (4.x)
- Apache Kafka 4.3.1 release announcement
- Apache Kafka CVE list
- apache/kafka README (Java build matrix)
- Apache Kafka: Eligible Leader Replicas operations doc
- Apache Kafka: KRaft operations doc
- Apache Kafka: Tiered storage doc
- KIP index (Apache wiki)