Kafka 4.0 Changes Streaming Platform Operations

Apache Kafka 4.0, released March 18, 2025, eliminates ZooKeeper, requiring all clusters to run in KRaft mode with embedded metadata management. This reduces infrastructure by removing separate ZooKeeper ensembles, speeds up controller failover, and simplifies Kubernetes deployments. Additionally, the new consumer rebalance protocol (KIP-848) replaces stop-the-world rebalances to reduce consumer lag spikes, and Queues for Kafka (KIP-932) enables point-to-point messaging. The article details operational implications, upgrade paths, removed APIs, and best practices for platform teams managing streaming data architectures.