Event-Driven Table Compaction with Agents

This article discusses event-driven table compaction coordinated by agents for lakehouse operators managing file layout and table health. It addresses the problem of small files causing planning, cost, and latency issues for both humans and agents. The architecture emphasizes compaction responding to file layout, workload pressure, and commit safety, with automated maintenance coordinating with concurrent writers. It covers specs from Apache Iceberg, Flink, and Kafka, and details a five-layer architecture: storage, catalog, execution, semantics, and agent interface. The article provides practical guidance on measuring outcomes, common failure modes, guardrails, and operational checklists for engineers, data owners, and executives.