PyIceberg at Scale Without Apache Spark

This technical article discusses the practical use of PyIceberg at scale without Apache Spark, targeting Python data engineers and platform teams. It emphasizes that while serverless functions can handle small batch appends and metadata operations, large backfills (e.g., 20 TB) should run on proper execution engines. The article covers architecture patterns, supported documentation, common failure modes, guardrails for agentic use, and operational checklists. It advocates for using PyIceberg for catalog operations, schema evolution, and lightweight automation, while reserving heavy analytical reads and rewrites for distributed engines. The content is grounded in official PyIceberg and Apache Iceberg specifications, avoiding hype and focusing on production-ready boundaries.