Processing data from SQLite hosted in an Azure File share, running in Azure Kubernetes Services

This article explores a proof-of-concept for querying a large SQLite database hosted on an Azure File share, accessed simultaneously from multiple containers in Azure Kubernetes Services (AKS). It covers mounting an Azure File share to containers, using a simplified C# console app as a container image, and scaling horizontally across nodes. Key considerations include read-only access to avoid concurrency issues, writing results to Azure Table Storage, and the importance of database indexes for performance. The author shares scale test results with 15 nodes and 750 containers, highlighting practical challenges and solutions for distributed data processing in a cloud-native environment.