Viability of local models for coding

This article, part of Thoughtworks' 'Exploring Gen AI' series, examines the current state of running local AI models for coding tasks. Authored by a Distinguished Engineer with over 20 years of experience, it details personal experiments over four weeks using Apple M3 Max and M5 Pro machines. The article focuses on agentic coding beyond autocomplete, assessing factors like model runnability, response speed, tool calling success, and quality of outcomes. It discusses hardware constraints (RAM, processing power, memory bandwidth), quantization, architecture, and context window size. The author provides a nuanced view of when local models are viable for developers, noting improvements but also challenges in setup and consistency.