Alex Strick van Linschoten • 2/7/2025

Notes on ‘AI Engineering’ chapter 9: Inference Optimisation

This article provides detailed notes on inference optimization for AI systems, based on Chapter 9 of Chip Huyen's 'AI Engineering' book. It covers core concepts like compute-bound and memory bandwidth-bound bottlenecks, inference APIs (online vs. batch), key performance metrics (latency, throughput), and the critical business importance of reducing inference costs, which can constitute up to 90% of ML expenses.

0 comments

#Machine Learning #llm #Inference Optimization