How to Monitor NVIDIA GPUs with DCGM Exporter, Prometheus, and Grafana

This article provides a comprehensive operational runbook for monitoring NVIDIA GPUs in Kubernetes environments using DCGM Exporter, Prometheus, and Grafana. It covers collecting device telemetry, scraping metrics, visualizing fleet and workload behavior, and setting up alerts. The guide emphasizes distinguishing low utilization from genuine performance issues, controlling metric cardinality, preserving per-pod context, and capturing evidence for escalation. It includes deployment scenarios with or without NVIDIA GPU Operator, validation gates, and common production questions like latency increases, training job slowdowns, and capacity planning.