🧠 Someone Asked Me to Add a 975B Model to a LocaLLMs, here’s Why we Can’t (Yet)

This article details the evaluation of adding Inkling, a 975B-parameter Mixture-of-Experts multimodal model, to the ElBruno.LocalLLMs .NET library. The author identifies four key barriers: MoE routing is unsupported by ONNX GenAI, multimodal input/output cannot pass through the text-only conversion pipeline, the model requires ~490GB+ of weights (data-center scale), and NVFP4 numerics with custom architecture prevent conversion. The conclusion is that the model is not viable for local inference, recommending hosted APIs and smaller alternatives like Phi-4, Qwen2.5-32B, and DeepSeek-R1-Distill-Qwen-14B. The article is a technical deep-dive into local LLM deployment constraints.