Reducing LLM epistemic slop

This article explores how to use large language models (LLMs) as an approximate joint probability distribution over tokens rather than as an expert system, aiming to reduce 'epistemic slop'—unreliable output. The author presents practical techniques such as multinomial/ordinal queries with grammar constraints (using Backus-Naur form in llama.cpp) to avoid errors from greedy recursive generation, enable uncertainty quantification via logits, and allow robust inference through invariant query reformulations. For binomial cases, the Beta distribution is used to combine information. The article includes code examples and emphasizes treating LLMs probabilistically for more reliable results.