LLM Evaluation articles

4/8/2024 • EN

Comparing LLMs on "Real-World" Retrieval

A developer compares 8 LLMs on a custom retrieval task using medical transcripts, analyzing performance on simple to complex questions.

Data Wrangling Instruction Tuning LLM Evaluation Model Comparison Retrieval Benchmarks

Shreya Shankar

3/31/2024 • EN

Task-Specific LLM Evals that Do & Don't Work

A guide to effective and ineffective evaluation methods for LLMs on tasks like classification, summarization, and translation, including practical metrics.

classification LLM Evaluation Summarization Toxicity Translation

Eugene Yan

3/5/2024 • EN

Evaluate LLMs with Hugging Face Lighteval on Amazon SageMaker

A tutorial on evaluating Large Language Models using Hugging Face's Lighteval library on Amazon SageMaker, focusing on benchmarks like TruthfulQA.

Amazon Sagemaker benchmarking Hugging Face Lighteval LLM Evaluation

Philipp Schmid

3/4/2024 • EN

Measuring and Mitigating Hallucinations in Large Language Models: A Multifaceted Approach

A technical paper exploring the causes, measurement, and mitigation strategies for hallucinations in Large Language Models (LLMs).

AI Safety Hallucination Mitigation large language models LLM Evaluation Model Alignment

Xavier Amatriain

10/30/2023 • EN

Evaluate LLMs and RAG a practical example using Langchain and Hugging Face

A hands-on guide to evaluating LLMs and RAG systems using Langchain and Hugging Face, covering criteria-based and pairwise evaluation methods.

Gpt 4 Hugging Face Langchain LLM Evaluation Rag

Philipp Schmid

9/3/2023 • EN

Evaluation & Hallucination Detection for Abstractive Summaries

Explores methods for evaluating abstractive text summaries and detecting hallucinations, covering key dimensions and metrics like NLI and QA.

Abstractive Summarization Hallucination Detection LLM Evaluation Natural Language Inference Question Answering