Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
gsm8k
Datasets & benchmarksHugging Face dataset by openai
openai/gsm8k -
gpqa
Datasets & benchmarksHugging Face dataset by Idavidrein
Idavidrein/gpqa -
Long-Horizon-Terminal-Bench
Datasets & benchmarksHugging Face dataset by IntelligenceLab
IntelligenceLab/Long-Horizon-Terminal-Bench -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
olmOCR-bench
Datasets & benchmarksHugging Face dataset by allenai
allenai/olmOCR-bench -
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
tweet_eval
Datasets & benchmarksHugging Face dataset by cardiffnlp
cardiffnlp/tweet_eval -
ReForm-Eval
Datasets & benchmarksAn benchmark for evaluating the capabilities of large vision-language models (LVLMs)
git clone https://github.com/FudanDISC/ReForm-Eval.git -
MMLU-Pro
Datasets & benchmarksHugging Face dataset by TIGER-Lab
TIGER-Lab/MMLU-Pro -
SWE-bench_Pro
Datasets & benchmarksHugging Face dataset by ScaleAI
ScaleAI/SWE-bench_Pro -
SWE-bench_Verified
Datasets & benchmarksHugging Face dataset by SWE-bench
SWE-bench/SWE-bench_Verified -
gamebot
Datasets & benchmarks[ACL 2025] GAMEBoT: Transparent Assessment of LLM Reasoning in Games
git clone https://github.com/Visual-AI/gamebot.git -
self-bench
Datasets & benchmarksBenchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.
git clone https://github.com/mupt-ai/self-bench.git -
agents-last-exam
Datasets & benchmarksHugging Face dataset by agents-last-exam
agents-last-exam/agents-last-exam -
typed-decisions
Datasets & benchmarksHugging Face dataset by LocalLLaMA
LocalLLaMA/typed-decisions -
apex-agents
Datasets & benchmarksHugging Face dataset by mercor
mercor/apex-agents -
openbench
Datasets & benchmarksSame model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost
git clone https://github.com/minghinmatthewlam/openbench.git -
ParseBench
Datasets & benchmarksHugging Face dataset by llamaindex
llamaindex/ParseBench -
AIRTBench-Code
Datasets & benchmarksCode Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
git clone https://github.com/dreadnode/AIRTBench-Code.git -
elimination_game
Datasets & benchmarksA multi-player tournament benchmark that tests LLMs in social reasoning, strategy, and deception. Players engage in public and private conversations, form alliances, and vote to eliminate each other
git clone https://github.com/lechmazur/elimination_game.git -
inferock-bench
Datasets & benchmarksLocal LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.
git clone https://github.com/inferock/inferock-bench.git -
seo-skill-bench
Datasets & benchmarksOpen benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.
git clone https://github.com/aleclindz/seo-skill-bench.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.