Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
olmOCR-bench
Datasets & benchmarksHugging Face dataset by allenai
allenai/olmOCR-bench -
wildjailbreak
Datasets & benchmarksHugging Face dataset by allenai
allenai/wildjailbreak -
tweet_eval
Datasets & benchmarksHugging Face dataset by cardiffnlp
cardiffnlp/tweet_eval -
ReForm-Eval
Datasets & benchmarksAn benchmark for evaluating the capabilities of large vision-language models (LVLMs)
git clone https://github.com/FudanDISC/ReForm-Eval.git -
PKU-SafeRLHF
Datasets & benchmarksHugging Face dataset by PKU-Alignment
PKU-Alignment/PKU-SafeRLHF -
gsm8k
Datasets & benchmarksHugging Face dataset by openai
openai/gsm8k -
MMLU-Pro
Datasets & benchmarksHugging Face dataset by TIGER-Lab
TIGER-Lab/MMLU-Pro -
SWE-bench_Pro
Datasets & benchmarksHugging Face dataset by ScaleAI
ScaleAI/SWE-bench_Pro -
SWE-bench_Verified
Datasets & benchmarksHugging Face dataset by SWE-bench
SWE-bench/SWE-bench_Verified -
autoguardrails
Datasets & benchmarksAlignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface
git clone https://github.com/SantanderAI/autoguardrails.git -
gamebot
Datasets & benchmarks[ACL 2025] GAMEBoT: Transparent Assessment of LLM Reasoning in Games
git clone https://github.com/Visual-AI/gamebot.git -
ai-village
Datasets & benchmarksHugging Face dataset by aidigestorg
aidigestorg/ai-village -
gpqa
Datasets & benchmarksHugging Face dataset by Idavidrein
Idavidrein/gpqa -
Omni-SafetyBench
Datasets & benchmarks[ACM MM 2026 Oral] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".
git clone https://github.com/THU-BPM/Omni-SafetyBench.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
self-bench
Datasets & benchmarksBenchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.
git clone https://github.com/mupt-ai/self-bench.git -
TrustLLM
Datasets & benchmarks[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
git clone https://github.com/HowieHwong/TrustLLM.git -
agents-last-exam
Datasets & benchmarksHugging Face dataset by agents-last-exam
agents-last-exam/agents-last-exam -
typed-decisions
Datasets & benchmarksHugging Face dataset by LocalLLaMA
LocalLLaMA/typed-decisions -
apex-agents
Datasets & benchmarksHugging Face dataset by mercor
mercor/apex-agents
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.