Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
BALROG
Datasets & benchmarksBenchmarking Agentic LLM and VLM Reasoning On Games
git clone https://github.com/balrog-ai/BALROG.git -
TurtleBench
Datasets & benchmarksTurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
git clone https://github.com/mazzzystar/TurtleBench.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
LongCite
Datasets & benchmarksLongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
git clone https://github.com/THUDM/LongCite.git -
active_genie
Datasets & benchmarksThe Lodash for GenAI: Real Value + Consistent + Model-Agnostic
git clone https://github.com/Roriz/active_genie.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
MobileWorld
Datasets & benchmarksBenchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (ACL 2026)
git clone https://github.com/Tongyi-MAI/MobileWorld.git -
geobench
Datasets & benchmarksGeoGuessr benchmark for language models
git clone https://github.com/ccmdi/geobench.git -
SciCode
Datasets & benchmarksA benchmark that challenges language models to code solutions for scientific problems
git clone https://github.com/scicode-bench/SciCode.git -
ping_pong_bench
Datasets & benchmarksA benchmark for role-playing language models
git clone https://github.com/IlyaGusev/ping_pong_bench.git -
PawBench
Datasets & benchmarksA benchmark for evaluating LLM × harness performance.
git clone https://github.com/agentscope-ai/PawBench.git -
MMLU-CF
Datasets & benchmarksA Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]
git clone https://github.com/microsoft/MMLU-CF.git -
minebench
Datasets & benchmarksMinecraft-style voxel benchmark for comparing AI models (Arena + Sandbox)
git clone https://github.com/Ammaar-Alam/minebench.git -
BizFinBench
Datasets & benchmarksA Business-Driven Real-World Financial Benchmark for Evaluating LLMs
git clone https://github.com/HiThink-Research/BizFinBench.git -
OSWorld
Datasets & benchmarks[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
git clone https://github.com/xlang-ai/OSWorld.git -
o11y-bench
Datasets & benchmarksOpen benchmark on observability tasks built on Harbor
git clone https://github.com/grafana/o11y-bench.git -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
LoopArena
Datasets & benchmarksBenchmarking models as runtime Controllers for Loop Engineering
git clone https://github.com/AMAP-ML/LoopArena.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
LLM-Agent-Benchmark-List
Datasets & benchmarksA banchmark list for evaluation of large language models.
git clone https://github.com/zhangxjohn/LLM-Agent-Benchmark-List.git -
turnkeyml
Datasets & benchmarksNo-code CLI designed for accelerating ONNX workflows
git clone https://github.com/onnx/turnkeyml.git -
LLM-RGB
Datasets & benchmarksLLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.
git clone https://github.com/babelcloud/LLM-RGB.git -
ParseBench
Datasets & benchmarksParseBench - A Document Parsing Benchmark for AI Agents
git clone https://github.com/run-llama/ParseBench.git -
ClawProBench
Datasets & benchmarksClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
git clone https://github.com/suyoumo/ClawProBench.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.