Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
BALROG
Datasets & benchmarksBenchmarking Agentic LLM and VLM Reasoning On Games
git clone https://github.com/balrog-ai/BALROG.git -
WideSearch
Datasets & benchmarksWideSearch: Benchmarking Agentic Broad Info-Seeking
git clone https://github.com/ByteDance-Seed/WideSearch.git -
SWE-bench_Pro
Datasets & benchmarksHugging Face dataset by ScaleAI
ScaleAI/SWE-bench_Pro -
DyCodeEval
Datasets & benchmarksOfficial repository of the ICML2025 paper “Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination”
git clone https://github.com/SeekingDream/DyCodeEval.git -
ParseBench
Datasets & benchmarksHugging Face dataset by llamaindex
llamaindex/ParseBench -
gpqa
Datasets & benchmarksHugging Face dataset by Idavidrein
Idavidrein/gpqa -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
SWE-bench_Verified
Datasets & benchmarksHugging Face dataset by SWE-bench
SWE-bench/SWE-bench_Verified -
MobileWorld
Datasets & benchmarksBenchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (ACL 2026)
git clone https://github.com/Tongyi-MAI/MobileWorld.git -
confabulations
Datasets & benchmarksHallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
git clone https://github.com/lechmazur/confabulations.git -
ClawProBench
Datasets & benchmarksClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
git clone https://github.com/suyoumo/ClawProBench.git -
ChemLLMBench
Datasets & benchmarksOfficial Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)
git clone https://github.com/ChemFoundationModels/ChemLLMBench.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
ParseBench
Datasets & benchmarksParseBench - A Document Parsing Benchmark for AI Agents
git clone https://github.com/run-llama/ParseBench.git -
Long-Horizon-Terminal-Bench
Datasets & benchmarksHugging Face dataset by IntelligenceLab
IntelligenceLab/Long-Horizon-Terminal-Bench -
Simreal-MLBench
Datasets & benchmarks[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
git clone https://github.com/Simreal-AI/Simreal-MLBench.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
deepmark
Datasets & benchmarksDeepmark AI enables a unique testing environment for language models (LLM) assessment on task-specific metrics and on your own data so your GenAI-powered solution has predictable and reliable performance.
git clone https://github.com/IngestAI/deepmark.git -
explicit-edit-benchmark
Datasets & benchmarksBenchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.
git clone https://github.com/alexshpunt/explicit-edit-benchmark.git -
AIRTBench-Code
Datasets & benchmarksCode Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
git clone https://github.com/dreadnode/AIRTBench-Code.git -
hle
Datasets & benchmarksHugging Face dataset by cais
cais/hle -
FDE-Gym
Datasets & benchmarksHugging Face dataset by DeepLethe
DeepLethe/FDE-Gym -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
ping_pong_bench
Datasets & benchmarksA benchmark for role-playing language models
git clone https://github.com/IlyaGusev/ping_pong_bench.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.