Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
Simreal-MLBench
Datasets & benchmarks[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
git clone https://github.com/Simreal-AI/Simreal-MLBench.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
hle
Datasets & benchmarksHugging Face dataset by cais
cais/hle -
FDE-Gym
Datasets & benchmarksHugging Face dataset by DeepLethe
DeepLethe/FDE-Gym -
primitive-bench
Datasets & benchmarksThe marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.
git clone https://github.com/primitive-bench/primitive-bench.git -
inferock-bench
Datasets & benchmarksLocal LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.
git clone https://github.com/inferock/inferock-bench.git -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
deepmark
Datasets & benchmarksDeepmark AI enables a unique testing environment for language models (LLM) assessment on task-specific metrics and on your own data so your GenAI-powered solution has predictable and reliable performance.
git clone https://github.com/IngestAI/deepmark.git -
mt_bench_human_judgments
Datasets & benchmarksHugging Face dataset by lmsys
lmsys/mt_bench_human_judgments -
nagi-bench
Datasets & benchmarksOne-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.
git clone https://github.com/nagi-studio/nagi-bench.git -
AIRTBench-Code
Datasets & benchmarksCode Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
git clone https://github.com/dreadnode/AIRTBench-Code.git -
SGI-Bench
Datasets & benchmarksProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
git clone https://github.com/InternScience/SGI-Bench.git -
role-play-bench
Datasets & benchmarksHugging Face dataset by MiniMaxAI
MiniMaxAI/role-play-bench -
explicit-edit-benchmark
Datasets & benchmarksBenchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.
git clone https://github.com/alexshpunt/explicit-edit-benchmark.git -
HallusionBench
Datasets & benchmarks[CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models
git clone https://github.com/tianyi-lab/HallusionBench.git -
typed-decisions
Datasets & benchmarksHugging Face dataset by LocalLLaMA
LocalLLaMA/typed-decisions -
DeepResearch-Bench-II
Datasets & benchmarksDeepResearch Bench II (DRB2) is the follow-up to DeepResearch Bench, with a stronger focus on measuring the gap between deep research systems and human experts. It does so by decomposing expert-written reports into hierarchical rubrics covering presentation, analysis, and evidence, and using them to evaluate model-generated.
git clone https://github.com/imlrz/DeepResearch-Bench-II.git -
ArtifactsBenchmark
Datasets & benchmarksgit clone https://github.com/Tencent-Hunyuan/ArtifactsBenchmark.git -
PawBench
Datasets & benchmarksA benchmark for evaluating LLM × harness performance.
git clone https://github.com/agentscope-ai/PawBench.git -
MMLU-CF
Datasets & benchmarksA Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]
git clone https://github.com/microsoft/MMLU-CF.git -
LoopArena
Datasets & benchmarksBenchmarking models as runtime Controllers for Loop Engineering
git clone https://github.com/AMAP-ML/LoopArena.git -
LLM-RGB
Datasets & benchmarksLLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.
git clone https://github.com/babelcloud/LLM-RGB.git -
WritingBench
Datasets & benchmarksWritingBench: A Comprehensive Benchmark for Generative Writing
git clone https://github.com/X-PLUG/WritingBench.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.