Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
BALROG
Datasets & benchmarksBenchmarking Agentic LLM and VLM Reasoning On Games
git clone https://github.com/balrog-ai/BALROG.git -
SWE-bench
Datasets & benchmarksHugging Face dataset by princeton-nlp
princeton-nlp/SWE-bench -
OneMillion-Bench
Datasets & benchmarks[NeurIPS 2026] $OneMillion-Bench: How Far are Language Agents from Human Experts?
git clone https://github.com/humanlaya/OneMillion-Bench.git -
leaderboard
Tools & appsHugging Face space by agent-memory-leaderboard
agent-memory-leaderboard/leaderboard -
ParseBench
Datasets & benchmarksHugging Face dataset by llamaindex
llamaindex/ParseBench -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
WideSearch
Datasets & benchmarksWideSearch: Benchmarking Agentic Broad Info-Seeking
git clone https://github.com/ByteDance-Seed/WideSearch.git -
DyCodeEval
Datasets & benchmarksOfficial repository of the ICML2025 paper “Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination”
git clone https://github.com/SeekingDream/DyCodeEval.git -
ClawProBench
Datasets & benchmarksClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
git clone https://github.com/suyoumo/ClawProBench.git -
MobileWorld
Datasets & benchmarksBenchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (ACL 2026)
git clone https://github.com/Tongyi-MAI/MobileWorld.git -
confabulations
Datasets & benchmarksHallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
git clone https://github.com/lechmazur/confabulations.git -
ParseBench
Datasets & benchmarksParseBench - A Document Parsing Benchmark for AI Agents
git clone https://github.com/run-llama/ParseBench.git -
ChemLLMBench
Datasets & benchmarksOfficial Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)
git clone https://github.com/ChemFoundationModels/ChemLLMBench.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
Simreal-MLBench
Datasets & benchmarks[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
git clone https://github.com/Simreal-AI/Simreal-MLBench.git -
beir
AI repositoriesA Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
git clone https://github.com/beir-cellar/beir.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
GLM-5.3-Flash × J-Space capability realization — benchmark presentation of the J-Space Cognition Suite
git clone https://github.com/Tiger3807861189/GLM-5.3-Flash-J-Space-Capability-Realization-Report.git -
hle
Datasets & benchmarksHugging Face dataset by cais
cais/hle -
FDE-Gym
Datasets & benchmarksHugging Face dataset by DeepLethe
DeepLethe/FDE-Gym -
primitive-bench
Datasets & benchmarksThe marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.
git clone https://github.com/primitive-bench/primitive-bench.git -
inferock-bench
Datasets & benchmarksLocal LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.
git clone https://github.com/inferock/inferock-bench.git -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
deepmark
Datasets & benchmarksDeepmark AI enables a unique testing environment for language models (LLM) assessment on task-specific metrics and on your own data so your GenAI-powered solution has predictable and reliable performance.
git clone https://github.com/IngestAI/deepmark.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.