Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
SystemChat
Datasets & benchmarksHugging Face dataset by abacusai
abacusai/SystemChat -
HaluMem
Datasets & benchmarksHaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.
git clone https://github.com/MemTensor/HaluMem.git -
typed-decisions
Datasets & benchmarksHugging Face dataset by LocalLLaMA
LocalLLaMA/typed-decisions -
Awesome-AI-Scientist
Datasets & benchmarks🧪 Awesome list of AI Scientist papers, systems, benchmarks, datasets and open-source platforms.
git clone https://github.com/Omni-Scientist/Awesome-AI-Scientist.git -
LLM-RGB
Datasets & benchmarksLLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.
git clone https://github.com/babelcloud/LLM-RGB.git -
meta-agents-research-environments
Datasets & benchmarksMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
git clone https://github.com/facebookresearch/meta-agents-research-environments.git -
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
multi-agents
Datasets & benchmarksMulti-agent teams for real work: run teams of coding agents (Claude Code, Codex, any model) in isolated git worktrees, and measure whether the team beats a solo agent on cost and results.
git clone https://github.com/BichengWang/multi-agents.git -
DecisionPivot
Datasets & benchmarksDecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
git clone https://github.com/StartLuxLabs/DecisionPivot.git -
fde-gym
Datasets & benchmarksThe online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.
git clone https://github.com/deeplethe/fde-gym.git -
DeepResearch-Bench-II
Datasets & benchmarksDeepResearch Bench II (DRB2) is the follow-up to DeepResearch Bench, with a stronger focus on measuring the gap between deep research systems and human experts. It does so by decomposing expert-written reports into hierarchical rubrics covering presentation, analysis, and evidence, and using them to evaluate model-generated.
git clone https://github.com/imlrz/DeepResearch-Bench-II.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.