Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to build an AI agent
-
pi-mono
Datasets & benchmarksHugging Face dataset by badlogicgames
badlogicgames/pi-mono -
multi-agents
Datasets & benchmarksMulti-agent teams for real work: run teams of coding agents (Claude Code, Codex, any model) in isolated git worktrees, and measure whether the team beats a solo agent on cost and results.
git clone https://github.com/BichengWang/multi-agents.git -
claude-fable-5-claude-code
Datasets & benchmarksHugging Face dataset by armand0e
armand0e/claude-fable-5-claude-code -
jupyter-agent-dataset
Datasets & benchmarksHugging Face dataset by jupyter-agent
jupyter-agent/jupyter-agent-dataset -
explicit-edit-benchmark
Datasets & benchmarksBenchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.
git clone https://github.com/alexshpunt/explicit-edit-benchmark.git -
AIRTBench-Code
Datasets & benchmarksCode Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
git clone https://github.com/dreadnode/AIRTBench-Code.git -
doc2harness
Datasets & benchmarksOfficial Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "
git clone https://github.com/YuyaoGe/doc2harness.git -
OSWorld
Datasets & benchmarks[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
git clone https://github.com/xlang-ai/OSWorld.git -
openbench
Datasets & benchmarksSame model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost
git clone https://github.com/minghinmatthewlam/openbench.git -
agent-duel
Datasets & benchmarksBlinded battles between coding agents: two agents, one prompt, you pick the winner. Desktop app, runs locally.
git clone https://github.com/bottomless/agent-duel.git -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
Open-SWE-Traces
Datasets & benchmarksHugging Face dataset by nvidia
nvidia/Open-SWE-Traces -
dataclaw-peteromallet
Datasets & benchmarksHugging Face dataset by peteromallet
peteromallet/dataclaw-peteromallet -
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
LoopArena
Datasets & benchmarksBenchmarking models as runtime Controllers for Loop Engineering
git clone https://github.com/AMAP-ML/LoopArena.git -
AgentTrove
Datasets & benchmarksHugging Face dataset by open-thoughts
open-thoughts/AgentTrove -
Step-3.5-Flash-SFT
Datasets & benchmarksHugging Face dataset by stepfun-ai
stepfun-ai/Step-3.5-Flash-SFT -
self-bench
Datasets & benchmarksBenchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.
git clone https://github.com/mupt-ai/self-bench.git -
OctoCodingBench
Datasets & benchmarksHugging Face dataset by MiniMaxAI
MiniMaxAI/OctoCodingBench -
agent-orchestra
Datasets & benchmarksAgents can work. They cannot declare success. A governed coordination protocol for multi-agent AI work: eight contracts (task, resource, ownership, handoff, verification, evidence, approval, recovery) enforced in code, plus a reproducible conformance benchmark. Apache-2.0 code, CC BY 4.0 specification, zero dependencies, Node 18+.
git clone https://github.com/3338902669-ops/agent-orchestra.git -
fde-gym
Datasets & benchmarksThe online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.
git clone https://github.com/deeplethe/fde-gym.git -
UltraData-SFT-Agent-2609
Datasets & benchmarksHugging Face dataset by openbmb
openbmb/UltraData-SFT-Agent-2609 -
ResearchClawBench
Datasets & benchmarks🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery
git clone https://github.com/InternScience/ResearchClawBench.git -
SGI-Bench
Datasets & benchmarksProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
git clone https://github.com/InternScience/SGI-Bench.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.