Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
DishwasherBench
Datasets & benchmarksA dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.
git clone https://github.com/composo-ai/DishwasherBench.git -
inferock-bench
Datasets & benchmarksLocal LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.
git clone https://github.com/inferock/inferock-bench.git -
primitive-bench
Datasets & benchmarksThe marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.
git clone https://github.com/primitive-bench/primitive-bench.git -
fde-gym
Datasets & benchmarksThe online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.
git clone https://github.com/deeplethe/fde-gym.git -
llm-bulls-and-cows-benchmark
Datasets & benchmarksA mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.
git clone https://github.com/stalkermustang/llm-bulls-and-cows-benchmark.git -
explicit-edit-benchmark
Datasets & benchmarksBenchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.
git clone https://github.com/alexshpunt/explicit-edit-benchmark.git -
nagi-bench
Datasets & benchmarksOne-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.
git clone https://github.com/nagi-studio/nagi-bench.git -
alem-env
Datasets & benchmarksCan LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).
git clone https://github.com/alem-world/alem-env.git -
mobilegym
Datasets & benchmarks[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training
git clone https://github.com/Purewhiter/mobilegym.git -
VibeSearchBench
Datasets & benchmarks🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.
git clone https://github.com/VibeBench/VibeSearchBench.git -
DeepResearch-Bench-II
Datasets & benchmarksDeepResearch Bench II (DRB2) is the follow-up to DeepResearch Bench, with a stronger focus on measuring the gap between deep research systems and human experts. It does so by decomposing expert-written reports into hierarchical rubrics covering presentation, analysis, and evidence, and using them to evaluate model-generated.
git clone https://github.com/imlrz/DeepResearch-Bench-II.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.