InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. DishwasherBench

    Datasets & benchmarks

    A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.

    Stale GitHub, composo-ai 71 ★MITUpdated 30 days ago

    git clone https://github.com/composo-ai/DishwasherBench.git
  2. inferock-bench

    Datasets & benchmarks

    Local LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.

    Stale GitHub, inferock 107 ★Custom licence (see repository)Updated 3 months ago

    git clone https://github.com/inferock/inferock-bench.git
  3. primitive-bench

    Datasets & benchmarks

    The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.

    Stale GitHub, primitive-bench 122 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/primitive-bench/primitive-bench.git
  4. fde-gym

    Datasets & benchmarks

    The online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.

    Stale GitHub, deeplethe 111 ★Apache-2.0Updated 18 days ago

    git clone https://github.com/deeplethe/fde-gym.git
  5. llm-bulls-and-cows-benchmark

    Datasets & benchmarks

    A mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.

    Stale GitHub, stalkermustang 235 ★MITUpdated 21 months ago

    git clone https://github.com/stalkermustang/llm-bulls-and-cows-benchmark.git
  6. explicit-edit-benchmark

    Datasets & benchmarks

    Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

    Stale GitHub, alexshpunt 38 ★MITUpdated 7 days ago

    git clone https://github.com/alexshpunt/explicit-edit-benchmark.git
  7. nagi-bench

    Datasets & benchmarks

    One-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.

    Indexed GitHub, nagi-studio 78 ★MITUpdated 4 days ago

    git clone https://github.com/nagi-studio/nagi-bench.git
  8. alem-env

    Datasets & benchmarks

    Can LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).

    Stale GitHub, alem-world 57 ★MITUpdated 11 days ago

    git clone https://github.com/alem-world/alem-env.git
  9. mobilegym

    Datasets & benchmarks

    [EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training

    Indexed GitHub, Purewhiter 866 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/Purewhiter/mobilegym.git
  10. VibeSearchBench

    Datasets & benchmarks

    🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.

    Stale GitHub, VibeBench 112 ★MITUpdated 5 months ago

    git clone https://github.com/VibeBench/VibeSearchBench.git
  11. DeepResearch-Bench-II

    Datasets & benchmarks

    DeepResearch Bench II (DRB2) is the follow-up to DeepResearch Bench, with a stronger focus on measuring the gap between deep research systems and human experts. It does so by decomposing expert-written reports into hierarchical rubrics covering presentation, analysis, and evidence, and using them to evaluate model-generated.

    Stale GitHub, imlrz 96 ★Apache-2.0Updated 28 days ago

    git clone https://github.com/imlrz/DeepResearch-Bench-II.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.