InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. o11y-bench

    Datasets & benchmarks

    Open benchmark on observability tasks built on Harbor

    Indexed GitHub, grafana 80 ★AGPL-3.0Updated today

    git clone https://github.com/grafana/o11y-bench.git
  2. humanities_data_benchmark

    Datasets & benchmarks

    LLM Benchmark Suite for Humanities Data

    Stale GitHub, RISE-UNIBAS 32 ★GPL-3.0Updated 12 days ago

    git clone https://github.com/RISE-UNIBAS/humanities_data_benchmark.git
  3. Gym

    AI repositories

    Evaluate and improve models and agents using environments

    Indexed GitHub, NVIDIA-NeMo 1,334 ★Apache-2.0Updated today

    git clone https://github.com/NVIDIA-NeMo/Gym.git
  4. Static-to-Dynamic-LLMEval

    Datasets & benchmarks

    The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"

    Stale GitHub, SeekingDream 493 ★Updated 49 days ago

    git clone https://github.com/SeekingDream/Static-to-Dynamic-LLMEval.git
  5. awesome-web-agents

    Docs & tutorials

    🔥 A list of tools, frameworks, and resources for building AI web agents

    Indexed GitHub, steel-dev 1,609 ★Custom licence (see repository)Updated 6 days ago

    git clone https://github.com/steel-dev/awesome-web-agents.git
  6. llm-price-compass

    Datasets & benchmarks

    This project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider price comparison, gpu benchmarks to price per token calculation, gpu benchmark table

    Stale GitHub, arc53 221 ★MITUpdated 23 months ago

    git clone https://github.com/arc53/llm-price-compass.git
  7. SWE-bench-Live

    Datasets & benchmarks

    [NeurIPS 2025 D&B] 🚀 SWE-bench Goes Live!

    Stale GitHub, microsoft 262 ★MITUpdated 8 days ago

    git clone https://github.com/microsoft/SWE-bench-Live.git
  8. Bittune

    Agents & frameworks

    Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.

    Indexed GitHub, bitcloud-ai 304 ★MITUpdated 3 days ago

    git clone https://github.com/bitcloud-ai/Bittune.git
  9. GDPevo

    Datasets & benchmarks

    A Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks

    Stale GitHub, Prism-Shadow 119 ★Apache-2.0Updated 18 days ago

    git clone https://github.com/Prism-Shadow/GDPevo.git
  10. frames-benchmark

    Datasets & benchmarks

    Hugging Face dataset by google

    Indexed Hugging Face Hub, google 276 ♥apache-2.0Updated 2 years ago

    google/frames-benchmark
  11. toon

    AI repositories

    🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

    Stale GitHub, toon-format 25.5k ★MITUpdated 38 days ago

    git clone https://github.com/toon-format/toon.git
  12. aa-agentperf-local

    Datasets & benchmarks

    Benchmark local LLM serving by replaying real agent trajectories

    Stale GitHub, ArtificialAnalysis 105 ★Apache-2.0Updated 31 days ago

    git clone https://github.com/ArtificialAnalysis/aa-agentperf-local.git
  13. nautilus-compass

    Datasets & benchmarks

    Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

    Indexed GitHub, chunxiaoxx 1,258 ★Custom licence (see repository)Updated 2 days ago

    git clone https://github.com/chunxiaoxx/nautilus-compass.git
  14. GAIA

    Datasets & benchmarks

    Hugging Face dataset by gaia-benchmark

    Indexed Hugging Face Hub, gaia-benchmark 971 ♥Updated 12 months ago 1 note

    gaia-benchmark/GAIA
  15. PhysGym

    Datasets & benchmarks

    A benchmark suite for evaluating LLM-based interactive scientific reasoning.

    Indexed GitHub, principia-ai 103 ★MITUpdated 3 days ago

    git clone https://github.com/principia-ai/PhysGym.git
  16. LLM-eval-survey

    Datasets & benchmarks

    The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

    Stale GitHub, MLGroupJLU 1,615 ★Updated 7 days ago

    git clone https://github.com/MLGroupJLU/LLM-eval-survey.git
  17. aurora

    Datasets & benchmarks

    [ICLR 2025] AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

    Stale GitHub, wenhaochai 149 ★Apache-2.0Updated 17 months ago

    git clone https://github.com/wenhaochai/aurora.git
  18. Awesome-Scientific-LLM-Benchmarks

    Datasets & benchmarks

    A curated, accuracy-first list of benchmarks for evaluating LLMs on scientific reasoning and discovery — math, physics, chemistry, materials, biology, and agentic science.

    Stale GitHub, subinium 42 ★MITUpdated 14 days ago

    git clone https://github.com/subinium/Awesome-Scientific-LLM-Benchmarks.git
  19. doc2harness

    Datasets & benchmarks

    Official Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "

    Stale GitHub, YuyaoGe 116 ★Custom licence (see repository)Updated 20 days ago

    git clone https://github.com/YuyaoGe/doc2harness.git
  20. GateMem

    Agents & frameworks

    GateMem: a benchmark and evaluation toolkit for memory governance in multi-principal shared-memory LLM agents.

    Stale GitHub, rzhub 139 ★MITUpdated 35 days ago

    git clone https://github.com/rzhub/GateMem.git
  21. Awesome-Datasets-Hub

    Datasets & benchmarks

    A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

    Stale GitHub, ahammadmejbah 149 ★Updated 4 months ago

    git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git
  22. sec-code-bench

    Datasets & benchmarks

    SecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).

    Stale GitHub, alibaba 135 ★Apache-2.0Updated 35 days ago

    git clone https://github.com/alibaba/sec-code-bench.git
  23. OSWorld-V2

    Datasets & benchmarks

    OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    Indexed GitHub, xlang-ai 421 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/xlang-ai/OSWorld-V2.git
  24. LongCite

    Datasets & benchmarks

    LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

    Stale GitHub, THUDM 523 ★Apache-2.0Updated 22 months ago

    git clone https://github.com/THUDM/LongCite.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.