InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. PawBench

    Datasets & benchmarks

    A benchmark for evaluating LLM × harness performance.

    Stale GitHub, agentscope-ai 118 ★Apache-2.0Updated 3 months ago

    git clone https://github.com/agentscope-ai/PawBench.git
  2. HallusionBench

    Datasets & benchmarks

    [CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models

    Stale GitHub, tianyi-lab 344 ★BSD-3-ClauseUpdated 13 months ago

    git clone https://github.com/tianyi-lab/HallusionBench.git
  3. MMLU-CF

    Datasets & benchmarks

    A Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]

    Stale GitHub, microsoft 127 ★MITUpdated 18 months ago

    git clone https://github.com/microsoft/MMLU-CF.git
  4. ArtifactsBenchmark

    Datasets & benchmarks

    Stale GitHub, Tencent-Hunyuan 280 ★Custom licence (see repository)Updated 10 months ago

    git clone https://github.com/Tencent-Hunyuan/ArtifactsBenchmark.git
  5. LoopArena

    Datasets & benchmarks

    Benchmarking models as runtime Controllers for Loop Engineering

    Stale GitHub, AMAP-ML 151 ★Apache-2.0Updated 32 days ago

    git clone https://github.com/AMAP-ML/LoopArena.git
  6. LLM-RGB

    Datasets & benchmarks

    LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

    Stale GitHub, babelcloud 165 ★MITUpdated 18 months ago

    git clone https://github.com/babelcloud/LLM-RGB.git
  7. typed-decisions

    Datasets & benchmarks

    Hugging Face dataset by LocalLLaMA

    Indexed Hugging Face Hub, LocalLLaMA 277 ♥apache-2.0Updated today

    LocalLLaMA/typed-decisions
  8. WritingBench

    Datasets & benchmarks

    WritingBench: A Comprehensive Benchmark for Generative Writing

    Stale GitHub, X-PLUG 202 ★Apache-2.0Updated 11 months ago

    git clone https://github.com/X-PLUG/WritingBench.git
  9. geobench

    Datasets & benchmarks

    GeoGuessr benchmark for language models

    Indexed GitHub, ccmdi 65 ★MITUpdated 3 days ago

    git clone https://github.com/ccmdi/geobench.git
  10. o11y-bench

    Datasets & benchmarks

    Open benchmark on observability tasks built on Harbor

    Indexed GitHub, grafana 80 ★AGPL-3.0Updated today

    git clone https://github.com/grafana/o11y-bench.git
  11. humanities_data_benchmark

    Datasets & benchmarks

    LLM Benchmark Suite for Humanities Data

    Stale GitHub, RISE-UNIBAS 32 ★GPL-3.0Updated 12 days ago

    git clone https://github.com/RISE-UNIBAS/humanities_data_benchmark.git
  12. Static-to-Dynamic-LLMEval

    Datasets & benchmarks

    The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"

    Stale GitHub, SeekingDream 493 ★Updated 49 days ago

    git clone https://github.com/SeekingDream/Static-to-Dynamic-LLMEval.git
  13. llm-price-compass

    Datasets & benchmarks

    This project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider price comparison, gpu benchmarks to price per token calculation, gpu benchmark table

    Stale GitHub, arc53 221 ★MITUpdated 23 months ago

    git clone https://github.com/arc53/llm-price-compass.git
  14. SWE-bench-Live

    Datasets & benchmarks

    [NeurIPS 2025 D&B] 🚀 SWE-bench Goes Live!

    Stale GitHub, microsoft 262 ★MITUpdated 8 days ago

    git clone https://github.com/microsoft/SWE-bench-Live.git
  15. GDPevo

    Datasets & benchmarks

    A Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks

    Stale GitHub, Prism-Shadow 119 ★Apache-2.0Updated 18 days ago

    git clone https://github.com/Prism-Shadow/GDPevo.git
  16. frames-benchmark

    Datasets & benchmarks

    Hugging Face dataset by google

    Indexed Hugging Face Hub, google 276 ♥apache-2.0Updated 2 years ago

    google/frames-benchmark
  17. aa-agentperf-local

    Datasets & benchmarks

    Benchmark local LLM serving by replaying real agent trajectories

    Stale GitHub, ArtificialAnalysis 105 ★Apache-2.0Updated 31 days ago

    git clone https://github.com/ArtificialAnalysis/aa-agentperf-local.git
  18. nautilus-compass

    Datasets & benchmarks

    Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

    Indexed GitHub, chunxiaoxx 1,258 ★Custom licence (see repository)Updated 2 days ago

    git clone https://github.com/chunxiaoxx/nautilus-compass.git
  19. GAIA

    Datasets & benchmarks

    Hugging Face dataset by gaia-benchmark

    Indexed Hugging Face Hub, gaia-benchmark 971 ♥Updated 12 months ago 1 note

    gaia-benchmark/GAIA
  20. PhysGym

    Datasets & benchmarks

    A benchmark suite for evaluating LLM-based interactive scientific reasoning.

    Indexed GitHub, principia-ai 103 ★MITUpdated 3 days ago

    git clone https://github.com/principia-ai/PhysGym.git
  21. LLM-eval-survey

    Datasets & benchmarks

    The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

    Stale GitHub, MLGroupJLU 1,615 ★Updated 7 days ago

    git clone https://github.com/MLGroupJLU/LLM-eval-survey.git
  22. aurora

    Datasets & benchmarks

    [ICLR 2025] AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

    Stale GitHub, wenhaochai 149 ★Apache-2.0Updated 17 months ago

    git clone https://github.com/wenhaochai/aurora.git
  23. Awesome-Scientific-LLM-Benchmarks

    Datasets & benchmarks

    A curated, accuracy-first list of benchmarks for evaluating LLMs on scientific reasoning and discovery — math, physics, chemistry, materials, biology, and agentic science.

    Stale GitHub, subinium 42 ★MITUpdated 14 days ago

    git clone https://github.com/subinium/Awesome-Scientific-LLM-Benchmarks.git
  24. doc2harness

    Datasets & benchmarks

    Official Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "

    Stale GitHub, YuyaoGe 116 ★Custom licence (see repository)Updated 20 days ago

    git clone https://github.com/YuyaoGe/doc2harness.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.