Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
o11y-bench
Datasets & benchmarksOpen benchmark on observability tasks built on Harbor
git clone https://github.com/grafana/o11y-bench.git -
humanities_data_benchmark
Datasets & benchmarksLLM Benchmark Suite for Humanities Data
git clone https://github.com/RISE-UNIBAS/humanities_data_benchmark.git -
Gym
AI repositoriesEvaluate and improve models and agents using environments
git clone https://github.com/NVIDIA-NeMo/Gym.git -
Static-to-Dynamic-LLMEval
Datasets & benchmarksThe official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"
git clone https://github.com/SeekingDream/Static-to-Dynamic-LLMEval.git -
awesome-web-agents
Docs & tutorials🔥 A list of tools, frameworks, and resources for building AI web agents
git clone https://github.com/steel-dev/awesome-web-agents.git -
llm-price-compass
Datasets & benchmarksThis project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider price comparison, gpu benchmarks to price per token calculation, gpu benchmark table
git clone https://github.com/arc53/llm-price-compass.git -
SWE-bench-Live
Datasets & benchmarks[NeurIPS 2025 D&B] 🚀 SWE-bench Goes Live!
git clone https://github.com/microsoft/SWE-bench-Live.git -
Bittune
Agents & frameworksLocal-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.
git clone https://github.com/bitcloud-ai/Bittune.git -
GDPevo
Datasets & benchmarksA Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks
git clone https://github.com/Prism-Shadow/GDPevo.git -
frames-benchmark
Datasets & benchmarksHugging Face dataset by google
google/frames-benchmark -
toon
AI repositories🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
git clone https://github.com/toon-format/toon.git -
aa-agentperf-local
Datasets & benchmarksBenchmark local LLM serving by replaying real agent trajectories
git clone https://github.com/ArtificialAnalysis/aa-agentperf-local.git -
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
GAIA
Datasets & benchmarksHugging Face dataset by gaia-benchmark
gaia-benchmark/GAIA -
PhysGym
Datasets & benchmarksA benchmark suite for evaluating LLM-based interactive scientific reasoning.
git clone https://github.com/principia-ai/PhysGym.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
aurora
Datasets & benchmarks[ICLR 2025] AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
git clone https://github.com/wenhaochai/aurora.git -
Awesome-Scientific-LLM-Benchmarks
Datasets & benchmarksA curated, accuracy-first list of benchmarks for evaluating LLMs on scientific reasoning and discovery — math, physics, chemistry, materials, biology, and agentic science.
git clone https://github.com/subinium/Awesome-Scientific-LLM-Benchmarks.git -
doc2harness
Datasets & benchmarksOfficial Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "
git clone https://github.com/YuyaoGe/doc2harness.git -
GateMem
Agents & frameworksGateMem: a benchmark and evaluation toolkit for memory governance in multi-principal shared-memory LLM agents.
git clone https://github.com/rzhub/GateMem.git -
Awesome-Datasets-Hub
Datasets & benchmarksA curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.
git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git -
sec-code-bench
Datasets & benchmarksSecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).
git clone https://github.com/alibaba/sec-code-bench.git -
OSWorld-V2
Datasets & benchmarksOSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
git clone https://github.com/xlang-ai/OSWorld-V2.git -
LongCite
Datasets & benchmarksLongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
git clone https://github.com/THUDM/LongCite.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.