Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
AI_Diplomacy
Datasets & benchmarksFrontier Models playing the board game Diplomacy.
git clone https://github.com/GoodStartLabs/AI_Diplomacy.git -
tau2-bench
Datasets & benchmarksτ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
git clone https://github.com/sierra-research/tau2-bench.git -
TheAgentCompany
Datasets & benchmarksAn agent benchmark with tasks in a simulated software company.
git clone https://github.com/TheAgentCompany/TheAgentCompany.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
SciCode
Datasets & benchmarksA benchmark that challenges language models to code solutions for scientific problems
git clone https://github.com/scicode-bench/SciCode.git -
o11y-bench
Datasets & benchmarksOpen benchmark on observability tasks built on Harbor
git clone https://github.com/grafana/o11y-bench.git -
NoLiMa
Datasets & benchmarksOfficial repository for "NoLiMa: Long-Context Evaluation Beyond Literal Matching"
git clone https://github.com/adobe-research/NoLiMa.git -
arena-ai-leaderboards
Datasets & benchmarks📊 Daily auto-updated snapshots of all Arena AI (LMSYS Chatbot Arena) leaderboards — LLM, Vision, Code, Video, Image & more. Structured JSON with historical tracking.
git clone https://github.com/oolong-tea-2026/arena-ai-leaderboards.git -
WritingBench
Datasets & benchmarksWritingBench: A Comprehensive Benchmark for Generative Writing
git clone https://github.com/X-PLUG/WritingBench.git -
AI Benchmark Index
MCP serversSearch 264 AI benchmarks by category, with data sources and licences for each.
-
GDPevo
Datasets & benchmarksA Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks
git clone https://github.com/Prism-Shadow/GDPevo.git -
Imagine AI Benchmark
MCP serversLinkedIn content benchmark for 900+ B2B companies and their teams, with an open scoring method.
-
somali-language-standard
AI repositoriesThe open, machine-readable standard for the Somali language — orthography, grammar, terminology, translation, and AI/benchmark resources, as a versioned, citable RFC-style catalog.
git clone https://github.com/somastllc/somali-language-standard.git -
ChinaTravel
Datasets & benchmarksChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning
git clone https://github.com/LAMDA-NeSy/ChinaTravel.git -
HaluMem
Datasets & benchmarksHaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.
git clone https://github.com/MemTensor/HaluMem.git -
mas-lab
Agents & frameworksSpecification-driven multi-agent systems with OpenTelemetry observability, agent traces, benchmarking, MCP, and A2A.
git clone https://github.com/ioc-project/mas-lab.git -
NIMStats
Datasets & benchmarks📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.
git clone https://github.com/MauroDruwel/NIMStats.git -
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
SkillEvaluator
SkillsMulti-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
git clone https://github.com/NVIDIA/SkillEvaluator.git -
VibeSearchBench
Datasets & benchmarks🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.
git clone https://github.com/VibeBench/VibeSearchBench.git -
agentic-ai
AI repositoriesAgentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.
git clone https://github.com/mahmoudrabie/agentic-ai.git -
benchmarks
Agents & frameworksReproducible voice-AI benchmarking — TTS / STT / S2S latency and accuracy.
git clone https://github.com/coval-ai/benchmarks.git -
bg3-agent-harness
MCP serversAn AI agent plays Baldur's Gate 3 on Linux through a Lua mod and an MCP server, seeing only what a player sees. Includes a model benchmark and a playthrough app.
git clone https://github.com/Bogzx/bg3-agent-harness.git -
hvilkenai-mcp
MCP serversDaily Scandinavian AI benchmark — Norwegian, Swedish, Danish. 12+ models tested.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.