InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. AI_Diplomacy

    Datasets & benchmarks

    Frontier Models playing the board game Diplomacy.

    Stale GitHub, GoodStartLabs 708 ★Custom licence (see repository)Updated 5 months ago

    git clone https://github.com/GoodStartLabs/AI_Diplomacy.git
  2. tau2-bench

    Datasets & benchmarks

    τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Indexed GitHub, sierra-research 2,442 ★MITUpdated 6 days ago

    git clone https://github.com/sierra-research/tau2-bench.git
  3. TheAgentCompany

    Datasets & benchmarks

    An agent benchmark with tasks in a simulated software company.

    Stale GitHub, TheAgentCompany 795 ★MITUpdated 12 months ago

    git clone https://github.com/TheAgentCompany/TheAgentCompany.git
  4. MedXpertQA

    Datasets & benchmarks

    [ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

    Stale GitHub, TsinghuaC3I 182 ★MITUpdated 2 months ago

    git clone https://github.com/TsinghuaC3I/MedXpertQA.git
  5. SciCode

    Datasets & benchmarks

    A benchmark that challenges language models to code solutions for scientific problems

    Stale GitHub, scicode-bench 246 ★Apache-2.0Updated 14 days ago

    git clone https://github.com/scicode-bench/SciCode.git
  6. o11y-bench

    Datasets & benchmarks

    Open benchmark on observability tasks built on Harbor

    Indexed GitHub, grafana 80 ★AGPL-3.0Updated today

    git clone https://github.com/grafana/o11y-bench.git
  7. NoLiMa

    Datasets & benchmarks

    Official repository for "NoLiMa: Long-Context Evaluation Beyond Literal Matching"

    Stale GitHub, adobe-research 205 ★Custom licence (see repository)Updated 16 months ago

    git clone https://github.com/adobe-research/NoLiMa.git
  8. arena-ai-leaderboards

    Datasets & benchmarks

    📊 Daily auto-updated snapshots of all Arena AI (LMSYS Chatbot Arena) leaderboards — LLM, Vision, Code, Video, Image & more. Structured JSON with historical tracking.

    Indexed GitHub, oolong-tea-2026 42 ★MITUpdated yesterday

    git clone https://github.com/oolong-tea-2026/arena-ai-leaderboards.git
  9. WritingBench

    Datasets & benchmarks

    WritingBench: A Comprehensive Benchmark for Generative Writing

    Stale GitHub, X-PLUG 202 ★Apache-2.0Updated 11 months ago

    git clone https://github.com/X-PLUG/WritingBench.git
  10. AI Benchmark Index

    MCP servers

    Search 264 AI benchmarks by category, with data sources and licences for each.

    Stale Official MCP Registry, com.aibenchmarkindex Updated 31 days ago 1 note

  11. GDPevo

    Datasets & benchmarks

    A Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks

    Stale GitHub, Prism-Shadow 119 ★Apache-2.0Updated 18 days ago

    git clone https://github.com/Prism-Shadow/GDPevo.git
  12. Imagine AI Benchmark

    MCP servers

    LinkedIn content benchmark for 900+ B2B companies and their teams, with an open scoring method.

    Stale Official MCP Registry, ai.useimagine Updated 37 days ago 1 note

  13. somali-language-standard

    AI repositories

    The open, machine-readable standard for the Somali language — orthography, grammar, terminology, translation, and AI/benchmark resources, as a versioned, citable RFC-style catalog.

    Stale GitHub, somastllc 108 ★MITUpdated 19 days ago

    git clone https://github.com/somastllc/somali-language-standard.git
  14. ChinaTravel

    Datasets & benchmarks

    ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning

    Stale GitHub, LAMDA-NeSy 133 ★MITUpdated 2 months ago

    git clone https://github.com/LAMDA-NeSy/ChinaTravel.git
  15. HaluMem

    Datasets & benchmarks

    HaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.

    Stale GitHub, MemTensor 178 ★Custom licence (see repository)Updated 19 days ago

    git clone https://github.com/MemTensor/HaluMem.git
  16. mas-lab

    Agents & frameworks

    Specification-driven multi-agent systems with OpenTelemetry observability, agent traces, benchmarking, MCP, and A2A.

    Indexed GitHub, ioc-project 65 ★Apache-2.0Updated yesterday

    git clone https://github.com/ioc-project/mas-lab.git
  17. NIMStats

    Datasets & benchmarks

    📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.

    Indexed GitHub, MauroDruwel 57 ★Updated today

    git clone https://github.com/MauroDruwel/NIMStats.git
  18. nautilus-compass

    Datasets & benchmarks

    Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

    Indexed GitHub, chunxiaoxx 1,258 ★Custom licence (see repository)Updated 2 days ago

    git clone https://github.com/chunxiaoxx/nautilus-compass.git
  19. Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

    Indexed GitHub, NVIDIA 1,963 ★Apache-2.0Updated today

    git clone https://github.com/NVIDIA/SkillEvaluator.git
  20. VibeSearchBench

    Datasets & benchmarks

    🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.

    Stale GitHub, VibeBench 112 ★MITUpdated 5 months ago

    git clone https://github.com/VibeBench/VibeSearchBench.git
  21. agentic-ai

    AI repositories

    Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

    Indexed GitHub, mahmoudrabie 193 ★MITUpdated 2 days ago

    git clone https://github.com/mahmoudrabie/agentic-ai.git
  22. benchmarks

    Agents & frameworks

    Reproducible voice-AI benchmarking — TTS / STT / S2S latency and accuracy.

    Stale GitHub, coval-ai 104 ★Custom licence (see repository)Updated 24 days ago

    git clone https://github.com/coval-ai/benchmarks.git
  23. bg3-agent-harness

    MCP servers

    An AI agent plays Baldur's Gate 3 on Linux through a Lua mod and an MCP server, seeing only what a player sees. Includes a model benchmark and a playthrough app.

    Stale GitHub, Bogzx 33 ★MITUpdated 7 days ago

    git clone https://github.com/Bogzx/bg3-agent-harness.git
  24. hvilkenai-mcp

    MCP servers

    Daily Scandinavian AI benchmark — Norwegian, Swedish, Danish. 12+ models tested.

    Stale Official MCP Registry, io.github.erorund Updated 5 months ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.