InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. DyCodeEval

    Datasets & benchmarks

    Official repository of the ICML2025 paper “Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination”

    Stale GitHub, SeekingDream 222 ★Updated 10 months ago

    git clone https://github.com/SeekingDream/DyCodeEval.git
  2. Hugging Face dataset by gorilla-llm

    Stale Hugging Face Hub, gorilla-llm 135 ♥apache-2.0Updated 6 months ago

    gorilla-llm/Berkeley-Function-Calling-Leaderboard
  3. confabulations

    Datasets & benchmarks

    Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

    Stale GitHub, lechmazur 249 ★Updated 15 months ago

    git clone https://github.com/lechmazur/confabulations.git
  4. awesome-ai-gateway

    Datasets & benchmarks

    ⚡ Awesome AI Gateway — pick an AI gateway from 240+ (LiteLLM, OpenRouter, Portkey, Kong, Higress, new-api, Bifrost) by cost, security & compliance, see what consolidated in 2026, learn how they actually work from an 8-chapter handbook, and verify every number yourself. Reproducible benchmarks, bilingual; stars & releases refreshed daily.

    Indexed GitHub, cuihuan 158 ★CC0-1.0Updated yesterday

    git clone https://github.com/cuihuan/awesome-ai-gateway.git
  5. xFinder

    Datasets & benchmarks

    [ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

    Stale GitHub, IAAR-Shanghai 178 ★Custom licence (see repository)Updated 12 months ago

    git clone https://github.com/IAAR-Shanghai/xFinder.git
  6. DecisionPivot

    Datasets & benchmarks

    DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.

    Stale GitHub, StartLuxLabs 144 ★Apache-2.0Updated 26 days ago

    git clone https://github.com/StartLuxLabs/DecisionPivot.git
  7. AICGSecEval

    Datasets & benchmarks

    A.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.

    Stale GitHub, Tencent 664 ★Custom licence (see repository)Updated 5 months ago

    git clone https://github.com/Tencent/AICGSecEval.git
  8. SWE-bench-Live

    Datasets & benchmarks

    [NeurIPS 2025 D&B] 🚀 SWE-bench Goes Live!

    Stale GitHub, microsoft 262 ★MITUpdated 8 days ago

    git clone https://github.com/microsoft/SWE-bench-Live.git
  9. xlam-function-calling-60k

    Datasets & benchmarks

    Hugging Face dataset by Salesforce

    Indexed Hugging Face Hub, Salesforce 764 ♥cc-by-4.0Updated 22 months ago 1 note

    Salesforce/xlam-function-calling-60k
  10. ChemLLMBench

    Datasets & benchmarks

    Official Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)

    Stale GitHub, ChemFoundationModels 176 ★Updated 2 years ago

    git clone https://github.com/ChemFoundationModels/ChemLLMBench.git
  11. SkyPile-150B

    Datasets & benchmarks

    Hugging Face dataset by Skywork

    Indexed Hugging Face Hub, Skywork 421 ♥Updated 2 years ago

    Skywork/SkyPile-150B
  12. civ6-mcp

    Datasets & benchmarks

    An MCP server that lets LLM agents play Civilization VI.

    Stale GitHub, lmwilki 218 ★MITUpdated 34 days ago

    git clone https://github.com/lmwilki/civ6-mcp.git
  13. HaluMem

    Datasets & benchmarks

    HaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.

    Stale GitHub, MemTensor 178 ★Custom licence (see repository)Updated 19 days ago

    git clone https://github.com/MemTensor/HaluMem.git
  14. ParseBench

    Datasets & benchmarks

    ParseBench - A Document Parsing Benchmark for AI Agents

    Indexed GitHub, run-llama 640 ★Apache-2.0Updated today

    git clone https://github.com/run-llama/ParseBench.git
  15. PawBench

    Datasets & benchmarks

    A benchmark for evaluating LLM × harness performance.

    Stale GitHub, agentscope-ai 118 ★Apache-2.0Updated 3 months ago

    git clone https://github.com/agentscope-ai/PawBench.git
  16. NextTransformer

    Datasets & benchmarks

    :sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.

    Indexed GitHub, Interactive-Active-AI 435 ★MITUpdated today

    git clone https://github.com/Interactive-Active-AI/NextTransformer.git
  17. inferock-bench

    Datasets & benchmarks

    Local LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.

    Stale GitHub, inferock 107 ★Custom licence (see repository)Updated 3 months ago

    git clone https://github.com/inferock/inferock-bench.git
  18. autoguardrails

    Datasets & benchmarks

    Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

    Stale GitHub, SantanderAI 129 ★Apache-2.0Updated 46 days ago

    git clone https://github.com/SantanderAI/autoguardrails.git
  19. MobileWorld

    Datasets & benchmarks

    Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (ACL 2026)

    Stale GitHub, Tongyi-MAI 300 ★Apache-2.0Updated 13 days ago

    git clone https://github.com/Tongyi-MAI/MobileWorld.git
  20. ChinaTravel

    Datasets & benchmarks

    ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning

    Stale GitHub, LAMDA-NeSy 133 ★MITUpdated 2 months ago

    git clone https://github.com/LAMDA-NeSy/ChinaTravel.git
  21. WideSearch

    Datasets & benchmarks

    WideSearch: Benchmarking Agentic Broad Info-Seeking

    Stale GitHub, ByteDance-Seed 156 ★MITUpdated 13 months ago

    git clone https://github.com/ByteDance-Seed/WideSearch.git
  22. minebench

    Datasets & benchmarks

    Minecraft-style voxel benchmark for comparing AI models (Arena + Sandbox)

    Indexed GitHub, Ammaar-Alam 343 ★MITUpdated 4 days ago

    git clone https://github.com/Ammaar-Alam/minebench.git
  23. FastEval

    Datasets & benchmarks

    Fast & more realistic evaluation of chat language models. Includes leaderboard.

    Stale GitHub, FastEval 187 ★Apache-2.0Updated 2 years ago

    git clone https://github.com/FastEval/FastEval.git
  24. strix-halo-guide

    Datasets & benchmarks

    Evidence-backed AMD Strix Halo local-AI setup and benchmarks: Qwen3.8, Ollama, llama.cpp, Vulkan/ROCm, large GGUFs, and cross-OEM results.

    Stale GitHub, hogeheer499-commits 373 ★MITUpdated 7 days ago

    git clone https://github.com/hogeheer499-commits/strix-halo-guide.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.