Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
TurtleBench
Datasets & benchmarksTurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
git clone https://github.com/mazzzystar/TurtleBench.git -
multi-agents
Datasets & benchmarksMulti-agent teams for real work: run teams of coding agents (Claude Code, Codex, any model) in isolated git worktrees, and measure whether the team beats a solo agent on cost and results.
git clone https://github.com/BichengWang/multi-agents.git -
awesome-ai-gateway
Datasets & benchmarks⚡ Awesome AI Gateway — pick an AI gateway from 240+ (LiteLLM, OpenRouter, Portkey, Kong, Higress, new-api, Bifrost) by cost, security & compliance, see what consolidated in 2026, learn how they actually work from an 8-chapter handbook, and verify every number yourself. Reproducible benchmarks, bilingual; stars & releases refreshed daily.
git clone https://github.com/cuihuan/awesome-ai-gateway.git -
WideSearch
Datasets & benchmarksWideSearch: Benchmarking Agentic Broad Info-Seeking
git clone https://github.com/ByteDance-Seed/WideSearch.git -
llmspeed
Datasets & benchmarksFeel how fast a local LLM runs before you buy the hardware
git clone https://github.com/nicorobinreal/llmspeed.git -
LoopArena
Datasets & benchmarksBenchmarking models as runtime Controllers for Loop Engineering
git clone https://github.com/AMAP-ML/LoopArena.git -
xVerify
Datasets & benchmarksxVerify: Efficient Answer Verifier for Reasoning Model Evaluations
git clone https://github.com/IAAR-Shanghai/xVerify.git -
aurora
Datasets & benchmarks[ICLR 2025] AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
git clone https://github.com/wenhaochai/aurora.git -
lora-speedrun
Datasets & benchmarksSpeedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.
git clone https://github.com/Saivineeth147/lora-speedrun.git -
Awesome-Datasets-Hub
Datasets & benchmarksA curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.
git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
ReForm-Eval
Datasets & benchmarksAn benchmark for evaluating the capabilities of large vision-language models (LVLMs)
git clone https://github.com/FudanDISC/ReForm-Eval.git -
DecisionPivot
Datasets & benchmarksDecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
git clone https://github.com/StartLuxLabs/DecisionPivot.git -
openbench
Datasets & benchmarksSame model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost
git clone https://github.com/minghinmatthewlam/openbench.git -
sec-code-bench
Datasets & benchmarksSecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).
git clone https://github.com/alibaba/sec-code-bench.git -
ChinaTravel
Datasets & benchmarksChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning
git clone https://github.com/LAMDA-NeSy/ChinaTravel.git -
autoguardrails
Datasets & benchmarksAlignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface
git clone https://github.com/SantanderAI/autoguardrails.git -
MMLU-CF
Datasets & benchmarksA Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]
git clone https://github.com/microsoft/MMLU-CF.git -
ping_pong_bench
Datasets & benchmarksA benchmark for role-playing language models
git clone https://github.com/IlyaGusev/ping_pong_bench.git -
primitive-bench
Datasets & benchmarksThe marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.
git clone https://github.com/primitive-bench/primitive-bench.git -
GDPevo
Datasets & benchmarksA Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks
git clone https://github.com/Prism-Shadow/GDPevo.git -
PawBench
Datasets & benchmarksA benchmark for evaluating LLM × harness performance.
git clone https://github.com/agentscope-ai/PawBench.git -
TwinFix
Datasets & benchmarksTwinFix — AI agent that diagnoses, reproduces and fixes bugs in a disposable twin environment. 在一次性孪生环境中自动诊断、复现并修复故障的 AI Agent。
git clone https://github.com/spacesky-cell/TwinFix.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.