Library
85,843 resources from 11 public sources. 51 checked by hand; the rest are unreviewed.
Results
-
ollama-benchmark
Datasets & benchmarksLLM Benchmark for Throughput via Ollama (Local LLMs)
git clone https://github.com/aidatatools/ollama-benchmark.git -
FastEval
Datasets & benchmarksFast & more realistic evaluation of chat language models. Includes leaderboard.
git clone https://github.com/FastEval/FastEval.git -
llm-colosseum
Datasets & benchmarksBenchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM
git clone https://github.com/OpenGenerativeAI/llm-colosseum.git -
BALROG
Datasets & benchmarksBenchmarking Agentic LLM and VLM Reasoning On Games
git clone https://github.com/balrog-ai/BALROG.git -
TurtleBench
Datasets & benchmarksTurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
git clone https://github.com/mazzzystar/TurtleBench.git -
MedXpertQA
Datasets & benchmarks[ICML 2025] MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
git clone https://github.com/TsinghuaC3I/MedXpertQA.git -
LongCite
Datasets & benchmarksLongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
git clone https://github.com/THUDM/LongCite.git -
active_genie
Datasets & benchmarksThe Lodash for GenAI: Real Value + Consistent + Model-Agnostic
git clone https://github.com/Roriz/active_genie.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
MobileWorld
Datasets & benchmarksBenchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (ACL 2026)
git clone https://github.com/Tongyi-MAI/MobileWorld.git -
geobench
Datasets & benchmarksGeoGuessr benchmark for language models
git clone https://github.com/ccmdi/geobench.git -
SciCode
Datasets & benchmarksA benchmark that challenges language models to code solutions for scientific problems
git clone https://github.com/scicode-bench/SciCode.git -
ping_pong_bench
Datasets & benchmarksA benchmark for role-playing language models
git clone https://github.com/IlyaGusev/ping_pong_bench.git -
PawBench
Datasets & benchmarksA benchmark for evaluating LLM × harness performance.
git clone https://github.com/agentscope-ai/PawBench.git -
MMLU-CF
Datasets & benchmarksA Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]
git clone https://github.com/microsoft/MMLU-CF.git -
minebench
Datasets & benchmarksMinecraft-style voxel benchmark for comparing AI models (Arena + Sandbox)
git clone https://github.com/Ammaar-Alam/minebench.git -
BizFinBench
Datasets & benchmarksA Business-Driven Real-World Financial Benchmark for Evaluating LLMs
git clone https://github.com/HiThink-Research/BizFinBench.git -
OSWorld
Datasets & benchmarks[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
git clone https://github.com/xlang-ai/OSWorld.git -
o11y-bench
Datasets & benchmarksOpen benchmark on observability tasks built on Harbor
git clone https://github.com/grafana/o11y-bench.git -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
LoopArena
Datasets & benchmarksBenchmarking models as runtime Controllers for Loop Engineering
git clone https://github.com/AMAP-ML/LoopArena.git -
chembench
Datasets & benchmarksHow good are LLMs at chemistry?
git clone https://github.com/lamalab-org/chembench.git -
LLM-Agent-Benchmark-List
Datasets & benchmarksA banchmark list for evaluation of large language models.
git clone https://github.com/zhangxjohn/LLM-Agent-Benchmark-List.git -
turnkeyml
Datasets & benchmarksNo-code CLI designed for accelerating ONNX workflows
git clone https://github.com/onnx/turnkeyml.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.