Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
ollama-benchmark
Datasets & benchmarksLLM Benchmark for Throughput via Ollama (Local LLMs)
git clone https://github.com/aidatatools/ollama-benchmark.git -
TheAgentCompany
Datasets & benchmarksAn agent benchmark with tasks in a simulated software company.
git clone https://github.com/TheAgentCompany/TheAgentCompany.git -
AI_Diplomacy
Datasets & benchmarksFrontier Models playing the board game Diplomacy.
git clone https://github.com/GoodStartLabs/AI_Diplomacy.git -
ClawBench
Datasets & benchmarksOpen-source benchmark for browser AI agents on daily tasks.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git -
swe-sweep
Datasets & benchmarksHow many bugs can LMs find & fix in large codebases?
git clone https://github.com/facebookresearch/swe-sweep.git -
tau2-bench
Datasets & benchmarksτ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
git clone https://github.com/sierra-research/tau2-bench.git -
agents-last-exam
Datasets & benchmarksHugging Face dataset by agents-last-exam
agents-last-exam/agents-last-exam -
gsm8k
Datasets & benchmarksHugging Face dataset by openai
openai/gsm8k -
WebsiteBench
Datasets & benchmarksCan AI agents rebuild websites through browser exploration?
git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git -
minebench
Datasets & benchmarksMinecraft-style voxel benchmark for comparing AI models (Arena + Sandbox)
git clone https://github.com/Ammaar-Alam/minebench.git -
llm-bulls-and-cows-benchmark
Datasets & benchmarksA mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.
git clone https://github.com/stalkermustang/llm-bulls-and-cows-benchmark.git -
olmOCR-bench
Datasets & benchmarksHugging Face dataset by allenai
allenai/olmOCR-bench -
llm-colosseum
Datasets & benchmarksBenchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM
git clone https://github.com/OpenGenerativeAI/llm-colosseum.git -
ollama-benchmark
Datasets & benchmarksOllama based Benchmark with detail I/O token per second. Python with Deepseek R1 example.
git clone https://github.com/LarHope/ollama-benchmark.git -
SciCode
Datasets & benchmarksA benchmark that challenges language models to code solutions for scientific problems
git clone https://github.com/scicode-bench/SciCode.git -
LawBench
Datasets & benchmarksBenchmarking Legal Knowledge of Large Language Models
git clone https://github.com/open-compass/LawBench.git -
LLM-Agent-Benchmark-List
Datasets & benchmarksA banchmark list for evaluation of large language models.
git clone https://github.com/zhangxjohn/LLM-Agent-Benchmark-List.git -
BizFinBench
Datasets & benchmarksA Business-Driven Real-World Financial Benchmark for Evaluating LLMs
git clone https://github.com/HiThink-Research/BizFinBench.git -
MMLU-Pro
Datasets & benchmarksHugging Face dataset by TIGER-Lab
TIGER-Lab/MMLU-Pro -
OSWorld
Datasets & benchmarks[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
git clone https://github.com/xlang-ai/OSWorld.git -
apex-agents
Datasets & benchmarksHugging Face dataset by mercor
mercor/apex-agents -
LongBench
Datasets & benchmarksLongBench v2 and LongBench (ACL 25'&24')
git clone https://github.com/THUDM/LongBench.git -
nyt-connections
Datasets & benchmarksBenchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words
git clone https://github.com/lechmazur/nyt-connections.git -
kg-gen
Datasets & benchmarks[NeurIPS '25] Knowledge Graph Generation from Any Text
git clone https://github.com/stair-lab/kg-gen.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.