Library
85,336 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
OSWorld
Datasets & benchmarks[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
git clone https://github.com/xlang-ai/OSWorld.git -
tau2-bench
Datasets & benchmarksτ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
git clone https://github.com/sierra-research/tau2-bench.git -
Awesome-LLM-Long-Context-Modeling
Datasets & benchmarks📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥
git clone https://github.com/Xnhyacinth/Awesome-LLM-Long-Context-Modeling.git -
PyHealth
Datasets & benchmarksA deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.
git clone https://github.com/sunlabuiuc/PyHealth.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
llm-colosseum
Datasets & benchmarksBenchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM
git clone https://github.com/OpenGenerativeAI/llm-colosseum.git -
kg-gen
Datasets & benchmarks[NeurIPS '25] Knowledge Graph Generation from Any Text
git clone https://github.com/stair-lab/kg-gen.git -
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
LongBench
Datasets & benchmarksLongBench v2 and LongBench (ACL 25'&24')
git clone https://github.com/THUDM/LongBench.git -
ClawBench
Datasets & benchmarksOpen-source benchmark for browser AI agents on daily tasks.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git -
github-code
Datasets & benchmarksHugging Face dataset by codeparrot
codeparrot/github-code -
mobilegym
Datasets & benchmarks[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training
git clone https://github.com/Purewhiter/mobilegym.git -
ClawProBench
Datasets & benchmarksClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
git clone https://github.com/suyoumo/ClawProBench.git -
TheAgentCompany
Datasets & benchmarksAn agent benchmark with tasks in a simulated software company.
git clone https://github.com/TheAgentCompany/TheAgentCompany.git -
LightCompress
Datasets & benchmarks[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
git clone https://github.com/ModelTC/LightCompress.git -
AI_Diplomacy
Datasets & benchmarksFrontier Models playing the board game Diplomacy.
git clone https://github.com/GoodStartLabs/AI_Diplomacy.git -
AICGSecEval
Datasets & benchmarksA.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team.
git clone https://github.com/Tencent/AICGSecEval.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
ParseBench
Datasets & benchmarksParseBench - A Document Parsing Benchmark for AI Agents
git clone https://github.com/run-llama/ParseBench.git -
TrustLLM
Datasets & benchmarks[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
git clone https://github.com/HowieHwong/TrustLLM.git -
meta-agents-research-environments
Datasets & benchmarksMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
git clone https://github.com/facebookresearch/meta-agents-research-environments.git -
LongCite
Datasets & benchmarksLongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
git clone https://github.com/THUDM/LongCite.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.