Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
LLM-Agent-Benchmark-List
Datasets & benchmarksA banchmark list for evaluation of large language models.
git clone https://github.com/zhangxjohn/LLM-Agent-Benchmark-List.git -
ArtifactsBenchmark
Datasets & benchmarksgit clone https://github.com/Tencent-Hunyuan/ArtifactsBenchmark.git -
Static-to-Dynamic-LLMEval
Datasets & benchmarksThe official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"
git clone https://github.com/SeekingDream/Static-to-Dynamic-LLMEval.git -
DyCodeEval
Datasets & benchmarksOfficial repository of the ICML2025 paper “Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination”
git clone https://github.com/SeekingDream/DyCodeEval.git -
ChemLLMBench
Datasets & benchmarksOfficial Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)
git clone https://github.com/ChemFoundationModels/ChemLLMBench.git -
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
Omni-SafetyBench
Datasets & benchmarks[ACM MM 2026 Oral] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".
git clone https://github.com/THU-BPM/Omni-SafetyBench.git -
Awesome-LLM-Long-Context-Modeling
Datasets & benchmarks📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥
git clone https://github.com/Xnhyacinth/Awesome-LLM-Long-Context-Modeling.git -
LawBench
Datasets & benchmarksBenchmarking Legal Knowledge of Large Language Models
git clone https://github.com/open-compass/LawBench.git -
aurora
Datasets & benchmarks[ICLR 2025] AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
git clone https://github.com/wenhaochai/aurora.git -
xFinder
Datasets & benchmarks[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
git clone https://github.com/IAAR-Shanghai/xFinder.git -
JBB-Behaviors
Datasets & benchmarksHugging Face dataset by JailbreakBench
JailbreakBench/JBB-Behaviors -
LightCompress
Datasets & benchmarks[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
git clone https://github.com/ModelTC/LightCompress.git -
AmericanStories
Datasets & benchmarksHugging Face dataset by dell-research-harvard
dell-research-harvard/AmericanStories -
UHGEval
Datasets & benchmarks[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
git clone https://github.com/IAAR-Shanghai/UHGEval.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
HallusionBench
Datasets & benchmarks[CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models
git clone https://github.com/tianyi-lab/HallusionBench.git -
ReForm-Eval
Datasets & benchmarksAn benchmark for evaluating the capabilities of large vision-language models (LVLMs)
git clone https://github.com/FudanDISC/ReForm-Eval.git -
TrustLLM
Datasets & benchmarks[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
git clone https://github.com/HowieHwong/TrustLLM.git -
sec-code-bench
Datasets & benchmarksSecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).
git clone https://github.com/alibaba/sec-code-bench.git -
meta-agents-research-environments
Datasets & benchmarksMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
git clone https://github.com/facebookresearch/meta-agents-research-environments.git -
Awesome-Scientific-LLM-Benchmarks
Datasets & benchmarksA curated, accuracy-first list of benchmarks for evaluating LLMs on scientific reasoning and discovery — math, physics, chemistry, materials, biology, and agentic science.
git clone https://github.com/subinium/Awesome-Scientific-LLM-Benchmarks.git -
ai-model-world
Datasets & benchmarksAI 大模型世界,把 556 个大模型拟人化成像素小人的可视化站点。进来就能看到此刻谁最聪明、谁最会写代码、谁最便宜、谁刚发布,往下是国内与国外分区的厂商广场、完整的发布时间线和多维排行榜。搜索认模型名、厂商和能力,输入「多模态」会直接列出全部多模态模型。数据取自 Epoch AI、models.dev、LiveBench 与 Hugging Face,每小时自动同步,所有文案由真实数据生成,不调用任何 LLM。Next.js 静态导出,零后端。
git clone https://github.com/liyupi/ai-model-world.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.