Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
FastEval
Datasets & benchmarksFast & more realistic evaluation of chat language models. Includes leaderboard.
git clone https://github.com/FastEval/FastEval.git -
TurtleBench
Datasets & benchmarksTurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
git clone https://github.com/mazzzystar/TurtleBench.git -
confabulations
Datasets & benchmarksHallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
git clone https://github.com/lechmazur/confabulations.git -
NextTransformer
Datasets & benchmarks:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
git clone https://github.com/Interactive-Active-AI/NextTransformer.git -
SciCode
Datasets & benchmarksA benchmark that challenges language models to code solutions for scientific problems
git clone https://github.com/scicode-bench/SciCode.git -
geobench
Datasets & benchmarksGeoGuessr benchmark for language models
git clone https://github.com/ccmdi/geobench.git -
AmericanStories
Datasets & benchmarksHugging Face dataset by dell-research-harvard
dell-research-harvard/AmericanStories -
finephrase
Datasets & benchmarksHugging Face dataset by HuggingFaceFW
HuggingFaceFW/finephrase -
UHGEval
Datasets & benchmarks[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
git clone https://github.com/IAAR-Shanghai/UHGEval.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
TrustLLM
Datasets & benchmarks[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
git clone https://github.com/HowieHwong/TrustLLM.git -
humanities_data_benchmark
Datasets & benchmarksLLM Benchmark Suite for Humanities Data
git clone https://github.com/RISE-UNIBAS/humanities_data_benchmark.git -
truthful_qa
Datasets & benchmarksHugging Face dataset by truthfulqa
truthfulqa/truthful_qa -
meta-agents-research-environments
Datasets & benchmarksMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
git clone https://github.com/facebookresearch/meta-agents-research-environments.git -
sec-code-bench
Datasets & benchmarksSecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).
git clone https://github.com/alibaba/sec-code-bench.git -
LongPage
Datasets & benchmarksHugging Face dataset by Pageshift-Entertainment
Pageshift-Entertainment/LongPage -
OSCAR-2201
Datasets & benchmarksHugging Face dataset by oscar-corpus
oscar-corpus/OSCAR-2201 -
Awesome-Scientific-LLM-Benchmarks
Datasets & benchmarksA curated, accuracy-first list of benchmarks for evaluating LLMs on scientific reasoning and discovery — math, physics, chemistry, materials, biology, and agentic science.
git clone https://github.com/subinium/Awesome-Scientific-LLM-Benchmarks.git -
ai-model-world
Datasets & benchmarksAI 大模型世界,把 556 个大模型拟人化成像素小人的可视化站点。进来就能看到此刻谁最聪明、谁最会写代码、谁最便宜、谁刚发布,往下是国内与国外分区的厂商广场、完整的发布时间线和多维排行榜。搜索认模型名、厂商和能力,输入「多模态」会直接列出全部多模态模型。数据取自 Epoch AI、models.dev、LiveBench 与 Hugging Face,每小时自动同步,所有文案由真实数据生成,不调用任何 LLM。Next.js 静态导出,零后端。
git clone https://github.com/liyupi/ai-model-world.git -
Ditto
Datasets & benchmarksA self-ailgnment method for role-play. Benchmark for role-play. Resources for "Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment".
git clone https://github.com/OFA-Sys/Ditto.git -
Awesome-AI-Scientist
Datasets & benchmarks🧪 Awesome list of AI Scientist papers, systems, benchmarks, datasets and open-source platforms.
git clone https://github.com/Omni-Scientist/Awesome-AI-Scientist.git -
Multi-User-LLM-Agent
Datasets & benchmarksOfficial code for the paper: "Multi-User Large Language Model Agents"
git clone https://github.com/Kordi-AI/Multi-User-LLM-Agent.git -
ai-hardware-fit
Datasets & benchmarksCan my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.
git clone https://github.com/jaeseok614/ai-hardware-fit.git -
AIRTBench-Code
Datasets & benchmarksCode Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models
git clone https://github.com/dreadnode/AIRTBench-Code.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.