Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
agentic_security
Agents & frameworksAgentic LLM Vulnerability Scanner / AI red teaming kit 🧪
git clone https://github.com/msoedov/agentic_security.git -
xFinder
Datasets & benchmarks[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
git clone https://github.com/IAAR-Shanghai/xFinder.git -
darwin-skill
Skills达尔文.skill —— 一个让你的Skill无限进化的系统:评估→改进→测试→保留或回滚 | Autoresearch-inspired autonomous skill optimization for Claude Code. Evaluate, improve, test, keep or revert.
git clone https://github.com/alchaincyf/darwin-skill.git -
Static-to-Dynamic-LLMEval
Datasets & benchmarksThe official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"
git clone https://github.com/SeekingDream/Static-to-Dynamic-LLMEval.git -
EnterpriseRAG-Bench
AI repositoriesDataset and benchmark for RAG on company internal documents.
git clone https://github.com/onyx-dot-app/EnterpriseRAG-Bench.git -
EvalForge
AI repositoriesEvaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
git clone https://github.com/jsdhwfmax/EvalForge.git -
VLMEvalKit
AI repositoriesOpen-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
git clone https://github.com/open-compass/VLMEvalKit.git -
OpenJudge
SkillsOpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
git clone https://github.com/agentscope-ai/OpenJudge.git -
ChainForge
PromptsAn open-source visual programming environment for battle-testing prompts to LLMs.
git clone https://github.com/ianarawjo/ChainForge.git -
FastEval
Datasets & benchmarksFast & more realistic evaluation of chat language models. Includes leaderboard.
git clone https://github.com/FastEval/FastEval.git -
evalstats
PromptsStatistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.
git clone https://github.com/ianarawjo/evalstats.git -
ClawProBench
Datasets & benchmarksClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
git clone https://github.com/suyoumo/ClawProBench.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
inspector
MCP serversTesting and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
git clone https://github.com/MCPJam/inspector.git -
ragbits
PromptsBuilding blocks for rapid development of GenAI applications
git clone https://github.com/deepsense-ai/ragbits.git -
TrustLLM
Datasets & benchmarks[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
git clone https://github.com/HowieHwong/TrustLLM.git -
PyRIT
AI repositoriesOpen-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
git clone https://github.com/microsoft/PyRIT.git -
BizFinBench
Datasets & benchmarksA Business-Driven Real-World Financial Benchmark for Evaluating LLMs
git clone https://github.com/HiThink-Research/BizFinBench.git -
NoLiMa
Datasets & benchmarksOfficial repository for "NoLiMa: Long-Context Evaluation Beyond Literal Matching"
git clone https://github.com/adobe-research/NoLiMa.git -
AI-Infra-Guard
AI repositoriesA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
git clone https://github.com/Tencent/AI-Infra-Guard.git -
WeKnora
AI repositoriesOpen-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
git clone https://github.com/Tencent/WeKnora.git -
UHGEval
Datasets & benchmarks[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
git clone https://github.com/IAAR-Shanghai/UHGEval.git -
prompty
AI repositoriesPrompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
git clone https://github.com/microsoft/prompty.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.