Library
85,336 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
phoenix
PromptsAI Observability & Evaluation
git clone https://github.com/Arize-ai/phoenix.git -
promptfoo
PromptsTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
git clone https://github.com/promptfoo/promptfoo.git -
trulens
Agents & frameworksEvaluation and Tracking for LLM Experiments and AI Agents
git clone https://github.com/truera/trulens.git -
LLM-eval-survey
Datasets & benchmarksThe official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
git clone https://github.com/MLGroupJLU/LLM-eval-survey.git -
giskard-oss
AI repositories🐢 Open-Source Evaluation & Testing library for LLM Agents
git clone https://github.com/Giskard-AI/giskard-oss.git -
Awesome-LLM-Eval
Datasets & benchmarksAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
git clone https://github.com/onejune2018/Awesome-LLM-Eval.git -
agent-skills-eval
SkillsA test runner for agentskills.io-style AI agent skills
git clone https://github.com/darkrishabh/agent-skills-eval.git -
agentic_security
Agents & frameworksAgentic LLM Vulnerability Scanner / AI red teaming kit 🧪
git clone https://github.com/msoedov/agentic_security.git -
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
opik
PromptsDebug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
git clone https://github.com/comet-ml/opik.git -
overmind
AI repositoriesThe platform for continuously improving AI agents.
git clone https://github.com/overmind-core/overmind.git -
LLM-Engineers-Handbook
AI repositoriesThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices
git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git -
BizFinBench
Datasets & benchmarksA Business-Driven Real-World Financial Benchmark for Evaluating LLMs
git clone https://github.com/HiThink-Research/BizFinBench.git -
WebCanvas
Agents & frameworksAll-in-one Web Agent framework for post-training. Start building with a few clicks!
git clone https://github.com/iMeanAI/WebCanvas.git -
gepa
PromptsOptimize prompts, code, and more with AI-powered Reflective Optimization
git clone https://github.com/gepa-ai/gepa.git -
GTA
Agents & frameworks[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2
git clone https://github.com/open-compass/GTA.git -
oumi
AI repositoriesEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
git clone https://github.com/oumi-ai/oumi.git -
Awesome-LLM-in-Social-Science
Agents & frameworksAwesome papers involving LLMs in Social Science.
git clone https://github.com/ValueByte-AI/Awesome-LLM-in-Social-Science.git -
lmnr
Tools & appsLaminar - open-source observability platform purpose-built for AI agents. YC S24.
git clone https://github.com/lmnr-ai/lmnr.git -
helicone
Prompts🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
git clone https://github.com/Helicone/helicone.git -
CivAgent
Agents & frameworksCivAgent is an LLM-based Human-like Agent acting as a Digital Player within the Strategy Game Unciv.
git clone https://github.com/fuxiAIlab/CivAgent.git -
AI-Engineer-Headquarters
AI repositoriesA collection of scientific methods, processes, algorithms, and systems to build stories & models.
git clone https://github.com/hemansnation/AI-Engineer-Headquarters.git -
cave-agent
Agents & frameworksStateful runtime management for LLM agents—inject, manipulate, and retrieve Python objects across turns.
git clone https://github.com/acodercat/cave-agent.git -
judgeval
PromptsThe Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
git clone https://github.com/JudgmentLabs/judgeval.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.