Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
ragas
AI repositoriesSupercharge Your LLM Application Evaluations 🚀
git clone https://github.com/vibrantlabsai/ragas.git -
opik
PromptsDebug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
git clone https://github.com/comet-ml/opik.git -
promptfoo
PromptsTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
git clone https://github.com/promptfoo/promptfoo.git -
SkillEvaluator
SkillsMulti-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
git clone https://github.com/NVIDIA/SkillEvaluator.git -
giskard-oss
AI repositories🐢 Open-Source Evaluation & Testing library for LLM Agents
git clone https://github.com/Giskard-AI/giskard-oss.git -
evaluation-guidebook
Docs & tutorialsSharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
git clone https://github.com/huggingface/evaluation-guidebook.git -
mlflow
PromptsThe open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
git clone https://github.com/mlflow/mlflow.git -
future-agi
Agents & frameworksOpen-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
git clone https://github.com/future-agi/future-agi.git -
phoenix
PromptsAI Observability & Evaluation
git clone https://github.com/Arize-ai/phoenix.git -
trulens
Agents & frameworksEvaluation and Tracking for LLM Experiments and AI Agents
git clone https://github.com/truera/trulens.git -
skill-up
SkillsEvaluate Agent Skills and environments. Evolve Skills with evidence.
git clone https://github.com/alibaba/skill-up.git -
helicone
Prompts🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
git clone https://github.com/Helicone/helicone.git -
evalscope
AI repositoriesA streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
git clone https://github.com/modelscope/evalscope.git -
Kiln
PromptsBuild, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
git clone https://github.com/Kiln-AI/Kiln.git -
langfuse
Prompts🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
git clone https://github.com/langfuse/langfuse.git -
tiny-universe
AI repositories《大模型白盒子构建指南》:一个全手搓的Tiny-Universe
git clone https://github.com/datawhalechina/tiny-universe.git -
text classification model by vectara · transformers
vectara/hallucination_evaluation_model -
lmnr
Tools & appsLaminar - open-source observability platform purpose-built for AI agents. YC S24.
git clone https://github.com/lmnr-ai/lmnr.git -
ClawBench
Datasets & benchmarksOpen-source benchmark for browser AI agents on daily tasks.
git clone https://github.com/TIGER-AI-Lab/ClawBench.git -
opencompass
Datasets & benchmarksOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
git clone https://github.com/open-compass/opencompass.git -
yao-meta-skill
Tools & appsYAO = Yielding AI Outcomes. A rigorous engineering, evaluation, governance, and portability system for reusable agent skills.
git clone https://github.com/yaojingang/yao-meta-skill.git -
Gym
AI repositoriesEvaluate and improve models and agents using environments
git clone https://github.com/NVIDIA-NeMo/Gym.git -
intelligent-audit-system
Agents & frameworksAuditPilot: auditable enterprise AI agents for evidence-grounded workflows, governed tools, evaluation harnesses, human review, and remediation delivery.
git clone https://github.com/Ricky-7-Yan/intelligent-audit-system.git -
oumi
AI repositoriesEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
git clone https://github.com/oumi-ai/oumi.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.