Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
ragas
AI repositoriesSupercharge Your LLM Application Evaluations 🚀
git clone https://github.com/vibrantlabsai/ragas.git -
giskard-oss
AI repositories🐢 Open-Source Evaluation & Testing library for LLM Agents
git clone https://github.com/Giskard-AI/giskard-oss.git -
evalscope
AI repositoriesA streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
git clone https://github.com/modelscope/evalscope.git -
mastra
AI repositoriesMastra is the modern TypeScript framework for AI-powered applications and agents.
git clone https://github.com/mastra-ai/mastra.git -
tiny-universe
AI repositories《大模型白盒子构建指南》:一个全手搓的Tiny-Universe
git clone https://github.com/datawhalechina/tiny-universe.git -
simba
AI repositoriesOpenSource Production ready Customer service with built in Evals and monitoring
git clone https://github.com/GitHamza0206/simba.git -
PyRIT
AI repositoriesOpen-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
git clone https://github.com/microsoft/PyRIT.git -
agentops
AI repositoriesPython SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
git clone https://github.com/AgentOps-AI/agentops.git -
EvalForge
AI repositoriesEvaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
git clone https://github.com/jsdhwfmax/EvalForge.git -
overmind
AI repositoriesThe platform for continuously improving AI agents.
git clone https://github.com/overmind-core/overmind.git -
oumi
AI repositoriesEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
git clone https://github.com/oumi-ai/oumi.git -
DreamLayer-Eval
AI repositoriesDreamLayer Eval: open-source benchmarking for image and video diffusion models. Automate prompts, seeds, and metrics for reproducible results. A research project from DreamLayer AI (dreamlayer.io).
git clone https://github.com/TheDesignFounder/DreamLayer-Eval.git -
Gym
AI repositoriesEvaluate and improve models and agents using environments
git clone https://github.com/NVIDIA-NeMo/Gym.git -
Tracely-ai
AI repositoriesTrace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
git clone https://github.com/Jwuthri/Tracely-ai.git -
VLMEvalKit
AI repositoriesOpen-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
git clone https://github.com/open-compass/VLMEvalKit.git -
EnterpriseRAG-Bench
AI repositoriesDataset and benchmark for RAG on company internal documents.
git clone https://github.com/onyx-dot-app/EnterpriseRAG-Bench.git -
ai-system-design-guide
AI repositoriesAI system design guide for engineers building production AI systems and evals.
git clone https://github.com/ombharatiya/ai-system-design-guide.git -
WeKnora
AI repositoriesOpen-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
git clone https://github.com/Tencent/WeKnora.git -
AI-Infra-Guard
AI repositoriesA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
git clone https://github.com/Tencent/AI-Infra-Guard.git -
easy-dataset
AI repositoriesA powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval
git clone https://github.com/ConardLi/easy-dataset.git -
prompty
AI repositoriesPrompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
git clone https://github.com/microsoft/prompty.git -
XRAG
AI repositoriesXRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
git clone https://github.com/DocAILab/XRAG.git -
LLM-Engineers-Handbook
AI repositoriesThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices
git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git -
gorilla
AI repositoriesGorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
git clone https://github.com/ShishirPatil/gorilla.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.