InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. ragas

    AI repositories

    Supercharge Your LLM Application Evaluations 🚀

    Indexed GitHub, vibrantlabsai 16.3k ★Apache-2.0Updated 2 days ago

    git clone https://github.com/vibrantlabsai/ragas.git
  2. giskard-oss

    AI repositories

    🐢 Open-Source Evaluation & Testing library for LLM Agents

    Indexed GitHub, Giskard-AI 5,954 ★Apache-2.0Updated today

    git clone https://github.com/Giskard-AI/giskard-oss.git
  3. evalscope

    AI repositories

    A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

    Indexed GitHub, modelscope 3,697 ★Apache-2.0Updated today

    git clone https://github.com/modelscope/evalscope.git
  4. mastra

    AI repositories

    Mastra is the modern TypeScript framework for AI-powered applications and agents.

    Indexed GitHub, mastra-ai 29.5k ★Custom licence (see repository)Updated today

    git clone https://github.com/mastra-ai/mastra.git
  5. tiny-universe

    AI repositories

    《大模型白盒子构建指南》:一个全手搓的Tiny-Universe

    Stale GitHub, datawhalechina 5,106 ★Updated 9 months ago

    git clone https://github.com/datawhalechina/tiny-universe.git
  6. simba

    AI repositories

    OpenSource Production ready Customer service with built in Evals and monitoring

    Stale GitHub, GitHamza0206 1,546 ★Apache-2.0Updated 5 months ago

    git clone https://github.com/GitHamza0206/simba.git
  7. PyRIT

    AI repositories

    Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

    Indexed GitHub, microsoft 4,869 ★MITUpdated today

    git clone https://github.com/microsoft/PyRIT.git
  8. agentops

    AI repositories

    Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

    Stale GitHub, AgentOps-AI 5,885 ★MITUpdated 4 months ago

    git clone https://github.com/AgentOps-AI/agentops.git
  9. EvalForge

    AI repositories

    Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

    Indexed GitHub, jsdhwfmax 194 ★MITUpdated today

    git clone https://github.com/jsdhwfmax/EvalForge.git
  10. overmind

    AI repositories

    The platform for continuously improving AI agents.

    Stale GitHub, overmind-core 753 ★AGPL-3.0Updated 7 days ago

    git clone https://github.com/overmind-core/overmind.git
  11. oumi

    AI repositories

    Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

    Stale GitHub, oumi-ai 9,408 ★Apache-2.0Updated 24 days ago

    git clone https://github.com/oumi-ai/oumi.git
  12. DreamLayer-Eval

    AI repositories

    DreamLayer Eval: open-source benchmarking for image and video diffusion models. Automate prompts, seeds, and metrics for reproducible results. A research project from DreamLayer AI (dreamlayer.io).

    Stale GitHub, TheDesignFounder 412 ★GPL-3.0Updated 17 days ago

    git clone https://github.com/TheDesignFounder/DreamLayer-Eval.git
  13. Gym

    AI repositories

    Evaluate and improve models and agents using environments

    Indexed GitHub, NVIDIA-NeMo 1,334 ★Apache-2.0Updated today

    git clone https://github.com/NVIDIA-NeMo/Gym.git
  14. Tracely-ai

    AI repositories

    Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

    Stale GitHub, Jwuthri 1,311 ★MITUpdated 8 days ago

    git clone https://github.com/Jwuthri/Tracely-ai.git
  15. VLMEvalKit

    AI repositories

    Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

    Indexed GitHub, open-compass 4,487 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/open-compass/VLMEvalKit.git
  16. EnterpriseRAG-Bench

    AI repositories

    Dataset and benchmark for RAG on company internal documents.

    Stale GitHub, onyx-dot-app 592 ★MITUpdated 2 months ago

    git clone https://github.com/onyx-dot-app/EnterpriseRAG-Bench.git
  17. ai-system-design-guide

    AI repositories

    AI system design guide for engineers building production AI systems and evals.

    Stale GitHub, ombharatiya 4,238 ★MITUpdated 6 days ago

    git clone https://github.com/ombharatiya/ai-system-design-guide.git
  18. WeKnora

    AI repositories

    Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

    Indexed GitHub, Tencent 37.5k ★Custom licence (see repository)Updated today

    git clone https://github.com/Tencent/WeKnora.git
  19. AI-Infra-Guard

    AI repositories

    A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.

    Stale GitHub, Tencent 7,471 ★Apache-2.0Updated 4 days ago

    git clone https://github.com/Tencent/AI-Infra-Guard.git
  20. easy-dataset

    AI repositories

    A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

    Stale GitHub, ConardLi 15.0k ★Custom licence (see repository)Updated 6 months ago

    git clone https://github.com/ConardLi/easy-dataset.git
  21. prompty

    AI repositories

    Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.

    Stale GitHub, microsoft 1,288 ★MITUpdated 7 days ago

    git clone https://github.com/microsoft/prompty.git
  22. XRAG

    AI repositories

    XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

    Stale GitHub, DocAILab 611 ★Apache-2.0Updated 15 days ago

    git clone https://github.com/DocAILab/XRAG.git
  23. LLM-Engineers-Handbook

    AI repositories

    The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

    Stale GitHub, PacktPublishing 5,377 ★MITUpdated 6 months ago

    git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git
  24. gorilla

    AI repositories

    Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

    Stale GitHub, ShishirPatil 13.1k ★Apache-2.0Updated 7 months ago

    git clone https://github.com/ShishirPatil/gorilla.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.