InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. Tracely-ai

    AI repositories

    Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

    Stale GitHub, Jwuthri 1,311 ★MITUpdated 8 days ago

    git clone https://github.com/Jwuthri/Tracely-ai.git
  2. swt-bench

    Datasets & benchmarks

    [NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation

    Stale GitHub, logic-star-ai 92 ★MITUpdated 12 days ago

    git clone https://github.com/logic-star-ai/swt-bench.git
  3. trpc-agent-go

    Agents & frameworks

    A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

    Indexed GitHub, trpc-group 1,931 ★Apache-2.0Updated yesterday

    git clone https://github.com/trpc-group/trpc-agent-go.git
  4. XRAG

    AI repositories

    XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

    Stale GitHub, DocAILab 611 ★Apache-2.0Updated 15 days ago

    git clone https://github.com/DocAILab/XRAG.git
  5. ai-agents-the-definitive-guide

    Agents & frameworks

    Repo for AI Agents The Definitive Guide

    Stale GitHub, Nicolepcx 2,747 ★Updated 3 months ago

    git clone https://github.com/Nicolepcx/ai-agents-the-definitive-guide.git
  6. One-Eval

    Datasets & benchmarks

    Automated system for LLM evaluation via agents. Doc as below:

    Stale GitHub, OpenDCAI 165 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/OpenDCAI/One-Eval.git
  7. KADATH

    Agents & frameworks

    Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.

    Stale GitHub, i3T4AN 253 ★Apache-2.0Updated 23 days ago

    git clone https://github.com/i3T4AN/KADATH.git
  8. nyt-connections

    Datasets & benchmarks

    Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words

    Indexed GitHub, lechmazur 250 ★Updated 5 days ago

    git clone https://github.com/lechmazur/nyt-connections.git
  9. overmind

    AI repositories

    The platform for continuously improving AI agents.

    Stale GitHub, overmind-core 753 ★AGPL-3.0Updated 7 days ago

    git clone https://github.com/overmind-core/overmind.git
  10. LLM-Engineers-Handbook

    AI repositories

    The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

    Stale GitHub, PacktPublishing 5,377 ★MITUpdated 6 months ago

    git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git
  11. DecisionPivot

    Datasets & benchmarks

    DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.

    Stale GitHub, StartLuxLabs 144 ★Apache-2.0Updated 26 days ago

    git clone https://github.com/StartLuxLabs/DecisionPivot.git
  12. WebsiteBench

    Datasets & benchmarks

    Can AI agents rebuild websites through browser exploration?

    Indexed GitHub, TIGER-AI-Lab 300 ★Apache-2.0Updated 6 days ago

    git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git
  13. litefuse

    Agents & frameworks

    Litefuse - Agent Observability and Evaluation Platform

    Indexed GitHub, litefuse 113 ★Custom licence (see repository)Updated 3 days ago

    git clone https://github.com/litefuse/litefuse.git
  14. AgentBench

    Agents & frameworks

    A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)

    Stale GitHub, THUDM 3,772 ★Apache-2.0Updated 9 months ago

    git clone https://github.com/THUDM/AgentBench.git
  15. promptpex

    Prompts

    Test Generation for Prompts

    Indexed GitHub, microsoft 165 ★CC-BY-4.0Updated 3 days ago

    git clone https://github.com/microsoft/promptpex.git
  16. dingo

    AI repositories

    Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

    Stale GitHub, MigoXLab 764 ★Apache-2.0Updated 13 days ago

    git clone https://github.com/MigoXLab/dingo.git
  17. langtrace

    Prompts

    Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

    Stale GitHub, Scale3-Labs 1,233 ★AGPL-3.0Updated 12 months ago

    git clone https://github.com/Scale3-Labs/langtrace.git
  18. gorilla

    AI repositories

    Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

    Stale GitHub, ShishirPatil 13.1k ★Apache-2.0Updated 7 months ago

    git clone https://github.com/ShishirPatil/gorilla.git
  19. gepa

    Prompts

    Optimize prompts, code, and more with AI-powered Reflective Optimization

    Stale GitHub, gepa-ai 7,394 ★MITUpdated 7 days ago

    git clone https://github.com/gepa-ai/gepa.git
  20. HaluMem

    Datasets & benchmarks

    HaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.

    Stale GitHub, MemTensor 178 ★Custom licence (see repository)Updated 19 days ago

    git clone https://github.com/MemTensor/HaluMem.git
  21. TerminalWorld

    Datasets & benchmarks

    TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 Evaluations and Datasets Track)

    Stale GitHub, terminalworld 54 ★Apache-2.0Updated 12 days ago

    git clone https://github.com/terminalworld/TerminalWorld.git
  22. railtracks

    AI repositories

    An agentic framework that helps developers build resilient agentic systems

    Indexed GitHub, RailtownAI 235 ★MITUpdated 3 days ago

    git clone https://github.com/RailtownAI/railtracks.git
  23. langwatch

    Prompts

    Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

    Indexed GitHub, langwatch 5,038 ★Apache-2.0Updated today

    git clone https://github.com/langwatch/langwatch.git
  24. Evidence-backed use cases, prompts, integrations, evaluations, and safety notes for OpenAI GPT-6 Astra.

    Stale GitHub, Anil-matcha 1,147 ★MITUpdated 44 days ago

    git clone https://github.com/Anil-matcha/awesome-gpt-6-astra.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.