Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
Tracely-ai
AI repositoriesTrace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
git clone https://github.com/Jwuthri/Tracely-ai.git -
swt-bench
Datasets & benchmarks[NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation
git clone https://github.com/logic-star-ai/swt-bench.git -
trpc-agent-go
Agents & frameworksA Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.
git clone https://github.com/trpc-group/trpc-agent-go.git -
XRAG
AI repositoriesXRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
git clone https://github.com/DocAILab/XRAG.git -
ai-agents-the-definitive-guide
Agents & frameworksRepo for AI Agents The Definitive Guide
git clone https://github.com/Nicolepcx/ai-agents-the-definitive-guide.git -
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
KADATH
Agents & frameworksEvolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.
git clone https://github.com/i3T4AN/KADATH.git -
nyt-connections
Datasets & benchmarksBenchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words
git clone https://github.com/lechmazur/nyt-connections.git -
overmind
AI repositoriesThe platform for continuously improving AI agents.
git clone https://github.com/overmind-core/overmind.git -
LLM-Engineers-Handbook
AI repositoriesThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices
git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git -
DecisionPivot
Datasets & benchmarksDecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
git clone https://github.com/StartLuxLabs/DecisionPivot.git -
WebsiteBench
Datasets & benchmarksCan AI agents rebuild websites through browser exploration?
git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git -
litefuse
Agents & frameworksLitefuse - Agent Observability and Evaluation Platform
git clone https://github.com/litefuse/litefuse.git -
AgentBench
Agents & frameworksA Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
git clone https://github.com/THUDM/AgentBench.git -
promptpex
PromptsTest Generation for Prompts
git clone https://github.com/microsoft/promptpex.git -
dingo
AI repositoriesDingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
git clone https://github.com/MigoXLab/dingo.git -
langtrace
PromptsLangtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊
git clone https://github.com/Scale3-Labs/langtrace.git -
gorilla
AI repositoriesGorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
git clone https://github.com/ShishirPatil/gorilla.git -
gepa
PromptsOptimize prompts, code, and more with AI-powered Reflective Optimization
git clone https://github.com/gepa-ai/gepa.git -
HaluMem
Datasets & benchmarksHaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.
git clone https://github.com/MemTensor/HaluMem.git -
TerminalWorld
Datasets & benchmarksTerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 Evaluations and Datasets Track)
git clone https://github.com/terminalworld/TerminalWorld.git -
railtracks
AI repositoriesAn agentic framework that helps developers build resilient agentic systems
git clone https://github.com/RailtownAI/railtracks.git -
langwatch
PromptsOpen-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
git clone https://github.com/langwatch/langwatch.git -
awesome-gpt-6-astra
PromptsEvidence-backed use cases, prompts, integrations, evaluations, and safety notes for OpenAI GPT-6 Astra.
git clone https://github.com/Anil-matcha/awesome-gpt-6-astra.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.