Library
85,336 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
promptfoo
PromptsTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
git clone https://github.com/promptfoo/promptfoo.git -
ai-agents-the-definitive-guide
Agents & frameworksRepo for AI Agents The Definitive Guide
git clone https://github.com/Nicolepcx/ai-agents-the-definitive-guide.git -
WritingBench
Datasets & benchmarksWritingBench: A Comprehensive Benchmark for Generative Writing
git clone https://github.com/X-PLUG/WritingBench.git -
AgentLab
Datasets & benchmarksAgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
git clone https://github.com/ServiceNow/AgentLab.git -
swt-bench
Datasets & benchmarks[NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation
git clone https://github.com/logic-star-ai/swt-bench.git -
agentic_security
Agents & frameworksAgentic LLM Vulnerability Scanner / AI red teaming kit 🧪
git clone https://github.com/msoedov/agentic_security.git -
Kiln
PromptsBuild, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
git clone https://github.com/Kiln-AI/Kiln.git -
future-agi
Agents & frameworksOpen-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
git clone https://github.com/future-agi/future-agi.git -
promptbench
PromptsA unified evaluation framework for large language models
git clone https://github.com/microsoftarchive/promptbench.git -
coder_eval
SkillsPlaywright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.
git clone https://github.com/UiPath/coder_eval.git -
FairMind
MCP serversRelease gate for AI systems from FairMind: evaluations, framework evidence and ledger snapshots.
fairmind-mcp@0.4.0a0 -
UHGEval
Datasets & benchmarks[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
git clone https://github.com/IAAR-Shanghai/UHGEval.git -
eval-genius
SkillsTeach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.
git clone https://github.com/alexgreensh/eval-genius.git -
skill-harness
SkillsHow do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”
git clone https://github.com/MrBinnacle/skill-harness.git -
skill-evolution
SkillsSelf-improvement feedback loop for AI agent skills — analyzes past sessions, drafts structured improvement proposals, and gates auto-apply behind an evaluation framework. Host-agnostic (Hermes, Claude Code, and any HostAdapter).
git clone https://github.com/Carlo1911/skill-evolution.git -
pydantic-ai-skills
SkillsProduction-ready Claude Code skills for building AI agents with Pydantic AI. Includes dependency injection, tools, validators, streaming, multi-agent orchestration, and evaluation framework patterns.
git clone https://github.com/Fuenfgeld/pydantic-ai-skills.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.