Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
evalstats
PromptsStatistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.
git clone https://github.com/ianarawjo/evalstats.git -
ChainForge
PromptsAn open-source visual programming environment for battle-testing prompts to LLMs.
git clone https://github.com/ianarawjo/ChainForge.git -
EvalForge
AI repositoriesEvaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
git clone https://github.com/jsdhwfmax/EvalForge.git -
PyRIT
AI repositoriesOpen-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
git clone https://github.com/microsoft/PyRIT.git -
confabulations
Datasets & benchmarksHallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
git clone https://github.com/lechmazur/confabulations.git -
Simreal-MLBench
Datasets & benchmarks[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
git clone https://github.com/Simreal-AI/Simreal-MLBench.git -
embodied-first-exam
Agents & frameworksThe first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
git clone https://github.com/mll-lab-nu/embodied-first-exam.git -
agentic-ai
AI repositoriesAgentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.
git clone https://github.com/mahmoudrabie/agentic-ai.git -
git clone https://github.com/microsoft/frontiertour27-ILL224-see-how-far-one-agent-goes-in-copilot-studio.git
-
conkurrence
MCP serversMulti-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.
conkurrence@1.0.5 -
Braintrust
MCP serversAI evaluation and observability platform — query experiments, logs, datasets, and prompts.
-
pydantic-ai-skills
SkillsProduction-ready Claude Code skills for building AI agents with Pydantic AI. Includes dependency injection, tools, validators, streaming, multi-agent orchestration, and evaluation framework patterns.
git clone https://github.com/Fuenfgeld/pydantic-ai-skills.git -
nasde-toolkit
SkillsCLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.
git clone https://github.com/NoesisVision/nasde-toolkit.git -
QACAT
MCP serversTranslation QA: automated checks, AI evaluation, linguistic review, and visual in-context testing.
-
rubrkit
MCP serversMCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
-
EvalGate
MCP serversRun AI evaluations, record agent traces and gate releases on evidence, with scoped org access.
-
futureagi-sdk
PromptsProduction-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.
git clone https://github.com/future-agi/futureagi-sdk.git -
Eval Engine API
MCP serversPay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
-
Rights-aware censorship, China-economic status, and tamper-evident AI evaluation tools.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.