InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. PyRIT

    AI repositories

    Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

    Indexed GitHub, microsoft 4,869 ★MITUpdated today

    git clone https://github.com/microsoft/PyRIT.git
  2. evalstats

    Prompts

    Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

    Indexed GitHub, ianarawjo 134 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/ianarawjo/evalstats.git
  3. ChainForge

    Prompts

    An open-source visual programming environment for battle-testing prompts to LLMs.

    Stale GitHub, ianarawjo 3,040 ★MITUpdated 18 days ago

    git clone https://github.com/ianarawjo/ChainForge.git
  4. Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

    Stale GitHub, Calibre-Labs 95 ★MITUpdated 9 days ago

    git clone https://github.com/Calibre-Labs/reforge-ai-evals.git
  5. EvalForge

    AI repositories

    Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

    Indexed GitHub, jsdhwfmax 194 ★MITUpdated today

    git clone https://github.com/jsdhwfmax/EvalForge.git
  6. ai-eval

    MCP servers

    Cloudflare Workers MCP server: ai-eval

    Stale Official MCP Registry, io.github.lazymac2x Updated 6 months ago 1 note

  7. confabulations

    Datasets & benchmarks

    Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

    Stale GitHub, lechmazur 249 ★Updated 15 months ago

    git clone https://github.com/lechmazur/confabulations.git
  8. Simreal-MLBench

    Datasets & benchmarks

    [Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

    Stale GitHub, Simreal-AI 301 ★Apache-2.0Updated 53 days ago

    git clone https://github.com/Simreal-AI/Simreal-MLBench.git
  9. benchmarks

    Agents & frameworks

    Reproducible voice-AI benchmarking — TTS / STT / S2S latency and accuracy.

    Stale GitHub, coval-ai 104 ★Custom licence (see repository)Updated 24 days ago

    git clone https://github.com/coval-ai/benchmarks.git
  10. future-agi

    Agents & frameworks

    Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

    Indexed GitHub, future-agi 2,267 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/future-agi/future-agi.git
  11. embodied-first-exam

    Agents & frameworks

    The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

    Stale GitHub, mll-lab-nu 236 ★MITUpdated 10 days ago

    git clone https://github.com/mll-lab-nu/embodied-first-exam.git
  12. agentic-ai

    AI repositories

    Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

    Indexed GitHub, mahmoudrabie 193 ★MITUpdated 2 days ago

    git clone https://github.com/mahmoudrabie/agentic-ai.git
  13. Stale GitHub, microsoft 11 ★MITUpdated 5 days ago

    git clone https://github.com/microsoft/frontiertour27-ILL224-see-how-far-one-agent-goes-in-copilot-studio.git
  14. conkurrence

    MCP servers

    Multi-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.

    Stale Official MCP Registry, io.github.AlligatorC0der Updated 44 days ago 1 note

    conkurrence@1.0.5
  15. Braintrust

    MCP servers

    AI evaluation and observability platform — query experiments, logs, datasets, and prompts.

    Stale Official MCP Registry, io.github.braintrustdata Updated 6 months ago 1 note

  16. Production-ready Claude Code skills for building AI agents with Pydantic AI. Includes dependency injection, tools, validators, streaming, multi-agent orchestration, and evaluation framework patterns.

    Stale GitHub, Fuenfgeld 10 ★MITUpdated 30 days ago

    git clone https://github.com/Fuenfgeld/pydantic-ai-skills.git
  17. nasde-toolkit

    Skills

    CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

    Stale GitHub, NoesisVision 14 ★MITUpdated 32 days ago

    git clone https://github.com/NoesisVision/nasde-toolkit.git
  18. QACAT

    MCP servers

    Translation QA: automated checks, AI evaluation, linguistic review, and visual in-context testing.

    Stale Official MCP Registry, io.github.dimawin22 Updated 3 months ago 1 note

  19. rubrkit

    MCP servers

    MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.

    Stale Official MCP Registry, com.rubrkit Updated 4 months ago 1 note

  20. EvalGate

    MCP servers

    Run AI evaluations, record agent traces and gate releases on evidence, with scoped org access.

    Stale Official MCP Registry, com.evalgate Updated 11 days ago 1 note

  21. futureagi-sdk

    Prompts

    Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.

    Stale GitHub, future-agi 52 ★Apache-2.0Updated 8 days ago

    git clone https://github.com/future-agi/futureagi-sdk.git
  22. evalgate

    MCP servers

    Statistical checks an agent runs before trusting an AI eval number (is #1 real, judge bias, more).

    Stale Official MCP Registry, io.github.ipezygj Updated 3 months ago 1 note

    eval-integrity@0.4.1
  23. Eval Engine API

    MCP servers

    Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.

    Stale Official MCP Registry, io.github.invertedhoologan Updated 4 months ago 1 note

  24. Rights-aware censorship, China-economic status, and tamper-evident AI evaluation tools.

    Stale Official MCP Registry, io.github.beepboop2025 Updated 2 months ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.