InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. evalstats

    Prompts

    Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

    Indexed GitHub, ianarawjo 134 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/ianarawjo/evalstats.git
  2. ChainForge

    Prompts

    An open-source visual programming environment for battle-testing prompts to LLMs.

    Stale GitHub, ianarawjo 3,040 ★MITUpdated 18 days ago

    git clone https://github.com/ianarawjo/ChainForge.git
  3. EvalForge

    AI repositories

    Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

    Indexed GitHub, jsdhwfmax 194 ★MITUpdated today

    git clone https://github.com/jsdhwfmax/EvalForge.git
  4. PyRIT

    AI repositories

    Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

    Indexed GitHub, microsoft 4,869 ★MITUpdated today

    git clone https://github.com/microsoft/PyRIT.git
  5. confabulations

    Datasets & benchmarks

    Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

    Stale GitHub, lechmazur 249 ★Updated 15 months ago

    git clone https://github.com/lechmazur/confabulations.git
  6. Simreal-MLBench

    Datasets & benchmarks

    [Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

    Stale GitHub, Simreal-AI 301 ★Apache-2.0Updated 53 days ago

    git clone https://github.com/Simreal-AI/Simreal-MLBench.git
  7. embodied-first-exam

    Agents & frameworks

    The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

    Stale GitHub, mll-lab-nu 236 ★MITUpdated 10 days ago

    git clone https://github.com/mll-lab-nu/embodied-first-exam.git
  8. agentic-ai

    AI repositories

    Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

    Indexed GitHub, mahmoudrabie 193 ★MITUpdated 2 days ago

    git clone https://github.com/mahmoudrabie/agentic-ai.git
  9. Stale GitHub, microsoft 11 ★MITUpdated 5 days ago

    git clone https://github.com/microsoft/frontiertour27-ILL224-see-how-far-one-agent-goes-in-copilot-studio.git
  10. conkurrence

    MCP servers

    Multi-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.

    Stale Official MCP Registry, io.github.AlligatorC0der Updated 44 days ago 1 note

    conkurrence@1.0.5
  11. Braintrust

    MCP servers

    AI evaluation and observability platform — query experiments, logs, datasets, and prompts.

    Stale Official MCP Registry, io.github.braintrustdata Updated 6 months ago 1 note

  12. Production-ready Claude Code skills for building AI agents with Pydantic AI. Includes dependency injection, tools, validators, streaming, multi-agent orchestration, and evaluation framework patterns.

    Stale GitHub, Fuenfgeld 10 ★MITUpdated 30 days ago

    git clone https://github.com/Fuenfgeld/pydantic-ai-skills.git
  13. nasde-toolkit

    Skills

    CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

    Stale GitHub, NoesisVision 14 ★MITUpdated 32 days ago

    git clone https://github.com/NoesisVision/nasde-toolkit.git
  14. QACAT

    MCP servers

    Translation QA: automated checks, AI evaluation, linguistic review, and visual in-context testing.

    Stale Official MCP Registry, io.github.dimawin22 Updated 3 months ago 1 note

  15. rubrkit

    MCP servers

    MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.

    Stale Official MCP Registry, com.rubrkit Updated 4 months ago 1 note

  16. EvalGate

    MCP servers

    Run AI evaluations, record agent traces and gate releases on evidence, with scoped org access.

    Stale Official MCP Registry, com.evalgate Updated 11 days ago 1 note

  17. futureagi-sdk

    Prompts

    Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.

    Stale GitHub, future-agi 52 ★Apache-2.0Updated 8 days ago

    git clone https://github.com/future-agi/futureagi-sdk.git
  18. Eval Engine API

    MCP servers

    Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.

    Stale Official MCP Registry, io.github.invertedhoologan Updated 4 months ago 1 note

  19. Rights-aware censorship, China-economic status, and tamper-evident AI evaluation tools.

    Stale Official MCP Registry, io.github.beepboop2025 Updated 2 months ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.