InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. 50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

    Stale GitHub, ContextJet-ai 46 ★Custom licence (see repository)Updated 5 days ago

    git clone https://github.com/ContextJet-ai/awesome-llm-observability.git
  2. Self-improvement feedback loop for AI agent skills — analyzes past sessions, drafts structured improvement proposals, and gates auto-apply behind an evaluation framework. Host-agnostic (Hermes, Claude Code, and any HostAdapter).

    Stale GitHub, Carlo1911 6 ★MITUpdated 29 days ago

    git clone https://github.com/Carlo1911/skill-evolution.git
  3. Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

    Stale GitHub, Calibre-Labs 95 ★MITUpdated 9 days ago

    git clone https://github.com/Calibre-Labs/reforge-ai-evals.git
  4. hermeneutic

    Skills

    Turn corrections from AI agent logs into guidance for similar tasks. Local memory with semantic retrieval, prompt-context hooks, and response checks.

    Stale GitHub, hermes-labs-ai 9 ★Apache-2.0Updated 27 days ago

    git clone https://github.com/hermes-labs-ai/hermeneutic.git
  5. OpenART

    Skills

    OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.

    Stale GitHub, AI45Lab 263 ★AGPL-3.0Updated 15 days ago

    git clone https://github.com/AI45Lab/OpenART.git
  6. retriEVAL

    MCP servers

    LLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination.

    Stale Official MCP Registry, io.github.hcarrillo001 Updated 2 months ago 2 notes

    ghcr.io/hcarrillo001/retrieval-mcp:1.0.0
  7. jev-judge-mcp

    MCP servers

    Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.

    Stale GitHub, PyModel 48 ★MITUpdated 32 days ago

    git clone https://github.com/PyModel/jev-judge-mcp.git
  8. Catch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic.

    Indexed GitHub, tonydzi 6 ★MITUpdated 2 days ago

    git clone https://github.com/tonydzi/verbatim-citation-gate.git
  9. PuzzleTide

    MCP servers

    Word search, crossword, and sudoku generators, printable PDF worksheets, and verifiable LLM evals.

    Stale Official MCP Registry, io.github.catorch Updated 4 months ago 1 note

    puzzletide-mcp@0.2.1
  10. Open-source skills for Claude Code and Codex, with an MCP server, a PyPI package and Suede Awesome Everything, a directory that scores AI agent skills 0 to 99. Created by Jason Colapietro, founder and CEO of Suede AI.

    Indexed GitHub, JasonColapietro 124 ★MITUpdated yesterday

    git clone https://github.com/JasonColapietro/suede-creator-skills.git
  11. eval-genius

    Skills

    Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.

    Stale GitHub, alexgreensh 24 ★Apache-2.0Updated 11 days ago

    git clone https://github.com/alexgreensh/eval-genius.git
  12. aiml-companion

    Agents & frameworks

    AI-ML Companion: an interactive platform to learn AI & ML by watching it work - 28 tracks, 400+ modules, live visualizations - plus 20+ end-to-end real world projects from classical ML, DL, Computer Vision to production LLM & multi-agent systems.

    Indexed GitHub, genieincodebottle 74 ★Updated 2 days ago

    git clone https://github.com/genieincodebottle/aiml-companion.git
  13. digline

    MCP servers

    Read, measure and explain LLM eval runs of a digline suite. Cannot promote a baseline.

    Indexed Official MCP Registry, dev.digline Updated today 1 note

    digline-mcp@0.10.0
  14. 🛡️ Autonomous GitHub Bounty Scanner & On-Chain Escrow Verifier — Find Real Crypto/USDC/wDOOM Bounties & Filter Fake Scraper Traps

    Stale GitHub, doomsday-network 8 ★Updated 22 days ago

    git clone https://github.com/doomsday-network/github-bounty-scanner.git
  15. bounty-guard

    Skills

    Zero-dependency Python CLI, GitHub Action, and MCP server scanning untrusted GitHub issues and repositories for prompt injection, CI secret exfiltration, and cryptographic bounty escrow.

    Stale GitHub, doomsday-network 15 ★MITUpdated 22 days ago

    git clone https://github.com/doomsday-network/bounty-guard.git
  16. archboard

    Skills

    Plan the next change with your agent on a map of the system, before it writes a line. Then follow any feature into the code, and see every place where the map and the code disagree.

    Indexed GitHub, miklschmidt 11 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/miklschmidt/archboard.git
  17. LLM Prover

    MCP servers

    LLM evaluation: compare models on cost and quality, score outputs, run benchmarks, analyse agents.

    Stale Official MCP Registry, com.pysolvr Updated 4 days ago 1 note

  18. Make AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and an independent verifier that never sees the worker's reasoning. Plain markdown, EN + RU.

    Stale GitHub, alexpiar 23 ★MITUpdated 22 days ago

    git clone https://github.com/alexpiar/be-ambitious-harness.git
  19. evalpair

    MCP servers

    Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.

    Stale Official MCP Registry, io.github.Srikarmk Updated 11 days ago 2 notes

    evalpair@0.1.1
  20. Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product specification, architecture, implementation, UI & device adaptation, debugging, verification, release readiness.

    Stale GitHub, NocodeMrLi 95 ★MITUpdated 13 days ago

    git clone https://github.com/NocodeMrLi/mini-program-engineering-skill-suite.git
  21. mcp-as-a-judge

    MCP servers

    MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

    Stale Official MCP Registry, io.github.OtherVibes Updated 14 months ago 1 note

    mcp-as-a-judge@0.3.20
  22. The math arm for LLMs: compute and check numbers from CSV and Excel data with pandas, SciPy and statsmodels, each with its calculation. Tool calling for Claude, Codex and local models (Ollama, vLLM, llama.cpp, LM Studio), Kaggle-style AutoML with --explain, and paper tables and figures.

    Stale GitHub, saineshnakra 35 ★MITUpdated 10 days ago

    git clone https://github.com/saineshnakra/automated-data-analyst.git
  23. LLM-driven evolution of prompts and pipelines. Evolves the prompt and the pipeline's parameters together against your labeled data, cuts losing candidates early, and shows every measurement behind the winner. Open source.

    Indexed GitHub, PromptPotter 7 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/PromptPotter/prompt-potter-optimizer.git
  24. skill-harness

    Skills

    How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”

    Stale GitHub, MrBinnacle 16 ★MITUpdated 14 days ago

    git clone https://github.com/MrBinnacle/skill-harness.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.