Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
awesome-llm-observability
Skills50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
git clone https://github.com/ContextJet-ai/awesome-llm-observability.git -
skill-evolution
SkillsSelf-improvement feedback loop for AI agent skills — analyzes past sessions, drafts structured improvement proposals, and gates auto-apply behind an evaluation framework. Host-agnostic (Hermes, Claude Code, and any HostAdapter).
git clone https://github.com/Carlo1911/skill-evolution.git -
reforge-ai-evals
PromptsOpen-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.
git clone https://github.com/Calibre-Labs/reforge-ai-evals.git -
hermeneutic
SkillsTurn corrections from AI agent logs into guidance for similar tasks. Local memory with semantic retrieval, prompt-context hooks, and response checks.
git clone https://github.com/hermes-labs-ai/hermeneutic.git -
OpenART
SkillsOpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.
git clone https://github.com/AI45Lab/OpenART.git -
retriEVAL
MCP serversLLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination.
ghcr.io/hcarrillo001/retrieval-mcp:1.0.0 -
jev-judge-mcp
MCP serversTyped judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
git clone https://github.com/PyModel/jev-judge-mcp.git -
verbatim-citation-gate
SkillsCatch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic.
git clone https://github.com/tonydzi/verbatim-citation-gate.git -
PuzzleTide
MCP serversWord search, crossword, and sudoku generators, printable PDF worksheets, and verifiable LLM evals.
puzzletide-mcp@0.2.1 -
suede-creator-skills
SkillsOpen-source skills for Claude Code and Codex, with an MCP server, a PyPI package and Suede Awesome Everything, a directory that scores AI agent skills 0 to 99. Created by Jason Colapietro, founder and CEO of Suede AI.
git clone https://github.com/JasonColapietro/suede-creator-skills.git -
eval-genius
SkillsTeach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.
git clone https://github.com/alexgreensh/eval-genius.git -
aiml-companion
Agents & frameworksAI-ML Companion: an interactive platform to learn AI & ML by watching it work - 28 tracks, 400+ modules, live visualizations - plus 20+ end-to-end real world projects from classical ML, DL, Computer Vision to production LLM & multi-agent systems.
git clone https://github.com/genieincodebottle/aiml-companion.git -
digline
MCP serversRead, measure and explain LLM eval runs of a digline suite. Cannot promote a baseline.
digline-mcp@0.10.0 -
github-bounty-scanner
Skills🛡️ Autonomous GitHub Bounty Scanner & On-Chain Escrow Verifier — Find Real Crypto/USDC/wDOOM Bounties & Filter Fake Scraper Traps
git clone https://github.com/doomsday-network/github-bounty-scanner.git -
bounty-guard
SkillsZero-dependency Python CLI, GitHub Action, and MCP server scanning untrusted GitHub issues and repositories for prompt injection, CI secret exfiltration, and cryptographic bounty escrow.
git clone https://github.com/doomsday-network/bounty-guard.git -
archboard
SkillsPlan the next change with your agent on a map of the system, before it writes a line. Then follow any feature into the code, and see every place where the map and the code disagree.
git clone https://github.com/miklschmidt/archboard.git -
LLM Prover
MCP serversLLM evaluation: compare models on cost and quality, score outputs, run benchmarks, analyse agents.
-
be-ambitious-harness
SkillsMake AI agents aim higher without losing control: a Claude Code skill with two bars per task, a cheap gate check before expensive work, and an independent verifier that never sees the worker's reasoning. Plain markdown, EN + RU.
git clone https://github.com/alexpiar/be-ambitious-harness.git -
evalpair
MCP serversRun LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.
evalpair@0.1.1 -
Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product specification, architecture, implementation, UI & device adaptation, debugging, verification, release readiness.
git clone https://github.com/NocodeMrLi/mini-program-engineering-skill-suite.git -
mcp-as-a-judge
MCP serversMCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations
mcp-as-a-judge@0.3.20 -
automated-data-analyst
SkillsThe math arm for LLMs: compute and check numbers from CSV and Excel data with pandas, SciPy and statsmodels, each with its calculation. Tool calling for Claude, Codex and local models (Ollama, vLLM, llama.cpp, LM Studio), Kaggle-style AutoML with --explain, and paper tables and figures.
git clone https://github.com/saineshnakra/automated-data-analyst.git -
prompt-potter-optimizer
SkillsLLM-driven evolution of prompts and pipelines. Evolves the prompt and the pipeline's parameters together against your labeled data, cuts losing candidates early, and shows every measurement behind the winner. Open source.
git clone https://github.com/PromptPotter/prompt-potter-optimizer.git -
skill-harness
SkillsHow do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”
git clone https://github.com/MrBinnacle/skill-harness.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.