Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
One-Eval
Datasets & benchmarksAutomated system for LLM evaluation via agents. Doc as below:
git clone https://github.com/OpenDCAI/One-Eval.git -
Awesome-Datasets-Hub
Datasets & benchmarksA curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.
git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git -
KADATH
Agents & frameworksEvolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.
git clone https://github.com/i3T4AN/KADATH.git -
seo-skill-bench
Datasets & benchmarksOpen benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.
git clone https://github.com/aleclindz/seo-skill-bench.git -
agentenv-framework
MCP serversCreating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
git clone https://github.com/scaleapi/agentenv-framework.git -
ai-reliability-copilot
PromptsTurn a production incident into a structured 9-section LLM response (severity, root cause, mitigation, postmortem). Ships with a 5-scenario regression suite + LLM-as-judge eval pipeline.
git clone https://github.com/YanpengQi7/ai-reliability-copilot.git -
wtfibought
Agents & frameworksA live arena where LLM agents trade real market data with virtual money — every thought, tool call, thesis and post-mortem is public. Bring your own model and key.
git clone https://github.com/mamawai/wtfibought.git -
skills
SkillsEdge finds task-fit agent skills, shows available evidence, and loads pinned versions into AI agents.
git clone https://github.com/getedgehq/skills.git -
dev-skills
SkillsDeveloper and product skills for Claude Code, Cursor and Codex: 1-file AI evals and Chrome extension store assets.
git clone https://github.com/Illyism/dev-skills.git -
prism
Datasets & benchmarksОткрытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением в 1С — Claude, GPT, Gemini, DeepSeek, YandexGPT, GigaChat.
git clone https://github.com/genlab-1c/prism.git -
skill-evolution
SkillsImprove Claude Code and OpenAI Codex skills from real feedback, with evidence, regression checks, and human review.
git clone https://github.com/rwang23/skill-evolution.git -
riftcoach-agent
AI repositoriesA quality-gated League of Legends post-game coaching agent built on deterministic match data, RAG, and LLM evaluation.
git clone https://github.com/123Cx330Yrx/riftcoach-agent.git -
agentic-loop-engineering-course
Docs & tutorialsAn 18 notebook course that isolates and measures each component of agentic loop engineering on real, industry standard software datasets.
git clone https://github.com/FareedKhan-dev/agentic-loop-engineering-course.git -
dokimi
SkillsDokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.
git clone https://github.com/intel/dokimi.git -
coding.sh
MCP serversOpen-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
git clone https://github.com/jackulau/coding.sh.git -
ai-rulez
SkillsAuthor, generate, validate and govern AI agent rules, skills, plugins and MCP configuration across 52 coding assistants. Review with verified model evidence, evaluate skills, lock and sign dependencies, and publish bundles. CLI, MCP server and Go API.
git clone https://github.com/Goldziher/ai-rulez.git -
awesome-llm-observability
Skills50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
git clone https://github.com/ContextJet-ai/awesome-llm-observability.git -
skill-evolution
SkillsSelf-improvement feedback loop for AI agent skills — analyzes past sessions, drafts structured improvement proposals, and gates auto-apply behind an evaluation framework. Host-agnostic (Hermes, Claude Code, and any HostAdapter).
git clone https://github.com/Carlo1911/skill-evolution.git -
reforge-ai-evals
PromptsOpen-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.
git clone https://github.com/Calibre-Labs/reforge-ai-evals.git -
hermeneutic
SkillsTurn corrections from AI agent logs into guidance for similar tasks. Local memory with semantic retrieval, prompt-context hooks, and response checks.
git clone https://github.com/hermes-labs-ai/hermeneutic.git -
OpenART
SkillsOpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.
git clone https://github.com/AI45Lab/OpenART.git -
jev-judge-mcp
MCP serversTyped judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
git clone https://github.com/PyModel/jev-judge-mcp.git -
verbatim-citation-gate
SkillsCatch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic.
git clone https://github.com/tonydzi/verbatim-citation-gate.git -
nasde-toolkit
SkillsCLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.
git clone https://github.com/NoesisVision/nasde-toolkit.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.