InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. One-Eval

    Datasets & benchmarks

    Automated system for LLM evaluation via agents. Doc as below:

    Stale GitHub, OpenDCAI 165 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/OpenDCAI/One-Eval.git
  2. Awesome-Datasets-Hub

    Datasets & benchmarks

    A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

    Stale GitHub, ahammadmejbah 149 ★Updated 4 months ago

    git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git
  3. KADATH

    Agents & frameworks

    Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.

    Stale GitHub, i3T4AN 253 ★Apache-2.0Updated 23 days ago

    git clone https://github.com/i3T4AN/KADATH.git
  4. seo-skill-bench

    Datasets & benchmarks

    Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.

    Stale GitHub, aleclindz 51 ★MITUpdated 32 days ago

    git clone https://github.com/aleclindz/seo-skill-bench.git
  5. agentenv-framework

    MCP servers

    Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.

    Indexed GitHub, scaleapi 441 ★Apache-2.0Updated today

    git clone https://github.com/scaleapi/agentenv-framework.git
  6. Turn a production incident into a structured 9-section LLM response (severity, root cause, mitigation, postmortem). Ships with a 5-scenario regression suite + LLM-as-judge eval pipeline.

    Stale GitHub, YanpengQi7 83 ★Updated 29 days ago

    git clone https://github.com/YanpengQi7/ai-reliability-copilot.git
  7. wtfibought

    Agents & frameworks

    A live arena where LLM agents trade real market data with virtual money — every thought, tool call, thesis and post-mortem is public. Bring your own model and key.

    Stale GitHub, mamawai 143 ★MITUpdated 18 days ago

    git clone https://github.com/mamawai/wtfibought.git
  8. skills

    Skills

    Edge finds task-fit agent skills, shows available evidence, and loads pinned versions into AI agents.

    Stale GitHub, getedgehq 5 ★Apache-2.0Updated 4 days ago

    git clone https://github.com/getedgehq/skills.git
  9. dev-skills

    Skills

    Developer and product skills for Claude Code, Cursor and Codex: 1-file AI evals and Chrome extension store assets.

    Stale GitHub, Illyism 22 ★Updated 4 days ago

    git clone https://github.com/Illyism/dev-skills.git
  10. prism

    Datasets & benchmarks

    Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением в 1С — Claude, GPT, Gemini, DeepSeek, YandexGPT, GigaChat.

    Stale GitHub, genlab-1c 55 ★MITUpdated 18 days ago

    git clone https://github.com/genlab-1c/prism.git
  11. Improve Claude Code and OpenAI Codex skills from real feedback, with evidence, regression checks, and human review.

    Stale GitHub, rwang23 5 ★MITUpdated 16 days ago

    git clone https://github.com/rwang23/skill-evolution.git
  12. riftcoach-agent

    AI repositories

    A quality-gated League of Legends post-game coaching agent built on deterministic match data, RAG, and LLM evaluation.

    Stale GitHub, 123Cx330Yrx 101 ★MITUpdated 25 days ago

    git clone https://github.com/123Cx330Yrx/riftcoach-agent.git
  13. An 18 notebook course that isolates and measures each component of agentic loop engineering on real, industry standard software datasets.

    Stale GitHub, FareedKhan-dev 59 ★MITUpdated 4 months ago

    git clone https://github.com/FareedKhan-dev/agentic-loop-engineering-course.git
  14. dokimi

    Skills

    Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.

    Indexed GitHub, intel 37 ★Apache-2.0Updated today

    git clone https://github.com/intel/dokimi.git
  15. coding.sh

    MCP servers

    Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.

    Indexed GitHub, jackulau 30 ★MITUpdated today

    git clone https://github.com/jackulau/coding.sh.git
  16. ai-rulez

    Skills

    Author, generate, validate and govern AI agent rules, skills, plugins and MCP configuration across 52 coding assistants. Review with verified model evidence, evaluate skills, lock and sign dependencies, and publish bundles. CLI, MCP server and Go API.

    Indexed GitHub, Goldziher 172 ★MITUpdated today

    git clone https://github.com/Goldziher/ai-rulez.git
  17. 50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

    Stale GitHub, ContextJet-ai 46 ★Custom licence (see repository)Updated 5 days ago

    git clone https://github.com/ContextJet-ai/awesome-llm-observability.git
  18. Self-improvement feedback loop for AI agent skills — analyzes past sessions, drafts structured improvement proposals, and gates auto-apply behind an evaluation framework. Host-agnostic (Hermes, Claude Code, and any HostAdapter).

    Stale GitHub, Carlo1911 6 ★MITUpdated 29 days ago

    git clone https://github.com/Carlo1911/skill-evolution.git
  19. Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

    Stale GitHub, Calibre-Labs 95 ★MITUpdated 9 days ago

    git clone https://github.com/Calibre-Labs/reforge-ai-evals.git
  20. hermeneutic

    Skills

    Turn corrections from AI agent logs into guidance for similar tasks. Local memory with semantic retrieval, prompt-context hooks, and response checks.

    Stale GitHub, hermes-labs-ai 9 ★Apache-2.0Updated 27 days ago

    git clone https://github.com/hermes-labs-ai/hermeneutic.git
  21. OpenART

    Skills

    OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.

    Stale GitHub, AI45Lab 263 ★AGPL-3.0Updated 15 days ago

    git clone https://github.com/AI45Lab/OpenART.git
  22. jev-judge-mcp

    MCP servers

    Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.

    Stale GitHub, PyModel 48 ★MITUpdated 32 days ago

    git clone https://github.com/PyModel/jev-judge-mcp.git
  23. Catch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic.

    Indexed GitHub, tonydzi 6 ★MITUpdated 2 days ago

    git clone https://github.com/tonydzi/verbatim-citation-gate.git
  24. nasde-toolkit

    Skills

    CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

    Stale GitHub, NoesisVision 14 ★MITUpdated 32 days ago

    git clone https://github.com/NoesisVision/nasde-toolkit.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.