InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. OpenART

    Skills

    OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.

    Stale GitHub, AI45Lab 263 ★AGPL-3.0Updated 15 days ago

    git clone https://github.com/AI45Lab/OpenART.git
  2. ai-agents-tutorial

    Agents & frameworks

    Learn AI Agents step by step, from scratch - from function calling to agent loops to multi-agent systems, orchestration, and evaluation.

    Indexed GitHub, amitshekhariitbhu 212 ★Apache-2.0Updated 2 days ago

    git clone https://github.com/amitshekhariitbhu/ai-agents-tutorial.git
  3. WritingBench

    Datasets & benchmarks

    WritingBench: A Comprehensive Benchmark for Generative Writing

    Stale GitHub, X-PLUG 202 ★Apache-2.0Updated 11 months ago

    git clone https://github.com/X-PLUG/WritingBench.git
  4. GTA

    Agents & frameworks

    [NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2

    Indexed GitHub, open-compass 153 ★Apache-2.0Updated 6 days ago

    git clone https://github.com/open-compass/GTA.git
  5. skill-reach

    Skills

    An evaluation suite for AI agent skill triggering and collision detection

    Stale GitHub, google 13 ★Apache-2.0Updated 23 days ago

    git clone https://github.com/google/skill-reach.git
  6. MMLU-Pro

    Datasets & benchmarks

    Hugging Face dataset by TIGER-Lab

    Indexed Hugging Face Hub, TIGER-Lab 602 ♥mitUpdated 6 months ago

    TIGER-Lab/MMLU-Pro
  7. AgentLab

    Datasets & benchmarks

    AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

    Stale GitHub, ServiceNow 653 ★Custom licence (see repository)Updated 41 days ago

    git clone https://github.com/ServiceNow/AgentLab.git
  8. agentops

    AI repositories

    Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

    Stale GitHub, AgentOps-AI 5,885 ★MITUpdated 4 months ago

    git clone https://github.com/AgentOps-AI/agentops.git
  9. self-bench

    Datasets & benchmarks

    Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests.

    Indexed GitHub, mupt-ai 34 ★MITUpdated today

    git clone https://github.com/mupt-ai/self-bench.git
  10. Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

    Stale GitHub, Calibre-Labs 95 ★MITUpdated 9 days ago

    git clone https://github.com/Calibre-Labs/reforge-ai-evals.git
  11. judgeval

    Prompts

    The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

    Indexed GitHub, JudgmentLabs 1,074 ★Apache-2.0Updated today

    git clone https://github.com/JudgmentLabs/judgeval.git
  12. xorcise

    Datasets & benchmarks

    Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the claim.

    Stale GitHub, xorcise-ai 33 ★Custom licence (see repository)Updated 17 days ago

    git clone https://github.com/xorcise-ai/xorcise.git
  13. 50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

    Stale GitHub, ContextJet-ai 46 ★Custom licence (see repository)Updated 5 days ago

    git clone https://github.com/ContextJet-ai/awesome-llm-observability.git
  14. instructor-xl

    Models

    sentence similarity model by hkunlp · sentence-transformers

    Stale Hugging Face Hub, hkunlp 587 ♥apache-2.0Updated 3 years ago

    hkunlp/instructor-xl
  15. CivAgent

    Agents & frameworks

    CivAgent is an LLM-based Human-like Agent acting as a Digital Player within the Strategy Game Unciv.

    Stale GitHub, fuxiAIlab 167 ★MPL-2.0Updated 20 months ago

    git clone https://github.com/fuxiAIlab/CivAgent.git
  16. folkbench

    AI repositories

    帮你的 AI 编程工具找到不降智、不断连、高性价比的好线路 · Live LLM relay rankings, model integrity probes & Tauri/Rust switcher for Claude Code, Codex & Gemini CLI

    Indexed GitHub, Folkbench 3,883 ★Custom licence (see repository)Updated today

    git clone https://github.com/Folkbench/folkbench.git
  17. t2v_metrics

    AI repositories

    Evaluating text-to-image/video/3D models with VQAScore

    Stale GitHub, linzhiqiu 603 ★Apache-2.0Updated 5 months ago

    git clone https://github.com/linzhiqiu/t2v_metrics.git
  18. tweakidea

    Skills

    Evaluate your startup idea from 1 to 5

    Indexed GitHub, eph5xx 57 ★MITUpdated yesterday

    git clone https://github.com/eph5xx/tweakidea.git
  19. rashomon

    Skills

    Measure prompt and skill improvements with blind A/B comparison.

    Stale GitHub, shinpr 19 ★MITUpdated 25 days ago

    git clone https://github.com/shinpr/rashomon.git
  20. sentence similarity model by hkunlp · sentence-transformers

    Stale Hugging Face Hub, hkunlp 525 ♥apache-2.0Updated 3 years ago

    hkunlp/instructor-large
  21. meta-agents-research-environments

    Datasets & benchmarks

    Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.

    Stale GitHub, facebookresearch 567 ★MITUpdated 47 days ago

    git clone https://github.com/facebookresearch/meta-agents-research-environments.git
  22. cave-agent

    Agents & frameworks

    Stateful runtime management for LLM agents—inject, manipulate, and retrieve Python objects across turns.

    Stale GitHub, acodercat 204 ★MITUpdated 2 months ago

    git clone https://github.com/acodercat/cave-agent.git
  23. MimirQ

    AI repositories

    中文优先的企业 RAG 知识库:可控解析、治理、切块、混合检索、重排、引用、图谱、评测与 Dify 接入。

    Stale GitHub, skygazer42 595 ★Apache-2.0Updated 5 days ago

    git clone https://github.com/skygazer42/MimirQ.git
  24. Awesome-Datasets-Hub

    Datasets & benchmarks

    A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

    Stale GitHub, ahammadmejbah 149 ★Updated 4 months ago

    git clone https://github.com/ahammadmejbah/Awesome-Datasets-Hub.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.