InnerVault

Library

85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. sacp

    Datasets & benchmarks

    State-Aware Collaboration Protocol: auditable work receipts for AI agents.

    Stale GitHub, aDragon0707 107 ★MITUpdated 32 days ago

    git clone https://github.com/aDragon0707/sacp.git
  2. HallusionBench

    Datasets & benchmarks

    [CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models

    Stale GitHub, tianyi-lab 344 ★BSD-3-ClauseUpdated 13 months ago

    git clone https://github.com/tianyi-lab/HallusionBench.git
  3. openbench

    Datasets & benchmarks

    Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost

    Stale GitHub, minghinmatthewlam 137 ★MITUpdated 2 months ago

    git clone https://github.com/minghinmatthewlam/openbench.git
  4. UHGEval

    Datasets & benchmarks

    [ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.

    Stale GitHub, IAAR-Shanghai 183 ★Apache-2.0Updated 17 months ago

    git clone https://github.com/IAAR-Shanghai/UHGEval.git
  5. TerminalWorld

    Datasets & benchmarks

    TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 Evaluations and Datasets Track)

    Stale GitHub, terminalworld 54 ★Apache-2.0Updated 12 days ago

    git clone https://github.com/terminalworld/TerminalWorld.git
  6. Long-Horizon-Terminal-Bench

    Datasets & benchmarks

    Hugging Face dataset by IntelligenceLab

    Stale Hugging Face Hub, IntelligenceLab 140 ♥apache-2.0Updated 36 days ago

    IntelligenceLab/Long-Horizon-Terminal-Bench
  7. mac-ml-speed-test

    Datasets & benchmarks

    A few quick scripts focused on testing TensorFlow/PyTorch/Llama 2 on macOS.

    Stale GitHub, mrdbourke 203 ★Updated 2 years ago

    git clone https://github.com/mrdbourke/mac-ml-speed-test.git
  8. Omni-SafetyBench

    Datasets & benchmarks

    [ACM MM 2026 Oral] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".

    Stale GitHub, THU-BPM 115 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/THU-BPM/Omni-SafetyBench.git
  9. meta-agents-research-environments

    Datasets & benchmarks

    Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.

    Stale GitHub, facebookresearch 567 ★MITUpdated 47 days ago

    git clone https://github.com/facebookresearch/meta-agents-research-environments.git
  10. AIRTBench-Code

    Datasets & benchmarks

    Code Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

    Stale GitHub, dreadnode 111 ★Apache-2.0Updated 32 days ago

    git clone https://github.com/dreadnode/AIRTBench-Code.git
  11. arena-ai-leaderboards

    Datasets & benchmarks

    📊 Daily auto-updated snapshots of all Arena AI (LMSYS Chatbot Arena) leaderboards — LLM, Vision, Code, Video, Image & more. Structured JSON with historical tracking.

    Indexed GitHub, oolong-tea-2026 42 ★MITUpdated yesterday

    git clone https://github.com/oolong-tea-2026/arena-ai-leaderboards.git
  12. TwinFix

    Datasets & benchmarks

    TwinFix — AI agent that diagnoses, reproduces and fixes bugs in a disposable twin environment. 在一次性孪生环境中自动诊断、复现并修复故障的 AI Agent。

    Indexed GitHub, spacesky-cell 117 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/spacesky-cell/TwinFix.git
  13. VisualNews-Repository

    Datasets & benchmarks

    [EMNLP'21] Visual News: Benchmark and Challenges in News Image Captioning

    Stale GitHub, FuxiaoLiu 106 ★Updated 2 years ago

    git clone https://github.com/FuxiaoLiu/VisualNews-Repository.git
  14. CodeX-2M-Thinking

    Datasets & benchmarks

    Hugging Face dataset by Modotte

    Stale Hugging Face Hub, Modotte 129 ♥apache-2.0Updated 9 months ago

    Modotte/CodeX-2M-Thinking
  15. ReForm-Eval

    Datasets & benchmarks

    An benchmark for evaluating the capabilities of large vision-language models (LVLMs)

    Stale GitHub, FudanDISC 145 ★Apache-2.0Updated 3 years ago

    git clone https://github.com/FudanDISC/ReForm-Eval.git
  16. NIMStats

    Datasets & benchmarks

    📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.

    Indexed GitHub, MauroDruwel 57 ★Updated today

    git clone https://github.com/MauroDruwel/NIMStats.git
  17. VibeSearchBench

    Datasets & benchmarks

    🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.

    Stale GitHub, VibeBench 112 ★MITUpdated 5 months ago

    git clone https://github.com/VibeBench/VibeSearchBench.git
  18. qwen38-strix-halo-harness

    Datasets & benchmarks

    AI coding agent harness for any hardware, local or cloud (OpenAI-compatible engines). Tuned for AMD Strix Halo: Qwen3.8 125B on the iGPU, NPU side-jobs, crash recovery, logged benchmarks.

    Stale GitHub, aic0d3r 38 ★MITUpdated 8 days ago

    git clone https://github.com/aic0d3r/qwen38-strix-halo-harness.git
  19. agent-duel

    Datasets & benchmarks

    Blinded battles between coding agents: two agents, one prompt, you pick the winner. Desktop app, runs locally.

    Stale GitHub, bottomless 35 ★Custom licence (see repository)Updated 5 days ago

    git clone https://github.com/bottomless/agent-duel.git
  20. multi-agents

    Datasets & benchmarks

    Multi-agent teams for real work: run teams of coding agents (Claude Code, Codex, any model) in isolated git worktrees, and measure whether the team beats a solo agent on cost and results.

    Stale GitHub, BichengWang 158 ★MITUpdated 8 days ago

    git clone https://github.com/BichengWang/multi-agents.git
  21. primitive-bench

    Datasets & benchmarks

    The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.

    Stale GitHub, primitive-bench 122 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/primitive-bench/primitive-bench.git
  22. xorcise

    Datasets & benchmarks

    Run cyber-AI agents against real missions in isolated environments. Record everything as OpenTelemetry evidence. Grade the evidence, not the claim.

    Stale GitHub, xorcise-ai 33 ★Custom licence (see repository)Updated 17 days ago

    git clone https://github.com/xorcise-ai/xorcise.git
  23. agent-orchestra

    Datasets & benchmarks

    Agents can work. They cannot declare success. A governed coordination protocol for multi-agent AI work: eight contracts (task, resource, ownership, handoff, verification, evidence, approval, recovery) enforced in code, plus a reproducible conformance benchmark. Apache-2.0 code, CC BY 4.0 specification, zero dependencies, Node 18+.

    Stale GitHub, 3338902669-ops 93 ★Apache-2.0Updated 24 days ago

    git clone https://github.com/3338902669-ops/agent-orchestra.git
  24. lasagna

    Datasets & benchmarks

    Swap one RAG pipeline stage, hold the rest fixed, and measure whether the gain survives composition. The framework behind "Retrieval Augmented Generation Compositionality": 450 configurations, eight benchmarks, corpora from 5K to 200K documents.

    Indexed GitHub, dolev31 35 ★Apache-2.0Updated yesterday

    git clone https://github.com/dolev31/lasagna.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.