Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
HaluMem
Datasets & benchmarksHaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.
git clone https://github.com/MemTensor/HaluMem.git -
confabulations
Datasets & benchmarksHallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
git clone https://github.com/lechmazur/confabulations.git -
Multi-User-LLM-Agent
Datasets & benchmarksOfficial code for the paper: "Multi-User Large Language Model Agents"
git clone https://github.com/Kordi-AI/Multi-User-LLM-Agent.git -
sacp
Datasets & benchmarksState-Aware Collaboration Protocol: auditable work receipts for AI agents.
git clone https://github.com/aDragon0707/sacp.git -
nagi-bench
Datasets & benchmarksOne-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.
git clone https://github.com/nagi-studio/nagi-bench.git -
PyHealth
Datasets & benchmarksA deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.
git clone https://github.com/sunlabuiuc/PyHealth.git -
Glyd
Datasets & benchmarksLossless AI compression: 33% less GPU memory, bit for bit.
git clone https://github.com/surya-koritala/Glyd.git -
NextTransformer
Datasets & benchmarks:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
git clone https://github.com/Interactive-Active-AI/NextTransformer.git -
ds4-on-spark
Datasets & benchmarksEntrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark, and full continuous batch support
git clone https://github.com/Entrpi/ds4-on-spark.git -
ResearchClawBench
Datasets & benchmarks🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery
git clone https://github.com/InternScience/ResearchClawBench.git -
Simreal-MLBench
Datasets & benchmarks[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
git clone https://github.com/Simreal-AI/Simreal-MLBench.git -
DecisionPivot
Datasets & benchmarksDecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
git clone https://github.com/StartLuxLabs/DecisionPivot.git -
mobilegym
Datasets & benchmarks[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training
git clone https://github.com/Purewhiter/mobilegym.git -
openbench
Datasets & benchmarksSame model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost
git clone https://github.com/minghinmatthewlam/openbench.git -
Omni-SafetyBench
Datasets & benchmarks[ACM MM 2026 Oral] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".
git clone https://github.com/THU-BPM/Omni-SafetyBench.git -
NIMStats
Datasets & benchmarks📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.
git clone https://github.com/MauroDruwel/NIMStats.git -
WebsiteBench
Datasets & benchmarksCan AI agents rebuild websites through browser exploration?
git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git -
xVerify
Datasets & benchmarksxVerify: Efficient Answer Verifier for Reasoning Model Evaluations
git clone https://github.com/IAAR-Shanghai/xVerify.git -
agent-duel
Datasets & benchmarksBlinded battles between coding agents: two agents, one prompt, you pick the winner. Desktop app, runs locally.
git clone https://github.com/bottomless/agent-duel.git -
TwinFix
Datasets & benchmarksTwinFix — AI agent that diagnoses, reproduces and fixes bugs in a disposable twin environment. 在一次性孪生环境中自动诊断、复现并修复故障的 AI Agent。
git clone https://github.com/spacesky-cell/TwinFix.git -
mac-ml-speed-test
Datasets & benchmarksA few quick scripts focused on testing TensorFlow/PyTorch/Llama 2 on macOS.
git clone https://github.com/mrdbourke/mac-ml-speed-test.git -
debate
Datasets & benchmarksAdversarial multi-turn benchmark for LLM debate quality, using side-swapped matchups and multi-model judging to rank models by judged debate performance.
git clone https://github.com/lechmazur/debate.git -
autoguardrails
Datasets & benchmarksAlignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface
git clone https://github.com/SantanderAI/autoguardrails.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.