InnerVault

Library

85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

  1. nautilus-compass

    Datasets & benchmarks

    Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

    Indexed GitHub, chunxiaoxx 1,258 ★Custom licence (see repository)Updated 3 days ago

    git clone https://github.com/chunxiaoxx/nautilus-compass.git
  2. HaluMem

    Datasets & benchmarks

    HaluMem is the first operation level hallucination evaluation benchmark tailored to agent memory systems.

    Stale GitHub, MemTensor 178 ★Custom licence (see repository)Updated 20 days ago

    git clone https://github.com/MemTensor/HaluMem.git
  3. confabulations

    Datasets & benchmarks

    Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

    Stale GitHub, lechmazur 249 ★Updated 15 months ago

    git clone https://github.com/lechmazur/confabulations.git
  4. Multi-User-LLM-Agent

    Datasets & benchmarks

    Official code for the paper: "Multi-User Large Language Model Agents"

    Stale GitHub, Kordi-AI 39 ★Apache-2.0Updated 8 days ago

    git clone https://github.com/Kordi-AI/Multi-User-LLM-Agent.git
  5. sacp

    Datasets & benchmarks

    State-Aware Collaboration Protocol: auditable work receipts for AI agents.

    Stale GitHub, aDragon0707 107 ★MITUpdated 33 days ago

    git clone https://github.com/aDragon0707/sacp.git
  6. nagi-bench

    Datasets & benchmarks

    One-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.

    Indexed GitHub, nagi-studio 78 ★MITUpdated 5 days ago

    git clone https://github.com/nagi-studio/nagi-bench.git
  7. PyHealth

    Datasets & benchmarks

    A deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.

    Indexed GitHub, sunlabuiuc 1,699 ★MITUpdated yesterday

    git clone https://github.com/sunlabuiuc/PyHealth.git
  8. Glyd

    Datasets & benchmarks

    Lossless AI compression: 33% less GPU memory, bit for bit.

    Stale GitHub, surya-koritala 321 ★Custom licence (see repository)Updated 15 days ago

    git clone https://github.com/surya-koritala/Glyd.git
  9. NextTransformer

    Datasets & benchmarks

    :sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.

    Indexed GitHub, Interactive-Active-AI 435 ★MITUpdated yesterday

    git clone https://github.com/Interactive-Active-AI/NextTransformer.git
  10. ds4-on-spark

    Datasets & benchmarks

    Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark, and full continuous batch support

    Stale GitHub, Entrpi 409 ★MITUpdated 2 months ago

    git clone https://github.com/Entrpi/ds4-on-spark.git
  11. ResearchClawBench

    Datasets & benchmarks

    🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery

    Stale GitHub, InternScience 275 ★MITUpdated 20 days ago

    git clone https://github.com/InternScience/ResearchClawBench.git
  12. Simreal-MLBench

    Datasets & benchmarks

    [Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

    Stale GitHub, Simreal-AI 301 ★Apache-2.0Updated 54 days ago

    git clone https://github.com/Simreal-AI/Simreal-MLBench.git
  13. DecisionPivot

    Datasets & benchmarks

    DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.

    Stale GitHub, StartLuxLabs 144 ★Apache-2.0Updated 27 days ago

    git clone https://github.com/StartLuxLabs/DecisionPivot.git
  14. mobilegym

    Datasets & benchmarks

    [EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training

    Indexed GitHub, Purewhiter 866 ★Apache-2.0Updated 4 days ago

    git clone https://github.com/Purewhiter/mobilegym.git
  15. openbench

    Datasets & benchmarks

    Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost

    Stale GitHub, minghinmatthewlam 137 ★MITUpdated 2 months ago

    git clone https://github.com/minghinmatthewlam/openbench.git
  16. Omni-SafetyBench

    Datasets & benchmarks

    [ACM MM 2026 Oral] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".

    Stale GitHub, THU-BPM 115 ★Apache-2.0Updated 2 months ago

    git clone https://github.com/THU-BPM/Omni-SafetyBench.git
  17. NIMStats

    Datasets & benchmarks

    📊 Automated hourly benchmarks for 20+ NVIDIA NIM models — interactive dashboard, zero infra, self-hostable. Open-source & community-driven.

    Indexed GitHub, MauroDruwel 57 ★Updated yesterday

    git clone https://github.com/MauroDruwel/NIMStats.git
  18. WebsiteBench

    Datasets & benchmarks

    Can AI agents rebuild websites through browser exploration?

    Indexed GitHub, TIGER-AI-Lab 300 ★Apache-2.0Updated 7 days ago

    git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git
  19. xVerify

    Datasets & benchmarks

    xVerify: Efficient Answer Verifier for Reasoning Model Evaluations

    Stale GitHub, IAAR-Shanghai 150 ★Custom licence (see repository)Updated 20 days ago

    git clone https://github.com/IAAR-Shanghai/xVerify.git
  20. agent-duel

    Datasets & benchmarks

    Blinded battles between coding agents: two agents, one prompt, you pick the winner. Desktop app, runs locally.

    Stale GitHub, bottomless 35 ★Custom licence (see repository)Updated 6 days ago

    git clone https://github.com/bottomless/agent-duel.git
  21. TwinFix

    Datasets & benchmarks

    TwinFix — AI agent that diagnoses, reproduces and fixes bugs in a disposable twin environment. 在一次性孪生环境中自动诊断、复现并修复故障的 AI Agent。

    Indexed GitHub, spacesky-cell 117 ★Apache-2.0Updated 4 days ago

    git clone https://github.com/spacesky-cell/TwinFix.git
  22. mac-ml-speed-test

    Datasets & benchmarks

    A few quick scripts focused on testing TensorFlow/PyTorch/Llama 2 on macOS.

    Stale GitHub, mrdbourke 203 ★Updated 2 years ago

    git clone https://github.com/mrdbourke/mac-ml-speed-test.git
  23. debate

    Datasets & benchmarks

    Adversarial multi-turn benchmark for LLM debate quality, using side-swapped matchups and multi-model judging to rank models by judged debate performance.

    Stale GitHub, lechmazur 38 ★Updated 22 days ago

    git clone https://github.com/lechmazur/debate.git
  24. autoguardrails

    Datasets & benchmarks

    Alignment-research scaffold (autoresearch-style) for LLM guardrails: search over a single policy.md surface

    Stale GitHub, SantanderAI 129 ★Apache-2.0Updated 47 days ago

    git clone https://github.com/SantanderAI/autoguardrails.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.