Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
judge0
Agents & frameworksRobust, fast, scalable, and sandboxed open-source online code execution system for humans and AI.
git clone https://github.com/judge0/judge0.git -
judgeval
PromptsThe Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
git clone https://github.com/JudgmentLabs/judgeval.git -
bigcode-models-leaderboard
Tools & appsHugging Face space by bigcode
bigcode/bigcode-models-leaderboard -
nautilus-compass
Datasets & benchmarksOpen memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
git clone https://github.com/chunxiaoxx/nautilus-compass.git -
mt_bench_human_judgments
Datasets & benchmarksHugging Face dataset by lmsys
lmsys/mt_bench_human_judgments -
dingo
AI repositoriesDingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
git clone https://github.com/MigoXLab/dingo.git -
jev-judge-mcp
MCP serversTyped judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
git clone https://github.com/PyModel/jev-judge-mcp.git -
xVerify
Datasets & benchmarksxVerify: Efficient Answer Verifier for Reasoning Model Evaluations
git clone https://github.com/IAAR-Shanghai/xVerify.git -
Mica-v0.5-4B
Modelstext classification model by sky7350
sky7350/Mica-v0.5-4B -
xFinder
Datasets & benchmarks[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
git clone https://github.com/IAAR-Shanghai/xFinder.git -
fde-gym
Datasets & benchmarksThe online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.
git clone https://github.com/deeplethe/fde-gym.git -
oumi
AI repositoriesEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
git clone https://github.com/oumi-ai/oumi.git -
styleseed
SkillsOpen-source design-method engine for Claude Code, Codex & Cursor. One agent skill with 22 internal workflows for fixed design judgment, multiple grammars, semantic palettes, reference compilation, and evidence-verified UI. MIT.
git clone https://github.com/bitjaru/styleseed.git -
Judge0 CE
Public APIsOnline code execution system
-
Fjc Judges
MCP serversFJC Judges — the Federal Judicial Center's Biographical Directory of Article III federal judges…
@pipeworx/mcp-fjc-judges@0.2.1 -
Wenshucha Court Judgments
MCP serversSearch 160M+ Chinese court judgments; verify case numbers and quotes.
-
judgevet
MCP serversAsk TypeSafe Jev yes/no, choice and score questions and receive structured model judgments.
judgevet@0.23.0 -
simmer
SkillsIterative artifact refinement with investigation-first judge board - constructs problem-specific judges that read the code, understand the problem, and propose evidence-based improvements
git clone https://github.com/2389-research/simmer.git -
judgment-pack
MCP serversOffline JPS validator, conformance tester, and experimental evaluator over stdio MCP; keyless.
ghcr.io/judgment-pack/judgment-pack:0.46.0 -
Judges Panel
MCP servers45 judges that evaluate AI-generated code for security, cost, and quality with built-in AST.
@kevinrabun/judges@3.129.9 -
ZTL Judge
MCP serversZero-trust logic judge: ZFL claims, algebra derivation steps, sets defined by a property.
-
mcp-as-a-judge
MCP serversMCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations
mcp-as-a-judge@0.3.20 -
Money Mind — the judge
MCP serversStatistical adjudication for agents: backtest judges, fact-checks. Pay per call in USDC via x402.
-
debate
Datasets & benchmarksAdversarial multi-turn benchmark for LLM debate quality, using side-swapped matchups and multi-model judging to rank models by judged debate performance.
git clone https://github.com/lechmazur/debate.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.