LLM Eval Harness
Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.
Open the source (opens github.com) https://github.com/razilzul/llm-eval-harness
Before you use it
- Runs on your computer with your permissions. Read what it can do before connecting it to an agent.
- Needs credentials: ANTHROPIC_API_KEY, GEMINI_API_KEY. Use a key with the least access you can.
What it is
MCP servers from io.github.razilzul.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
oci package (stdio)
ghcr.io/razilzul/llm-eval-harness:1.0.0
From the server's entry in the official MCP Registry
Settings it asks for
ANTHROPIC_API_KEYsecret · Only needed to evaluate anthropic:* models. Not needed for mock models.GEMINI_API_KEYsecret · Only needed to evaluate gemini:* models. Not needed for mock models.
Related
-
mcp-eval-runner
MCP serversA standardized testing harness for MCP servers and agent workflows
mcp-eval-runner@1.0.1 -
Human Creative AI Workflow & Eval Lab
MCP serversRemote MCP tools for 3-vendor AI workflow diagnostics, HAR sandboxes, 50v audits, and rubrics.
-
coding.sh
MCP serversOpen-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
git clone https://github.com/jackulau/coding.sh.git -
cap-evolve
MCP serversOptimize any AI agent’s skills, tools/MCP, and prompts against your own evals.
git clone https://github.com/skillberry-ai/cap-evolve.git -
javis-os
MCP serversgit clone https://github.com/blogminhquy/javis-os.git -
tuui
MCP serversA desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.
git clone https://github.com/AI-QL/tuui.git