InnerVault

LLM Eval Harness

Stale Official MCP Registry, io.github.razilzul

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Open the source (opens github.com) https://github.com/razilzul/llm-eval-harness

Before you use it

  • Runs on your computer with your permissions. Read what it can do before connecting it to an agent.
  • Needs credentials: ANTHROPIC_API_KEY, GEMINI_API_KEY. Use a key with the least access you can.

What it is

MCP servers from io.github.razilzul.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

oci package (stdio)

ghcr.io/razilzul/llm-eval-harness:1.0.0

From the server's entry in the official MCP Registry

Settings it asks for

  • ANTHROPIC_API_KEY secret · Only needed to evaluate anthropic:* models. Not needed for mock models.
  • GEMINI_API_KEY secret · Only needed to evaluate gemini:* models. Not needed for mock models.

Related

  1. mcp-eval-runner

    MCP servers

    A standardized testing harness for MCP servers and agent workflows

    Stale Official MCP Registry, io.github.dbsectrainer Updated 12 days ago 2 notes

    mcp-eval-runner@1.0.1
  2. Remote MCP tools for 3-vendor AI workflow diagnostics, HAR sandboxes, 50v audits, and rubrics.

    Stale Official MCP Registry, io.github.jdurey Updated 2 days ago 1 note

  3. coding.sh

    MCP servers

    Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.

    Indexed GitHub, jackulau 30 ★MITUpdated today

    git clone https://github.com/jackulau/coding.sh.git
  4. cap-evolve

    MCP servers

    Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.

    Indexed GitHub, skillberry-ai 70 ★Apache-2.0Updated today

    git clone https://github.com/skillberry-ai/cap-evolve.git
  5. javis-os

    MCP servers

    Indexed GitHub, blogminhquy 827 ★MITUpdated today

    git clone https://github.com/blogminhquy/javis-os.git
  6. tuui

    MCP servers

    A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.

    Stale GitHub, AI-QL 1,157 ★Apache-2.0Updated 6 months ago

    git clone https://github.com/AI-QL/tuui.git