Testing Framework

Evals

Check that real models pick the right tool with the right arguments.

View .mdEdit on GitHub
// tests/evals/moon.eval.json
{
  "models": ["claude-opus-5-5", "gpt-5"],
  "runs": 3,
  "passRate": 1,
  "system": "You are a helpful astronomy assistant.",
  "cases": [
    { "prompt": "what does the moon look like on 2026-11-24?", "expect": { "tool": "show-moon", "args": { "date": "2026-11-24" } } },
    { "prompt": "quick: moon phase on 2026-12-09, just text", "expect": { "tool": "moon-facts" } },
    { "prompt": "write me a haiku about autumn", "expect": { "tool": null } }
  ]
}
ANTHROPIC_API_KEY=... OPENAI_API_KEY=... moonpeak test --eval

For every case and model, moonpeak sends your tool list (names, descriptions, input schemas) and the prompt, runs times, and checks which tool the model called:

expect Passes when
"tool": "name" the model called that tool
"tool": null the model called no tool
"args": {...} the arguments partially match

A case passes when at least passRate of its runs pass (default: all of them). Tools with visibility: ["app"] are never sent, because hosts never show them to the model.

Providers

Model names Needs
claude-* ANTHROPIC_API_KEY and @anthropic-ai/sdk in your project
gpt-*, o1/o3/o4-*, chatgpt-* OPENAI_API_KEY and openai in your project

ANTHROPIC_BASE_URL and OPENAI_BASE_URL are respected, so evals can run against a proxy or a gateway. Evals are the one mode that costs money; keep runs small and run them on demand rather than on every push.