runSuiteFile

The upcoming SDK 10, CLI 6, and Vitest 0.3 releases require an MCPJam login or sk_ API key for every eval, including BYOK runs with uploads disabled. These versions must ship after the backend run-start endpoint is deployed. Earlier versions keep their existing behavior.

runSuiteFile(sourceText, options) runs an MCPJam suite file locally and decides it with the v2 verdict policy. It is the engine behind mcpjam test, exposed for code that wants the same run without the CLI. It takes explicit configuration: server bindings, provider keys, and an optional callback that returns an MCPJam connection. Every run resolves MCPJam identity; when the callback is absent, it uses the SDK credential resolver. It does not discover suite files, print, write artifacts or exit the process. It never uploads results and sends no usage telemetry — that belongs to the caller.

import { readFile } from "node:fs/promises";
import { runSuiteFile, SuiteFileRunError } from "@mcpjam/sdk";

const source = await readFile(".mcpjam/evals/example.yaml", "utf8");

try {
  const result = await runSuiteFile(source, {
    servers: {
      notes: { config: { command: "node", args: ["./notes-server.js"] }, source: "code" },
    },
    inference: {
      mode: "byok",
      providerKeys: { anthropic: process.env.ANTHROPIC_API_KEY! },
    },
    caseIds: ["c_reads_a_note"],
  });
  console.log(result.verdict); // "passed" | "failed" | "inconclusive" | "notEstablished"
} catch (error) {
  if (error instanceof SuiteFileRunError) {
    console.error(error.code, error.phase, error.category, error.message);
  }
  throw error;
}

Options

Option	Default	Description
`servers`	required	Target server bindings by name: `{ config: MCPServerConfig, source?: string }`. `source` is a non-secret label shown in reports. The runner creates and owns its own `MCPClientManager`.
`caseIds`	every enabled case	Exact authored case ids. Unknown, duplicate and disabled ids are refused.
`inference.mode`	`"auto"`	`"auto"`, `"byok"` or `"mcpjam"` — see Inference.
`inference.providerKeys`	`{}`	BYOK keys by provider id (`anthropic`, `openai`, `google`, `deepseek`, `mistral`, `openrouter`, `xai`).
`inference.baseUrls`	—	Non-secret provider base URLs (`anthropic`, `openai`, `ollama`, …).
`inference.resolveMcpjam`	—	`() => Promise<{ baseUrl, projectId, getAuth, headers? }>`, called once before every run, including BYOK. `baseUrl` is the app origin (not `/api/v1`), `projectId` a concrete id, `getAuth` returns the current bearer and is re-read for every mint, retry and revoke. `headers` go to MCPJam’s API only.
`hostTemplateId`	—	One host template to emulate (`claude-code`, `chatgpt`, …).
`importApprovals`	—	`{ caseId, reason }[]` approving `approximated` imported cases for this run.
`concurrency`	`1`	Iterations of one case in flight at once. Cases run sequentially.
`iterationTimeoutMs`	`120000`	Bounds each iteration’s executor run (not grading, not setup).
`maxSteps`	`10`	Model steps per prompt.
`setupTimeoutMs`	`30000`	Bounds each setup operation — a connection, the tool catalog, a lease mint.
`signal`	—	Cooperative cancellation. After execution begins, an abort returns partial evidence instead of throwing.
`onProgress`	—	Observer for setup, case and iteration progress. A throwing observer is recorded as a warning.

Result

Field	Description
`verdict`	`"passed"` or `"failed"` only for a completed run; `"inconclusive"` when the validity policy withheld a verdict; `"notEstablished"` for an interrupted or undecidable run.
`passed`	`verdict === "passed"`.
`termination`	`"completed"`, `"aborted"` (the signal) or `"stopped"` (a credential, billing, or organization AI-key policy refusal stopped scheduling).
`complete`	Whether every planned iteration reached a terminal state.
`decision`	The validated v2 `EvalVerdictDecision` over the planned population. For an interrupted run it is partial evidence, never a completed gate — read `verdict`.
`cases`	Per case: effective model and rail, configured iterations and threshold, the evaluator-config hash, judge state, import evidence, per-iteration evidence (lifecycle status, task verdict, evaluator error, attributed refusal, tool calls, policy blocks, scores, stage chain) and the case’s row of the decision.
`toolPolicy`	The declared policy, the launch snapshot and every block.
`warnings`, `issues`	Warnings, and run-affecting problems (credentials, billing, integrity, cleanup) by phase and category.
`report`	The validated `eval-local-run` structured report — pass it to `renderStructuredRunJson`, `renderStructuredRunJUnitXml`, `renderStructuredRunHtml`, or `formatLocalEvalRunSummary`.

Nothing in the result carries a provider key, a platform token, a lease, server headers or environment objects. Every credential the run was handed — provider keys, platform tokens, server credentials, and any header, environment or URL value whose name or shape marks it as one (Authorization, GITHUB_TOKEN, ?sig=, a ghp_… token) — is replaced with [REDACTED] wherever observed text repeats it: execution errors, tool-call arguments, evaluator and predicate reasons, issues and warnings. Ordinary configuration such as NODE_ENV=production is not treated as a secret. Identifiers (case ids, server and tool names) and fixed vocabularies such as statuses are never rewritten.

Errors

Input that cannot run and environments that cannot be set up throw SuiteFileRunError with a stable code, a phase (validation, setup, execution, reporting) and a category. Validation and setup refusals are thrown before any model or tool call. A refusal that found concrete problems also lists every one found at that stage in details.problems; other errors carry only their message, so treat problems as optional. One error comes after execution: REPORT_INVALID (phase reporting, category integrity) means the cases ran — model calls were made and may have been billed — but the run’s own report failed validation, so no result is returned. Retrying repeats that inference.

`category`	Examples
`usage`	Invalid file, selection or options; conflicting inference intent; an environment-only target
`unsupported`	Direct `toolCall` steps, widget/render/discovery assertions, gating tool-result and latency assertions, gating judges; organization AI-key policy refused the model (`AI_POLICY_REFUSED`)
`import`	An ineligible imported case or an approval that applies to nothing
`policy`	A tool-policy `deny` name that matches no tool
`credentials`	A missing or rejected provider key or platform credential
`billing`	MCPJam or the provider refused inference for a billing reason (credits, quota, a spend budget); not the same as an organization AI-key policy refusal
`setup`	A missing binding, a failed connection or tool catalog, duplicate tool names across servers
`cancelled`	The signal fired before execution began
`integrity`	The run’s own report failed validation, after the cases ran

An iteration that failed its assertions is evidence, not an error: it comes back in the result. A provider refusal during execution is attributed on the iteration (refusal) and never counted as a failing assertion. Refusal attributions include credentials, billing, orgPolicy (organization AI-key policy), rateLimited, and unavailable. A terminal refusal (credentials, billing, or orgPolicy) stops scheduling further iterations.

Eval authentication

For the upcoming releases, see eval identity and local data for credential precedence, CI setup, failure behavior, and the exact fields sent before an eval.

Get Started

Concepts

Reference

Options

Result

Errors

Eval authentication