# Append a session note
Source: https://docs.mcpjam.com/api-reference/agent-browsers/append-a-session-note

/reference/openapi.json post /browser-sessions/note
Records a note without driving the browser. commandId provides idempotency. Request bodies are limited to 64,000 characters.



# Close a browser session
Source: https://docs.mcpjam.com/api-reference/agent-browsers/close-a-browser-session

/reference/openapi.json post /browser-sessions/close
Closes the durable session and releases its desktop. Use a new runKey for another browser. Request bodies are limited to 64,000 characters.



# Execute a browser command
Source: https://docs.mcpjam.com/api-reference/agent-browsers/execute-a-browser-command

/reference/openapi.json post /browser-sessions/command
Returns an executed, refused or unknown result. Reusing commandId with the same payload returns its receipt without replay; pending receipts return unknown. Page content is untrusted data. Request bodies are limited to 64,000 characters.



# Fetch a browser screenshot
Source: https://docs.mcpjam.com/api-reference/agent-browsers/fetch-a-browser-screenshot

/reference/openapi.json post /browser-sessions/artifact
Returns screenshot as base64 PNG, or null when absent. artifactId is the command ID from the result. Request bodies are limited to 64,000 characters.



# List your browser sessions
Source: https://docs.mcpjam.com/api-reference/agent-browsers/list-your-browser-sessions

/reference/openapi.json post /browser-sessions/sessions
Returns a sessions array of sessions owned by the caller in this project. Request bodies are limited to 64,000 characters.



# Open a cloud browser session
Source: https://docs.mcpjam.com/api-reference/agent-browsers/open-a-cloud-browser-session

/reference/openapi.json post /browser-sessions/session
Returns session identity, engine, bootId and runKey. Same runKey and policy reuse the session; omit runKey for a new isolated browser. Request bodies are limited to 64,000 characters.



# Read command receipts
Source: https://docs.mcpjam.com/api-reference/agent-browsers/read-command-receipts

/reference/openapi.json post /browser-sessions/trace
Returns entries ordered by sequence, optionally filtered by commandId. Screenshot payloads are fetched separately. Request bodies are limited to 64,000 characters.



# Run one headless agent turn
Source: https://docs.mcpjam.com/api-reference/agent/run-one-headless-agent-turn

/reference/openapi.json post /projects/{projectId}/agent
Runs ONE assistant turn over the supplied message history and responds synchronously with the final assistant text, the operations it invoked, references to any resources it created, and any actions awaiting human approval.

The caller owns conversation state: resend the full history each turn. The model is pinned server-side (hosted catalog) and billed to the project.

Tools are TIERED. Read operations and non-spending writes execute directly. Three categories are PROPOSAL-ONLY: operations that SPEND once (running a suite or a case, generating cases, cancelling a run), operations whose spend RECURS (setting a suite's schedule), and operations whose effects MCPJam cannot describe or undo because they leave MCPJam entirely (`call_server_tool`, which runs a tool on the caller's own MCP server). For each, the turn validates against the operation's real schema, persists a proposal, and returns it in `proposedActions` for a person to approve; `confirmSeverity` says which hazard the approver should be warned about. MCPJam-side deletions are excluded from the surface entirely — a proposal makes spend deliberate, but it does not make a deletion recoverable. Proposals are only offered when the request supplies a `conversationId`; without somewhere to collect a click there is nothing to approve, so the tools are withheld rather than offered and then refused.

Every tool invocation is hard-clamped to the path `projectId`. Turns are capped at 4 concurrent per organization (`429 RATE_LIMITED`; enforced per server instance) and ~90s wall clock (`504 TIMEOUT`). Guest callers are denied. Requires a hosted MCPJam deployment (`422 FEATURE_NOT_SUPPORTED` otherwise).

When a turn fails or times out AFTER work already persisted, the error body's `details.createdResources` and `details.proposedActions` carry what survived — check them before retrying.

**Retry policy:** send `idempotencyKey` as a STABLE identity for the triggering event (not a fresh uuid per attempt). Every write the turn performs derives its own key from it, so a retried turn's mutations land on the rows the first attempt created instead of duplicating them. The key makes a retry SAFE; it does not make one free — dedupe at your own trigger so most retries never re-run the turn at all. A retry whose model authors materially different arguments hashes differently and is correctly treated as a different write.



# Export project traces as OTLP/JSON
Source: https://docs.mcpjam.com/api-reference/catalog/export-project-traces-as-otlpjson

/reference/openapi.json get /trace-exports/otlp
Returns an OTLP `ExportTraceServiceRequest`. Pagination rides in RESPONSE HEADERS rather than the body so the payload stays a valid OTLP document: read `x-mcpjam-next-cursor` and feed it back as `cursor`, and treat `x-mcpjam-export-complete: true` as the end of the stream.



# Get the authenticated account
Source: https://docs.mcpjam.com/api-reference/catalog/get-the-authenticated-account

/reference/openapi.json get /me
The user behind the API key (keys act as their creator, scoped to one organization).



# Get the host-compat catalog
Source: https://docs.mcpjam.com/api-reference/catalog/get-the-host-compat-catalog

/reference/openapi.json get /host-catalog
The versioned host-compatibility catalog backing `mcpjam compat` verdicts. Public and unauthenticated: static host facts and creation config with no project or user scope. Always returns a catalog — `source` is `live` when the backend publish was reachable and `bundled` when serving the SDK's built-in fallback.



# List a project's eval suites
Source: https://docs.mcpjam.com/api-reference/catalog/list-a-projects-eval-suites

/reference/openapi.json get /projects/{projectId}/eval-suites
Eval suites in the project with latest-run summaries and pass-rate trends — the `suiteId`s the eval-run routes take.



# List a project's servers
Source: https://docs.mcpjam.com/api-reference/catalog/list-a-projects-servers

/reference/openapi.json get /projects/{projectId}/servers
The MCP servers saved in the project — the `serverId`s every other route takes. STDIO command/args/env and raw headers are never exposed.



# List chat sessions
Source: https://docs.mcpjam.com/api-reference/catalog/list-chat-sessions

/reference/openapi.json get /chat-sessions
Chat sessions visible to the caller (personal + project-shared merged). Top-level rather than project-nested because `projectId` is an optional filter, not an owning scope.



# List organizations
Source: https://docs.mcpjam.com/api-reference/catalog/list-organizations

/reference/openapi.json get /organizations
The organizations the caller belongs to — the source of the `organizationId` that `GET /projects` filters by and `POST /projects` accepts. An API key is bound to one organization and only ever sees that one. Read-only: organization, member, role and billing changes are not part of the public API.



# List projects
Source: https://docs.mcpjam.com/api-reference/catalog/list-projects

/reference/openapi.json get /projects
Projects the caller can access. API keys only ever see projects inside the key's organization.



# Read the organization's spend budget
Source: https://docs.mcpjam.com/api-reference/catalog/read-the-organizations-spend-budget

/reference/openapi.json get /organizations/{organizationId}/spend-budget
The ceiling on what this organization may spend on MCPJam-billed work — chat, evals, judges and computer time — in the current billing window, and how much of it has been spent.

Any member may read it: a member who cannot raise the ceiling still needs to know it exists, because it is what refused their run. `capUsd: null` means the organization is uncapped, which is the default.

Runs that use your own provider keys are never counted against the budget and never blocked by it. Neither is the free daily allowance — it is not credits.



# Remove the organization's spend budget
Source: https://docs.mcpjam.com/api-reference/catalog/remove-the-organizations-spend-budget

/reference/openapi.json delete /organizations/{organizationId}/spend-budget
Leave the organization uncapped. **Requires organization owner or admin.**

The window's spend counter survives: it is a record of spend, not of the budget, and a budget set again later measures against the spend already made this period.



# Search or list unified sessions
Source: https://docs.mcpjam.com/api-reference/catalog/search-or-list-unified-sessions

/reference/openapi.json get /projects/{projectId}/sessions
The unified, cross-surface sessions feed for one project: Playground, user-testing, eval, and swarm transcripts in one list, newest first. Project-NESTED (unlike `/chat-sessions`) because `projectId` owns the scope here rather than filtering it.

Supplying `q` at all switches to relevance-ordered search; `scope` then chooses what is searched. Row-level visibility applies in every mode: a caller never receives another member's private Playground session.

The response echoes the honored `scope`. A deployment predating that parameter omits the echo — clients requesting a non-default scope MUST treat its absence as "unsupported" rather than accepting the results.



# Set the organization's spend budget
Source: https://docs.mcpjam.com/api-reference/catalog/set-the-organizations-spend-budget

/reference/openapi.json put /organizations/{organizationId}/spend-budget
Set or replace the ceiling. **Requires organization owner or admin.**

When the budget is reached, new MCPJam-billed work — chat turns, eval runs, computer starts — is refused for the whole organization until the window resets or an owner or admin raises the cap. Work already in flight finishes, and running computers keep going until they hibernate.

`capUsd` is stored as whole credits (1 credit = 1 cent), so it is rounded to the cent; the response reports what was kept. Changing the cap mid-window does not reset the spend counter, and does not re-announce a threshold the organization was already told about.



# The MCPJam-provided model catalog
Source: https://docs.mcpjam.com/api-reference/catalog/the-mcpjam-provided-model-catalog

/reference/openapi.json get /models
PUBLIC: this route carries no user or project data and requires no bearer token, matching the upstream catalog it proxies. Mounted before authentication for that reason.



# Add a note to the API session browser
Source: https://docs.mcpjam.com/api-reference/chat-sessions/add-a-note-to-the-api-session-browser

/reference/openapi.json post /chat-sessions/{sessionId}/browser/note
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Close the API session browser
Source: https://docs.mcpjam.com/api-reference/chat-sessions/close-the-api-session-browser

/reference/openapi.json post /chat-sessions/{sessionId}/browser/close
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Command the API session browser
Source: https://docs.mcpjam.com/api-reference/chat-sessions/command-the-api-session-browser

/reference/openapi.json post /chat-sessions/{sessionId}/browser/command
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Create an API session shell and open its browser
Source: https://docs.mcpjam.com/api-reference/chat-sessions/create-an-api-session-shell-and-open-its-browser

/reference/openapi.json post /chat-sessions/browser
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Open the API session browser
Source: https://docs.mcpjam.com/api-reference/chat-sessions/open-the-api-session-browser

/reference/openapi.json post /chat-sessions/{sessionId}/browser/open
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Read a chat session's execution trace
Source: https://docs.mcpjam.com/api-reference/chat-sessions/read-a-chat-sessions-execution-trace

/reference/openapi.json get /chat-sessions/{sessionId}/trace
Per-turn execution spans: per-tool-call latency, token usage, and indices into the transcript. Same span shape as the eval iteration trace.

**Incremental by default** — returns the LATEST turn, not the whole session, because a session's spans are the largest thing this API emits and the turn you just took is the one you are debugging. Reach older turns with `turnId` or `afterPromptIndex`; use `includeSpans=false` for cheap per-turn summaries.

A turn whose spans could not be read reports `spansUnavailable: true` rather than an empty `spans` array: "made no calls" and "could not fetch" are opposite conclusions.



# Read a chat session's messages
Source: https://docs.mcpjam.com/api-reference/chat-sessions/read-a-chat-sessions-messages

/reference/openapi.json get /chat-sessions/{sessionId}
Session metadata plus a bounded window of its raw messages.

`index` on each message is its ABSOLUTE position in the stored transcript, not its position in the returned page: trace spans reference messages positionally, so this read and `/trace` join on that number.

A transcript that could not be read reports `transcriptUnavailable: true` and a **null** `messageCount` — never 0, which would claim the conversation is empty.



# Read an API session browser screenshot URL
Source: https://docs.mcpjam.com/api-reference/chat-sessions/read-an-api-session-browser-screenshot-url

/reference/openapi.json post /chat-sessions/{sessionId}/browser/artifact
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Read the API session browser command trace
Source: https://docs.mcpjam.com/api-reference/chat-sessions/read-the-api-session-browser-command-trace

/reference/openapi.json post /chat-sessions/{sessionId}/browser/trace
Same-user, API-origin session browser operation. sessionId is the public chatSessions id. Screenshot evidence is returned as URLs; browser identity persists across turns.



# Send one agent Playground message
Source: https://docs.mcpjam.com/api-reference/chat-sessions/send-one-agent-playground-message

/reference/openapi.json post /chat-sessions/messages
Send one message to a project's MCP servers and get the model's reply plus the telemetry a participant in the conversation could not see: which tools ran, with what arguments, what each returned, per-call latency, and token usage.

**Spends model credits per call.** `idempotencyKey` is required and must be STABLE for the triggering intent — a fresh key per HTTP attempt deduplicates nothing, so a timeout-and-retry would run and bill the turn twice. With a stable key, a retry replays the completed turn.

Omit `sessionId` to start a session; pass the one this returns to continue it. Configuration (`modelId`, target, `systemPrompt`, `toolMode`) pins on the FIRST turn — a continuation that resends any of it is refused with `details.reason: "CONFIG_ON_CONTINUATION"`.

Only sessions created through this endpoint may be continued through it (`CONTINUATION_NOT_ALLOWED`): appending to a human's live Playground session would interleave two writers on one transcript.

`toolMode` defaults to `read_only`, which advertises only tools the server annotated `readOnlyHint: true`. That hint is server-asserted, so `read_only` is a policy this host applies, not a guarantee it can verify. `auto` advertises everything and **may cause real external side effects** through arbitrary third-party tools.

Pass `hostId` — or target an `environmentId`, which pins its own host — to run the turn AS a saved host. That is what selects the ENGINE: a host declaring an agent harness runs the real runtime, and one that cannot run here is refused BEFORE it spends (422, `details.reason: "HARNESS_UNAVAILABLE"`, with `details.kind` naming the rule that refused) rather than quietly downgraded to the emulated engine. Every response names what actually ran in `engine`. `hostId` is per-turn and cannot be pinned, so a continuation of a session that named ONLY a host must re-send it — omitting it is a 400 with `details.reason: "HOST_TARGET_REQUIRED"`, never a quiet fall back to the emulated engine.



# Create a client (from a template or a full config)
Source: https://docs.mcpjam.com/api-reference/clients/create-a-client-from-a-template-or-a-full-config

/reference/openapi.json post /projects/{projectId}/clients
Creates a client in the project and responds `201` with the new client's detail. Seed the config one of two ways: pass `template` (a built-in template id, optional `theme`) to seed server-side from the live backend client-template catalog, falling back to the bundled SDK catalog snapshot if the live catalog is unavailable, OR pass `config` (a full client config v2). Exactly one of `template` or `config` is required.

Guest callers are denied (client creation is a write).



# Delete a client
Source: https://docs.mcpjam.com/api-reference/clients/delete-a-client

/reference/openapi.json delete /projects/{projectId}/clients/{client}
Permanently deletes the client. Refused while a live environment, an active legacy journey or an eval suite still holds it — the error names what to detach first. Guest callers are denied (a write).



# Duplicate a client
Source: https://docs.mcpjam.com/api-reference/clients/duplicate-a-client

/reference/openapi.json post /projects/{projectId}/clients/{client}/duplicate
Creates a new client carrying the selected client's current config, and responds `201` with the copy's detail. Purely additive — the source is untouched.



# Get a client
Source: https://docs.mcpjam.com/api-reference/clients/get-a-client

/reference/openapi.json get /projects/{projectId}/clients/{client}
One client's full settings, including its resolved config, its `configId` (the concurrency token every write takes), and `impact` — what a config edit would follow. The path segment accepts a client **name or ID**. A client that exists but belongs to a different project reads as `404 NOT_FOUND`.



# List a project's clients
Source: https://docs.mcpjam.com/api-reference/clients/list-a-projects-clients

/reference/openapi.json get /projects/{projectId}/clients
The clients saved in the project — the named, reusable configurations that define how MCPJam connects to and talks to your MCP servers. Returns the `id`s and `configId`s the detail and write routes take. Clients that exist only as the private backing of a User Testing scenario are omitted unless `includePrivateBacking=true`.



# Replace a client's servers
Source: https://docs.mcpjam.com/api-reference/clients/replace-a-clients-servers

/reference/openapi.json post /projects/{projectId}/clients/{client}/servers
Replaces the client's required and optional saved-server attachments without round-tripping the whole config. Composed server-side from the client's current config, so every other setting is preserved — and `expectedConfigId` is required so two concurrent server replacements cannot silently lose one.



# Update a client
Source: https://docs.mcpjam.com/api-reference/clients/update-a-client

/reference/openapi.json patch /projects/{projectId}/clients/{client}
Edit a client's display name and/or its config. Two ways to change the config: `config` replaces it wholesale, and `set` changes named fields over the config read inside the write transaction — prefer `set`, because a whole-config round-trip composed from a stale read reverts whatever landed in between.

Config edits require `expectedConfigId` and renames require `expectedName`. A stale token is `409 CONFLICT` whose `details` carries the current value: re-read the client and retry. A config edit that resolves to byte-identical settings writes nothing at all.

Guest callers are denied (a write).



# Fetch a bounded conformance report projection
Source: https://docs.mcpjam.com/api-reference/conformance-runs/fetch-a-bounded-conformance-report-projection

/reference/openapi.json get /projects/{projectId}/conformance-runs/{runId}/report
Failing checks only, failed before could-not-run, capped at 50. `totalCases` is every case; `totalFailingCases` is the failing population `truncated` is computed from. `pending` on a check means this profile reported it but did not score it. `404` when the run has no stored report.



# Finalize an incrementally ingested conformance run
Source: https://docs.mcpjam.com/api-reference/conformance-runs/finalize-an-incrementally-ingested-conformance-run

/reference/openapi.json post /projects/{projectId}/conformance-ingest/runs/finalize
How conformance runs executed OUTSIDE the platform (local CLI, GitHub Actions) reach the Conformance history. Authenticate like any other `/api/v1` route (typically an `sk_` key). The `{projectId}` segment always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project. STATUS AND BODY PASS THROUGH VERBATIM.

Recomputes the pooled outcome from uploaded cases. Idempotent.



# Get a persisted conformance run
Source: https://docs.mcpjam.com/api-reference/conformance-runs/get-a-persisted-conformance-run

/reference/openapi.json get /projects/{projectId}/conformance-runs/{runId}
Status, outcome, and score are three different answers. A run whose `projectId` is not the path's is `404`. Per-suite `profileId` / `pending` are present when the backend has stored them.



# Ingest a completed conformance run in one call
Source: https://docs.mcpjam.com/api-reference/conformance-runs/ingest-a-completed-conformance-run-in-one-call

/reference/openapi.json post /projects/{projectId}/conformance-ingest/report
How conformance runs executed OUTSIDE the platform (local CLI, GitHub Actions) reach the Conformance history. Authenticate like any other `/api/v1` route (typically an `sk_` key). The `{projectId}` segment always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project. STATUS AND BODY PASS THROUGH VERBATIM.

The whole-run variant: submit a finished composite report. This is what the SDK reporter uses by default.



# Keep a long-running uploaded conformance run alive
Source: https://docs.mcpjam.com/api-reference/conformance-runs/keep-a-long-running-uploaded-conformance-run-alive

/reference/openapi.json post /projects/{projectId}/conformance-ingest/runs/heartbeat
How conformance runs executed OUTSIDE the platform (local CLI, GitHub Actions) reach the Conformance history. Authenticate like any other `/api/v1` route (typically an `sk_` key). The `{projectId}` segment always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project. STATUS AND BODY PASS THROUGH VERBATIM.



# List persisted conformance runs
Source: https://docs.mcpjam.com/api-reference/conformance-runs/list-persisted-conformance-runs

/reference/openapi.json get /projects/{projectId}/conformance-runs
Newest first. Optionally narrow to one saved server. `pending` counts checks the run's profile reported but did not score.



# Open a conformance run for incremental ingestion
Source: https://docs.mcpjam.com/api-reference/conformance-runs/open-a-conformance-run-for-incremental-ingestion

/reference/openapi.json post /projects/{projectId}/conformance-ingest/runs/start
How conformance runs executed OUTSIDE the platform (local CLI, GitHub Actions) reach the Conformance history. Authenticate like any other `/api/v1` route (typically an `sk_` key). The `{projectId}` segment always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project. STATUS AND BODY PASS THROUGH VERBATIM.

Opens (or idempotently reuses) a run that suite reports are appended to.



# Start a persisted conformance run
Source: https://docs.mcpjam.com/api-reference/conformance-runs/start-a-persisted-conformance-run

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/conformance-runs
Grades the saved server against the protocol, apps, and tasks suites and returns `202` with a run id. The target comes from the saved server the path names: a body cannot supply a URL. OAuth is not startable here. Poll the run detail for the verdict. Status, outcome, and score are three different answers.



# Upload one suite report onto a started conformance run
Source: https://docs.mcpjam.com/api-reference/conformance-runs/upload-one-suite-report-onto-a-started-conformance-run

/reference/openapi.json post /projects/{projectId}/conformance-ingest/runs/reports
How conformance runs executed OUTSIDE the platform (local CLI, GitHub Actions) reach the Conformance history. Authenticate like any other `/api/v1` route (typically an `sk_` key). The `{projectId}` segment always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project. STATUS AND BODY PASS THROUGH VERBATIM.

One requested suite per call. Content-hash conflicts 409.



# Cancel an in-flight readiness run
Source: https://docs.mcpjam.com/api-reference/directory-readiness/cancel-an-in-flight-readiness-run

/reference/openapi.json post /projects/{projectId}/readiness-runs/{runId}/cancel
The executing node learns about this on its next heartbeat and aborts the run in flight, which matters more than the row's status: the thing being stopped is traffic to somebody else's server.



# Fetch a readiness run's full report
Source: https://docs.mcpjam.com/api-reference/directory-readiness/fetch-a-readiness-runs-full-report

/reference/openapi.json get /projects/{projectId}/readiness-runs/{runId}/report
Every finding, with its class, provenance, publisher-documentation citation and remediation. AI-derived findings carry `provenance` of `llm` and are confined to the experience lane as `heuristic` or `manual-review`; they can never decide a lane. `404` when the run has no stored report: it may be in flight, it may have failed, or its report may have aged past retention.



# List readiness runs
Source: https://docs.mcpjam.com/api-reference/directory-readiness/list-readiness-runs

/reference/openapi.json get /projects/{projectId}/readiness-runs



# Read one readiness run
Source: https://docs.mcpjam.com/api-reference/directory-readiness/read-one-readiness-run

/reference/openapi.json get /projects/{projectId}/readiness-runs/{runId}
Lane statuses, coverage and the observation axis. The full report — every finding with its citation and provenance — is a separate fetch.



# Start a Claude directory-readiness run
Source: https://docs.mcpjam.com/api-reference/directory-readiness/start-a-claude-directory-readiness-run

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/readiness-runs/claude
Grades the saved server against Anthropic's connector-directory requirements and returns `202` with a run id. The target comes from the saved server the path names: a body cannot supply a URL, an actor, an organization, a model or a cost. Poll the run detail for the verdict. Deterministic grading is free.



# Start an OpenAI plugin-directory readiness run
Source: https://docs.mcpjam.com/api-reference/directory-readiness/start-an-openai-plugin-directory-readiness-run

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/readiness-runs/openai
Grades the saved server against OpenAI's plugin-directory requirements and returns `202` with a run id. `submissionMode` is required and never inferred: inferring it would read a forgotten package as `mcp-only` and report the package lane not-applicable, turning a missing input into a clean bill of health. Hosted runs grade the two wire-safe shapes; the package shapes need an upload this API cannot receive and are available from the CLI.



# Archive an environment
Source: https://docs.mcpjam.com/api-reference/environments/archive-an-environment

/reference/openapi.json post /projects/{projectId}/environments/{environmentId}/archive
Archive an environment. It stops being selectable for runs and frees its name for a new one, but the row is kept and can be restored — this is why archive is a sub-action rather than a DELETE. Requires `expectedRevision` — the revision you last read. If the environment changed since, the write is rejected with 409 rather than overwriting the concurrent edit. Requires project admin.



# Check what this deployment's environment surface supports
Source: https://docs.mcpjam.com/api-reference/environments/check-what-this-deployments-environment-surface-supports

/reference/openapi.json get /projects/{projectId}/environments/capabilities
Report which environment features this MCPJam deployment accepts. Call it before sending a model override: clients ship independently of the platform, and a field an older deployment does not know is a hard validation error there rather than a silently ignored one. A deployment too old to answer reports `false` for everything, which is the correct assumption.



# Create an environment
Source: https://docs.mcpjam.com/api-reference/environments/create-an-environment

/reference/openapi.json post /projects/{projectId}/environments
Create a project environment from a host plus, optionally, a standalone server group, a pinned skill selection, and pinned plugin versions. The name must be unique among the project's live environments. Requires project admin.



# Get or create an unnamed environment for a composed stack
Source: https://docs.mcpjam.com/api-reference/environments/get-or-create-an-unnamed-environment-for-a-composed-stack

/reference/openapi.json post /projects/{projectId}/environments/ensure-adhoc
GET-OR-CREATE an UNNAMED, content-addressed environment for a composed stack — a host plus an optional server group, model override, computer image and pinned skills.

Distinct from `POST /environments`, which mints a NAMED row that lands in the project's environment list forever. A composed stack is a throwaway: the caller wants to run this exact combination, not to add a permanent entry someone else has to reason about.

Deduplicated by a server-side fingerprint of the stack, so the same stack always returns the same environment. `created` says which happened — the status is `200` either way, because get-or-create cannot honestly answer `201` on the dedupe path.

A deployment that predates ad-hoc environments answers `400` with `details.reason: "ADHOC_UNAVAILABLE"`.

Requires project membership (admin only when the stack pins plugin versions). Promote one to a named environment with `POST /environments/{environmentId}/name`.



# List a project's environments
Source: https://docs.mcpjam.com/api-reference/environments/list-a-projects-environments

/reference/openapi.json get /projects/{projectId}/environments
The project environments saved in the project. Archived environments are excluded unless `includeArchived=true` — you need that to find one to restore.



# List a project's server groups
Source: https://docs.mcpjam.com/api-reference/environments/list-a-projects-server-groups

/reference/openapi.json get /projects/{projectId}/server-groups
A server group is an immutable snapshot of a set of saved servers. Pinning one on an environment is what keeps a run off its host's live server list, so a composed run keeps testing the servers it was composed against even after the shared host is edited.



# Preview what an environment resolves to
Source: https://docs.mcpjam.com/api-reference/environments/preview-what-an-environment-resolves-to

/reference/openapi.json get /projects/{projectId}/environments/{environmentId}/resolve
Resolve an environment to the exact execution inputs a run would use right now: the host's current config, the closed server set (including servers contributed by pinned plugin versions), and the resolved plugin versions. Returns 409 when the environment cannot currently produce a runnable configuration — for example a pinned plugin was disabled or deleted; `details.code` carries the specific reason (`ENV_PLUGIN_UNAVAILABLE`, `ENV_NO_SERVERS`, `ENV_HOST_MISSING`, …).



# Promote an unnamed environment
Source: https://docs.mcpjam.com/api-reference/environments/promote-an-unnamed-environment

/reference/openapi.json post /projects/{projectId}/environments/{environmentId}/name
Give an UNNAMED (ad-hoc) environment a name, promoting it IN PLACE — the same environment, now a permanent entry in the project's list, with the same id every existing run still points at.

The ONLY promotion path. `PATCH /environments/{environmentId}` renames an already-named environment and is admin-gated; this is member-gated and refuses a row that already has a name (`409`). The platform keeps them apart on purpose: routing promotion through the rename would either open it to members or leave ad-hoc rows unnameable.

Promotion also drops the content fingerprint, because a named environment is mutable: keeping it would let a later identical composition deduplicate onto a row that has since been edited into something else.



# Restore an archived environment
Source: https://docs.mcpjam.com/api-reference/environments/restore-an-archived-environment

/reference/openapi.json post /projects/{projectId}/environments/{environmentId}/restore
Restore an archived environment. Returns 409 if another live environment took its name in the meantime. Plugin pins whose version row no longer exists at all are dropped on the way back to live — compare the returned `pluginVersionIds` against what you archived to detect that. Requires `expectedRevision` — the revision you last read. If the environment changed since, the write is rejected with 409 rather than overwriting the concurrent edit. Requires project admin.



# Show one environment
Source: https://docs.mcpjam.com/api-reference/environments/show-one-environment

/reference/openapi.json get /projects/{projectId}/environments/{environmentId}
Read one environment, including the `revision` you pass back as `expectedRevision` when writing to it.



# Snapshot servers into a server group
Source: https://docs.mcpjam.com/api-reference/environments/snapshot-servers-into-a-server-group

/reference/openapi.json post /projects/{projectId}/server-groups
Creates a server group holding exactly the given servers and responds `201` with its detail. Groups are create-only — there is no update route, because an editable group would reintroduce the drift that pinning exists to prevent. Names are unique per project, so a clash responds `409`. Servers must belong to this project and must not be plugin-managed.



# Update an environment
Source: https://docs.mcpjam.com/api-reference/environments/update-an-environment

/reference/openapi.json patch /projects/{projectId}/environments/{environmentId}
Edit an environment. Only the fields you send change; send `null` for `serverAttachmentId`, `skillSelection`, or `pluginVersionIds` to clear them. Requires `expectedRevision` — the revision you last read. If the environment changed since, the write is rejected with 409 rather than overwriting the concurrent edit. Requires project admin.



# Append iterations to an open run
Source: https://docs.mcpjam.com/api-reference/eval-runs/append-iterations-to-an-open-run

/reference/openapi.json post /projects/{projectId}/eval-ingest/runs/iterations
Appends one batch of iteration results to a run opened by `runs/start`.

How eval runs executed OUTSIDE the platform (local dev, CI) reach the Evals dashboard. Authenticate like any other `/api/v1` route (typically an `sk_` key); the gateway swaps in a delegated org-scoped token so the backend's fail-closed org scoping applies.

The `{projectId}` segment declares where results land and always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project — the zero-config CI case.

STATUS AND BODY PASS THROUGH VERBATIM: success shapes are the legacy `{ ok: true, ... }` envelopes the SDK reporter parses, not the v1 resource envelope.



# Cancel a run
Source: https://docs.mcpjam.com/api-reference/eval-runs/cancel-a-run

/reference/openapi.json post /projects/{projectId}/eval-runs/{runId}/cancel
Request cancellation of an in-flight run; marks the run and its pending/running iterations `cancelled`. A run held in `grading` for its gating judge CAN be cancelled — it lands `cancelled` with no verdict, which is the honest outcome for a run nobody waited out. A no-op success when the run is already cancelled; returns `409` when the run already reached a terminal status (`completed`/`failed`/`timed_out`).



# Close an open ingestion run
Source: https://docs.mcpjam.com/api-reference/eval-runs/close-an-open-ingestion-run

/reference/openapi.json post /projects/{projectId}/eval-ingest/runs/finalize
Marks the run complete and computes its rollups. Until this is called the run stays open in the dashboard.

How eval runs executed OUTSIDE the platform (local dev, CI) reach the Evals dashboard. Authenticate like any other `/api/v1` route (typically an `sk_` key); the gateway swaps in a delegated org-scoped token so the backend's fail-closed org scoping applies.

The `{projectId}` segment declares where results land and always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project — the zero-config CI case.

STATUS AND BODY PASS THROUGH VERBATIM: success shapes are the legacy `{ ok: true, ... }` envelopes the SDK reporter parses, not the v1 resource envelope.



# Compare a run against a baseline
Source: https://docs.mcpjam.com/api-reference/eval-runs/compare-a-run-against-a-baseline

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/compare
Compare this run against a baseline run: per-case status (`regressed`, `fixed`, `new_case`, `removed_case`, `changed`), per-scorer pass-rate and mean deltas from the evaluation contract, and whether the evaluation config changed. Omit `baseRunId` to compare against the nearest earlier **completed** run in the same suite. Returns `404` with `details.reason` = `BASELINE_NOT_FOUND` when there is no comparable predecessor — that is an incomplete comparison, not a failing one. A scorer whose `definitionChanged` is `true` was graded by a different definition on each side, so its delta is not a regression.



# Create an eval case
Source: https://docs.mcpjam.com/api-reference/eval-runs/create-an-eval-case

/reference/openapi.json post /projects/{projectId}/eval-suites/{suiteId}/cases
Adds one case to the suite. Both `title` and a non-empty `steps` array are required — `steps` is optional on the shared case shape so PATCH can be partial, but a case persisted without executable steps could never run. When `models` is omitted the suite's configured model is used.



# Create an eval run (async)
Source: https://docs.mcpjam.com/api-reference/eval-runs/create-an-eval-run-async

/reference/openapi.json post /projects/{projectId}/eval-runs
Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`.

A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`.

For a suite with attached project environments, pass `environmentId` to choose which one the run uses; the `202` echoes the resolved `environment` triple, and `GET /eval-runs/{runId}` reports the same triple for the life of the run.

To make the run readable as something more specific than `API` in the Runs table, send `x-mcpjam-launcher` (and `x-mcpjam-ci` from a CI job). Both are labels: `source` is stamped `api` either way.



# Create an eval suite (author-only, does not run)
Source: https://docs.mcpjam.com/api-reference/eval-runs/create-an-eval-suite-author-only-does-not-run

/reference/openapi.json post /projects/{projectId}/eval-suites
Creates a runnable eval suite — the suite record plus its test cases — and responds `201` **synchronously**, WITHOUT executing anything. Use this to author a suite, then run it later with `POST /eval-runs` (passing the returned `suiteId`).

This is distinct from `POST /eval-runs`, which creates a run and **detaches execution**, responding `202` with a `runId`. There is no concurrency cap here (no run is started).

The body uses an ergonomic authoring shape: a suite-level default `model` (and optional `provider`) applies to every test unless the test overrides it; `provider` is derived from a `provider/model` id when neither is supplied. Each test's case body is an ordered `steps` array (prompt / toolCall / interact / assert).

Guest callers are denied (suite creation is a write).



# Create several eval cases
Source: https://docs.mcpjam.com/api-reference/eval-runs/create-several-eval-cases

/reference/openapi.json post /projects/{projectId}/eval-suites/{suiteId}/cases/batch
Adds up to 100 cases to the suite in one call — the bulk form of the single create, and the way to import a suite or convert a repo's test files without a round trip per case. Every entry needs a `title` and a non-empty `steps` array. Cases are validated together and reported individually: a `failed` entry does not roll back its committed siblings, so the response is `201` even when some entries were refused — read `failed` rather than branching on the status alone. Send an `Idempotency-Key` header to make a retry land on the same rows.



# Delete an eval case
Source: https://docs.mcpjam.com/api-reference/eval-runs/delete-an-eval-case

/reference/openapi.json delete /projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}
Removes the case from the suite. Past runs that executed it are not affected.

A case in a CI-owned suite (see `EvalSuiteDetail.managedBy`) refuses this with `409`; send the suite file's `suite.id` as `declaredSuiteId` to delete as the file.



# Delete an eval suite
Source: https://docs.mcpjam.com/api-reference/eval-runs/delete-an-eval-suite

/reference/openapi.json delete /projects/{projectId}/eval-suites/{suiteId}
Deletes the suite and its cases. Past runs are not deleted.

A CI-owned suite (see `EvalSuiteDetail.managedBy`) refuses this with `409`; send its file's `suite.id` as `declaredSuiteId` to delete as the file.



# Enable, repoint, or disable a suite's schedule
Source: https://docs.mcpjam.com/api-reference/eval-runs/enable-repoint-or-disable-a-suites-schedule

/reference/openapi.json patch /projects/{projectId}/eval-suites/{suiteId}/schedule
A schedule fires exactly ONE run per interval, so an environment-based suite must pin exactly one of its attached environments. Omitting `environmentId` on a single-environment suite means that environment; omitting it on a multi-environment suite is a VALIDATION_ERROR. Disabling preserves the existing pin. Responds with the full updated suite.



# Generate eval cases from the suite's tools
Source: https://docs.mcpjam.com/api-reference/eval-runs/generate-eval-cases-from-the-suites-tools

/reference/openapi.json post /projects/{projectId}/eval-suites/{suiteId}/cases/generate
Discovers the suite's server tools over a live MCP connection, generates cases against them, and persists them — the only edit route that connects to a server, and the only one that SPENDS ORG CREDITS. Synchronous: connect, generate, persist, disconnect, respond.

An environment-based suite generates against that environment's closed server set, so the cases match the tools its runs will actually see.

Pass `x-mcpjam-idempotency-key` to make a retry safe: drafts are recorded backend-side BEFORE any case is persisted, so a replay reuses them instead of spending credits again, and each case is persisted under a derived per-item key so the loop is resumable.



# Get a run's decision summary
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-a-runs-decision-summary

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/decision-summary
The canonical, versioned run decision contract: the verdict and where it came from, the counts with the population they count, the run's own verdict decision when it has one, and one page of per-trial diagnostics carrying the user-value chain, the first failed stage, the failure category, evidence scoped to that stage, and one next action.

ADDITIVE and composed: it is the same two reads a caller could make by hand (`GET …/eval-runs/{runId}` and `GET …/iterations`), assembled once here so every client shares one reading of a run instead of inventing its own. `cursor` and `limit` page the DIAGNOSTICS using the same cursors the iterations endpoint issues, and `diagnostics.complete` says honestly whether the page you got is the run's whole non-passing set.



# Get a run's gate waiver
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-a-runs-gate-waiver

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/gate-waivers
The waiver in force over this run, or `null`. Available to anyone who can VIEW the run, not only to those who can grant one — a waiver its readers cannot see is not a visible waiver.

A client gating on this run does not need this endpoint: `GET /eval-runs/{runId}` already carries `gateWaiver`, so a gate can fold one in without a second round trip.



# Get an iteration trace
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-an-iteration-trace

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/iterations/{iterationId}/trace
Full trace envelope for one iteration: conversation messages, expected-vs-actual tool call analysis, and spans. The shape is rich and may evolve — treat it as an open document. Returns `404` with `details.reason: "TRACE_NOT_AVAILABLE"` when the iteration finished without a stored trace.



# Get an iteration's step results
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-an-iterations-step-results

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/iterations/{iterationId}/steps
One row per authored test step, in order, with `status` (`ok`/`fail`/`skipped`/`pending`), a `reason`, and any `evidence` (screenshots, replay-video offset, widget tool calls). The fastest way to see which step failed and why. Unlike `/trace`, a missing trace is not a `404` — step verdicts still return, just without evidence.



# Get an upload URL for a run artifact
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-an-upload-url-for-a-run-artifact

/reference/openapi.json post /projects/{projectId}/eval-ingest/artifacts/upload-url
Mints a short-lived URL for uploading a widget blob or other artifact referenced by an ingested run.

How eval runs executed OUTSIDE the platform (local dev, CI) reach the Evals dashboard. Authenticate like any other `/api/v1` route (typically an `sk_` key); the gateway swaps in a delegated org-scoped token so the backend's fail-closed org scoping applies.

The `{projectId}` segment declares where results land and always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project — the zero-config CI case.

STATUS AND BODY PASS THROUGH VERBATIM: success shapes are the legacy `{ ok: true, ... }` envelopes the SDK reporter parses, not the v1 resource envelope.



# Get one eval case
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-one-eval-case

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}
A case that belongs to this suite and project. A case in another suite is a 404, not a 403.



# Get one eval suite's configuration
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-one-eval-suites-configuration

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}
The suite's full settings — attached environments, hosts, execution config, judge, schedule. The list route returns run rollups instead; this is the shape the edit routes read and write.



# Get one run's stage analytics
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-one-runs-stage-analytics

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/stage-analytics
ONE run's materialized stage-analytics document: where its trials fell out of the six-stage user-value chain (connection, discovery, selection, tool call, response, user value), overall and sliced marginally by intent, model and host.

The same document the suite listing returns, addressed by run. Use this rather than paging `/eval-suites/{suiteId}/stage-analytics` to reach a known run: that listing is newest-first, so walking it costs work proportional to how long ago the run finished and cannot answer once the run falls outside the pages read.

Counts are returned; RATES are not. A zero denominator is `notMeasured`, never `0`, and every excluded observation is counted under a named class so a denominator can never shrink silently.

`404` covers BOTH "this run has no document" and "this run is not visible to you", and the two are deliberately not distinguished — separating them would confirm that a run exists in a project you cannot see. NOT backfilled either: a run that completed before this shipped has no document, and that absence means UNMEASURED, never a funnel of zeros.



# Get one run's suite quality-gate report
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-one-runs-suite-quality-gate-report

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/gate
ONE run's stored suite quality-gate report (`SuiteGateReportV1`): passed, failed, non_gateable, or not_configured.

`not_configured` is a 200 report — the suite has no active conditions — never a missing route. A 404 means the run is not visible. A deployment that predates the evaluator answers 501 FEATURE_NOT_SUPPORTED; that is a fact about the deployment, not proof the suite has no policy. A run waiver never covers this report.



# Get run status
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-run-status

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}
Run status, result, and summary. Poll until `status` is terminal (`completed`, `failed`, `cancelled` or `timed_out`). `grading` is not terminal — every trial has finished and the run is held for its gating judge, with `result` still `pending` — so keep polling.



# Get the pre-run disclosure for a suite launch plan
Source: https://docs.mcpjam.com/api-reference/eval-runs/get-the-pre-run-disclosure-for-a-suite-launch-plan

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/run-disclosure
What happens to a run's content BEFORE you launch it: which models it calls and where they route, which LLM analyzers/judges can fire and where their evidence goes, capture/retention/region facts, and the subprocessors engaged. Read-only — never launches or gates a run. Keyed by the SAME destination-affecting subset a launch selects (`caseIds`/`environmentId`/`environmentIds`); pass the selectors you would pass to a run so what this discloses is what that run would do. `run_eval_suite` already fetches this itself and returns it on the receipt's `disclosure` field — call this route directly when you need the disclosure BEFORE deciding to launch. A deployment that predates this contract answers 422 `FEATURE_NOT_SUPPORTED` with `details.reason: "contract_unavailable"`, never a partial payload — this is a GUARANTEE only when the deployment's missing-function error reaches this route unredacted. Production Convex can redact that same failure to a generic "Server Error" indistinguishable from a genuine handler crash on a suite the caller CAN see; this route has no independent way to tell the two apart in that one case, so it answers 502 instead of guessing 422 — a caller cannot rely on this code alone to detect an old production deployment, and a 502 does not imply the contract is available either. `projectId` is positional only, for REST consistency with sibling routes — this endpoint authorizes per-suite, not per-project, so any `projectId` returns the same disclosure for a `suiteId` you can reach.



# Ingest a completed eval run in one call
Source: https://docs.mcpjam.com/api-reference/eval-runs/ingest-a-completed-eval-run-in-one-call

/reference/openapi.json post /projects/{projectId}/eval-ingest/report
The whole-run variant: submit a finished run and its iterations in a single request. This is what the SDK reporter uses by default.

How eval runs executed OUTSIDE the platform (local dev, CI) reach the Evals dashboard. Authenticate like any other `/api/v1` route (typically an `sk_` key); the gateway swaps in a delegated org-scoped token so the backend's fail-closed org scoping applies.

The `{projectId}` segment declares where results land and always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project — the zero-config CI case.

STATUS AND BODY PASS THROUGH VERBATIM: success shapes are the legacy `{ ok: true, ... }` envelopes the SDK reporter parses, not the v1 resource envelope.



# Launch one run per target as a group (async)
Source: https://docs.mcpjam.com/api-reference/eval-runs/launch-one-run-per-target-as-a-group-async

/reference/openapi.json post /projects/{projectId}/eval-run-groups
Launches ONE run per target — attached project environments, or attached named hosts — under a single **server-minted** `runGroupId`, then detaches execution and responds `202` with a per-target receipt.

This is the only endpoint with grouped-launch semantics. `POST /eval-runs` also accepts a `runGroupId`, but purely as a display label: it groups sibling rows in the UI and gives N separate launches no group treatment. Here the server bounds the fan-out (max 10 targets), validates EVERY target before launching any of them (an unattached target, a heterogeneous target list, or a target whose host selects an unavailable harness returns `400` with zero runs started), and holds exactly ONE organization concurrency slot for the whole group — released only when the last sibling finishes.

Targets fan out along ONE axis: all `environmentId` or all `namedHostId`, never a mix. Duplicates are deduplicated by id, preserving order.

A per-target failure does NOT abort its siblings, so read `outcome` rather than treating the `202` as "everything started": `started` means every target launched, `partial` means some did, `failed` means none did.

Send `Idempotency-Key` (or `idempotencyKey`) to make the launch replayable: the group id is DERIVED from the key, and each target carries a run key derived from that, so a retry after a crash mid-launch returns the original run ids instead of double-launching the targets that already started.

`x-mcpjam-launcher` and `x-mcpjam-ci` are read once and applied to EVERY sibling: one fan-out is one launch by one process, so the targets badge identically.



# List a suite's eval cases
Source: https://docs.mcpjam.com/api-reference/eval-runs/list-a-suites-eval-cases

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/cases
Every case in the suite, in the public steps-first shape.



# List a suite's per-run stage analytics
Source: https://docs.mcpjam.com/api-reference/eval-runs/list-a-suites-per-run-stage-analytics

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/stage-analytics
One materialized stage-analytics document per RUN, newest completion first: where trials fell out of the six-stage user-value chain (connection, discovery, selection, tool call, response, user value), overall and sliced marginally by intent, model and host.

Each item is one run's COMPLETE document and stands alone. There is deliberately no cross-run aggregate — two funnels averaged together describe no run — so compare runs by rendering them side by side, never by summing these documents.

Counts are returned; RATES are not. A zero denominator is `notMeasured`, never `0`, and every excluded observation is counted under a named class so a denominator can never shrink silently.

NOT backfilled: a run that completed before this shipped has no document at all, and that absence means UNMEASURED — it is never a funnel of zeros.



# List a suite's runs
Source: https://docs.mcpjam.com/api-reference/eval-runs/list-a-suites-runs

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/runs
Recent runs for a suite, newest first.



# List run iterations
Source: https://docs.mcpjam.com/api-reference/eval-runs/list-run-iterations

/reference/openapi.json get /projects/{projectId}/eval-runs/{runId}/iterations
Per-iteration results: actual tool calls, structured token usage, and latency. Cursor-paginated.



# Open a run for incremental ingestion
Source: https://docs.mcpjam.com/api-reference/eval-runs/open-a-run-for-incremental-ingestion

/reference/openapi.json post /projects/{projectId}/eval-ingest/runs/start
Opens a run that iterations are appended to, for a long CI job that reports as it goes. Pair with `runs/iterations` and `runs/finalize`.

How eval runs executed OUTSIDE the platform (local dev, CI) reach the Evals dashboard. Authenticate like any other `/api/v1` route (typically an `sk_` key); the gateway swaps in a delegated org-scoped token so the backend's fail-closed org scoping applies.

The `{projectId}` segment declares where results land and always wins over any `projectId` in the body. The literal `default` resolves to the key org's Default project — the zero-config CI case.

STATUS AND BODY PASS THROUGH VERBATIM: success shapes are the legacy `{ ok: true, ... }` envelopes the SDK reporter parses, not the v1 resource envelope.



# Resolve or create a file-owned eval suite
Source: https://docs.mcpjam.com/api-reference/eval-runs/resolve-or-create-a-file-owned-eval-suite

/reference/openapi.json post /projects/{projectId}/eval-suites/from-file
Resolve a file-owned suite by declared id within the project, or create one. Lookup is by `(projectId, declaredSuiteId)` and never by name. A UI-authored suite has no declared id and cannot be claimed. The inspector parses the suite file; this body is the declared identity, source hash, optional provenance, and hosted settings — not the raw file.

`201` on first create; `200` on a later update of the same declared id.



# Revoke a gate waiver
Source: https://docs.mcpjam.com/api-reference/eval-runs/revoke-a-gate-waiver

/reference/openapi.json delete /projects/{projectId}/eval-runs/{runId}/gate-waivers/{waiverId}
End a waiver early, putting the gate and the GitHub Check Run back. Requires the manage tier.

IDEMPOTENT, and `already_revoked` is a SUCCESS: it reports the ORIGINAL revocation rather than restamping it, so a retry cannot overwrite the record of who actually ended the waiver. An already-expired waiver may still be revoked — the audit trail distinguishes "this was wrong" from "this ran out".



# Update an eval case
Source: https://docs.mcpjam.com/api-reference/eval-runs/update-an-eval-case

/reference/openapi.json patch /projects/{projectId}/eval-suites/{suiteId}/cases/{caseId}
Partial update — omitted fields are left as they are. Supplying `steps` REPLACES the case's definition rather than merging into it. `matchOptions` and `checks` accept `null` to clear the case-level override and fall back to the suite's.



# Update an eval suite's settings
Source: https://docs.mcpjam.com/api-reference/eval-runs/update-an-eval-suites-settings

/reference/openapi.json patch /projects/{projectId}/eval-suites/{suiteId}
Partial update — omitted fields are left as they are. `environmentIds` is tri-state: a non-empty array sets/replaces, `null` clears (reverting the suite to legacy config), and `[]` is rejected rather than silently treated as a clear. Responds with the full updated suite.



# Waive a run gate
Source: https://docs.mcpjam.com/api-reference/eval-runs/waive-a-run-gate

/reference/openapi.json post /projects/{projectId}/eval-runs/{runId}/gate-waivers
Override a FAILING run's release gate until an expiry you name. This does NOT make the run pass: the run keeps its `result`, the waiver is a separate audited record, and every surface that honors it — the GitHub Check Run and the CLI's `eval gate` — reports the gate as waived, by whom, why, and until when.

Requires the manage tier; whoever launched the run gets no exception for having launched it. `reason` is stored UNREDACTED and readable by anyone who can see the suite, for as long as the suite exists.

`400` covers five distinct refusals, each with a message written for the caller: the suite belongs to no organization, the reason is blank, the reason exceeds 500 characters, the expiry is not in the future, or the expiry is more than 30 days out. `403` means the caller can see the suite but lacks the manage tier — deliberately not collapsed to `404`, which would send a legitimate member hunting for a run sitting in front of them.



# Attach one environment to a suite (atomic append)
Source: https://docs.mcpjam.com/api-reference/eval-suites/attach-one-environment-to-a-suite-atomic-append

/reference/openapi.json post /projects/{projectId}/eval-suites/{suiteId}/environments
APPEND one project environment to the suite's attachments, if it is not already there.

Distinct from `PATCH /eval-suites/{suiteId}` with `environmentIds`, which REPLACES the whole list. An append built on the replace door is a read-modify-write across two round trips, and a concurrent attach landing in between is silently DETACHED. The compose-and-run path attaches on every launch, which makes that race ordinary rather than theoretical, so the append happens inside one transaction instead.

Idempotent: attaching an already-attached environment reports `attached: false` and changes nothing, which is what lets a retried launch converge instead of erroring. Subject to the same cap, membership and schedule-pin rules as the replace door — growing a single-environment suite to two is exactly the edit that strands an enabled unpinned schedule.



# List a suite's settings history
Source: https://docs.mcpjam.com/api-reference/eval-suites/list-a-suites-settings-history

/reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId}/revisions
One entry per committed edit to the suite's settings, newest first: who made it, which STORED fields moved, the note they left, how many runs were launched against it, and the revision group that ties one request's writes together. Rows carry no configuration snapshots. Send a `revisionNumber` back as `expectedRevisionNumber` on `PATCH /eval-suites/{suiteId}` to make that edit a compare-and-set.



# Grade an eval run with LLM as Judge
Source: https://docs.mcpjam.com/api-reference/evals/grade-an-eval-run-with-llm-as-judge

/reference/openapi.json post /projects/{projectId}/eval-runs/{runId}/judge
**Spends.** Runs the goal-completion judge over the finished run, scoring each case's final answer against its expected output.

`202`: scheduled, not done. Read the grades from the run detail's `judges.goalCompletion` rather than re-requesting — a second POST only spends again.

A run's grading config is pinned when the run is created, so turning the judge on for the suite does not reach an already-recorded run: `enable: true` is what grades one, and it changes nothing beyond that run. Omitting `model` and `threshold` clears any override a previous request left on the run.



# List GitHub Checks repositories
Source: https://docs.mcpjam.com/api-reference/evals/list-github-checks-repositories

/reference/openapi.json get /organizations/{organizationId}/eval-check-repos
The repositories in this organization whose pull requests run an eval suite, plus the repositories the MCPJam GitHub App can reach (the choices a connect has).

`available: false` means GitHub Checks is not enabled for this organization at all — connecting a repository will not help. `connectable: null` means the App could not be asked (GitHub unavailable, or no installation), which is different from an empty list meaning it reaches nothing; the already-connected list needs no GitHub call and is returned either way.



# Request insights for an eval run
Source: https://docs.mcpjam.com/api-reference/evals/request-insights-for-an-eval-run

/reference/openapi.json post /projects/{projectId}/eval-runs/{runId}/insights
**Spends.** Runs a model over the finished run and draws on the same shared `insightsPerDay` ledger as swarm and user-testing insights.

`202`: scheduled, not done. Poll the run detail's `insights` envelope rather than re-requesting — the envelope's `status` distinguishes `pending` from `not_requested`, which a second POST would not.



# Run an eval suite on a repository's pull requests
Source: https://docs.mcpjam.com/api-reference/evals/run-an-eval-suite-on-a-repositorys-pull-requests

/reference/openapi.json post /organizations/{organizationId}/eval-check-repos
Connect a repository so every pull request to it runs one eval suite and reports a GitHub check. **Affects everyone who opens a pull request on that repository**, and with `outagePolicy: fail_closed` it can block their merges.

The repository is verified against the organization's GitHub App installation before it is stored. A repository that does not exist and one the App cannot see answer identically, on purpose: answering differently would make this an oracle for private repository names.

Deliberately narrow: retargeting a repository at a different suite, pausing it, and disconnecting are repo-level decisions and live in the app's Settings → Integrations, where every connected repository is visible at once.



# Call a tool
Source: https://docs.mcpjam.com/api-reference/execution/call-a-tool

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/tools/call
Executes a tool on the server and returns the MCP `CallToolResult` plus an additive `durationMs` (wall-clock time of `executeTool`). Tool-level failures (`isError: true` in the result) are **successful calls** — the server answered; only transport/auth errors use the error envelope.



# Render a prompt
Source: https://docs.mcpjam.com/api-reference/execution/render-a-prompt

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/prompts/get
Renders a prompt with arguments and returns the MCP `GetPromptResult` directly (`description?`, `messages`).



# Render an MCP App widget headlessly
Source: https://docs.mcpjam.com/api-reference/execution/render-an-mcp-app-widget-headlessly

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/widgets/render
Call an MCP App tool and mount its `ui://` widget in real headless Chromium running the production host bridge, then report whether it rendered, what it logged, what it was blocked from fetching, and the widget as an accessibility tree with addressable elements.

**Executes the tool**, so it has whatever side effects that tool has — the render is what happens afterwards.

Stateless: connect, call, render, read, dispose, all inside one request. Interactive widget sessions are a local-Inspector capability (`/api/mcp/widget-session`) because they hold a live browser in process, which a no-affinity hosted plane cannot serve.

Each render launches a browser, so the endpoint carries a per-replica concurrency cap and answers 429 when it is full.



# Export a server snapshot
Source: https://docs.mcpjam.com/api-reference/export/export-a-server-snapshot

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/export
Lists tools, resources, and prompts in one call and returns a single JSON snapshot — handy for diffing a server's surface over time in CI.



# List a harness's native built-in tools
Source: https://docs.mcpjam.com/api-reference/harness/list-a-harnesss-native-built-in-tools

/reference/openapi.json get /harness/{harnessId}/builtin-tools
The native tools an agent harness (e.g. `claude-code`) runs INSIDE its sandbox — Bash, Read, Edit, Glob, Grep, WebSearch, and the like. Display-only: these execute via the harness's own agent loop and are NOT callable through MCPJam. Static published-package metadata; no project scope.



# Import OAuth tokens
Source: https://docs.mcpjam.com/api-reference/oauth/import-oauth-tokens

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/oauth/import-tokens
Stores OAuth tokens you obtained yourself (e.g. via the SDK's `runOAuthLogin` — interactive loopback, headless, or client-credentials) for this server, scoped to your user, project, and server. Subsequent API calls against the server inject the stored access token automatically, and `401`s from the server trigger a server-side refresh — no further caller involvement.

This closes the loop after an `OAUTH_REQUIRED` error.



# Get a plugin version
Source: https://docs.mcpjam.com/api-reference/plugins/get-a-plugin-version

/reference/openapi.json get /plugin-versions/{pluginVersionId}
One imported plugin version with its component projections: declared MCP servers (placement, auth timing, materialized server id) and declared skills (namespaced model refs). Addressed by the version id alone — access is membership of the version's own project, and historical versions of uninstalled plugins stay readable because eval snapshots and stale environment pins reference them.



# List a project's plugins
Source: https://docs.mcpjam.com/api-reference/plugins/list-a-projects-plugins

/reference/openapi.json get /projects/{projectId}/plugins
The live (installed, non-uninstalled) Agent Plugins in the project, disabled ones included (marked `enabled: false`). Read-only: import, activation, enable/disable and uninstall are app flows, not API operations.



# Get a server skill
Source: https://docs.mcpjam.com/api-reference/primitives/get-a-server-skill

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/skills/get
Fetches one skill by URI and verifies it before returning a byte: the SKILL.md digest against the manifest, the fetched frontmatter against what the listing advertised, and the name against the URI. Reaches skills a partial listing never mentioned. When a check fails the response carries a `refusal` naming it, not an error.



# List prompts
Source: https://docs.mcpjam.com/api-reference/primitives/list-prompts

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/prompts
Returns the server's prompts as a collection page.



# List resources
Source: https://docs.mcpjam.com/api-reference/primitives/list-resources

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/resources
Returns the server's resources as a collection page.



# List server skills
Source: https://docs.mcpjam.com/api-reference/primitives/list-server-skills

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/skills
Returns the Agent Skills a connected server serves over the MCP skills extension (SEP-2640), including skills it advertises that MCPJam declines to load. Not paginated: the catalog is drained server-side, because duplicate detection spans the whole listing.



# List tools
Source: https://docs.mcpjam.com/api-reference/primitives/list-tools

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/tools
Returns the server's tools as a collection page. Each item is the MCP tool definition (`name`, `description`, `inputSchema`, ...).



# Read a resource
Source: https://docs.mcpjam.com/api-reference/primitives/read-a-resource

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/resources/read
Reads a single resource by URI and returns its contents.



# Read a server skill's supporting file
Source: https://docs.mcpjam.com/api-reference/primitives/read-a-server-skills-supporting-file

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/skills/read-file
Reads one supporting file belonging to a skill, checking its byte length and digest against the skill's own manifest. The manifest is the read allowlist and is re-fetched here rather than accepted from the caller, so a request cannot authorize its own read.



# Create a project
Source: https://docs.mcpjam.com/api-reference/projects/create-a-project

/reference/openapi.json post /projects
Creates a project in the caller's organization and responds `201` with its detail. Requires organization membership and counts against the plan's `maxProjects` limit. Guest callers are denied.



# Delete a project and everything scoped to it
Source: https://docs.mcpjam.com/api-reference/projects/delete-a-project-and-everything-scoped-to-it

/reference/openapi.json delete /projects/{projectId}
IRREVERSIBLE, and it cascades: project access grants and invites, the mapped workspace and its memberships, every saved server in the project (each soft-deleted asynchronously, which also rotates the host configs that referenced it), and hosted tasks. Requires project admin. The request body must be empty.



# Update project metadata
Source: https://docs.mcpjam.com/api-reference/projects/update-project-metadata

/reference/openapi.json patch /projects/{projectId}
Updates name, description, icon or visibility. Requires project admin; a non-member gets `404` rather than `403`, so the response never confirms that a project exists. The project's server map is not updatable here — see the per-server routes.



# What the caller may do here
Source: https://docs.mcpjam.com/api-reference/projects/what-the-caller-may-do-here

/reference/openapi.json get /projects/{projectId}/capabilities
Role, gate state, plan limits and a set of derived booleans — so an agent on a static surface can check before it acts instead of attempting a write and reading the failure.

Descriptive, never authoritative: see the schema.



# Get one directory server
Source: https://docs.mcpjam.com/api-reference/registry/get-one-directory-server

/reference/openapi.json get /registry/directory-servers/{idOrName}
Resolve by Convex `catalogServerId` (long unhyphenated id) or by `serverName`. Pass `source` to disambiguate a name that exists in more than one directory. The response includes `latestContentHash` — pass it as `expectedContentHash` on install to freeze the row.



# Install a curated or org registry card
Source: https://docs.mcpjam.com/api-reference/registry/install-a-curated-or-org-registry-card

/reference/openapi.json post /projects/{projectId}/registry/installs
Same install-not-connect contract as the directory install. Pass `expectedUpdatedAt` from `listRegistryServers` to freeze the card.



# Install a directory server into a project
Source: https://docs.mcpjam.com/api-reference/registry/install-a-directory-server-into-a-project

/reference/openapi.json post /projects/{projectId}/registry/directory-installs
Writes a `servers` row and catalog provenance. SUCCESS MEANS INSTALLED, NOT CONNECTED — follow with `get_project_server_connection_status` and, for OAuth, the browser connect-link handoff. Pass `expectedContentHash` from `GET /registry/directory-servers/{id}` to reject a row that moved (TOCTOU). There is no catalog-uninstall route: delete the project server.



# List curated and org registry cards
Source: https://docs.mcpjam.com/api-reference/registry/list-curated-and-org-registry-cards

/reference/openapi.json get /projects/{projectId}/registry/servers
Merges global approved cards with the project's organization cards. Guest-denied. `scope` defaults to all.



# List directory sources
Source: https://docs.mcpjam.com/api-reference/registry/list-directory-sources

/reference/openapi.json get /registry/directory-sources
Discovery for the free-string `source` filter. Sources are data, not an enum — do not hardcode `claude`/`chatgpt` client-side.



# List registry installs in a project
Source: https://docs.mcpjam.com/api-reference/registry/list-registry-installs-in-a-project

/reference/openapi.json get /projects/{projectId}/registry/connections
Merged catalog + card provenance. Guest-denied. Rows whose `servers` row is deleted are filtered out.



# Search the scraped MCP directories
Source: https://docs.mcpjam.com/api-reference/registry/search-the-scraped-mcp-directories

/reference/openapi.json get /registry/directory-servers
Cross-source search over Claude, ChatGPT, and any future directory source. Omit `source` (or pass `all`) to search every source. The `cursor` is filter-bound. Prefer a matching curated card from `listRegistryServers` (global scope) when one exists — those carry vetted OAuth config. Bearer always required (guest tokens allowed). Not in PUBLIC_OPERATIONS.



# Uninstall a registry card
Source: https://docs.mcpjam.com/api-reference/registry/uninstall-a-registry-card

/reference/openapi.json delete /projects/{projectId}/registry/installs/{registryServerId}
Card-only. Directory uninstall is `DELETE /projects/{projectId}/servers/{serverId}` — provenance is cleaned with the server.



# Boot your computer from a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/boot-your-computer-from-a-sandbox-image

/reference/openapi.json post /projects/{projectId}/images/{imageId}/use
Attach the sandbox image to the caller's computer, which re-provisions it from the pinned image (installed files are wiped). The image must have a `ready` build, and the computer provider must match the build's. Bodyless. Guest callers are denied (a write).



# Build a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/build-a-sandbox-image

/reference/openapi.json post /projects/{projectId}/images/{imageId}/build
Trigger a build of the sandbox image's image and respond `202`. The build runs asynchronously — poll the builds list for status. Bodyless. Guest callers are denied (a write).



# Create a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/create-a-sandbox-image

/reference/openapi.json post /projects/{projectId}/images
Create a personal-draft sandbox image from a blueprint and respond `201` with its detail. Build it (`POST .../build`) before a computer can boot from it. Guest callers are denied (a write).



# Delete a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/delete-a-sandbox-image

/reference/openapi.json delete /projects/{projectId}/images/{imageId}
Permanently delete a sandbox image. Computers booted from it fall back to the base image. Deleting a project-shared image requires project admin. Bodyless — any field is rejected. Guest callers are denied (a write).



# Get a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/get-a-sandbox-image

/reference/openapi.json get /projects/{projectId}/images/{imageId}
One sandbox image's blueprint, sharing, and latest build. An image that exists but belongs to a different project reads as `404 NOT_FOUND`.



# List a project's sandbox images
Source: https://docs.mcpjam.com/api-reference/sandbox-images/list-a-projects-sandbox-images

/reference/openapi.json get /projects/{projectId}/images
The custom Computer images (digest-pinned Dockerfiles built into immutable images) saved in the project — your own personal drafts plus the project-shared ones.



# List a sandbox image's builds
Source: https://docs.mcpjam.com/api-reference/sandbox-images/list-a-sandbox-images-builds

/reference/openapi.json get /projects/{projectId}/images/{imageId}/builds
A sandbox image's builds, newest first, each with its status and capped log preview.



# Reset your computer to its image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/reset-your-computer-to-its-image

/reference/openapi.json post /projects/{projectId}/computer/reset
Reset the caller's computer back to its current image, wiping mutable state. Bodyless. Guest callers are denied (a write).



# Share a sandbox image with the project
Source: https://docs.mcpjam.com/api-reference/sandbox-images/share-a-sandbox-image-with-the-project

/reference/openapi.json post /projects/{projectId}/images/{imageId}/promote
Promote a personal-draft sandbox image to a project-shared one. Requires project admin. Bodyless. Guest callers are denied (a write).



# Update a sandbox image
Source: https://docs.mcpjam.com/api-reference/sandbox-images/update-a-sandbox-image

/reference/openapi.json patch /projects/{projectId}/images/{imageId}
Edit a sandbox image's name and/or Dockerfile. Re-build it for the changes to take effect on a computer. Guest callers are denied (a write).



# Validate a blueprint
Source: https://docs.mcpjam.com/api-reference/sandbox-images/validate-a-blueprint

/reference/openapi.json post /projects/{projectId}/images/validate
Lint blueprint YAML without saving it. Returns 200 with `ok: false` + structured errors for an invalid blueprint; the authoritative rejection still happens at create/update/build.



# Get a scenario
Source: https://docs.mcpjam.com/api-reference/scenarios/get-a-scenario

/reference/openapi.json get /projects/{projectId}/scenarios/{scenarioId}
One scenario's full read-only settings: model, system prompt, tool-approval policy, attached servers, and share link. A scenario that exists but belongs to a different project reads as `404 NOT_FOUND`.



# List a project's scenarios
Source: https://docs.mcpjam.com/api-reference/scenarios/list-a-projects-scenarios

/reference/openapi.json get /projects/{projectId}/scenarios
The scenarios published from this project — name, access mode, attached servers, and share link. Read-only.



# Create a secret
Source: https://docs.mcpjam.com/api-reference/secrets/create-a-secret

/reference/openapi.json post /projects/{projectId}/secrets
Store a credential so environments can grant it to runs. THE VALUE TRAVELS IN THE REQUEST BODY and becomes visible to whatever makes the call. The response is metadata only. Creating a project-shared secret requires project admin; a personal one does not.



# Delete a secret
Source: https://docs.mcpjam.com/api-reference/secrets/delete-a-secret

/reference/openapi.json delete /projects/{projectId}/secrets/{secretId}
Delete a stored credential. HARD — the row and the encrypted value both go, and MCPJam stops delivering it. Deliberately NOT blocked when an environment still selects it: refusing would leave a leaked credential undeletable until someone edited every environment naming it, and deletion must never wait on cleanup. Runs already in flight keep the value they were handed, and this does NOT revoke the credential at the provider that issued it.



# Get a secret
Source: https://docs.mcpjam.com/api-reference/secrets/get-a-secret

/reference/openapi.json get /projects/{projectId}/secrets/{secretId}
One secret's metadata: delivery mode, host binding, sharing, and when it was last handed to a run. Never its value. A secret from another project — and another member's personal secret — both read as 404.



# List secrets
Source: https://docs.mcpjam.com/api-reference/secrets/list-secrets

/reference/openapi.json get /projects/{projectId}/secrets
The project's credentials as METADATA ONLY — no value is returned by this or any other route. Shows the project-shared secrets plus the caller's own personal ones; another member's personal secret does not appear at all, not even its name.



# Rotate or re-bind a secret
Source: https://docs.mcpjam.com/api-reference/secrets/rotate-or-re-bind-a-secret

/reference/openapi.json patch /projects/{projectId}/secrets/{secretId}
Rotate the value and/or change how it is delivered. A rotation reaches NEW RUNS ONLY: a session already running holds the old value — materialized in its box's environment, or inside an egress policy that cannot be read back — and there is no safe way to replace it mid-run.



# Cancel a connection request
Source: https://docs.mcpjam.com/api-reference/server-connections/cancel-a-connection-request

/reference/openapi.json post /server-connections/{requestId}/cancel
Stops a request and invalidates its handoff link. Cancelling an already-terminal request is a no-op.



# Connect an MCP server
Source: https://docs.mcpjam.com/api-reference/server-connections/connect-an-mcp-server

/reference/openapi.json post /server-connections
Starts a connection request for an MCP server URL. Discovery, browser consent, and validation happen asynchronously; poll the returned `connectionRequestId` until the status is terminal. When the next step belongs to a person, the response carries a `handoffUrl` to open.



# Get a connection request
Source: https://docs.mcpjam.com/api-reference/server-connections/get-a-connection-request

/reference/openapi.json get /server-connections/{requestId}
Current state of one connection request. Metered on its own poll budget rather than the shared per-caller one, so polling on a short interval is expected and will not spend the budget your other calls need. A 429 here means the poll interval itself is too fast; honour Retry-After.



# Retry validation
Source: https://docs.mcpjam.com/api-reference/server-connections/retry-validation

/reference/openapi.json post /server-connections/{requestId}/retry-validation
Re-arms a stalled validation immediately instead of waiting out its backoff. Does not revive a terminal request — after `failed`, `expired`, or `cancelled`, start a new one.



# Check OAuth requirement
Source: https://docs.mcpjam.com/api-reference/server-diagnostics/check-oauth-requirement

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/check-oauth
Lightweight, no MCP connection: reports whether the server is configured to require an OAuth grant.



# Run the doctor
Source: https://docs.mcpjam.com/api-reference/server-diagnostics/run-the-doctor

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/doctor
Runs the full doctor workflow — probe → connect → initialize → capabilities → primitives — and returns a step-by-step report. The richest signal for "is this server healthy, and why not."



# Validate a server
Source: https://docs.mcpjam.com/api-reference/server-diagnostics/validate-a-server

/reference/openapi.json post /projects/{projectId}/servers/{serverId}/validate
Connects to the server, initializes the MCP session, and returns a connection snapshot (server info, negotiated capabilities). The same check the hosted inspector runs when you connect a server.



# Delete a saved server
Source: https://docs.mcpjam.com/api-reference/servers/delete-a-saved-server

/reference/openapi.json delete /projects/{projectId}/servers/{serverId}
Soft-deletes the server and schedules its cascade: MCP Apps views, hosted OAuth credentials, encrypted secret objects, and a rotation of every host config that referenced it. The cascade is ASYNCHRONOUS, so a read immediately after this call may still observe the row. The request body must be empty.



# Get one saved server
Source: https://docs.mcpjam.com/api-reference/servers/get-one-saved-server

/reference/openapi.json get /projects/{projectId}/servers/{serverId}
The server's detail, projected toward the hosted shape. Secrets are never included — `hasClientSecret` reports presence only. A server id from another project reads as `404`.



# Save a server into a project
Source: https://docs.mcpjam.com/api-reference/servers/save-a-server-into-a-project

/reference/openapi.json post /projects/{projectId}/servers
Creates a saved MCP server and responds `201` with its detail. This is the provisioning entry point: every eval, host and environment route addresses servers that were created here. Names are unique per workspace, so a clash responds `409` rather than silently returning the existing row. Secret-bearing fields (`env`, `headers`, `clientSecret`) are encrypted at rest and never returned by any read.



# Update a saved server
Source: https://docs.mcpjam.com/api-reference/servers/update-a-saved-server

/reference/openapi.json patch /projects/{projectId}/servers/{serverId}
Sparse update: omitted fields are unchanged. Renaming re-checks per-workspace uniqueness (`409` on a clash). Editing a server shared as an OAuth connection requires project admin. `clientSecret` and `clearClientSecret` are mutually exclusive.



# Get a skill
Source: https://docs.mcpjam.com/api-reference/skills/get-a-skill

/reference/openapi.json get /projects/{projectId}/skills/{skillId}
One skill, including its SKILL.md body. The body is mutable and an edit overwrites the previous one in place, so `aggregateHash` is the only handle on which content this read returned.



# List a project's skills
Source: https://docs.mcpjam.com/api-reference/skills/list-a-projects-skills

/reference/openapi.json get /projects/{projectId}/skills
The Cloud Skills visible to the caller in this project: the project-shared ones plus the caller's own drafts. Each row reports `pinnability`, which is what decides whether its id is usable in an environment's `skillSelection`.



# Cancel a pending wave-insights generation
Source: https://docs.mcpjam.com/api-reference/swarm-insights/cancel-a-pending-wave-insights-generation

/reference/openapi.json delete /projects/{projectId}/waves/{waveId}/insights
Stops a generation that is still `pending`. A wave with no generation in progress answers 409 — there is nothing to cancel, and reporting success would tell a caller it stopped something it did not.



# Dismiss a finding
Source: https://docs.mcpjam.com/api-reference/swarm-insights/dismiss-a-finding

/reference/openapi.json post /projects/{projectId}/journey-findings/{findingId}/dismiss
Hides a finding from your view. ORTHOGONAL to `status`: dismissing does not claim the problem stopped happening, and a dismissed finding that fires again stays `recurring`.

Addressed entirely by the path `findingId` — there is no body.



# Get a run's rubric scorecard
Source: https://docs.mcpjam.com/api-reference/swarm-insights/get-a-runs-rubric-scorecard

/reference/openapi.json get /projects/{projectId}/journey-runs/{runId}/scorecard
The deterministic rubric result for one run. No model, no spend.

Divide by `sessionsGraded`, never `sessionsTotal`, and keep `failedGradingCount` separate from `failCount` — a crashed judge is not a regression.



# Get wave insights
Source: https://docs.mcpjam.com/api-reference/swarm-insights/get-wave-insights

/reference/openapi.json get /projects/{projectId}/waves/{waveId}/insights
Poll this after requesting insights. **404 means nobody asked** — it is deliberately not an empty `status: "none"` body, so a caller in a polling loop cannot mistake "never requested" for "asked and still working".



# List swarm findings
Source: https://docs.mcpjam.com/api-reference/swarm-insights/list-swarm-findings

/reference/openapi.json get /projects/{projectId}/journey-findings
Criteria that keep failing, tracked across waves. One finding is one problem OVER TIME — `occurrenceCount` and `status` carry the history, so do not treat repeat appearances as separate problems.



# Project swarm roll-up
Source: https://docs.mcpjam.com/api-reference/swarm-insights/project-swarm-roll-up

/reference/openapi.json get /projects/{projectId}/journeys-overview
Recent runs with their goal-completion rates and repeat-failure findings, plus a project-wide trend. Deterministic and free — start here.



# Request wave insights
Source: https://docs.mcpjam.com/api-reference/swarm-insights/request-wave-insights

/reference/openapi.json post /projects/{projectId}/waves/{waveId}/insights
Answers **202**: generation is scheduled, not done. Poll the GET above rather than re-requesting.

**SPENDS.** This runs models on your organization's account and draws on the `insightsPerDay` ledger, which is SHARED with eval-run and user-testing insights — burning it here takes it from there too.

Three refusals, and they mean different things:
- `429` with `code: "RATE_LIMITED"` and a short `Retry-After` — the per-minute burst brake. Wait seconds.
- `429` naming `insightsPerDay` in `details` — the daily ledger. `Retry-After` counts to UTC midnight.
- `403` — the feature is not available to your organization. Waiting will never help; collapsing this into the 429s would send someone who hit today's cap shopping for a plan they already have.



# Undismiss a finding
Source: https://docs.mcpjam.com/api-reference/swarm-insights/undismiss-a-finding

/reference/openapi.json post /projects/{projectId}/journey-findings/{findingId}/undismiss
Reverses a dismissal. Bodyless, like its twin.



# Cancel a journey run
Source: https://docs.mcpjam.com/api-reference/swarm-runs/cancel-a-journey-run

/reference/openapi.json post /projects/{projectId}/journey-runs/{runId}/cancel
Stops a run and settles its in-flight attempts. IDEMPOTENT: cancelling an already-cancelled run answers 200 with `alreadyCanceled: true` and `finalized: 0`, not a 409.

Addressed entirely by the path `runId` — there is no body.

Deliberately NOT behind the beta gate: cancelling REDUCES exposure and spend, so an organization that loses the flag can still stop a run that is under way.



# Get a journey run
Source: https://docs.mcpjam.com/api-reference/swarm-runs/get-a-journey-run

/reference/openapi.json get /projects/{projectId}/journey-runs/{runId}
The full record, including per-session `attempts` and `targetSummaries`.

Read `canceled` before reporting a failure: a stopped run carries `status: "failed"`.



# Launch a journey
Source: https://docs.mcpjam.com/api-reference/swarm-runs/launch-a-journey

/reference/openapi.json post /projects/{projectId}/journeys/{journeyId}/runs
Answers **202**: the run row exists and its fan-out has started. It is not finished — poll `GET /projects/{projectId}/journey-runs/{runId}` until `status` leaves `running`.

**SPENDS.** A launch fans out into `targets × sessionsPerTarget` chat sessions, every one of which runs a model on your organization's account.

Two different 429s can come back and they mean different waits: the per-minute burst brake (retry in seconds) and your plan's daily launch cap (resets at UTC midnight). Both normally carry `Retry-After` — honor it when it is there, and back off on your own when it is not, rather than blocking on a header that may not come. A retried `Idempotency-Key` consumes NO quota.



# List a journey's runs
Source: https://docs.mcpjam.com/api-reference/swarm-runs/list-a-journeys-runs

/reference/openapi.json get /projects/{projectId}/journeys/{journeyId}/runs
Newest first. Compact records — `attempts` is on the single-run read.



# List a run's sessions
Source: https://docs.mcpjam.com/api-reference/swarm-runs/list-a-runs-sessions

/reference/openapi.json get /projects/{projectId}/journey-runs/{runId}/sessions
One entry per chat session the run produced. Summaries, not transcripts — read a transcript through the chat-session routes using `id`.



# Archive a journey
Source: https://docs.mcpjam.com/api-reference/swarms/archive-a-journey

/reference/openapi.json delete /projects/{projectId}/journeys/{journeyId}
ARCHIVES rather than deletes, and the response says so. Existing runs keep pointing at the definition they executed, so their history stays readable.



# Archive a swarm
Source: https://docs.mcpjam.com/api-reference/swarms/archive-a-swarm

/reference/openapi.json delete /projects/{projectId}/swarms/{swarmId}
ARCHIVES the container. The journeys authored under it are not touched — they keep their own definitions and stay launchable.



# Create a journey
Source: https://docs.mcpjam.com/api-reference/swarms/create-a-journey

/reference/openapi.json post /projects/{projectId}/journeys
Creates one journey: a persona pursuing a `goal` against one or more environments. To draft several with a model first, use `POST /projects/{projectId}/journeys/generate` — that route persists nothing.

`personaId` and `swarmId` must belong to THIS project. Naming one from another project is a `404`, not a silent cross-project reference.

Creating a journey does not run it. Launch it with `POST /projects/{projectId}/journeys/{journeyId}/runs`.



# Create a persona
Source: https://docs.mcpjam.com/api-reference/swarms/create-a-persona

/reference/openapi.json post /projects/{projectId}/personas
Creates one persona. To draft several with a model first, use `POST /projects/{projectId}/personas/generate` — that route persists nothing.



# Create a swarm
Source: https://docs.mcpjam.com/api-reference/swarms/create-a-swarm

/reference/openapi.json post /projects/{projectId}/swarms
A swarm is an authoring container holding DEFAULTS for the journeys made under it. Creating one starts nothing.



# Delete a persona
Source: https://docs.mcpjam.com/api-reference/swarms/delete-a-persona

/reference/openapi.json delete /projects/{projectId}/personas/{personaId}
Refused with 409 while journeys still reference this persona — deleting it would strand them.



# Draft journeys for a persona
Source: https://docs.mcpjam.com/api-reference/swarms/draft-journeys-for-a-persona

/reference/openapi.json post /projects/{projectId}/journeys/generate
Returns DRAFTS and persists nothing — see `POST /projects/{projectId}/personas/generate` for why, and for what this spends.



# Draft personas with a model
Source: https://docs.mcpjam.com/api-reference/swarms/draft-personas-with-a-model

/reference/openapi.json post /projects/{projectId}/personas/generate
Returns DRAFTS and persists nothing. That separation is deliberate: generation is the step most likely to produce something nobody wanted, and a route that both invented and saved would make "let me see what you'd suggest" indistinguishable from "fill my project with these".

It is also why there is no idempotency key — a call with no effect has nothing to de-duplicate, and offering one would imply the drafts are stable across retries when they are not.

**SPENDS.** This runs models on your organization's account and is metered two ways: a per-minute burst brake (429, retry in seconds) and your plan's daily budget (429, resets at UTC midnight). Both normally carry `Retry-After` — honor it when it is there, and back off on your own when it is not.



# Get a journey
Source: https://docs.mcpjam.com/api-reference/swarms/get-a-journey

/reference/openapi.json get /projects/{projectId}/journeys/{journeyId}



# Get a persona
Source: https://docs.mcpjam.com/api-reference/swarms/get-a-persona

/reference/openapi.json get /projects/{projectId}/personas/{personaId}
A persona that lives in a DIFFERENT project answers 404, never 403 — the route is not an existence oracle for a project you cannot see.



# Get a swarm
Source: https://docs.mcpjam.com/api-reference/swarms/get-a-swarm

/reference/openapi.json get /projects/{projectId}/swarms/{swarmId}



# List journeys
Source: https://docs.mcpjam.com/api-reference/swarms/list-journeys

/reference/openapi.json get /projects/{projectId}/journeys
The project's journeys, newest first. Archived journeys are excluded.



# List personas
Source: https://docs.mcpjam.com/api-reference/swarms/list-personas

/reference/openapi.json get /projects/{projectId}/personas
The project's personas, newest first. Archived personas are excluded.



# List swarms
Source: https://docs.mcpjam.com/api-reference/swarms/list-swarms

/reference/openapi.json get /projects/{projectId}/swarms



# Update a journey
Source: https://docs.mcpjam.com/api-reference/swarms/update-a-journey

/reference/openapi.json patch /projects/{projectId}/journeys/{journeyId}
Editing a journey never disturbs a run already under way: a run pins its own immutable snapshot of the definition at launch.



# Update a persona
Source: https://docs.mcpjam.com/api-reference/swarms/update-a-persona

/reference/openapi.json patch /projects/{projectId}/personas/{personaId}
Partial update. Journeys reference the persona by id, so renaming one does not disturb anything pointing at it.



# Update a swarm
Source: https://docs.mcpjam.com/api-reference/swarms/update-a-swarm

/reference/openapi.json patch /projects/{projectId}/swarms/{swarmId}
Changes the container's DEFAULTS. Journeys already authored under it keep the values they were created with — this is not a cascade.



# Create (or revive) a relay tunnel for a named project server
Source: https://docs.mcpjam.com/api-reference/tunnels/create-or-revive-a-relay-tunnel-for-a-named-project-server

/reference/openapi.json post /projects/{projectId}/tunnels
Registers a server record named `name` if missing, mints a relay tunnel grant for it, and **persists the tunnel bearer URL (including the plaintext `?k=` secret) onto the server record's `url`** so evals and scenarios can target the tunnel like any remote server. The plaintext persistence is a deliberate trade-off of the current tunnel MVP — the backend otherwise stores only a hash of the secret — mitigated by rotation: **every call rotates the secret, revokes the previous grant at the edge (disconnecting any live tunnel session for the server), and updates the stored URL**, so re-calling this route is also the rotation/recovery path.

The caller hosts the tunnel itself: connect a WebSocket to `relayWsUrl` (subprotocol `mcpjam-tunnel.v1`, `Authorization: Bearer <connectToken>`) and serve the relayed requests — this is what `mcpjam cloud tunnel` does. When the host disconnects, the server record stays and calls to the public URL fail fast at the edge.



# Revoke a tunnel's live grant
Source: https://docs.mcpjam.com/api-reference/tunnels/revoke-a-tunnels-live-grant

/reference/openapi.json post /projects/{projectId}/tunnels/{serverId}/close
Revokes the grant at the control plane and edge: the public URL stops working immediately and any live tunnel session is disconnected. The server record — including its now-dead `url` — is intentionally left untouched, so the next create revives the tunnel with the same slug.



# Cancel a window's generation
Source: https://docs.mcpjam.com/api-reference/user-testing/cancel-a-windows-generation

/reference/openapi.json delete /projects/{projectId}/user-testing/scenarios/{scenarioId}/insights
Releases a generation that is stuck `pending`. Idempotent against an already-finished one.



# Current window signals
Source: https://docs.mcpjam.com/api-reference/user-testing/current-window-signals

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/signals
What the miner extracted from the current window — and where you get the window id the insights read takes.

`404` when no window has been analyzed yet.



# Dismiss a finding
Source: https://docs.mcpjam.com/api-reference/user-testing/dismiss-a-finding

/reference/openapi.json post /projects/{projectId}/user-testing/scenarios/{scenarioId}/findings/{findingId}/dismiss
Keyed on the finding's stable remediation id, so the same problem stays dismissed when insights regenerate. Bodyless: the path names the finding.



# Get a scenario
Source: https://docs.mcpjam.com/api-reference/user-testing/get-a-scenario

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}
The scenario, plus the insights envelope over its latest analyzed window when the caller may have it.

Project members only. This route is deliberately outside the guest allowlist, and the envelope additionally needs workspace membership — share-link visitors can never read other visitors' evidence.



# Get a session transcript
Source: https://docs.mcpjam.com/api-reference/user-testing/get-a-session-transcript

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/sessions/{sessionId}
The projected transcript, paged. Tool payloads and blobs are dropped, and the stored blob URL is never returned.

Check `transcriptUnavailable` before trusting `messageCount`: an unreadable transcript reports `null`, never `0`.



# Invite a member
Source: https://docs.mcpjam.com/api-reference/user-testing/invite-a-member

/reference/openapi.json put /projects/{projectId}/user-testing/scenarios/{scenarioId}/members
Upsert by email, so re-inviting somebody is not an error. Only meaningful while `mode` is `invited_only`.



# List a scenario's sessions
Source: https://docs.mcpjam.com/api-reference/user-testing/list-a-scenarios-sessions

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/sessions
What visitors did, as summaries. Project members only — this route is deliberately absent from the guest allowlist, so a share-link visitor can never read other visitors' sessions.



# List scenario findings
Source: https://docs.mcpjam.com/api-reference/user-testing/list-scenario-findings

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/findings
Every finding raised over this scenario's windows, dismissed ones included — dismissal is a state on the finding, not a deletion.



# Point a scenario at a different environment
Source: https://docs.mcpjam.com/api-reference/user-testing/point-a-scenario-at-a-different-environment

/reference/openapi.json post /projects/{projectId}/user-testing/scenarios/{scenarioId}/rebind
Keeps the scenario's LINK, its members and its session history while changing what visitors actually reach. The alternative — unpublish and republish — mints a new link, which means re-sharing it with everyone.

The new environment must belong to THIS project; one that does not answers 404 rather than confirming it exists elsewhere.



# Publish an environment for user testing
Source: https://docs.mcpjam.com/api-reference/user-testing/publish-an-environment-for-user-testing

/reference/openapi.json put /projects/{projectId}/environments/{environmentId}/scenario
PUT, not POST, because publishing is IDEMPOTENT: one scenario per environment, and publishing an already-published environment returns the existing one rather than minting a second. `created` says which happened.

Requires project ADMIN.



# Read a window's insights
Source: https://docs.mcpjam.com/api-reference/user-testing/read-a-windows-insights

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/windows/{windowId}/insights
Poll this after requesting a generation.

`404` means NOBODY ASKED — deliberately distinct from a `pending` status, so a caller looping on this cannot mistake "never requested" for "requested and still working".



# Remove a member
Source: https://docs.mcpjam.com/api-reference/user-testing/remove-a-member

/reference/openapi.json delete /projects/{projectId}/user-testing/scenarios/{scenarioId}/members/{memberIdOrEmail}
Removal NARROWS access, so it bumps `accessVersion` and cuts off any session that member already had — it takes effect at once rather than at expiry.



# Request insights for the current window
Source: https://docs.mcpjam.com/api-reference/user-testing/request-insights-for-the-current-window

/reference/openapi.json post /projects/{projectId}/user-testing/scenarios/{scenarioId}/insights
**Spends.** Runs a model over the scenario's current window and draws on the organization's `insightsPerDay` ledger, which is SHARED with eval-run and swarm wave insights — spending it here takes it from there.

`202`: scheduled, not done. Poll `GET .../windows/{windowId}/insights` with the returned `windowId` rather than re-requesting.

`409` when the current window has not been mined yet: the request was well formed and retrying it verbatim will not help until sessions have been processed.



# Restore a dismissed finding
Source: https://docs.mcpjam.com/api-reference/user-testing/restore-a-dismissed-finding

/reference/openapi.json post /projects/{projectId}/user-testing/scenarios/{scenarioId}/findings/{findingId}/undismiss
Undoes a dismissal. Bodyless: the path names the finding.



# Rotate the share link
Source: https://docs.mcpjam.com/api-reference/user-testing/rotate-the-share-link

/reference/openapi.json post /projects/{projectId}/user-testing/scenarios/{scenarioId}/rotate-link
Mints a new share link and stops the old URL from granting access. Everyone you shared it with needs the new one.

**Rotation does NOT evict anyone who already redeemed the old link.** Their grant is a stored row, not a property of the URL, and it survives rotation — `accessVersion` is deliberately not bumped here. So rotation alone does not undo a leak: rotate to close the door, then `DELETE .../members/{memberIdOrEmail}` for anyone who already walked through it.

Bodyless: the scenario is the whole request, and the new secret is minted server-side by definition.



# Session metrics
Source: https://docs.mcpjam.com/api-reference/user-testing/session-metrics

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/metrics
Volumes, completion and feedback rates over the scenario's sessions.

`404` when the scenario has produced no sessions yet — an empty scenario has no metrics, which is different from metrics that are all zero.



# Set the guest-execution caps
Source: https://docs.mcpjam.com/api-reference/user-testing/set-the-guest-execution-caps

/reference/openapi.json put /projects/{projectId}/user-testing/scenarios/{scenarioId}/guest-execution
The spend dial for anonymous visitors. A FULL REPLACEMENT — send every field. These caps only mean something as a set, and a partial update that raised one while leaving another stale would produce a combination nobody chose.

Requires project ADMIN: it is the one exposure control that genuinely does, because it authorizes strangers to spend.



# Unpublish an environment
Source: https://docs.mcpjam.com/api-reference/user-testing/unpublish-an-environment

/reference/openapi.json delete /projects/{projectId}/environments/{environmentId}/scenario
Takes the scenario down. Deliberately NOT behind the beta gate that guards publishing: an organization that loses the flag must still be able to take a live scenario down.

Unpublishing something already unpublished answers 200 with `deleted: false` — a no-op, not an error.



# Update a scenario
Source: https://docs.mcpjam.com/api-reference/user-testing/update-a-scenario

/reference/openapi.json patch /projects/{projectId}/user-testing/scenarios/{scenarioId}
Change a scenario's identity or its exposure — but not both in one call. **Send `mode` on its own.** Identity and exposure are separate operations upstream, and applying them in sequence could leave the scenario live in a mode you did not ask for.

Narrowing `mode` bumps `accessVersion` upstream and invalidates sessions minted under the old one.



# Usage breakdown
Source: https://docs.mcpjam.com/api-reference/user-testing/usage-breakdown

/reference/openapi.json get /projects/{projectId}/user-testing/scenarios/{scenarioId}/usage
Which models, tools and servers the scenario's sessions used.

**Check `scan.truncated` before quoting a rate.** `404` when there is no usage data yet.



# Changelog
Source: https://docs.mcpjam.com/changelog/overview

Track the latest updates, features, and improvements to MCPJam Inspector

## Overview

This changelog tracks all notable changes to MCPJam Inspector. We follow [Semantic Versioning](https://semver.org/) and keep our changelog in the spirit of [Keep a Changelog](https://keepachangelog.com/).

## Latest Releases

### 2026-09-09

#### Added

* **Findings tab on User Testing scenario detail** — opening a User Testing scenario now lands on a **Findings** tab instead of Insights. The tab groups sessions by how users came across — Gave up, Frustrated, Neutral, Satisfied, or Uncategorized — and lists the goal clusters each group engaged with, ordered worst-first. Expanding a goal shows the per-goal user-value chain (Connection → Discovery → Selection → Call → Response → Value) once stage data is available. Click any goal's session count to jump to Sessions with the matching transcript pre-selected. Insights and Sessions remain reachable from the tab strip.

#### Changed

* **User Testing scenarios analyze themselves** — opening a User Testing scenario's Findings or Insights tab when sessions exist but no analysis has run now starts the analysis automatically. The "Analyze sessions" button and the "These sessions haven't been analyzed yet" state are gone; the tab shows an analyzing banner instead and fills in once the run completes. The cluster tuning control and the freshness chip still require an explicit action. Swarm and Benchmark scopes are unchanged — Swarm already queues analysis when a run settles, and Benchmark flow analysis remains explicitly requested.

### 2026-09-02

#### Added

* **GPT-5.6 models available with your own OpenAI key** — `gpt-5.6-luna`, `gpt-5.6-sol`, and `gpt-5.6-terra` are now selectable in the model picker under **Your providers → OpenAI** when you have an OpenAI API key configured. Each model has a 1,050,000-token context window. These models were already available in the free hosted catalog; this change makes them reachable for BYOK users as well.

### 2026-08-31

#### Changed

* **Public score runner redesign** — the score runner landing page has been rebuilt with a new design. After you paste a public MCP server URL and click **Score this server**, you are now prompted for an email address before the scan starts. The scorecard is delivered to that address once the run completes. The email is preserved across OAuth redirects, so servers that require authorization before scanning still collect it. Legacy OAuth resume records that predate the email step are kept on the email prompt rather than bypassing it. The featured scores section on the landing page now shows real MCP endpoints (Linear, GitHub Copilot, Notion), and each row populates the URL field when clicked. URL examples default to Streamable HTTP at `/mcp`.

### 2026-06-26

#### Added

* **Compatibility destination** — a new Inspector page that evaluates your connected MCP server against a catalog of AI hosts (Claude, ChatGPT, Cursor, Copilot, Codex, and others). Shows conformance gates, per-host findings split into Apps and Server lanes, and live widget rendering. Accessible from the left sidebar when a server is connected.

* **OTLP/OpenInference trace export** — export eval run traces as OTLP JSON from the run detail page. Compatible with Arize Phoenix, Datadog, and any OTLP-compatible observability backend. Supports exporting a single run or the whole project, with an opt-in to include content and artifacts.

* **`@mcpjam/sdk/host-compat` subpackage** — the shared host-compatibility engine is now importable directly. Use `deriveServerRequirements`, `evaluateHostCompat`, `evaluateAllHosts`, `scanWidgetSource`, and `detectHostCompatBridgeFromMeta` to build your own compatibility checks in CI or custom tooling.

### Version 1.1.0

#### Improved

* **Code Quality**
  * Improved code consistency and maintainability

### Recent Updates

#### Added

* **VS Code and Notion market hosts (`@mcpjam/sdk` minor)**
  * **VS Code** (GitHub Copilot chat panel) is now a market host in the SDK's host-compat catalog. It renders MCP UI inline and supports all display modes (`inline`, `fullscreen`, `pip`). Capabilities mirror the VS Code client template; `updateModelContext` and `message` are not advertised. Provenance: probe.
  * **Notion** is now a market host. It is a tools-only/headless client (no widget rendering) that supports protocol version `2025-11-25`. Provenance: probe.
  * Both hosts are available via `buildMarketHostProfiles()` / `evaluateMarketHosts()` and carry logos in the inspector's host-compat UI.

* **`imageSupport` dimension on all market hosts (`@mcpjam/sdk` minor)**

  * Every market host now carries an `imageSupport` field on its `HostCompatProfile` describing how it handles MCP tool-result images.
  * Three image sources are tracked per host: `toolImageContent` (direct MCP image blocks), `embeddedResourceImages` (image blobs inside MCP resources), and `resourceLinkImages` (image links resolved via `resources/read`).
  * Each source records `model` (whether the image is passed to the model) and `ui` (whether the host UI renders it).
  * A single `placement` value (`"inline"` | `"collapsed"` | `"none"`) describes where rendered previews appear in the host UI.
  * New exported types: `HostImageSupport`, `ImageSourceSupport`, `ImagePlacement` from `@mcpjam/sdk/host-compat`.
  * New capability matrix: `MCP_APPS_VSCODE` exported from `@mcpjam/sdk/host-compat`.

  | Host       | Tool image (model / UI) | Placement |
  | ---------- | ----------------------- | --------- |
  | Claude     | ✓ / ✓                   | inline    |
  | ChatGPT    | ✓ / ✓                   | inline    |
  | Cursor     | ✓ / ✗                   | none      |
  | Copilot    | ✓ / ✗                   | none      |
  | VS Code    | ✓ / ✓                   | inline    |
  | Notion     | ✗ / ✓                   | collapsed |
  | Goose      | ✓ / ✓                   | collapsed |
  | Codex      | ✓ / ✓                   | collapsed |
  | Cline      | ✓ / ✗                   | none      |
  | Slack      | ✗ / ✗                   | none      |
  | Mistral    | ✗ / ✗                   | none      |
  | n8n        | ✗ / ✗                   | none      |
  | Perplexity | ✗ / ✗                   | none      |

#### Changed

* **Stateless MCP protocol version updated to `2026-07-28` RC**
  * The stateless transport protocol version has been updated from the retired `DRAFT-2026-v1` placeholder to the upstream RC literal `2026-07-28`, matching the canonical pin in `modelcontextprotocol/main`.
  * The protocol version picker in the inspector UI now shows **Latest (2025-11-25)** and **2026 RC (2026-07-28)** instead of "Latest" and "Draft".
  * The SDK constant `STATELESS_DRAFT_2026_V1` has been renamed to `LATEST_STATELESS_PROTOCOL_VERSION` and now equals `"2026-07-28"`.
  * **Migration required**: If you had servers pinned to `DRAFT-2026-v1`, you must re-select the protocol version in the inspector UI. Stored `DRAFT-2026-v1` pins are rejected by spec-conforming servers with a `-32004 UnsupportedProtocolVersionError`.

#### Added

* **OAuth Protocol 2025-03-26 Support**
  * Added support for original MCP OAuth specification (2025-03-26)
  * Direct RFC8414 discovery from MCP server base URL with fallback endpoints
  * PKCE required for all clients
  * DCR and pre-registered client registration strategies

* **OAuth Debugger Enhancements**
  * Edit Config button to modify server settings without reconnecting
  * Copy logs feature with guide and raw formats for debugging
  * Deprecated transport detection with migration guidance for HTTP+SSE servers
  * RFC 8707 compliant resource URL canonicalization

* **GPT-5 Model Support**
  * Added support for GPT-5 model variants: `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5-chat-latest`, `gpt-5-pro`, `gpt-5-codex`
  * Organization verification notice for GPT-5 access

#### Improved

* **OAuth Flow**
  * Refined diagram focusing and zoom behavior for better step visualization
  * Improved error handling for expected 4xx responses during transport detection
  * Enhanced header merging in OAuth state machines across all protocol versions

* **Error Handling**
  * Enhanced streaming error handling in chat interface
  * Errors now display as inline alerts instead of failing silently
  * Better error messages from AI providers

### Version 1.0.0

#### Added

* **MCP Server Connection Management**
  * Support for STDIO, SSE, and Streamable HTTP transports
  * Multi-server connection support
  * Real-time connection status monitoring

* **LLM Playground**
  * OpenAI integration (GPT-3.5/4)
  * Anthropic Claude integration (Claude 2/3)
  * DeepSeek AI support (DeepSeek R1)
  * Ollama local model compatibility
  * Interactive chat interface with streaming responses

* **Tools & Resources Testing**
  * Tool execution and validation
  * Resource schema verification
  * Prompt testing interface
  * Real-time parameter validation

* **OAuth 2.0 Testing**
  * Guided OAuth flow setup
  * Token management
  * Scope verification
  * Refresh token handling

* **MCP Evals**
  * Automated compliance testing
  * Custom evaluation framework
  * Test result reporting
  * Performance benchmarking

* **Developer Tools**
  * Comprehensive logging system
  * Request/response tracing
  * Error reporting and analysis
  * Performance monitoring

#### Improved

* Enhanced UI/UX with modern design
* Better error messages and debugging support
* Optimized performance for large MCP responses
* Improved documentation and examples

#### Fixed

* Various bug fixes and stability improvements
* Memory leak fixes in long-running sessions
* Edge cases in MCP protocol handling

## Coming Soon

We're working on:

* Enhanced evaluation templates
* More LLM provider integrations
* Advanced debugging tools
* Performance analytics dashboard
* CI/CD integration support

## Stay Updated

* Follow our [GitHub repository](https://github.com/MCPJam/inspector) for the latest updates
* Join our [Discord community](https://discord.gg/JEnDtz8X6z) for announcements
* Check our [website](https://mcpjam.com) for release notes

## Reporting Issues

Found a bug or have a feature request? Please [open an issue](https://github.com/MCPJam/inspector/issues/new) on GitHub.


# MCP Apps Conformance
Source: https://docs.mcpjam.com/cli/apps-conformance

Validate MCP Apps tool metadata and ui:// resource wiring from the CLI

The `apps conformance` command validates the server-side MCP Apps surface your server exposes through tools and `ui://` resources.

Use `tools call --ui` when you want to execute one UI-capable tool and render the completed result in the local Inspector App Builder.

<Note>
  This is currently **server-side** MCP Apps conformance only. A passing run does
  **not** prove full SEP-1865 host behavior such as `ui/initialize`,
  `ui/notifications/tool-input` ordering, sandbox proxy behavior, or host
  display-mode handling.
</Note>

## Quick start

```bash theme={"theme":"css-variables"}
mcpjam apps conformance --url https://your-server.com/mcp
```

For a local stdio server:

```bash theme={"theme":"css-variables"}
mcpjam apps conformance --command node --args server.js --cwd /path/to/project
```

You can emit CI-friendly JUnit XML from either the single-run or suite command:

```bash theme={"theme":"css-variables"}
mcpjam apps conformance \
  --url https://your-server.com/mcp \
  --reporter junit-xml > apps-report.xml
```

To reproduce one tool result in the local Inspector:

```bash theme={"theme":"css-variables"}
mcpjam tools call \
  --url https://your-server.com/mcp \
  --tool-name create_view \
  --tool-args @params.json \
  --ui \
  --quiet \
  --format json
```

## What it checks

The current runner validates:

1. MCP Apps tools are present.
2. Tool metadata uses valid `_meta.ui.resourceUri` and `visibility` values.
3. Tool `inputSchema` is a non-null JSON Schema object.
4. Listed UI resources use `ui://` URIs and `text/html;profile=mcp-app`.
5. Referenced UI resources can be fetched with `resources/read`.
6. Resource contents provide exactly one HTML payload via `text` or `blob`.
7. `_meta.ui.csp`, `permissions`, `domain`, and `prefersBorder` use valid shapes.

## Example output

```bash theme={"theme":"css-variables"}
mcpjam apps conformance \
  --url https://your-server.com/mcp \
  --format json
```

Typical success summary:

```json theme={"theme":"css-variables"}
{
  "passed": true,
  "outcome": "passed",
  "summary": "7/7 checks passed, 0 failed, 0 could not run, 0 not applicable",
  "discovery": {
    "toolCount": 3,
    "uiToolCount": 1,
    "listedResourceCount": 1,
    "listedUiResourceCount": 1,
    "checkedUiResourceCount": 1
  }
}
```

## Suite runner

Use `apps conformance-suite` when you want a named CI matrix from a config file.

Example config:

```json theme={"theme":"css-variables"}
{
  "name": "Apps CI",
  "target": {
    "url": "https://your-server.com/mcp",
    "timeout": 30000
  },
  "defaults": {
    "checkIds": ["ui-tools-present", "ui-tool-metadata-valid"]
  },
  "runs": [
    {
      "label": "metadata"
    },
    {
      "label": "resources",
      "checkIds": [
        "ui-resources-readable",
        "ui-resource-contents-valid",
        "ui-resource-meta-valid"
      ]
    }
  ]
}
```

Run it with:

```bash theme={"theme":"css-variables"}
mcpjam apps conformance-suite \
  --config ./apps-conformance.json \
  --reporter junit-xml > apps-report.xml
```

`target` uses the same shared server shape as the rest of the CLI, so it can point at either:

* an HTTP target with `url`
* a stdio target with `command`, optional `args`, optional `env`, and optional `cwd`

## Categories and check ids

Two categories are available:

* `tools`
* `resources`

Specific check ids:

* `ui-tools-present`
* `ui-tool-metadata-valid`
* `ui-tool-input-schema-valid`
* `ui-listed-resources-valid`
* `ui-resources-readable`
* `ui-resource-contents-valid`
* `ui-resource-meta-valid`

Use `--category` to run a subset by category, or `--check-id` to run specific checks.

```bash theme={"theme":"css-variables"}
# Tool-only checks
mcpjam apps conformance \
  --url https://your-server.com/mcp \
  --category tools

# Specific checks
mcpjam apps conformance \
  --url https://your-server.com/mcp \
  --check-id ui-resources-readable \
  --check-id ui-resource-contents-valid
```

<Note>
  If you pass both `--category` and `--check-id`, the explicit `--check-id`
  selection wins.
</Note>

## Render one app result

Use `tools list` to find UI-capable tools. A tool has interactive UI when its metadata includes `_meta.ui.resourceUri`, deprecated `_meta["ui/resourceUri"]`, or `openai/outputTemplate` in `toolsMetadata`.

Then use `tools call --ui` when you need to inspect the output of one UI-capable tool call. The CLI starts or attaches to the local Inspector backend, opens Inspector by default in a TTY, connects the target server, focuses App Builder in the active Inspector browser client, injects the already-completed tool result, and requests a UI snapshot. `--inspector-url` is the Inspector backend/API URL; `--frontend-url` is the browser/client URL and skips frontend discovery. Use `--no-open` when browser automation already opened `http://127.0.0.1:6274/#app-builder`; use `--attach-only` to disallow startup, browser opening, and discovery. Default non-TTY `--ui` runs do not open a browser unless `--open` is passed. TTY stderr runs print the App Builder URL and initial wait progress unless `--quiet` is set; the elapsed-seconds heartbeat only appears when stderr is a TTY.

```bash theme={"theme":"css-variables"}
mcpjam tools call \
  --command node --args server.js --cwd /path/to/project \
  --tool-name create_view \
  --tool-args @params.json \
  --ui \
  --protocol mcp-apps \
  --device desktop \
  --theme light \
  --format json
```

<Note>
  `tools call --ui` executes the server tool once. Inspector receives the
  completed result through `renderToolResult`, so the UI render path does not
  call the tool a second time. The rendered state lives in the active Inspector
  client; opening the URL in a fresh tab later does not hydrate the old render.
</Note>

If `inspectorRender.status` is `skipped`, the tool call succeeded but Inspector had no active browser client, a render precondition was missing, or the render wait timed out; the envelope includes a stable root `warning` and `inspectorRender.warning` with one of the stable codes `no_active_client`, `timeout`, `disconnected_server`, or `unsupported_in_mode`, plus an `inspectorRender.remediation` hint. Non-quiet runs print a stderr tip, and the command keeps the tool-call exit code unless `--require-render` is set. If the command returns `success: false` with an `error`, check whether the failure came from `result` or from an Inspector render command.

Raw `ui://` resource HTML can be inspected directly:

```bash theme={"theme":"css-variables"}
mcpjam resources read \
  --url https://your-server.com/mcp \
  --resource-uri ui://widget/app.html \
  --quiet \
  --format json
```

## Shared connection flags

`apps conformance` uses the same shared target flags as the rest of the CLI:

| Flag                           | Description                                                   |
| ------------------------------ | ------------------------------------------------------------- |
| `--transport <transport>`      | Explicit transport type (`http` or `stdio`)                   |
| `--url <url>`                  | HTTP MCP server URL                                           |
| `--access-token <token>`       | Bearer access token                                           |
| `--oauth-access-token <token>` | OAuth bearer access token                                     |
| `--refresh-token <token>`      | OAuth refresh token                                           |
| `--client-id <id>`             | OAuth client ID (with `--refresh-token`)                      |
| `--client-secret <secret>`     | OAuth client secret (with `--refresh-token`)                  |
| `--header <header>`            | HTTP header in `Key: Value` format (repeatable)               |
| `--client-capabilities <json>` | Client capabilities as inline JSON, `@path`, or `-` for stdin |
| `--command <command>`          | Command for a stdio server                                    |
| `--args <arg...>`              | Preferred stdio command arguments                             |
| `--command-args <arg>`         | Legacy stdio command argument (repeatable)                    |
| `-e, --env <env...>`           | Stdio environment `KEY=VALUE` values                          |
| `--cwd <path>`                 | Working directory for the stdio child process                 |

Stdio child processes inherit the parent shell environment by default. Use
`-e/--env` to add values or override inherited ones when the app server needs
project-specific configuration.
`--transport` is optional; without it, `--url` implies HTTP and `--command`
implies stdio.

## Outcomes: passed, failed, incomplete

A run reports one of three outcomes, and `passed` is `true` only for the first:

| `outcome`    | Exit code | Meaning                                                                                                      |
| ------------ | --------- | ------------------------------------------------------------------------------------------------------------ |
| `passed`     | `0`       | Every selected check either ran and passed, or does not apply to this server.                                |
| `failed`     | `1`       | At least one check found a violation.                                                                        |
| `incomplete` | `3`       | Nothing failed, but at least one selected check could not be run, so the run does not establish conformance. |

Every skipped check carries a `skipReason`:

* `not-applicable` — the check cannot apply here (no MCP Apps tools discovered, no UI resources listed). These never hold a run back.
* `could-not-run` — the check applies but was never exercised (tools/list did not complete, connection failed). These make the run `incomplete`.

An incomplete run carries a root `incompleteReason` naming the checks that did not run and what to change; non-quiet runs also print it to stderr. For a suite, the exit code is the worst of all runs (failure outranks incomplete).

## Notes

* The runner always advertises the MCP Apps UI extension capability so servers do not hide their MCP Apps surface when custom client capabilities are provided.
* Deprecated `_meta["ui/resourceUri"]` is accepted but reported as a warning.
* Tool name SHOULD validations (length, character set, uniqueness) surface as warnings, not failures.
* `permissions` are validated against the SEP-1865 object shape, for example `geolocation: {}` instead of booleans.
* Exit codes are CI-friendly: `0` when all selected checks pass, `1` when any check fails, `2` for invalid command usage or config files, and `3` when the run is incomplete because a selected check could not be run.

## Related commands

* [Server inspection](/cli/server-inspection) for breadth-first connectivity and capability triage
* [Tools, resources & prompts](/cli/tools-resources-prompts) for direct connected checks
* [Command reference](/cli/reference) for the full flag list


# CI / CD
Source: https://docs.mcpjam.com/cli/ci

Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run `mcpjam` in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

## GitHub Actions

### Authentication

There are three ways to authenticate in CI, depending on your server setup.

#### Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json
```

#### Option 2: Refresh token

Best when you already have a refresh token from a previous `oauth login`. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

**Secrets needed:**

| Secret              | Description                                         |
| ------------------- | --------------------------------------------------- |
| `MCP_SERVER_URL`    | Your MCP server URL                                 |
| `MCP_REFRESH_TOKEN` | OAuth refresh token from a previous login           |
| `MCP_CLIENT_ID`     | OAuth client ID (required with refresh tokens)      |
| `MCP_CLIENT_SECRET` | OAuth client secret (if the client is confidential) |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json
```

<Tip>
  To get a refresh token, run `mcpjam oauth login` locally with `--format json` and grab `.credentials.refreshToken` from the output.
</Tip>

#### Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |
| `MCP_API_KEY`    | Static API key      |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json
```

#### Option 4: No auth

Some servers don't require authentication at all.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

```yaml theme={"theme":"css-variables"}
      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)
```

### OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

```yaml theme={"theme":"css-variables"}
      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

### Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

```yaml theme={"theme":"css-variables"}
      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml
```

### MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

```yaml theme={"theme":"css-variables"}
      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml
```

Single-run `protocol conformance`, `oauth conformance`, and `apps conformance` also accept `--reporter junit-xml` when you only need one target/check selection instead of a suite config file.

***

## GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and `.gitlab-ci.yml` syntax.

### Authentication

#### Headless OAuth login

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Refresh token

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Static API key

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

```yaml theme={"theme":"css-variables"}
mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json
```

### OAuth conformance suite

```yaml theme={"theme":"css-variables"}
mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

***

## Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a **hosted eval run** with the CLI, or run evals **locally with the SDK** and upload the results. Both authenticate with an MCPJam API key (`sk_…`) from **Settings → API keys**.

### Trigger a hosted eval suite

`mcpjam cloud eval run` starts an asynchronous run of a suite that lives in your MCPJam project. Without `--wait`, it prints a launch receipt and returns immediately. In CI, add `--wait` and `--out` to write a structured JSON report after every launched run reaches a terminal state.

**Secrets needed:**

| Secret           | Description             |
| ---------------- | ----------------------- |
| `MCPJAM_API_KEY` | MCPJam API key (`sk_…`) |

```yaml theme={"theme":"css-variables"}
      - name: Run hosted eval
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest cloud eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --wait \
            --out eval-report.json \
            --format json > eval-result.json
          echo "Completed run $(jq -r '.runs[0].id' eval-result.json)"

      - name: Gate and write JUnit
        # Always run this step, even if the run above exited 1 on a failed
        # verdict: a default `if:` is `success()`, and skipping this step
        # would lose the gate's configurable policy and its JUnit report
        # exactly when the run failed. The job still ends up failed either
        # way — the run step above already exited nonzero.
        if: always()
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runs[0].id' eval-result.json)
          npx -y @mcpjam/cli@latest cloud eval gate \
            --run "$RUN_ID" \
            --project "My project" \
            --wait \
            --min-pass-rate-percent 100 \
            --reporter junit-xml \
            --out eval-report.xml

      - name: Upload eval reports
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: hosted-eval
          path: |
            eval-report.json
            eval-report.xml
```

<Note>
  **Runs launched from Actions badge as GitHub, and fill the CI column.** The CLI
  declares what it is on every launch, so a run started from a workflow shows
  **GitHub** in the Runs table rather than the generic **API** that every hosted
  launch used to show, and its commit, branch and job appear alongside it. Nothing
  to configure: the declaration is derived from the standard `GITHUB_*`
  environment, exactly as conformance uploads already do it. Outside CI the same
  command declares **CLI**.

  That commit is also what `eval compare --baseline-sha` resolves against, so a
  run launched by the CLI in CI can now be compared to a previous commit's run the
  same way an SDK-reported one can.

  This is a *declared* label, deliberately: the platform still stamps the run's
  own `source` server-side and records the verified credential separately, so a
  badge can never claim more than "the client said so". No secret that could prove
  it can live in a public npm package.
</Note>

In human format, `eval run` prints a `View:` line after the payload so you can open the run directly from the terminal:

```text theme={"theme":"css-variables"}
View: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>
```

This line is only emitted in human format — `--format json` output is unchanged, so scripts that parse the JSON stream are unaffected.

Use `--wait-timeout <ms>` to replace the 10-minute default. `--out` defaults to the structured JSON format; add `--reporter junit-xml` to write JUnit XML instead, or `--reporter html` for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When `--reporter` is present, the same report is also written to stdout.

<Warning>
  **`eval run --wait` sets a verdict-based exit code.** `0` pass, `1` a completed run's verdict failed (the ONLY condition that produces `1`), `2` usage error or an invalid suite file, `3` auth failed (no credential, or the platform rejected it, at launch or mid-wait), `4` a connection/setup failure this CLI itself observed before evaluation ran (or a local `--out` write failure), `5` no valid verdict — `inconclusive`, a null/unrecognized result, a run status of failed/cancelled/timed-out, or a wait that hit its deadline. A multi-target launch merges these worst-of across every waited run, in the order `1 > 3 > 4 > 5 > 0`.

  Retry guidance: `4` and `5` mean infrastructure, or an absence of observation — nothing here says the server is wrong, so retrying the CI job is reasonable. But a bare re-run is not automatically safe: `eval run --suite` only dedupes against an in-flight or already-completed launch when you pass a **stable** `--idempotency-key`, and exit `5` can mean the run is still running (a wait that hit its deadline) — without that key, a retry can start a second paid run alongside the first rather than resuming it. Pass `--idempotency-key` (or poll/resume the run id already in the receipt) before retrying on `4` or `5`. `3` means fix the credential first; it poisons every other observation in the same launch. Never retry blindly on `1` — that code is reserved for a run the platform actually graded as failed.

  Either `eval run --wait` or `eval gate` already fails the job on its own — the `eval gate` step above adds a configurable pass/fail *policy* (thresholds, per-scorer gates) and baseline comparison on top of the same verdict, so keep it when you want more than "did this run's own verdict pass".
</Warning>

<Note>
  `eval gate` sets a verdict-based exit code, and writes its report before doing so: `0` passed **or waived**, `1` an eval verdict failed, `2` usage error, `3` incomplete or non-gateable. Infrastructure conditions never map to `1`, so retrying on `3` is safe. This is a **different, four-code contract from `eval run --wait`** above — `gate`'s `3` means "incomplete", not the six-code scheme's `3` ("auth failed"), and the two are deliberately not unified (see the [CLI reference](/cli/reference#cloud-eval-gate) for why). `eval status` also prints a `View:` line in human format, identical to the one `eval run` prints.
</Note>

#### Gating on cost

Cost is the question people actually have about a change: *did this make the
suite more expensive?* Gate it relatively rather than absolutely — an absolute
ceiling goes stale on every prompt and model change, while a percentage
against the previous run keeps meaning the same thing.

```bash theme={"theme":"css-variables"}
npx -y @mcpjam/cli@latest cloud eval compare --run "$RUN_ID" \
  --project "My project" \
  --max-cost-increase-percent 10
```

With no `--base-run` or `--base-sha`, the baseline is the nearest earlier
completed run in the same suite — which is what "did my change make this more
expensive?" usually means.

An absolute ceiling is also available, for a suite whose budget is fixed:

```bash theme={"theme":"css-variables"}
npx -y @mcpjam/cli@latest cloud eval gate --run "$RUN_ID" \
  --project "My project" \
  --max-cost-usd 0.50
```

Both report **non-gateable** (exit `3`) rather than passing when the cost is
unknown or only partly measured — a run on your own API keys, a harness run,
or a run only some of whose iterations MCPJam priced. Exit `3` is safe to retry
and safe to treat as "no opinion"; what it never does is let an unpriced run
through as if it were cheap.

#### Waiving a gate

A run whose gate failed can be overridden by an authorized user until an expiry
they name, so a release is not blocked while a known regression is being fixed:

```bash theme={"theme":"css-variables"}
mcpjam cloud eval gate waive --run "$RUN_ID" --reason "hotfix ships today; tracked in ENG-4821" --expires-in 3d
```

`eval gate` then exits `0` and reports the outcome as `waived`. It is **not**
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a `<skipped>` element, so it neither fails the build nor renders as
a clean green row).

Only a real verdict failure is waivable. A cancelled run, a `--wait` timeout, or
a network failure still exits `3` with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.

Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform's answer. `mcpjam cloud eval gate
unwaive --run "$RUN_ID"` ends one early.

<Warning>
  The waiver reason is stored **unredacted** and readable by anyone who can see
  the suite, for as long as the suite exists. Never put secrets, tokens, or
  customer data in it.
</Warning>

#### Decision summary

`eval run --wait`, `eval status`, `eval gate` and `eval compare` all read one versioned object — the **run decision summary** — and every output format restates it. Where each command puts it:

| Command           | `--format json`                          | `--format human`                           | `--out` / `--reporter`          |
| ----------------- | ---------------------------------------- | ------------------------------------------ | ------------------------------- |
| `eval run --wait` | `decisionSummary` on the stdout receipt  | block on stdout, after the receipt         | `decisionSummary` on the report |
| `eval status`     | `decisionSummary` on the stdout document | block on stdout, above the `View:` line    | —                               |
| `eval gate`       | `decisionSummary` beside `gate`          | block on **stderr**, under the gate report | `decisionSummary` on the report |
| `eval compare`    | `decisionSummary` beside `compare`       | block on **stderr**, under the gate report | `decisionSummary` on the report |

Two scoping rules that are easy to miss. `eval run --wait` attaches a summary only when the invocation launched **one** run: a fan-out has several, and labelling a receipt about N runs with the decision of one would be a false claim rather than a partial one. `eval compare` reports the **compare side's** decision only — the baseline's failures are a different run's diagnostics, and printing them here would read as this run's.

```text theme={"theme":"css-variables"}
Decision summary: failed (verdict policy v2) — 2/3 case variants passed, 1 failed
  Why: a case did not meet its pass threshold
  Diagnostics: 2 non-passing of 9 iterations examined (the complete set)
  First break: Tool call — the call arguments did not match what the case expects (1 of 2 measured iterations)
  Fetch order (c_orders, iteration 2) — failed
    First failed stage: Tool call — the call arguments did not match what the case expects
    Failure category: call arguments
    Expected tool calls: fetch_order
    Observed failure: server rejected arguments
    Evidence at Tool call: span ids span-call-2; reasons order_id must be a string
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_2/trace
    Next action: review the authored arguments against the tool input schema
  Setup abort (c_setup, iteration 5) — failed
    First failed stage: none was established — the run never reached the server's stages
    Failure category: setup
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_5/trace
    Next action: check the server connection and environment configuration
```

Four things about it are worth knowing before you script against it.

**The counts carry the population they count.** `measurementUnit` is `caseVariant` under verdict policy v2 — one case under one provider/model, with repetitions as *iterations inside it* — and `trial` on a legacy percent-threshold run. A 3-case suite with 5 repetitions is legitimately "3" under one unit and "15" under the other, so a count quoted without its unit is not a fact.

**The summary explains the verdict; it never re-decides it.** Under policy v2 the run's own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on `decision`. The per-iteration diagnostics sit *underneath* that: a case can pass with a failing iteration in it, so tallying the diagnostics gives a different answer than the platform reached.

**`notEstablished` is not a failure.** It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. `undecided.reason` says which. It is also not `inconclusive`, which *is* a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.

**A page of diagnostics says whether it is the whole story.** `diagnostics.complete` is true only when the listed iterations are the run's entire non-passing set, and `scannedIterations` says how many were examined — so an empty list from a complete page ("nothing failed") is distinguishable from an empty list from a partial one ("we did not look").

Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage's row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.

`eval status` prints the block only when a terminal run did not pass — a clean pass has nothing to diagnose. `--format json` stays exactly one parseable document in every case: the summary rides *inside* it, never as a second block appended after it. If the summary cannot be fetched, it is omitted rather than failing the command.

**The human block leads with where the chain broke.** Under the diagnostics headline, before any per-iteration detail, a non-passing run gets one line naming the earliest stage at which a readable iteration stopped, why, and how many iterations stopped there:

```text theme={"theme":"css-variables"}
  First break: Tool call — the call arguments did not match what the case expects (2 of 3 measured iterations)
```

"First" means earliest in chain order — `connection → discovery → selection → call → response → userValue` — never "most common", so the count beside it is what tells you whether the run had one problem or several. When the breaks are spread the line says so (`earliest of 3 stages that broke`), and when some chains could not be read it says that too (`1 more had no readable chain`), because otherwise the denominator quietly shrinks to the iterations that happened to validate. A run that reached no stage at all — a setup abort, an evaluator error — names its bucket instead of inventing a location.

The line is **not** a diagnosis. A first failed stage is a location and a failure category is a bucket; neither on its own says what to change.

`eval status --stages` expands each failing iteration to all six chain rows with their states and reasons. It is off by default: six rows per iteration is a lot of terminal on a run with twenty failures, and the first-break line above already carries the answer. Human output is not a stable contract — script against `--format json`, where the enums travel as enums.

Hosted runs execute LLM iterations on the platform and consume your organization's credits or configured provider keys. See the [`cloud eval` command reference](/cli/reference#cloud-eval-commands) for the full surface, including `cloud eval judge` (request LLM-as-judge grading on a finished run), `cloud eval validate` (offline suite-file validation), `cloud eval export` (write a hosted suite to a local file), `cloud eval github list/connect` (GitHub checks integration), and more.

### Upload SDK eval results

If you instead run evals inside your own CI job with [`@mcpjam/sdk`](/sdk/concepts/running-evals) (`EvalTest` / `EvalSuite`), set `MCPJAM_API_KEY` and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

```yaml theme={"theme":"css-variables"}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/
```

See [Save Results to MCPJam](/sdk/concepts/saving-results) for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).


# MCPJam as an MCP Server
Source: https://docs.mcpjam.com/cli/mcp-server

Run MCPJam as a local stdio MCP server so agents can connect to, exercise, and debug other MCP servers

`mcpjam mcp` runs MCPJam itself as an MCP server over stdio. Add it to any MCP client — Claude Desktop, Claude Code, Cursor, or your own agent — and the agent gets MCPJam's local testing engine as tools: connect to a server under test, call its tools, read its resources and prompts, watch the notifications it emits, and run diagnostic sweeps.

```bash theme={"theme":"css-variables"}
npx -y @mcpjam/cli@latest mcp
```

Everything runs locally in the spawned process. No account, login, or hosted backend is involved, and the server under test can be a local stdio process — something hosted tooling can't reach.

## Why use this instead of the CLI?

Agents with shell access can already run `mcpjam` commands directly, and for one-shot checks that is often the better choice. The MCP server mode adds two things the CLI cannot do:

* **Clients without a shell.** Claude Desktop and other chat-style MCP clients can't run CLI commands. This is the only way to give them MCPJam's testing engine.
* **Persistent sessions.** Each CLI invocation reconnects to the target server. `mcpjam mcp` holds connections open across tool calls, so the agent can observe `notifications/message` logs, `list_changed` events, resource updates, and other session behavior that one-shot commands miss.

## Setup

<Tabs>
  <Tab title="Claude Desktop">
    Add to `claude_desktop_config.json` (Settings → Developer → Edit Config):

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "mcpjam": {
          "command": "npx",
          "args": ["-y", "@mcpjam/cli@latest", "mcp"]
        }
      }
    }
    ```
  </Tab>

  <Tab title="Claude Code">
    ```bash theme={"theme":"css-variables"}
    claude mcp add mcpjam -- npx -y @mcpjam/cli@latest mcp
    ```
  </Tab>

  <Tab title="Cursor">
    Add to `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json`):

    ```json theme={"theme":"css-variables"}
    {
      "mcpServers": {
        "mcpjam": {
          "command": "npx",
          "args": ["-y", "@mcpjam/cli@latest", "mcp"]
        }
      }
    }
    ```
  </Tab>

  <Tab title="Other clients">
    Any client that supports stdio servers can launch:

    ```bash theme={"theme":"css-variables"}
    npx -y @mcpjam/cli@latest mcp
    ```

    With a global install (`npm i -g @mcpjam/cli`), use `mcpjam mcp` as the command instead.
  </Tab>
</Tabs>

<Note>
  In MCP server mode all status output goes to stderr; stdout carries only
  JSON-RPC. The `--timeout <ms>` global flag sets the default per-request
  timeout against target servers (individual tool calls can override it with
  `timeoutMs`).
</Note>

## Tools

### Connections

| Tool                | Description                                                                                                                                                                                                  |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `connect_server`    | Open a persistent connection to a target server: `url` (+ optional `accessToken`, `headers`) for HTTP, or `command` (+ optional `args`, `env`, `cwd`) for stdio. Returns the negotiated initialization info. |
| `disconnect_server` | Close a connection and discard its buffered notifications                                                                                                                                                    |
| `list_servers`      | List open connections with status and a redacted target summary                                                                                                                                              |
| `server_info`       | Initialization info for a connection: protocol version, transport, server version, capabilities, instructions                                                                                                |
| `ping_server`       | MCP ping with round-trip latency                                                                                                                                                                             |

### Exercising the target

| Tool                | Description                                                                                                                                            |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `list_tools`        | List the target's tools (with pagination cursor)                                                                                                       |
| `call_tool`         | Call a tool and return the target's raw `CallToolResult` — check the payload's `isError` for tool-level failures                                       |
| `list_resources`    | List resources, or resource templates with `templates: true`                                                                                           |
| `read_resource`     | Read a resource by URI                                                                                                                                 |
| `list_prompts`      | List prompts                                                                                                                                           |
| `get_prompt`        | Fetch a named prompt with optional string arguments                                                                                                    |
| `get_notifications` | Notifications buffered since connect: log messages, progress, `list_changed` events, resource updates. Filter by `server`/`method`, optionally `clear` |

### Stateless diagnostics

| Tool            | Description                                                                                                        |
| --------------- | ------------------------------------------------------------------------------------------------------------------ |
| `server_doctor` | One-shot diagnostic sweep (HTTP or stdio target) — same checks as [`mcpjam server doctor`](/cli/server-inspection) |
| `probe_server`  | HTTP-only probe: transport selection, auth requirements, OAuth metadata discovery — same as `mcpjam server probe`  |

## Skills over MCP

Alongside its tools, `mcpjam mcp` serves two of MCPJam's Agent Skills over the [`io.modelcontextprotocol/skills` extension](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2640) (SEP-2640):

| Skill                | What it is for                                                                                           |
| -------------------- | -------------------------------------------------------------------------------------------------------- |
| `mcp-inspector`      | Interpreting probe, doctor, OAuth, and conformance output conservatively — the tools this server exposes |
| `mcpjam-eval-import` | Converting an existing test corpus into an MCPJam eval suite                                             |

A client that advertises the extension sees them in `skills/list` and can load them with no install step — you do not need `npx skills add` for these two when you are connected this way. Each skill's files are also readable as ordinary resources under `skill://mcpjam/<name>/…`, which is why `resources/list` now returns rows for a server that previously exposed none.

Every file carries a SHA-256 digest and a byte size, and a conforming host verifies both before loading. The listing deliberately carries **no** `ttlMs`: because the documented launch command is `npx -y @mcpjam/cli@latest mcp`, the bytes behind an identical command change whenever a release lands, so this server issues no cache-freshness directive it could not honour.

`mcpjam-eval-import` is served by the [hosted MCP server](/cli/mcp-server#relationship-to-the-hosted-mcp-server) too, under the same URI. If you are connected to both at once, address it by server (`<server>/mcpjam-eval-import`) rather than by bare URI.

## Example session

A typical agent flow while developing a stdio server:

1. `connect_server` with `{ "name": "dev", "command": "node", "args": ["dist/server.js"] }`
2. `list_tools` with `{ "server": "dev" }` — review names, descriptions, schemas
3. `call_tool` with `{ "server": "dev", "tool": "search", "arguments": { "query": "test" } }`
4. `get_notifications` with `{ "server": "dev" }` — did the server log what you expected? Did it emit `notifications/tools/list_changed`?
5. Iterate on the server code, then `disconnect_server` / `connect_server` to pick up the rebuild
6. `server_doctor` with the same `command` for a final health sweep

Tool results are JSON payloads describing the target server. Errors come back as structured `{ "error": { "code", "message" } }` payloads with `isError: true` — for example `USAGE_ERROR` for invalid input or `SERVER_UNREACHABLE` when the target is down.

## Relationship to the hosted MCP server

MCPJam also operates a hosted MCP server at `mcp.mcpjam.com/mcp` that exposes your MCPJam **account** — projects, saved servers, eval suites and runs — behind OAuth. The two are complementary. `mcpjam mcp` protocol `serverInfo.name` is `mcpjam`; its title is `MCPJam CLI`. The hosted server identifies as `MCPJam MCP`.

|                     | `mcpjam mcp` (this page)                                           | `mcp.mcpjam.com`                         |
| ------------------- | ------------------------------------------------------------------ | ---------------------------------------- |
| Runs                | Locally, spawned by your MCP client                                | Hosted                                   |
| Auth                | None                                                               | MCPJam account (OAuth)                   |
| Scope               | Test any server reachable from your machine, including local stdio | Servers and evals saved in your projects |
| Transport to MCPJam | stdio                                                              | Streamable HTTP                          |
| Protocol name       | `mcpjam`                                                           | `MCPJam MCP`                             |
| Title               | `MCPJam CLI`                                                       | `MCPJam MCP`                             |

## Troubleshooting

* **Client says the server failed to start** — run `npx -y @mcpjam/cli@latest mcp` in a terminal; you should see `MCPJam CLI MCP server listening on stdio` on stderr. First runs may be slow while `npx` downloads the package; pre-install with `npm i -g @mcpjam/cli` and use `mcpjam mcp` to avoid the download.
* **`connect_server` fails with `SERVER_UNREACHABLE`** — the target URL or command is wrong, or the target crashed on startup. Try `server_doctor` against the same target for a structured diagnosis.
* **A connection name is already taken** — `connect_server` refuses to overwrite an existing name; `disconnect_server` it first or pass a different `name`.


# Migrating to CLI 4.0
Source: https://docs.mcpjam.com/cli/migration

Command-path and flag changes when account-bound mcpjam commands moved under mcpjam cloud

`@mcpjam/cli` 4.0 separates local MCP operations from account-bound Cloud operations. Most account-bound commands live under `mcpjam cloud`; hosted readiness is the intentional root-level exception because it targets publisher directory checks and does not use Cloud project links. There are no compatibility aliases for the old Cloud command paths.

```bash theme={"theme":"css-variables"}
mcpjam server probe --url https://example.com/mcp       # local
mcpjam oauth login --url https://example.com/mcp        # local MCP OAuth
mcpjam cloud login                                      # MCPJam Cloud account
mcpjam cloud projects list                              # MCPJam Cloud
mcpjam cloud eval run --suite smoke                     # MCPJam Cloud
mcpjam cloud tunnel --server dev -- npm start           # local target, Cloud registration
```

## What did not change

* Frozen local commands: `server`, `tools`, `resources`, `prompts`, `subscriptions`, `tasks`, `apps`, `oauth`, `protocol`, `xaa`, `compat`, `inspector`, `mcp`, `telemetry`
* Local commands still reject `--api-key` / `--api-url` / `--project`
* Hosted `readiness start|status|list|cancel|report` stays under the root `readiness` command (leaf `--api-key`; no Cloud project link)
* Local `mcpjam mcp` protocol `serverInfo.name` is still `mcpjam` (title is now `MCPJam CLI`)

## Command paths

| 3.x                                                                      | 4.0                                                              |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------- |
| `mcpjam login` / `logout` / `whoami`                                     | `mcpjam cloud login` / `logout` / `whoami`                       |
| `mcpjam organizations`                                                   | `mcpjam cloud organizations`                                     |
| `mcpjam projects …`                                                      | `mcpjam cloud projects …`                                        |
| `mcpjam eval …`                                                          | `mcpjam cloud eval …`                                            |
| `mcpjam chat-sessions list`                                              | `mcpjam cloud sessions list`                                     |
| `mcpjam sessions search`                                                 | `mcpjam cloud sessions search`                                   |
| `mcpjam hosts …`                                                         | `mcpjam cloud clients …` (`cloud hosts` still works as an alias) |
| `mcpjam environments …`                                                  | `mcpjam cloud environments …`                                    |
| `mcpjam capabilities`                                                    | `mcpjam cloud projects capabilities`                             |
| `mcpjam projects server …`                                               | `mcpjam cloud projects servers …` (plural only)                  |
| `mcpjam personas` / `journeys` / `scenarios` / `swarms` / `user-testing` | `mcpjam cloud …` (same names)                                    |
| `mcpjam images …`                                                        | `mcpjam cloud images …`                                          |
| `mcpjam tunnel …`                                                        | `mcpjam cloud tunnel …`                                          |

## Flags

| 3.x                                          | 4.0                                                                                             |       |            |      |         |       |       |            |                            |                                                   |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------- | ----- | ---------- | ---- | ------- | ----- | ----- | ---------- | -------------------------- | ------------------------------------------------- |
| `--api-key` / `--api-url` on each Cloud leaf | Declared on `mcpjam cloud`; work before or after descendants                                    |       |            |      |         |       |       |            |                            |                                                   |
| `projects list --organization-id`            | `mcpjam cloud projects list --org`                                                              |       |            |      |         |       |       |            |                            |                                                   |
| `tunnel --id <name>`                         | `mcpjam cloud tunnel --server <name>` (`--id` still works, hidden)                              |       |            |      |         |       |       |            |                            |                                                   |
| \`eval status                                | cancel                                                                                          | judge | iterations | gate | compare | trace | steps | screenshot | video --project\` required | Optional. Same precedence as other Cloud commands |
| `sessions list` with no `--project`          | Project-scoped (link / env / automatic). Pass `--all-projects` for the old all-projects listing |       |            |      |         |       |       |            |                            |                                                   |

Project-selection precedence: `--project` > explicit `project` in `--file`/`--json` > `MCPJAM_PROJECT` > nearest `.mcpjam/project.json` > automatic (most recently updated). `MCPJAM_PROJECT_ID` does not select a Cloud CLI project.

`mcpjam cloud link` writes `.mcpjam/project.json`; use `mcpjam cloud link --remove` to remove the nearest link. `mcpjam cloud status` is zero-network.

## Local MCP identity

`mcpjam mcp` still announces protocol name `mcpjam`. The display title is `MCPJam CLI`. Instructions say this is the local testing engine, not the hosted Cloud MCP at `mcp.mcpjam.com`.


# OAuth Conformance
Source: https://docs.mcpjam.com/cli/oauth-conformance

Test your MCP server's OAuth implementation across all registration methods and protocol versions

The OAuth conformance module runs real OAuth handshakes against your MCP server — the same flows Claude Desktop, ChatGPT, and Claude Code perform — and tells you exactly where they break.

## Why you need this

* **Each MCP client exercises OAuth slightly differently.** Passing conformance means your server works for all of them.
* **OAuth bugs are silent.** A valid-looking token can be rejected by MCP authentication downstream. Users see "failed to connect" with no context. `--verify-tools` catches this.
* **3 registration strategies x 4 protocol versions x 3 auth modes** = 36 possible flow combinations. The suite runner covers the matrix in one invocation.
* **CI-ready output.** `--reporter junit-xml` for dashboards. Exit 0/1/2 for scripts.

## Quick start

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --verify-tools
```

The default `--auth-mode` is `interactive` — a browser opens for consent. Pass `--auth-mode headless` for CI against auto-consenting auth servers, or `--auth-mode client_credentials` for M2M.

**What success looks like:**

```text theme={"theme":"css-variables"}
OAuth conformance: PASSED
Server: https://your-server.com/mcp
Flow: 2025-11-25 / dcr
Summary: OAuth conformance passed for https://your-server.com/mcp (2025-11-25, dcr)
Duration: 4312ms
Steps: 12 passed steps, 0 failed steps, 0 skipped steps

Verification
listTools: PASS (8 tools)
```

## What it tests

Each conformance run walks through:

1. **Initial request** — sends an unauthenticated request to trigger a 401
2. **Discovery** — fetches resource metadata and authorization server metadata
3. **Registration** — registers the client via CIMD, DCR, or pre-registered credentials
4. **Authorization** — obtains an authorization code (interactive, headless, or client\_credentials)
5. **Token exchange** — exchanges the code for access/refresh tokens
6. **Authenticated request** — retries the MCP request with the token
7. **Post-auth verification** (optional) — connects via MCP and lists tools / calls a tool

When `--conformance-checks` is enabled, six additional negative checks run after the flow:

* **DCR redirect URI policy** — attempts dynamic client registration with a non-loopback `http://` redirect URI and expects rejection under the MCP authorization profile
* **Invalid client** — confirms the auth server rejects an unknown client ID
* **Invalid redirect at the authorization endpoint** — sends an authorization request with a mismatched `redirect_uri` and looks for rejection before the server redirects back to it
* **Invalid token** — confirms the MCP server rejects an obviously invalid bearer token with HTTP 401
* **Invalid redirect** — attempts a token request with a mismatched redirect URI to look for exact-match enforcement; this check may be skipped if the request is rejected for a different reason first
* **Token format** — validates the token response has the expected fields

## Scenarios

### I just added OAuth to my MCP server

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url http://localhost:8080/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --verify-tools
```

### CI conformance testing without a browser

```bash theme={"theme":"css-variables"}
# Option A: headless (requires auto-consenting auth server)
mcpjam oauth conformance \
  --url $MCP_SERVER_URL \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --auth-mode headless \
  --verify-tools

# Option B: client_credentials (M2M, no browser)
mcpjam oauth conformance \
  --url $MCP_SERVER_URL \
  --protocol-version 2025-11-25 \
  --registration preregistered \
  --auth-mode client_credentials \
  --client-id "$M2M_CLIENT_ID" \
  --client-secret "$M2M_CLIENT_SECRET" \
  --verify-tools
```

<Note>
  Always pass `--auth-mode` explicitly in CI. The default is `interactive`
  and will attempt to open a browser.
</Note>

### Test multiple protocol versions at once

Create a config file:

```json theme={"theme":"css-variables"}
{
  "name": "Full Protocol Matrix",
  "serverUrl": "https://your-server.com/mcp",
  "defaults": {
    "auth": { "mode": "headless" },
    "verification": { "listTools": true }
  },
  "flows": [
    { "label": "2025-03-26 / dcr", "protocolVersion": "2025-03-26", "registrationStrategy": "dcr" },
    { "label": "2025-06-18 / dcr", "protocolVersion": "2025-06-18", "registrationStrategy": "dcr" },
    { "label": "2025-11-25 / dcr", "protocolVersion": "2025-11-25", "registrationStrategy": "dcr" },
    { "label": "2025-11-25 / cimd", "protocolVersion": "2025-11-25", "registrationStrategy": "cimd" },
    { "label": "2026-07-28 / cimd", "protocolVersion": "2026-07-28", "registrationStrategy": "cimd" }
  ]
}
```

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance-suite --config ./oauth-tests.json
```

### Users report tokens work but tool calls fail

This is exactly what `--verify-call-tool` catches. OAuth can succeed (valid token issued) while the MCP layer rejects it (wrong audience, missing scope, session binding).

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --verify-call-tool your_tool_name
```

`--verify-call-tool` implies `--verify-tools`.

### Test specific scopes

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --scopes "read:tools write:tools" \
  --verify-tools
```

<Note>
  `--scopes` sets what you **request** from the auth server. The runner does
  not currently verify that granted scopes match requested scopes. When
  `--scopes` is omitted, the runner prefers the `scope=` value from the initial
  `WWW-Authenticate` challenge and falls back to `scopes_supported`.
</Note>

### Drive consent via Playwright or automation

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --print-url \
  --verify-tools
```

`--print-url` writes the consent URL to stderr (`OAUTH_CONSENT_URL: https://...`) instead of launching a browser. The local callback listener still runs — your Playwright script opens the URL, drives consent, and the CLI catches the redirect.

### Run the OAuth negative checks

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --conformance-checks \
  --verify-tools
```

This adds 6 checks after the main flow: DCR redirect URI policy, invalid client rejection, invalid redirect enforcement at the authorization endpoint, invalid bearer token rejection at the MCP server, invalid redirect enforcement at the token endpoint, and token format validation.

## Registration strategies

| Strategy        | Description                                                              | Protocol versions      |
| --------------- | ------------------------------------------------------------------------ | ---------------------- |
| `cimd`          | Client ID Metadata Document — the client publishes its identity at a URL | 2025-11-25, 2026-07-28 |
| `dcr`           | Dynamic Client Registration — the client registers itself at runtime     | All                    |
| `preregistered` | Pre-registered client ID and optional secret                             | All                    |

**When to choose what:**

* **CIMD** (2025-11-25, 2026-07-28): preferred for production when the auth server supports it — avoids mutable DCR state.
* **DCR**: the default for quick testing — works everywhere, no pre-configuration needed.
* **Preregistered**: use when the auth server doesn't support DCR, or for `client_credentials` M2M flows.

When `cimd` is used without `--client-metadata-url`, the runner falls back to mcpjam's public metadata document at `https://www.mcpjam.com/.well-known/oauth/client-metadata.json`.

## Auth modes

| Mode                    | Use case                              | User interaction |
| ----------------------- | ------------------------------------- | ---------------- |
| `interactive` (default) | Local dev — opens browser for consent | Browser popup    |
| `headless`              | CI with auto-consenting auth servers  | None             |
| `client_credentials`    | Machine-to-machine service accounts   | None             |

### Compatibility

| Registration                    | `headless` | `interactive` |       `client_credentials`       |
| ------------------------------- | :--------: | :-----------: | :------------------------------: |
| `cimd` (2025-11-25, 2026-07-28) |     yes    |      yes      |                no                |
| `dcr`                           |     yes    |      yes      |       yes (secret from DCR)      |
| `preregistered`                 |     yes    |      yes      | yes (requires `--client-secret`) |

`cimd` + `client_credentials` is rejected with a clear error: *"CIMD is a browser-based registration flow and only works with --auth-mode headless or --auth-mode interactive."*

## Post-auth verification

OAuth conformance proves a token was issued. Verification proves the token is **usable for MCP operations**.

Three silent-failure modes this catches:

1. **Token audience mismatch** — OAuth flow succeeds but `tools/list` returns 401 because `aud` doesn't match the MCP resource URL.
2. **Scope mismatch** — token issued but the MCP tool handler requires different scopes.
3. **Session binding** — some servers require a separate session init after auth.

| Flag                        | What it does                                                 |
| --------------------------- | ------------------------------------------------------------ |
| `--verify-tools`            | After OAuth, connect to the MCP server and call `tools/list` |
| `--verify-call-tool <name>` | Also call the named tool (implies `--verify-tools`)          |

## Reuse credentials safely

Use `--credentials-out <path>` when a conformance run should hand credentials to
later connected commands. The file is written with `0600` permissions and can be
passed to commands such as `tools list`, `apps conformance`, `resources list`,
and `server doctor` with `--credentials-file <path>`.

```bash theme={"theme":"css-variables"}
mcpjam oauth conformance \
  --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --credentials-out creds.json

mcpjam tools list \
  --url https://your-server.com/mcp \
  --credentials-file creds.json
```

Raw JSON output redacts OAuth secrets by default. `--credentials-out` is the
supported way to persist live tokens.

## Troubleshooting

| Step that failed                                                            | Likely cause                                                                                                                                                   | Fix                                                                                                                          |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `request_without_token` returned 200                                        | Server allows anonymous access or missing `WWW-Authenticate`                                                                                                   | Return 401 with `WWW-Authenticate: Bearer resource_metadata="..."`                                                           |
| `request_resource_metadata` 404                                             | `/.well-known/oauth-protected-resource` missing or wrong Content-Type                                                                                          | Ensure the endpoint exists and returns `application/json`                                                                    |
| `received_resource_metadata` — resource validation failed                   | Protected Resource Metadata is missing its required `resource` field, uses a non-HTTPS URL, or advertises a resource on a different origin than the server URL | Ensure your PRM document includes a `resource` field that is an HTTPS URL on the same origin as the MCP server (RFC 9728 §2) |
| `received_resource_metadata` — AS URL mismatch                              | `authorization_servers` points to a URL without AS metadata                                                                                                    | Verify the AS URL has `/.well-known/oauth-authorization-server`                                                              |
| `request_client_registration` 400                                           | DCR endpoint rejected the registration metadata                                                                                                                | Check required fields; use SDK `dynamicRegistration` overrides for custom metadata                                           |
| `received_authorization_code` — HTML login page                             | Auth server requires interactive login but you used `--auth-mode headless`                                                                                     | Switch to `--auth-mode interactive`                                                                                          |
| `received_authorization_code` — state mismatch                              | Stale popup or concurrent authorization flows                                                                                                                  | Close other browser tabs and retry                                                                                           |
| `token_request` 401 `invalid_client`                                        | Wrong `token_endpoint_auth_method` or client\_id mismatch                                                                                                      | Check DCR response for `token_endpoint_auth_method`; CIMD uses `none`                                                        |
| `authenticated_mcp_request` 401                                             | Token `aud` doesn't match MCP resource URL                                                                                                                     | Ensure your auth server sets `aud` correctly                                                                                 |
| `verify_list_tools` auth error                                              | Token works for OAuth but MCP layer rejects it — scope issue                                                                                                   | Check that the token's scope matches what tool handlers require                                                              |
| DCR + `client_credentials`: "Dynamic registration produced a public client" | DCR returned `token_endpoint_auth_method: "none"`                                                                                                              | Use `--registration preregistered` with explicit credentials instead                                                         |

## CI/CD integration

### GitHub Actions

```yaml theme={"theme":"css-variables"}
- name: OAuth Conformance
  run: |
    npx -y @mcpjam/cli@latest oauth conformance \
      --url ${{ secrets.MCP_SERVER_URL }} \
      --protocol-version 2025-11-25 \
      --registration preregistered \
      --auth-mode client_credentials \
      --client-id ${{ secrets.M2M_CLIENT_ID }} \
      --client-secret ${{ secrets.M2M_CLIENT_SECRET }} \
      --verify-tools
```

### GitLab CI (JUnit output)

```yaml theme={"theme":"css-variables"}
oauth-conformance:
  stage: test
  script:
    - npx -y @mcpjam/cli@latest oauth conformance-suite
        --config ./oauth-tests.json
        --reporter junit-xml > report.xml
  artifacts:
    reports:
      junit: report.xml
```

## All flags

### `oauth conformance`

| Flag                          | Required | Default        | Description                                                              |
| ----------------------------- | -------- | -------------- | ------------------------------------------------------------------------ |
| `--url <url>`                 | Yes      |                | MCP server URL                                                           |
| `--protocol-version <v>`      | Yes      |                | `2025-03-26`, `2025-06-18`, `2025-11-25`, or `2026-07-28`                |
| `--registration <s>`          | Yes      |                | `cimd`, `dcr`, or `preregistered`                                        |
| `--auth-mode <m>`             | No       | `interactive`  | `headless`, `interactive`, or `client_credentials`                       |
| `--client-id <id>`            | No       |                | OAuth client ID (required for `preregistered`)                           |
| `--client-secret <s>`         | No       |                | OAuth client secret                                                      |
| `--client-metadata-url <url>` | No       |                | CIMD metadata document URL                                               |
| `--redirect-url <url>`        | No       | Auto-generated | OAuth redirect URL                                                       |
| `--scopes <scopes>`           | No       |                | Space-separated scope string                                             |
| `--header <header>`           | No       |                | HTTP header `Key: Value` (repeatable)                                    |
| `--step-timeout <ms>`         | No       | `30000`        | Per-step timeout                                                         |
| `--verify-tools`              | No       |                | After OAuth, connect and list tools                                      |
| `--verify-call-tool <name>`   | No       |                | Also call the named tool (implies `--verify-tools`)                      |
| `--conformance-checks`        | No       |                | Run additional negative OAuth checks after the main flow                 |
| `--credentials-out <path>`    | No       |                | Write OAuth credentials to file (mode 0600); stdout has secrets redacted |
| `--print-url`                 | No       |                | Print consent URL to stderr instead of launching a browser               |
| `--reporter <reporter>`       | No       |                | `json-summary` or `junit-xml` CI report output                           |

<Note>
  For interactive conformance runs, a custom `--redirect-url` must still be an
  `http://localhost` or `http://127.0.0.1` loopback URL. Custom callback paths
  are supported.
</Note>

### `oauth conformance-suite`

| Flag                        | Required | Default | Description                                                                              |
| --------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------- |
| `--config <path>`           | Yes      |         | Path to JSON config file                                                                 |
| `--verify-tools`            | No       |         | Enable tool listing on all flows                                                         |
| `--verify-call-tool <name>` | No       |         | Call the named tool after listing                                                        |
| `--credentials-out <path>`  | No       |         | Write OAuth credentials from the first flow that returns credentials to file (mode 0600) |
| `--reporter <reporter>`     | No       |         | `json-summary` or `junit-xml` CI report output                                           |

<Note>
  `--verify-tools` and `--verify-call-tool` on the suite command are forced
  onto **every** flow. Per-flow `verification` entries in the config file
  cannot disable them once the CLI flag is set.
</Note>

## Related

* [OAuth Login & Debugging](/cli/oauth-login) — get tokens and debug OAuth flows
* [Server Inspection](/cli/server-inspection) — probe and diagnose before testing conformance
* [OAuth Conformance SDK Reference](/sdk/reference/oauth-conformance) — programmatic usage with `OAuthConformanceTest`


# OAuth Login & Debugging
Source: https://docs.mcpjam.com/cli/oauth-login

Authenticate with MCP servers and debug OAuth flows

The `oauth` command group handles authentication against MCP servers that require OAuth. Use `oauth login` to obtain tokens interactively, and the proxy/metadata commands to debug discovery and endpoint behavior.

## Quick start

```bash theme={"theme":"css-variables"}
# Interactive login — opens a browser for consent
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr
```

On success, the command prints credentials such as an access token, refresh token, client ID, and sometimes client secret. You can pass access tokens to other commands via `--oauth-access-token` or `--access-token`.

<Note>
  `oauth login` validates the server's Protected Resource Metadata (PRM) at the discovery step. If the PRM document is missing its required `resource` field, or if the advertised resource is on a different origin than the server URL, the login fails before any authorization request is made. Fix the PRM document on your server to resolve this. The OAuth Debugger accepts these values with a warning so you can still observe the flow.
</Note>

## Commands

### `oauth login`

Runs a full OAuth flow: discovery, registration, authorization, token exchange. Supports all three registration strategies and auth modes.

```bash theme={"theme":"css-variables"}
# DCR + interactive (most common for local dev)
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr

# CIMD + interactive (2025-11-25 and 2026-07-28)
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration cimd

# Preregistered + client_credentials (M2M, no browser)
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration preregistered \
  --auth-mode client_credentials \
  --client-id "$CLIENT_ID" \
  --client-secret "$CLIENT_SECRET"
```

<Warning>
  Prefer environment variables for `--client-secret`. Passing it inline leaks
  to shell history and process listings.
</Warning>

The `--credentials-out <path>` flag saves OAuth credentials to a file with `0600` permissions. Depending on the flow, the file may include an access token, refresh token, client ID, and client secret. Secret fields are redacted from stdout output. Use the saved file with `--credentials-file` on other commands.

```bash theme={"theme":"css-variables"}
mcpjam oauth login --url https://your-server.com/mcp \
  --credentials-out creds.json

# Then reuse across commands
mcpjam tools list --url https://your-server.com/mcp --credentials-file creds.json
mcpjam server doctor --url https://your-server.com/mcp --credentials-file creds.json
```

The `--debug-out <path>` flag captures a structured debug artifact with full HTTP traces — useful for filing bug reports or handing off to another engineer.

```bash theme={"theme":"css-variables"}
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 \
  --registration dcr \
  --debug-out oauth-debug.json
```

### `oauth metadata`

Fetch and display OAuth metadata from a URL. Useful for verifying your server's discovery endpoints are correct.

```bash theme={"theme":"css-variables"}
mcpjam oauth metadata --url https://your-server.com/.well-known/oauth-protected-resource
mcpjam oauth metadata --url https://auth.example.com/.well-known/oauth-authorization-server
```

### `oauth proxy`

Send an arbitrary HTTP request through the SSRF-hardened OAuth proxy. Useful for testing individual OAuth endpoints (registration, token, etc.) without building raw HTTP requests.

Private targets — loopback, your LAN, or a hostname that resolves to one — are reachable by default, since the CLI runs on your machine. Add `--https-only` to apply the hosted app's stricter policy instead. Neither mode will dial a link-local or cloud-metadata address, and neither sends a plaintext request to a public host.

```bash theme={"theme":"css-variables"}
# Test a DCR registration
mcpjam oauth proxy \
  --url https://auth.example.com/register \
  --method POST \
  --header "Content-Type: application/json" \
  --body '{"redirect_uris":["http://localhost:8080/callback"],"client_name":"test"}'

# Fetch an endpoint
mcpjam oauth proxy --url https://auth.example.com/v1/oauth2/token --method GET
```

### `oauth debug-proxy`

Same as `oauth proxy` but routes through the debug proxy path. Use when testing endpoints that behave differently for debug vs. production requests.

## Login flags

| Flag                          | Required | Default        | Description                                                                           |
| ----------------------------- | -------- | -------------- | ------------------------------------------------------------------------------------- |
| `--url <url>`                 | Yes      |                | MCP server URL                                                                        |
| `--protocol-version <v>`      | Yes      |                | `2025-03-26`, `2025-06-18`, `2025-11-25`, or `2026-07-28`                             |
| `--registration <s>`          | Yes      |                | `cimd`, `dcr`, or `preregistered`                                                     |
| `--auth-mode <m>`             | No       | `interactive`  | `headless`, `interactive`, or `client_credentials`                                    |
| `--client-id <id>`            | No       |                | OAuth client ID (required for `preregistered`)                                        |
| `--client-secret <s>`         | No       |                | OAuth client secret                                                                   |
| `--client-metadata-url <url>` | No       |                | CIMD metadata document URL                                                            |
| `--redirect-url <url>`        | No       | Auto-generated | OAuth redirect URL                                                                    |
| `--scopes <scopes>`           | No       |                | Space-separated scope string                                                          |
| `--header <header>`           | No       |                | HTTP header `Key: Value` (repeatable)                                                 |
| `--step-timeout <ms>`         | No       | `30000`        | Per-step timeout                                                                      |
| `--verify-tools`              | No       |                | After login, connect and list tools                                                   |
| `--verify-call-tool <name>`   | No       |                | After listing, also call a named tool                                                 |
| `--credentials-out <path>`    | No       |                | Write OAuth credentials to file (mode 0600); stdout output has secret fields redacted |
| `--debug-out <path>`          | No       |                | Write debug artifact to file                                                          |

<Note>
  For interactive login, a custom `--redirect-url` must still be an
  `http://localhost` or `http://127.0.0.1` loopback URL. Custom callback paths
  are supported.
</Note>

## Using tokens from login

### Credentials file (interactive & agent workflows)

For interactive or agent workflows where you login once and run multiple commands, save credentials to a file:

```bash theme={"theme":"css-variables"}
# Login and save credentials
mcpjam oauth login --url https://your-server.com/mcp --credentials-out creds.json

# Reuse across commands
mcpjam tools list --url https://your-server.com/mcp --credentials-file creds.json
mcpjam server doctor --url https://your-server.com/mcp --credentials-file creds.json
mcpjam protocol conformance --url https://your-server.com/mcp --credentials-file creds.json
```

The credentials file includes the server URL and any reusable OAuth credentials returned by the login flow. The CLI validates that the file's server URL matches the target and checks token expiry before use.

Connected commands such as `tools list`, `resources list`, `prompts list`, and `server doctor` can use refresh-token credentials from the file when the saved access token is expired. `protocol conformance` and `server probe` only read a non-expired access token from the file; if the access token is expired, run `oauth login --credentials-out` again or pass a fresh token directly.

<Note>
  `--credentials-file` cannot be combined with `--access-token`, `--oauth-access-token`, `--refresh-token`, `--client-id`, or `--client-secret`. The CLI rejects conflicting auth sources upfront.
</Note>

### Direct token passing (CI/CD & M2M)

When you already have a token from an environment variable, a secret store, or a `client_credentials` flow, pass it directly:

```bash theme={"theme":"css-variables"}
# From an environment variable or secret store
mcpjam tools list --url https://your-server.com/mcp --access-token "$TOKEN"
mcpjam server doctor --url https://your-server.com/mcp --oauth-access-token "$TOKEN"

# M2M: preregistered client_credentials — token comes back on stdout
TOKEN=$(mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 --registration preregistered \
  --auth-mode client_credentials \
  --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" \
  --format json | jq -r '.credentials.accessToken')

mcpjam tools list --url https://your-server.com/mcp --access-token "$TOKEN"
```

You can also use `--refresh-token` with `--client-id` (and optionally `--client-secret`) for commands that support automatic token refresh.


# CLI Overview
Source: https://docs.mcpjam.com/cli/overview

Local MCP testing and mcpjam cloud: probing, debugging, OAuth, conformance, and hosted evals from your terminal

The `mcpjam` CLI is two invocations. Local MCP testing stays at the top level: inspect, debug, and test MCP servers from the command line or CI (connectivity, OAuth, MCP Apps, tools/resources/prompts, protocol conformance). Most local commands are one-shot; `mcpjam mcp` and `subscriptions listen` keep a process open. Account-bound commands live under `mcpjam cloud` and talk to your MCPJam account. There are no compatibility aliases for the old Cloud paths — see [Migrating to CLI 4.0](/cli/migration).

## Recommended: Use with Agent Skills

The easiest way to use the CLI is through the [MCPJam skill](https://github.com/MCPJam/inspector/tree/main/skills/mcp-inspector), which gives your agent full context on every command and workflow.

```bash theme={"theme":"css-variables"}
npx skills add mcpjam/inspector --skill mcp-inspector
```

To convert existing promptfoo, pytest, Jest, or CSV tests, use the [MCP eval import skill](https://github.com/MCPJam/inspector/tree/main/skills/mcpjam-eval-import).

```bash theme={"theme":"css-variables"}
npx skills add mcpjam/inspector --skill mcpjam-eval-import
```

To drive MCPJam's hosted eval tools — launch a run, poll it to a verdict, and triage a failure down to the step — use the [run MCPJam evals skill](https://github.com/MCPJam/inspector/tree/main/skills/run-mcpjam-evals). This one is about running suites that already exist, not writing them.

```bash theme={"theme":"css-variables"}
npx skills add mcpjam/inspector --skill run-mcpjam-evals
```

Once installed, your agent will know how and when to invoke `mcpjam` commands automatically.

## Install

```bash theme={"theme":"css-variables"}
# Global (gives you the mcpjam command)
npm i -g @mcpjam/cli

# Or run without installing
npx -y @mcpjam/cli@latest --help
```

<Note>
  All examples in these docs use `mcpjam` directly. If you installed via `npx`,
  replace `mcpjam` with `npx -y @mcpjam/cli@latest`.
</Note>

## Command groups

### Local MCP testing

These stay at the program root. They reject `--api-key` / `--api-url` / `--project`.

| Group                                                   | Purpose                                                                                                                                                      | Key commands                                                                  |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| [`server`](/cli/server-inspection)                      | Triage connectivity and capabilities                                                                                                                         | `probe`, `doctor`, `info`, `validate`, `ping`, `capabilities`, `export`       |
| [`tools`](/cli/tools-resources-prompts)                 | Exercise the tool surface                                                                                                                                    | `list`, `call`                                                                |
| [`resources`](/cli/tools-resources-prompts)             | Read resources and templates                                                                                                                                 | `list`, `read`, `templates`                                                   |
| [`prompts`](/cli/tools-resources-prompts)               | Fetch prompts                                                                                                                                                | `list`, `get`                                                                 |
| [`subscriptions`](/cli/reference)                       | Open a long-lived MCP subscription and stream notifications                                                                                                  | `listen`                                                                      |
| [`tasks`](/cli/tasks)                                   | Drive MCP Tasks and validate the tasks wire                                                                                                                  | `list`, `get`, `cancel`, `conformance`                                        |
| [`apps`](/cli/apps-conformance)                         | Validate MCP Apps metadata and resource wiring                                                                                                               | `conformance`, `conformance-suite`                                            |
| [`oauth`](/cli/oauth-conformance)                       | Test OAuth flows and conformance against an MCP server                                                                                                       | `conformance`, `conformance-suite`, `login`, `metadata`, `proxy`              |
| [`protocol`](/cli/reference)                            | MCP protocol conformance checks                                                                                                                              | `conformance`, `conformance-suite`                                            |
| [`readiness`](/cli/reference#readiness-hosted-commands) | Grade a server against a publisher's directory. Hosted `start` / `status` / `list` / `cancel` / `report` stay here (leaf `--api-key`; no Cloud project link) | `check`, `start claude`, `start openai`, `status`, `list`, `cancel`, `report` |
| [`xaa`](/cli/xaa)                                       | Run the Cross-App Access (ID-JAG) debugger against an MCP server                                                                                             | `run`                                                                         |
| [`compat`](/cli/reference#compat-command)               | Check whether a server's tools and widgets work on each AI host                                                                                              | `compat`                                                                      |
| [`inspector`](/cli/reference#inspector-commands)        | Start, open, or stop the local Inspector from the CLI                                                                                                        | `open`, `start`, `stop`                                                       |
| [`mcp`](/cli/mcp-server)                                | Run MCPJam as a stdio MCP server for agents (Claude Desktop, Claude Code, Cursor, ...)                                                                       | `mcp`                                                                         |
| [`telemetry`](/cli/telemetry)                           | Inspect and configure anonymous CLI telemetry                                                                                                                | `status`, `disable`, `enable`                                                 |

### MCPJam Cloud

Account-bound commands. Declare `--api-key` / `--api-url` on `mcpjam cloud` (before or after descendants). `mcpjam oauth login` is MCP-server OAuth; `mcpjam cloud login` is your MCPJam account.

| Group                                                                      | Purpose                                                                                                                                                              | Key commands                                                                                      |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [`cloud`](/cli/reference#cloud-workspace)                                  | Account, project link, and workspace                                                                                                                                 | `login`, `logout`, `whoami`, `link`, `status`, `organizations`, `projects`, `sessions`            |
| [`cloud eval`](/cli/reference#cloud-eval-commands)                         | Author and run eval suites in your hosted MCPJam projects                                                                                                            | `create`, `list`, `run`, `status`                                                                 |
| [`cloud tunnel`](/cli/reference#cloud-tunnel)                              | Expose a local MCP server through a public MCPJam tunnel URL, registered as a server in your project                                                                 | `tunnel --server <name> <url>`, `tunnel --server <name> -- <command>`                             |
| [`cloud clients`](/cli/reference#cloud-clients-commands)                   | Manage clients in your hosted MCPJam projects (`cloud hosts` is a deprecated alias)                                                                                  | `list`, `get`, `create`, `update`, `servers`, `duplicate`, `delete`, `templates`                  |
| [`cloud environments`](/cli/reference#cloud-environments-commands)         | Manage project environments — the host + servers + pinned skills/plugins bundles eval suites and journeys run against                                                | `list`, `get`, `resolve`, `create`, `update`, `archive`, `restore`                                |
| [`cloud images`](/cli/reference#cloud-images-commands)                     | Manage custom Computer sandbox images (blueprints)                                                                                                                   | `list`, `get`, `create`, `validate`, `edit`, `build`, `logs`, `use`, `reset`, `promote`, `delete` |
| [`cloud skills`](/cli/reference#cloud-skills-commands)                     | List and read Cloud Skills in your hosted MCPJam projects — discover the skill IDs that `--compose-skill` and `--skill` require (read-only; authoring is in the app) | `list`, `get`                                                                                     |
| [`cloud projects servers`](/cli/reference#cloud-projects-servers-commands) | Connect MCP servers to a project and manage pending connection requests                                                                                              | `connect`, `connect-status`, `connect-cancel`                                                     |

Also under `mcpjam cloud`: `journeys`, `scenarios`, `personas`, `swarms`, `user-testing`. Full flag tables are in the [command reference](/cli/reference).

## Cloud credentials and API URL

`--api-key` and `--api-url` are declared on `mcpjam cloud` and work before or after descendants (`mcpjam cloud --api-key sk_… eval list` and `mcpjam cloud eval list --api-key sk_…` are the same). Local commands, including hosted `readiness`, reject those flags at the program root. Hosted `readiness` still takes leaf `--api-key`.

Credential precedence for Cloud commands:

1. `--api-key` — must be an `sk_` key. A `mcpjam_…` legacy key is a usage error.
2. `MCPJAM_API_KEY` — same `sk_` rule. A legacy value in this env var is ignored (with a warning) so SDK eval reporting can still use it.
3. Stored OAuth from `mcpjam cloud login`
4. Missing — Cloud commands fail with login guidance before they hit the network (`mcpjam cloud status` reports `missing` instead)

API URL precedence:

1. `--api-url`
2. `MCPJAM_API_URL`
3. The `apiUrl` stored with the OAuth login (only when that login is the credential)
4. `https://app.mcpjam.com/api/v1`

An explicit `--api-url` or `MCPJAM_API_URL` that is not an `http(s)` URL is a usage error for commands that talk to the network (exit `2`). `mcpjam cloud status` uses the same checks but stays network-free: it still emits the complete JSON report, sets `ok: false`, includes `credential.valid` / `deployment.valid` plus an `error` string, and exits `1`. Missing credentials are informational (`credential.valid: null`) and successful.

Successful Cloud commands print an audience line to stderr (`Using MCPJam Cloud as … · project: … · <apiUrl>`). `--quiet` suppresses it. Machine-readable JSON stays on stdout. `cloud status` redacts API keys.

## Cloud project scope

Most Cloud commands pick one project:

1. `--project <id-or-name>`
2. An explicit `project` field in `--file` / `--json`
3. `MCPJAM_PROJECT`
4. The nearest valid `.mcpjam/project.json`
5. Automatic — the most recently updated project you can see

Empty `--project` or `MCPJAM_PROJECT` is a usage error. `MCPJAM_PROJECT_ID` does **not** select a Cloud CLI project (it is SDK eval reporting only).

```bash theme={"theme":"css-variables"}
mcpjam cloud login
mcpjam cloud link                  # writes .mcpjam/project.json (no secrets)
mcpjam cloud status                # zero-network: credential, deployment, selector
mcpjam cloud eval list             # uses the link
mcpjam cloud link --remove
```

`mcpjam cloud eval` run-inspection commands (`status`, `cancel`, `judge`, `iterations`, `gate`, `compare`, `trace`, `steps`, `screenshot`, `video`) follow the same rule — `--project` is optional. `mcpjam cloud sessions list` does too; `--all-projects` restores a cross-project listing and cannot be combined with `--project`. Hosted `readiness` never reads the project link.

## What `--host` means

`--host` is not one flag. The meaning depends on the command:

| Where                                                    | What it selects                                                                                                                                  |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `server`, `tools`, `resources`, `prompts`                | An **AI client** (`claude`, `chatgpt`, `cursor`, …). The CLI sends that host's `clientInfo`, capabilities, and protocol version in `initialize`. |
| `compat --host`                                          | Which AI clients to report. Repeatable; default is all.                                                                                          |
| `cloud eval --host`, `cloud journeys`, `cloud scenarios` | A **saved project Host** (`<id-or-name>`), the client config an eval or journey is stamped with.                                                 |
| `cloud environments --host-id`                           | The project Host id the environment runs as. Distinct from eval's `--host`.                                                                      |

`--host` on a local command is mutually exclusive with `--client-capabilities`. A Cloud `--host` is mutually exclusive with `--server` / `--environment` when those already supply a closed server set.

## Global flags

These flags apply to every command. Cloud credentials `--api-key` / `--api-url` are declared on `mcpjam cloud`, not the program root.

| Flag                | Default                           | Description                                                                |
| ------------------- | --------------------------------- | -------------------------------------------------------------------------- |
| `--timeout <ms>`    | `30000`                           | Request timeout in milliseconds                                            |
| `--rpc`             | off                               | Include raw JSON-RPC request/response logs in JSON output under `_rpcLogs` |
| `--quiet`           | off                               | Suppress non-result progress output on stderr                              |
| `--no-telemetry`    | off                               | Disable anonymous telemetry for this invocation                            |
| `--format <format>` | `human` on TTY, `json` when piped | Raw output format (`json` or `human`)                                      |
| `-v, --version`     |                                   | Print the CLI version                                                      |

## Output formats

The CLI auto-detects whether stdout is a terminal:

* **Interactive terminal** — defaults to `--format human`. For most commands this is pretty-printed JSON; `server doctor` and the OAuth conformance commands provide dedicated human-readable summaries.
* **Piped or redirected** (CI, `| jq`, agent invocation) — defaults to `--format json`, the full structured result.
* **Conformance in CI** — `protocol conformance`, `protocol conformance-suite`, `oauth conformance`, `oauth conformance-suite`, `apps conformance`, and `apps conformance-suite` support `--reporter junit-xml` and `--reporter json-summary`.
* **Explicit `--format`** always wins over the auto-detected default.

**For agents**: raw JSON is the source of truth. Human format is a lossy presentation layer. If you're parsing output programmatically, pass `--quiet --format json`.

JSON-valued flags accept inline JSON, `@path`, or `-` for stdin. `tools call` also accepts `--tool-args-stdin` as a shorthand for `--tool-args -`. Use files or stdin for large payloads to avoid shell escaping issues:

```bash theme={"theme":"css-variables"}
echo '{"key":"value"}' | mcpjam tools call --url $URL --access-token $TOKEN \
  --tool-name my_tool --tool-args - --quiet --format json
```

## Connecting to servers

The CLI supports two transport modes, selected by which flags you pass:

* `--url ...` selects HTTP.
* `--command ...` selects stdio.
* `--transport http|stdio` is optional and acts as an explicit override/validator when you want the CLI to reject mismatched flags early.

### HTTP (Streamable HTTP / SSE)

```bash theme={"theme":"css-variables"}
mcpjam server doctor --url https://your-server.com/mcp
```

Add auth when needed:

```bash theme={"theme":"css-variables"}
# Static bearer token
mcpjam server doctor --url https://your-server.com/mcp --access-token $TOKEN

# OAuth tokens from a prior login
mcpjam server doctor --url https://your-server.com/mcp --oauth-access-token $TOKEN

# Custom headers
mcpjam server doctor --url https://your-server.com/mcp --header "X-API-Key: $KEY"
```

### Stdio (local subprocess)

```bash theme={"theme":"css-variables"}
mcpjam server doctor --command node --args server.js --cwd /path/to/project -e API_KEY=$KEY
```

Stdio child processes inherit the parent shell environment by default. Use
`-e/--env` to add values or override inherited ones for the spawned server.
Structured debug artifacts only record the explicit env keys you pass through
`-e/--env`; inherited shell variables are not enumerated.

<Note>
  `oauth ...` and `protocol ...` are HTTP-only command groups. They do
  not accept stdio targets.
</Note>

## Update notifications

After a successful command, the CLI checks whether a newer version of `@mcpjam/cli` is available and prints a notice to stderr when one is found:

```
Update available: @mcpjam/cli 1.2.0 -> 1.3.0
Run npm install -g @mcpjam/cli to update
```

The check is non-blocking: version data is fetched in a detached background process and cached locally for 24 hours, so it never slows down your commands. The notice is written to stderr only and never touches stdout or JSON output, so it is safe for both interactive terminals and AI agents (Claude Code, Cursor, etc.) that capture stderr. It is suppressed automatically in CI.

To opt out, set one of these environment variables before running any command:

| Variable                   | Effect                                           |
| -------------------------- | ------------------------------------------------ |
| `MCPJAM_NO_UPDATE_CHECK=1` | Disable update checks for `@mcpjam/cli`          |
| `NO_UPDATE_NOTIFIER=1`     | Disable update checks (conventional npm opt-out) |
| `CI=true`                  | Automatically suppressed in CI environments      |

## Exit codes

| Code | Meaning                                                                         |
| ---- | ------------------------------------------------------------------------------- |
| `0`  | Success / all checks passed                                                     |
| `1`  | Command ran but reported a failure (e.g., server unhealthy, conformance failed) |
| `2`  | Invalid arguments or configuration                                              |

## Quick triage workflow

The fastest path from "I have a server URL" to "I know what's wrong":

```bash theme={"theme":"css-variables"}
# 1. One-shot health check (probe + connect + sweep)
mcpjam server doctor --url https://your-server.com/mcp

# 2. If oauth_required: get a token
mcpjam oauth login --url https://your-server.com/mcp \
  --protocol-version 2025-11-25 --registration dcr

# 3. Re-run doctor with the token
mcpjam server doctor --url https://your-server.com/mcp --oauth-access-token $TOKEN

# 4. Exercise tools directly
mcpjam tools list --url https://your-server.com/mcp --access-token $TOKEN
mcpjam tools call --url https://your-server.com/mcp --access-token $TOKEN \
  --tool-name my_tool --tool-args @params.json --quiet --format json

# 5. If the server exposes MCP Apps, validate the ui:// surface
mcpjam apps conformance --url https://your-server.com/mcp --access-token $TOKEN

# 6. Render one UI-capable tool result in Inspector's App Builder.
# --ui opens Inspector by default in a TTY; add --no-open if browser automation already opened it.
mcpjam tools call --url https://your-server.com/mcp --access-token $TOKEN \
  --tool-name my_app_tool --tool-args @params.json --ui --quiet --format json
```

## Use from an MCP client instead of a shell

If your agent speaks MCP but can't run shell commands (Claude Desktop, chat clients), run the same testing engine as a stdio MCP server:

```bash theme={"theme":"css-variables"}
npx -y @mcpjam/cli@latest mcp
```

Connections to servers under test stay open between tool calls, so agents can also observe notifications and session behavior that one-shot CLI commands can't. See [MCPJam as an MCP server](/cli/mcp-server).

## What's next

* [MCPJam as an MCP server](/cli/mcp-server) — give MCP clients the testing engine over stdio
* [Server inspection](/cli/server-inspection) — probe, doctor, and diagnostics
* [OAuth conformance](/cli/oauth-conformance) — test your OAuth implementation
* [OAuth login](/cli/oauth-login) — authenticate and debug OAuth flows
* [MCP Apps conformance](/cli/apps-conformance) — validate `_meta.ui.resourceUri` and `ui://` resource wiring
* [Tools, resources & prompts](/cli/tools-resources-prompts) — exercise the connected surface
* [Migrating to CLI 4.0](/cli/migration) — command-path and flag changes under `mcpjam cloud`
* [Full command reference](/cli/reference) — every flag for every command


# Command Reference
Source: https://docs.mcpjam.com/cli/reference

Complete flag reference for every mcpjam CLI command

Complete flag tables for every `mcpjam` command. For guides and recipes, see the individual command pages.

Account-bound Cloud commands live under `mcpjam cloud`. Local MCP testing stays at the top level (`mcpjam server`, `mcpjam oauth login`, …). Credential flags `--api-key` / `--api-url` / `--api-header` are declared on `mcpjam cloud` and work before or after descendants. Hosted `readiness` stays at the root and still takes leaf `--api-key`. See [Migrating to CLI 4.0](/cli/migration).

## Global flags

| Flag                | Default                           | Description                                               |
| ------------------- | --------------------------------- | --------------------------------------------------------- |
| `--timeout <ms>`    | `30000`                           | Request timeout in milliseconds                           |
| `--rpc`             | off                               | Include raw JSON-RPC logs in JSON output under `_rpcLogs` |
| `--quiet`           | off                               | Suppress non-result progress output on stderr             |
| `--no-telemetry`    | off                               | Disable anonymous telemetry for this invocation           |
| `--format <format>` | `human` on TTY, `json` when piped | Raw output format (`json` or `human`)                     |
| `-v, --version`     |                                   | Print the CLI version                                     |

***

## `server` commands

All server commands accept the shared connection flags below, plus command-specific options.

### Shared connection flags

| Flag                           | Description                                                                                                            |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `--transport <transport>`      | Explicit transport type (`http` or `stdio`)                                                                            |
| `--url <url>`                  | HTTP MCP server URL                                                                                                    |
| `--access-token <token>`       | Bearer access token                                                                                                    |
| `--oauth-access-token <token>` | OAuth bearer access token                                                                                              |
| `--refresh-token <token>`      | OAuth refresh token                                                                                                    |
| `--client-id <id>`             | OAuth client ID (with `--refresh-token`)                                                                               |
| `--client-secret <secret>`     | OAuth client secret (with `--refresh-token`)                                                                           |
| `--credentials-file <path>`    | Load OAuth credentials from a file created by `oauth login --credentials-out` or `oauth conformance --credentials-out` |
| `--header <header>`            | HTTP header `Key: Value` (repeatable)                                                                                  |
| `--client-capabilities <json>` | Client capabilities as inline JSON, `@path`, or `-` for stdin                                                          |
| `--command <command>`          | Stdio server command                                                                                                   |
| `--args <arg...>`              | Preferred stdio command arguments                                                                                      |
| `--command-args <arg>`         | Legacy stdio command argument (repeatable)                                                                             |
| `-e, --env <env...>`           | Stdio environment `KEY=VALUE` values                                                                                   |
| `--cwd <path>`                 | Working directory for the stdio child process                                                                          |

Transport selection is inferred from `--url` vs `--command` when
`--transport` is omitted. Use `--transport http|stdio` when you want an
explicit validation step.

For stdio targets, child processes inherit the parent shell environment by
default. `-e/--env` adds or overrides child env values, and structured debug
artifacts only record the explicit env keys you passed on the command line.

`--credentials-file` cannot be combined with individual token flags
(`--access-token`, `--oauth-access-token`, `--refresh-token`, `--client-id`,
`--client-secret`). The CLI rejects conflicting auth sources upfront.

### `server probe`

No additional flags beyond shared connection flags.

### `server doctor`

| Flag           | Description                              |
| -------------- | ---------------------------------------- |
| `--out <path>` | Write the doctor JSON artifact to a file |

### `server info`

No additional flags.

### `server validate`

No additional flags.

### `server ping`

No additional flags.

### `server capabilities`

No additional flags.

### `server export`

No additional flags.

***

## `tools` commands

### `tools list`

Uses shared connection flags, plus:

| Flag                 | Description                                                                                                                                                                                                                                    |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`        | Connect as this host (e.g. `claude`, `chatgpt`, `cursor`) — sends its `clientInfo`, `clientCapabilities`, and protocol version in `initialize`, and hides app-only tools its model can't see. Mutually exclusive with `--client-capabilities`. |
| `--cursor <cursor>`  | Pagination cursor                                                                                                                                                                                                                              |
| `--model-id <model>` | Model ID used for token counting                                                                                                                                                                                                               |

When `--host` is set, the output includes a `host` field and a `toolsDroppedVisibility` count. Hosts that opt out of visibility filtering (e.g. `cursor`) keep all tools.

### `tools call`

| Flag                    | Description                                                                                                                                                                                                     |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`           | Connect as this host — sends its identity/capabilities/protocol in `initialize`, and rejects app-only tools the host's model can't call. Mutually exclusive with `--client-capabilities`.                       |
| `--tool-name <name>`    | Name of the tool to call                                                                                                                                                                                        |
| `--name <name>`         | Legacy alias for `--tool-name`                                                                                                                                                                                  |
| `--interactive`         | Drive the `input_required` multi-round-trip loop: render embedded elicitations to the terminal and collect responses from stdin. Cannot be combined with `--tool-args -`, `--tool-args-stdin`, or `--params -`. |
| `--yes`                 | With `--interactive`, decline every embedded input request without prompting (non-interactive mode).                                                                                                            |
| `--tool-args <json>`    | Tool arguments as inline JSON, `@path`, or `-` for stdin                                                                                                                                                        |
| `--params <json>`       | Legacy alias for `--tool-args`                                                                                                                                                                                  |
| `--tool-args-stdin`     | Read tool arguments JSON from stdin                                                                                                                                                                             |
| `--validate-response`   | Validate the MCP tool-call envelope returned by the server                                                                                                                                                      |
| `--expect-success`      | Fail when the tool result reports `isError`                                                                                                                                                                     |
| `--reporter <reporter>` | `json-summary`, `junit-xml`, or `html` validation report output                                                                                                                                                 |
| `--debug-out <path>`    | Write debug artifact to file                                                                                                                                                                                    |
| `--ui`                  | Attach to Inspector and render the completed tool result in App Builder; opens a browser by default in a TTY                                                                                                    |
| `--require-render`      | Treat skipped Inspector renders as errors (requires `--ui`)                                                                                                                                                     |
| `--open`                | Open Inspector in the system browser before rendering (default with `--ui` in a TTY)                                                                                                                            |
| `--no-open`             | Start/use Inspector without opening a system browser                                                                                                                                                            |
| `--attach-only`         | Require an already-running Inspector browser client; do not start or open Inspector                                                                                                                             |
| `--inspector-url <url>` | Local Inspector backend/API base URL                                                                                                                                                                            |
| `--frontend-url <url>`  | Inspector frontend/browser base URL; skips frontend discovery                                                                                                                                                   |
| `--server-name <name>`  | Server name to use inside Inspector                                                                                                                                                                             |
| `--protocol <protocol>` | Render protocol: `mcp-apps` or `openai-sdk`                                                                                                                                                                     |
| `--device <device>`     | Render device: `mobile`, `tablet`, `desktop`, or `custom`                                                                                                                                                       |
| `--theme <theme>`       | Render theme: `light` or `dark`                                                                                                                                                                                 |
| `--locale <locale>`     | Render locale                                                                                                                                                                                                   |
| `--time-zone <iana>`    | Render IANA timezone                                                                                                                                                                                            |

Plus shared connection flags.

Without `--ui`, `tools call` returns the raw tool result. When that result is a JSON object (the usual MCP `CallToolResult`), the default JSON output also includes `_durationMs` — wall-clock milliseconds spent in the tool call itself, excluding connection setup and the `--host` visibility probe, so it measures the same window as the `durationMs` returned by `POST /v1/projects/{projectId}/servers/{serverId}/tools/call`. Arrays and non-objects are left unchanged so a list-shaped payload is not rewritten. `--reporter` keeps its own `durationMs`, which is end-to-end (connection setup plus the call, stopped before validation, Inspector render, and debug I/O), and does not get `_durationMs`. With `--ui`, it opens Inspector by default in a TTY and returns a compact envelope with `result`, `inspectorBrowserUrl`, and `inspectorRender` status. Object-shaped `result` values carry `_durationMs`; the envelope root does not. `inspectorRender.status` is `rendered` when Inspector accepted the render, `skipped` when the tool succeeded but Inspector had no active browser client, an unsatisfied render precondition, or a render timeout, and `error` for non-recoverable render command failures. `inspectorRender.remediation` is always present and is one of `open_browser`, `retry`, `reconnect_server`, or `none`. Skipped renders are emitted as a stable root `warning` plus `inspectorRender.warning`, both with the shape `{ code, message, remediation, browserUrl?, hasActiveClient?, inspectorStarted? }`. Stable skipped-render codes are `no_active_client`, `timeout`, `disconnected_server`, and `unsupported_in_mode`. Skipped renders keep the tool-call exit code unless `--require-render` is set; tool failures, validation failures, non-skippable render command errors, and `--require-render` skipped renders all exit nonzero. `--attach-only` is an exception to the skipped-render rule for `no_active_client`: by default a missing browser client yields `inspectorRender.status = "skipped"` with `inspectorRender.remediation = "open_browser"`, but when `--attach-only` is set, `no_active_client` is treated as non-skippable, surfaces as a root `error` (not a downgraded `warning`), and exits nonzero like other non-skippable render failures. `--inspector-url` points to the Inspector backend/API; pass `--frontend-url` when you already know the browser/client URL and want to skip health-advertised frontend checks and local dev port discovery. Use `--no-open` when browser automation already opened `inspectorBrowserUrl`; use `--attach-only` when startup, browser opening, and discovery should all be disallowed. Default non-TTY `--ui` runs do not open a browser unless `--open` is passed. When `--open` is in effect (default in TTYs, opt-in elsewhere), the App Builder URL and the initial browser-client wait progress are emitted to stderr unless `--quiet` is set, regardless of whether stderr is a TTY; only the elapsed-seconds heartbeat is gated on stderr being a TTY. The Inspector path injects the completed tool result through `renderToolResult`; it does not call the tool a second time. Fresh tabs do not hydrate the injected render state; use the active Inspector client that received the render. Use `--debug-out` for the full render envelope including params and command responses. `--ui` cannot be combined with `--reporter`.

### Reading `tools call --ui` output as an agent

Treat the tool result and the Inspector render as separate outcomes. An exit code of `0` means the tool call succeeded and no hard render error occurred; it does not, by itself, prove the UI rendered. Confirm UI delivery with `inspectorRender.status === "rendered"`. If `inspectorRender.status === "skipped"`, branch on `inspectorRender.remediation` or the stable root `warning.code`. If `--require-render` is set, the same skipped-render issue moves from root `warning` to root `error` and the command exits with code `1`.

```json theme={"theme":"css-variables"}
{
  "success": true,
  "command": "tools call",
  "inspectorUi": true,
  "inspectorBrowserUrl": "http://127.0.0.1:6274/#app-builder",
  "result": {
    "content": [{ "type": "text", "text": "view created" }],
    "_durationMs": 42
  },
  "inspectorRender": {
    "status": "skipped",
    "remediation": "open_browser",
    "mode": "active-client",
    "urlHydratesRender": false,
    "browserUrl": "http://127.0.0.1:6274/#app-builder",
    "hasActiveClient": false,
    "inspectorStarted": false,
    "warning": {
      "code": "no_active_client",
      "message": "Inspector has no active browser client. Open the Inspector App Builder URL in your browser, then rerun `tools call --ui`; or pass `--open` to let the CLI open a system browser.",
      "remediation": "open_browser",
      "browserUrl": "http://127.0.0.1:6274/#app-builder",
      "hasActiveClient": false,
      "inspectorStarted": false
    }
  },
  "warning": {
    "code": "no_active_client",
    "message": "Inspector has no active browser client. Open the Inspector App Builder URL in your browser, then rerun `tools call --ui`; or pass `--open` to let the CLI open a system browser.",
    "remediation": "open_browser",
    "browserUrl": "http://127.0.0.1:6274/#app-builder",
    "hasActiveClient": false,
    "inspectorStarted": false
  }
}
```

***

## `resources` commands

### `resources list`

Uses shared connection flags, plus:

| Flag                | Description                                                                                                                       |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`       | Connect as this host — sends its identity/capabilities/protocol in `initialize`. Mutually exclusive with `--client-capabilities`. |
| `--cursor <cursor>` | Pagination cursor                                                                                                                 |

### `resources read`

| Flag                   | Description                                                                                                                       |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`          | Connect as this host — sends its identity/capabilities/protocol in `initialize`. Mutually exclusive with `--client-capabilities`. |
| `--resource-uri <uri>` | URI of the resource to read                                                                                                       |
| `--uri <uri>`          | Legacy alias for `--resource-uri`                                                                                                 |
| `--interactive`        | Drive the `input_required` multi-round-trip loop: render embedded elicitations to the terminal and collect responses from stdin.  |
| `--yes`                | With `--interactive`, decline every embedded input request without prompting (non-interactive mode).                              |

Plus shared connection flags.

### `resources templates`

Uses shared connection flags, plus:

| Flag                | Description                                                                                                                       |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`       | Connect as this host — sends its identity/capabilities/protocol in `initialize`. Mutually exclusive with `--client-capabilities`. |
| `--cursor <cursor>` | Pagination cursor                                                                                                                 |

***

## `prompts` commands

### `prompts list`

Uses shared connection flags, plus:

| Flag                | Description                                                                                                                       |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`       | Connect as this host — sends its identity/capabilities/protocol in `initialize`. Mutually exclusive with `--client-capabilities`. |
| `--cursor <cursor>` | Pagination cursor                                                                                                                 |

### `prompts get`

| Flag                   | Description                                                                                                                                                                 |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>`          | Connect as this host — sends its identity/capabilities/protocol in `initialize`. Mutually exclusive with `--client-capabilities`.                                           |
| `--prompt-name <name>` | Name of the prompt                                                                                                                                                          |
| `--name <name>`        | Legacy alias for `--prompt-name`                                                                                                                                            |
| `--interactive`        | Drive the `input_required` multi-round-trip loop: render embedded elicitations to the terminal and collect responses from stdin. Cannot be combined with `--prompt-args -`. |
| `--yes`                | With `--interactive`, decline every embedded input request without prompting (non-interactive mode).                                                                        |
| `--prompt-args <json>` | Prompt arguments as inline JSON, `@path`, or `-` for stdin                                                                                                                  |

Plus shared connection flags.

***

## `xaa` commands

### `xaa run`

Run the Cross-App Access (ID-JAG) debugger: self-issue an ID-JAG, redeem it at the target authorization server (RFC 7523), and call the MCP server with the resulting access token. See the [XAA Debugger guide](/cli/xaa) for the trust setup, registration strategies, and CI usage.

| Flag                                    | Required | Default                  | Description                                                                                                                                                                                      |
| --------------------------------------- | -------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--url <url>`                           | Yes      |                          | Target MCP server URL (the protected resource)                                                                                                                                                   |
| `--issuer-base-url <url>`               | Yes      |                          | Origin the local mock IdP issues from. It must publish the CLI's local signing key — typically a running [local inspector](/cli/xaa#before-you-run-make-the-issuer-reachable), not your real IdP |
| `--sub <subject>`                       | Yes      |                          | Simulated end-user subject identifier                                                                                                                                                            |
| `--client-id <id>`                      | No       |                          | OAuth client ID. Required for `preregistered`; rejected for `dcr`/`cimd`                                                                                                                         |
| `--registration <method>`               | No       | `preregistered`          | `preregistered`, `dcr`, or `cimd`                                                                                                                                                                |
| `--client-metadata-url <url>`           | No       |                          | CIMD only: the Client ID Metadata Document URL to present as the `client_id`                                                                                                                     |
| `--client-auth <method>`                | No       | `none`                   | CIMD only: `none` (public) or `private-key-jwt` (confidential — generates a local EC P-256 key, publishes it via the hosted reflector, and signs a `client_assertion`)                           |
| `--cimd-metadata-origin <url>`          | No       | `https://app.mcpjam.com` | Confidential CIMD only: bare origin hosting the metadata-document reflector. An `http://` loopback origin is a dev-only opt-in; rejected when `--https-only` is set                              |
| `--authz-server-issuer <issuer>`        | No       |                          | Target AS issuer. When set, protected-resource metadata discovery is skipped                                                                                                                     |
| `--token-endpoint <url>`                | No       |                          | AS token endpoint. When set, AS-metadata discovery is skipped. Not valid with `dcr` or `cimd`                                                                                                    |
| `--email <email>`                       | No       |                          | Simulated end-user email claim                                                                                                                                                                   |
| `--client-secret <secret>`              | No       |                          | OAuth client secret presented at redemption                                                                                                                                                      |
| `--token-endpoint-auth-method <method>` | No       |                          | `client_secret_basic`, `client_secret_post`, or `none`. Not valid with `dcr` or `cimd`                                                                                                           |
| `--assertion-format <format>`           | No       | `oidc`                   | Identity assertion format: `oidc` (ID token) or `saml` (SAML 2.0 assertion)                                                                                                                      |
| `--scopes <scopes>`                     | No       |                          | Space-separated scope string                                                                                                                                                                     |
| `--https-only`                          | No       |                          | Reject non-HTTPS / private targets                                                                                                                                                               |

#### Registration strategies

| Strategy        | Description                                                                             |
| --------------- | --------------------------------------------------------------------------------------- |
| `preregistered` | Supply a pre-registered `--client-id` (and optional `--client-secret`)                  |
| `dcr`           | Dynamic Client Registration (RFC 7591) — the CLI registers a client at the RAS each run |
| `cimd`          | Client ID Metadata Document — the client publishes its identity at a URL                |

#### Confidential CIMD (`--client-auth private-key-jwt`)

When `--registration cimd --client-auth private-key-jwt` is used, the CLI loads or generates a local EC P-256 key pair (stored in `~/.mcpjam`). The public key is encoded into a reflector URL on `app.mcpjam.com`, which becomes the `client_id`. The CLI signs a `client_assertion` with the private key at token-redemption time; the private key never leaves the machine.

```bash theme={"theme":"css-variables"}
mcpjam inspector start   # serves the issuer at http://localhost:6274/api/mcp/xaa

mcpjam xaa run \
  --url http://localhost:8080/mcp \
  --issuer-base-url http://localhost:6274/api/mcp \
  --sub user@example.com \
  --registration cimd \
  --client-auth private-key-jwt
```

For a cloud authorization server, expose the inspector origin through a tunnel and pass the public origin as `--issuer-base-url` — see [making the issuer reachable](/cli/xaa#before-you-run-make-the-issuer-reachable).

**Key rotation:** the key is the identity. Deleting `~/.mcpjam/xaa-client-private.pem` or changing the `XAA_CLIENT_PRIVATE_KEY` environment variable generates a new `client_id`; any RAS-side allowlisting must be updated after rotation.

***

## `oauth` commands

### `oauth login`

| Flag                          | Required | Default        | Description                                                              |
| ----------------------------- | -------- | -------------- | ------------------------------------------------------------------------ |
| `--url <url>`                 | Yes      |                | MCP server URL                                                           |
| `--protocol-version <v>`      | Yes      |                | `2025-03-26`, `2025-06-18`, or `2025-11-25`                              |
| `--registration <s>`          | Yes      |                | `cimd`, `dcr`, or `preregistered`                                        |
| `--auth-mode <m>`             | No       | `interactive`  | `headless`, `interactive`, or `client_credentials`                       |
| `--client-id <id>`            | No       |                | OAuth client ID                                                          |
| `--client-secret <s>`         | No       |                | OAuth client secret                                                      |
| `--client-metadata-url <url>` | No       |                | CIMD metadata document URL                                               |
| `--redirect-url <url>`        | No       | Auto-generated | OAuth redirect URL                                                       |
| `--scopes <scopes>`           | No       |                | Space-separated scope string                                             |
| `--header <header>`           | No       |                | HTTP header `Key: Value` (repeatable)                                    |
| `--step-timeout <ms>`         | No       | `30000`        | Per-step timeout                                                         |
| `--verify-tools`              | No       |                | After login, list tools                                                  |
| `--verify-call-tool <name>`   | No       |                | Also call the named tool                                                 |
| `--credentials-out <path>`    | No       |                | Write OAuth credentials to file (mode 0600); stdout has secrets redacted |
| `--debug-out <path>`          | No       |                | Write debug artifact to file                                             |

### `oauth conformance`

| Flag                          | Required | Default        | Description                                                                                          |
| ----------------------------- | -------- | -------------- | ---------------------------------------------------------------------------------------------------- |
| `--url <url>`                 | Yes      |                | MCP server URL                                                                                       |
| `--protocol-version <v>`      | Yes      |                | `2025-03-26`, `2025-06-18`, or `2025-11-25`                                                          |
| `--registration <s>`          | Yes      |                | `cimd`, `dcr`, or `preregistered`                                                                    |
| `--auth-mode <m>`             | No       | `interactive`  | `headless`, `interactive`, or `client_credentials`                                                   |
| `--client-id <id>`            | No       |                | OAuth client ID                                                                                      |
| `--client-secret <s>`         | No       |                | OAuth client secret                                                                                  |
| `--client-metadata-url <url>` | No       |                | CIMD metadata document URL                                                                           |
| `--redirect-url <url>`        | No       | Auto-generated | OAuth redirect URL                                                                                   |
| `--scopes <scopes>`           | No       |                | Space-separated scope string                                                                         |
| `--header <header>`           | No       |                | HTTP header `Key: Value` (repeatable)                                                                |
| `--step-timeout <ms>`         | No       | `30000`        | Per-step timeout                                                                                     |
| `--verify-tools`              | No       |                | After OAuth, list tools                                                                              |
| `--verify-call-tool <name>`   | No       |                | Also call the named tool                                                                             |
| `--conformance-checks`        | No       |                | Run additional negative OAuth checks, including DCR redirect URI policy and redirect-mismatch probes |
| `--credentials-out <path>`    | No       |                | Write OAuth credentials to file (mode 0600); stdout has secrets redacted                             |
| `--print-url`                 | No       |                | Print consent URL to stderr (interactive only)                                                       |
| `--reporter <reporter>`       | No       |                | `json-summary` or `junit-xml` CI report output                                                       |

### `oauth conformance-suite`

| Flag                        | Required | Default | Description                                                                              |
| --------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------- |
| `--config <path>`           | Yes      |         | Path to JSON config file                                                                 |
| `--verify-tools`            | No       |         | Enable tool listing on all flows                                                         |
| `--verify-call-tool <name>` | No       |         | Call the named tool after listing                                                        |
| `--credentials-out <path>`  | No       |         | Write OAuth credentials from the first flow that returns credentials to file (mode 0600) |
| `--reporter <reporter>`     | No       |         | `json-summary` or `junit-xml` CI report output                                           |

### `oauth metadata`

| Flag           | Required | Description                                                   |
| -------------- | -------- | ------------------------------------------------------------- |
| `--url <url>`  | Yes      | OAuth metadata URL to fetch                                   |
| `--https-only` | No       | Apply the hosted policy: reject non-HTTPS and private targets |

### `oauth proxy` / `oauth debug-proxy`

| Flag                | Required | Default | Description                                                   |
| ------------------- | -------- | ------- | ------------------------------------------------------------- |
| `--url <url>`       | Yes      |         | OAuth request URL                                             |
| `--method <method>` | No       | `GET`   | HTTP method                                                   |
| `--header <header>` | No       |         | HTTP header `Key: Value` (repeatable)                         |
| `--body <value>`    | No       |         | Request body as JSON, raw string, `@path`, or `-` for stdin   |
| `--https-only`      | No       |         | Apply the hosted policy: reject non-HTTPS and private targets |

These three commands reach private destinations by default — a server on
loopback, on your LAN, or on a hostname that resolves to one — because the CLI
runs on your own machine. `--https-only` opts in to the policy the hosted web
app enforces, which is what you want when reproducing a hosted failure. Neither
mode will dial a link-local or cloud-metadata address.

***

## `protocol` commands

### `protocol conformance`

| Flag                        | Required | Default | Description                                                                                                            |
| --------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--url <url>`               | Yes      |         | MCP server URL                                                                                                         |
| `--access-token <token>`    | No       |         | Bearer access token                                                                                                    |
| `--credentials-file <path>` | No       |         | Load OAuth credentials from a file created by `oauth login --credentials-out` or `oauth conformance --credentials-out` |
| `--header <header>`         | No       |         | HTTP header `Key: Value` (repeatable)                                                                                  |
| `--check-timeout <ms>`      | No       | `15000` | Per-check timeout in milliseconds                                                                                      |
| `--category <category>`     | No       | all     | Restrict checks to one or more categories                                                                              |
| `--check-id <id>`           | No       | all     | Restrict checks to one or more check IDs                                                                               |
| `--reporter <reporter>`     | No       |         | `json-summary` or `junit-xml` CI report output                                                                         |

Use `--format json|human` for raw output and `--reporter json-summary|junit-xml` for CI reports.

### `protocol conformance-suite`

| Flag                    | Required | Default | Description                                    |
| ----------------------- | -------- | ------- | ---------------------------------------------- |
| `--config <path>`       | Yes      |         | Path to JSON config file                       |
| `--reporter <reporter>` | No       |         | `json-summary` or `junit-xml` CI report output |

***

## `apps` commands

### Shared connection flags

| Flag                           | Description                                                                                                            |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `--transport <transport>`      | Explicit transport type (`http` or `stdio`)                                                                            |
| `--url <url>`                  | HTTP MCP server URL                                                                                                    |
| `--access-token <token>`       | Bearer access token                                                                                                    |
| `--oauth-access-token <token>` | OAuth bearer access token                                                                                              |
| `--refresh-token <token>`      | OAuth refresh token                                                                                                    |
| `--client-id <id>`             | OAuth client ID (with `--refresh-token`)                                                                               |
| `--client-secret <secret>`     | OAuth client secret (with `--refresh-token`)                                                                           |
| `--credentials-file <path>`    | Load OAuth credentials from a file created by `oauth login --credentials-out` or `oauth conformance --credentials-out` |
| `--header <header>`            | HTTP header `Key: Value` (repeatable)                                                                                  |
| `--client-capabilities <json>` | Client capabilities as inline JSON, `@path`, or `-` for stdin                                                          |
| `--command <command>`          | Stdio server command                                                                                                   |
| `--args <arg...>`              | Preferred stdio command arguments                                                                                      |
| `--command-args <arg>`         | Legacy stdio command argument (repeatable)                                                                             |
| `-e, --env <env...>`           | Stdio environment `KEY=VALUE` values                                                                                   |
| `--cwd <path>`                 | Working directory for the stdio child process                                                                          |

Apps commands share the same transport inference rules as the rest of the CLI:
`--url` implies HTTP, `--command` implies stdio, and `--transport` is an
optional explicit override.

### `apps conformance`

MCP Apps server-side conformance checks. Uses shared connection flags plus:

| Flag                    | Description                                              |
| ----------------------- | -------------------------------------------------------- |
| `--category <category>` | Check category to run (`tools`, `resources`). Repeatable |
| `--check-id <id>`       | Specific check id to run. Repeatable                     |
| `--reporter <reporter>` | `json-summary` or `junit-xml` CI report output           |

Use `--format json|human` for raw output and `--reporter json-summary|junit-xml` for CI reports.

### `apps conformance-suite`

| Flag                    | Required | Default | Description                                    |
| ----------------------- | -------- | ------- | ---------------------------------------------- |
| `--config <path>`       | Yes      |         | Path to JSON config file                       |
| `--reporter <reporter>` | No       |         | `json-summary` or `junit-xml` CI report output |

***

## `compat` command

Check whether an MCP server's tools and widgets work on each AI host. Uses shared server connection flags (see [`server` commands](#server-commands)) plus:

| Flag          | Description                                                                                                                                    |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `--host <id>` | Only report this host ID. Repeat for several. Default: all. Valid IDs: `claude`, `chatgpt`, `cursor`, `copilot`, `codex`, `goose`, and others. |

```bash theme={"theme":"css-variables"}
mcpjam compat --url https://your-server.com/mcp
mcpjam compat --url https://your-server.com/mcp --host claude --host chatgpt
```

The result includes a per-host `verdict` (`works`, `degraded`, `blocked`, or `unknown`), a `findings` list explaining each verdict, and a top-level `summary` count. When the tool list is truncated (more than 50 pages), verdicts that would otherwise be `works` are downgraded to `unknown`.

***

## `cloud clients` commands

Manage the clients saved in your hosted MCPJam projects — the named, reusable
configurations that define how MCPJam connects to and talks to your MCP servers.
Every `clients` command that contacts Cloud requires an `sk_` API key (via
`--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior
`mcpjam cloud login`. `clients templates` is the exception: it prints the
built-in template list from the bundled SDK and needs no credential.

`mcpjam cloud hosts …` still works as a command alias, and every command accepts
`--host` as a deprecated spelling of `--client`. Passing both `--client` and
`--host` is a usage error rather than a silent precedence rule.

### Editing safely: `configId` and `expectedConfigId`

A client's config is immutable and content-addressed: `configId` identifies the
exact settings, so the same id means byte-identical settings. Every config edit
takes the `configId` you last read as `expectedConfigId`, and a rename takes the
`name` you last read as `expectedName`. If someone else edited the client since
you read it, the write is refused with a `409` naming the current value instead
of silently reverting their edit.

The loop is always the same:

```bash theme={"theme":"css-variables"}
mcpjam cloud clients get --client "Claude"        # note configId + name
mcpjam cloud clients update --client "Claude" \
  --set temperature=0.2 --expected-config-id hc_01J8...
```

On a conflict, re-run `get` and retry with the fresh value.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `cloud clients templates`

List the built-in templates usable with `mcpjam cloud clients create --template`. No additional flags.

### `cloud clients list`

| Flag                     | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| `--project <id-or-name>` | Project name or ID (defaults to the most recently updated project) |

Clients that exist only as the private backing of a User Testing scenario are
omitted, matching the Clients tab.

### `cloud clients get`

| Flag                     | Required | Description                                        |
| ------------------------ | -------- | -------------------------------------------------- |
| `--client <id-or-name>`  | Yes      | Client name or ID (`--host` is a deprecated alias) |
| `--project <id-or-name>` | No       | Project name or ID                                 |

Returns the resolved config, the `configId` every edit takes, and `impact` — the
live environments, scenario attachments and active legacy journeys a config edit
would follow.

### `cloud clients create`

| Flag                     | Required | Description                                                                        |
| ------------------------ | -------- | ---------------------------------------------------------------------------------- |
| `--name <name>`          | Yes      | Display name for the new client                                                    |
| `--project <id-or-name>` | No       | Project name or ID                                                                 |
| `--template <id>`        | No       | Built-in template ID (see `clients templates`), e.g. `claude`, `chatgpt`, `cursor` |
| `--theme <theme>`        | No       | Theme for the seeded config: `light` or `dark` (template only)                     |
| `--file <path>`          | No       | Client config v2 JSON file (or `-` for stdin)                                      |
| `--json <json>`          | No       | Inline client config v2 JSON (or `@file`, or `-`)                                  |

Provide either `--template` or `--file`/`--json`, not both.

### `cloud clients update`

| Flag                        | Required             | Description                                                                 |
| --------------------------- | -------------------- | --------------------------------------------------------------------------- |
| `--client <id-or-name>`     | Yes                  | Client name or ID (`--host` is a deprecated alias)                          |
| `--project <id-or-name>`    | No                   | Project name or ID                                                          |
| `--name <name>`             | No                   | New display name (requires `--expected-name`)                               |
| `--expected-name <name>`    | With `--name`        | The `name` you last read                                                    |
| `--expected-config-id <id>` | With any config edit | The `configId` you last read                                                |
| `--set <key=value>`         | No                   | Set one field. Repeatable                                                   |
| `--unset <key>`             | No                   | Clear an optional field, or reset a required one to its default. Repeatable |
| `--file <path>`             | No                   | Replacement client config v2 JSON (or `-` for stdin)                        |
| `--json <json>`             | No                   | Inline replacement client config v2 JSON (or `@file`, or `-`)               |

Use either `--set`/`--unset` or `--file`/`--json`, not both. Prefer `--set`: a
whole-config replacement composed from a stale read reverts anything that landed
in between.

#### `--set` field types

Values are parsed by the FIELD, never guessed from what the value looks like —
so `--set systemPrompt='{"a":1}'` sets that literal string, and a
`--set temperature=0.2x` typo fails locally instead of reaching the API as a
string.

| Field                         | Value                             | `--unset`                                                           |
| ----------------------------- | --------------------------------- | ------------------------------------------------------------------- |
| `modelId`                     | string                            | Not allowed — a client cannot be edited into one that pins no model |
| `systemPrompt`                | string                            | Resets to `""`                                                      |
| `temperature`                 | number                            | Resets to `0.7`                                                     |
| `requireToolApproval`         | `true` \| `false`                 | Resets to `false`                                                   |
| `connectionDefaults`          | JSON object                       | Resets to the platform defaults                                     |
| `respectToolVisibility`       | `true` \| `false`                 | Clears                                                              |
| `progressiveToolDiscovery`    | `true` \| `false`                 | Clears                                                              |
| `harness`                     | string (`claude-code` \| `codex`) | Clears                                                              |
| `computer`                    | JSON object                       | Clears                                                              |
| `builtInToolIds`              | JSON array                        | Clears                                                              |
| `skillSelection`              | JSON object                       | Clears                                                              |
| `modelVisibleMcpToolResults`  | JSON object                       | Clears                                                              |
| `mcpToolResultImageRendering` | JSON object                       | Clears                                                              |
| `mcpProfile`                  | JSON object                       | Clears                                                              |
| `hostCapabilitiesOverride`    | JSON object                       | Clears                                                              |
| `chatUiOverride`              | JSON object                       | Clears                                                              |

Object-valued fields are whole-object replacements, not merges. To change a deep
knob, read the client, overlay the sub-object, and send the whole object back.

### `cloud clients servers`

| Flag                             | Required | Description                                        |
| -------------------------------- | -------- | -------------------------------------------------- |
| `--client <id-or-name>`          | Yes      | Client name or ID (`--host` is a deprecated alias) |
| `--server-ids <id,...>`          | Yes      | Comma-separated required server IDs                |
| `--expected-config-id <id>`      | Yes      | The `configId` you last read                       |
| `--optional-server-ids <id,...>` | No       | Comma-separated optional server IDs                |
| `--project <id-or-name>`         | No       | Project name or ID                                 |

This REPLACES the server set: every server you leave out is detached.

### `cloud clients duplicate`

| Flag                     | Required | Description                                        |
| ------------------------ | -------- | -------------------------------------------------- |
| `--client <id-or-name>`  | Yes      | Client name or ID (`--host` is a deprecated alias) |
| `--name <name>`          | No       | Name for the new client                            |
| `--project <id-or-name>` | No       | Project name or ID                                 |

### `cloud clients delete`

| Flag                     | Required | Description                                        |
| ------------------------ | -------- | -------------------------------------------------- |
| `--client <id-or-name>`  | Yes      | Client name or ID (`--host` is a deprecated alias) |
| `--project <id-or-name>` | No       | Project name or ID                                 |

***

## `cloud environments` commands

Manage **project environments** in your hosted MCPJam projects. A project environment is a named, live-editable execution bundle — one host, optionally a standalone server group, optionally a pinned skill selection and pinned plugin versions — that eval suites and journeys run against.

<Note>
  A project environment is not a Computer sandbox image (those are `mcpjam cloud images`), and not the STDIO environment variables you pass with `--env`.
</Note>

All `environments` commands require an `sk_` API key (via `--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior `mcpjam cloud login`. Reading requires project membership; **creating, updating, archiving, and restoring require project admin** — a key bound to a non-admin gets a `FORBIDDEN` error on those.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### The revision workflow

Environments use optimistic concurrency. Every write takes `--expected-revision`, the `revision` you last read:

```bash theme={"theme":"css-variables"}
mcpjam cloud environments get --environment Staging          # note the "revision" field
mcpjam cloud environments update --environment Staging --expected-revision 3 --name Prod
```

If someone else changed the environment in between, the write fails with a `CONFLICT` error (HTTP 409) instead of overwriting their edit — re-read it and retry. `CONFLICT` also covers a duplicate name and archive-state errors (archiving something already archived, editing something archived), so read the message.

### `cloud environments list`

| Flag                     | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| `--project <id-or-name>` | Project name or ID (defaults to the most recently updated project) |
| `--include-archived`     | Include archived environments (needed to find one to restore)      |

### `cloud environments get`

| Flag                         | Required | Description            |
| ---------------------------- | -------- | ---------------------- |
| `--environment <id-or-name>` | Yes      | Environment name or ID |
| `--project <id-or-name>`     | No       | Project name or ID     |

### `cloud environments resolve`

Preview what the environment resolves to right now: the host's current config, the closed server set, and the pinned plugin versions. Fails with `CONFLICT` when the environment can't currently produce a runnable configuration (for example a pinned plugin was disabled); `details.code` carries the specific reason.

| Flag                         | Required | Description            |
| ---------------------------- | -------- | ---------------------- |
| `--environment <id-or-name>` | Yes      | Environment name or ID |
| `--project <id-or-name>`     | No       | Project name or ID     |

### `cloud environments create`

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--name <name>`          | Yes\*    | Display name; must be unique among the project's live environments |
| `--host-id <id>`         | Yes\*    | ID of the host this environment runs against                       |
| `--project <id-or-name>` | No       | Project name or ID                                                 |
| `--description <text>`   | No       | Optional description                                               |
| `--file <path>`          | No       | Environment JSON file (or `-` for stdin)                           |
| `--json <json>`          | No       | Inline environment JSON (or `@file`, or `-`)                       |

\* `--name` and `--host-id` may instead be supplied inside `--file`/`--json`; explicit flags override the same key in the JSON body. Use the JSON body for the structured fields that have no flag — `serverAttachmentId`, `skillSelection`, `secretSelection`, and `pluginVersionIds`:

```bash theme={"theme":"css-variables"}
mcpjam cloud environments create --project Acme --json '{
  "name": "Staging",
  "hostId": "h_123",
  "skillSelection": { "mode": "explicit", "skillIds": ["sk_1"] }
}'
```

### `cloud environments update`

| Flag                         | Required | Description                                                   |
| ---------------------------- | -------- | ------------------------------------------------------------- |
| `--environment <id-or-name>` | Yes      | Environment name or ID                                        |
| `--expected-revision <n>`    | Yes      | The revision you last read (from `environments get`)          |
| `--project <id-or-name>`     | No       | Project name or ID                                            |
| `--name <name>`              | No       | New display name                                              |
| `--host-id <id>`             | No       | New host                                                      |
| `--description <text>`       | No       | New description (empty string clears it)                      |
| `--file <path>`              | No       | Environment JSON with the fields to change (or `-` for stdin) |
| `--json <json>`              | No       | Inline environment JSON (or `@file`, or `-`)                  |

Only the fields you pass change. To **clear** `serverAttachmentId`, `skillSelection`, `secretSelection`, or `pluginVersionIds`, send an explicit `null` in the JSON body — an empty array is rejected, it is not a way to clear:

```bash theme={"theme":"css-variables"}
mcpjam cloud environments update --environment Staging --expected-revision 3 \
  --json '{ "pluginVersionIds": null }'
```

### `cloud environments archive`

Archiving is reversible and frees the name for a new environment; the row is kept.

| Flag                         | Required | Description                |
| ---------------------------- | -------- | -------------------------- |
| `--environment <id-or-name>` | Yes      | Environment name or ID     |
| `--expected-revision <n>`    | Yes      | The revision you last read |
| `--project <id-or-name>`     | No       | Project name or ID         |

### `cloud environments restore`

Fails with `CONFLICT` if another live environment took the name while this one was archived. Plugin pins whose version no longer exists at all are dropped on restore — compare the returned `pluginVersionIds` against what you archived.

| Flag                         | Required | Description                                                             |
| ---------------------------- | -------- | ----------------------------------------------------------------------- |
| `--environment <id-or-name>` | Yes      | Environment name or ID                                                  |
| `--expected-revision <n>`    | Yes      | The revision you last read (use `environments list --include-archived`) |
| `--project <id-or-name>`     | No       | Project name or ID                                                      |

### `cloud environments ensure-adhoc`

Get or create an **unnamed** environment for a composed stack. Deduplicated by content: the same stack always returns the same environment, with `created: false` on the second call. Ad-hoc environments do not appear in `environments list` — they exist so a one-off combination can be run and reproduced without adding a name to the project's list.

Requires project membership, not admin (pinning plugin versions still requires admin).

| Flag                      | Required | Description                                                                   |
| ------------------------- | -------- | ----------------------------------------------------------------------------- |
| `--host <id-or-name>`     | Yes      | Host the stack runs as — the client whose configuration a run is stamped with |
| `--server-group <id>`     | No       | Standalone server group to pin (omit to use the host's own servers)           |
| `--model <id>`            | No       | Model to run instead of the host's pinned one                                 |
| `--computer <id-or-name>` | No       | Project-shared sandbox image to pin, so runs boot a fresh computer from it    |
| `--skill <id...>`         | No       | Project-shared skill IDs to pin                                               |
| `--secret <id...>`        | No       | Project secret IDs the stack grants to its runs (`secrets list` shows them)   |
| `--project <id-or-name>`  | No       | Project name or ID                                                            |

### `cloud environments name`

Promote an ad-hoc environment to a named one **in place** — the same id every existing run already points at, so history stays attached. Fails with `CONFLICT` if the environment already has a name.

| Flag                      | Required | Description                                                                                   |
| ------------------------- | -------- | --------------------------------------------------------------------------------------------- |
| `--environment <id>`      | Yes      | The ad-hoc environment to promote, by ID (an unnamed environment has no name to select it by) |
| `--name <name>`           | Yes      | Display name for the promoted environment                                                     |
| `--expected-revision <n>` | Yes      | The revision you last read                                                                    |
| `--description <text>`    | No       | Optional description                                                                          |
| `--project <id-or-name>`  | No       | Project name or ID                                                                            |

### What can be pinned

The pin fields are narrower than they look:

* **`skillSelection`** accepts project-shared skills only. Personal skills, skills with supporting files or extra frontmatter, and plugin-component skills are all rejected.
* **`pluginVersionIds`** accepts a plugin version only when its plugin is installed and enabled, the version is `ready`, at most one version per plugin is pinned, and none of the version's skills carry supporting files.
* **`secretSelection`** is a GRANT, not a pin: the environment is the boundary, so a run receives the secrets it lists and nothing else. There is no "all of them" mode, an empty list is rejected (revoke with `"secretSelection": null` on update), and a `sharing: "user"` secret still reaches only sessions its owner started.

### Deployment support for model overrides and secret grants

<Note>
  `--model` / `"modelId"` and `--secret` / `"secretSelection"` are answered by
  the deployment, not by your account: a self-hosted or older MCPJam whose
  environment surface predates a field rejects it outright. The CLI asks first
  — `GET /projects/:id/environments/capabilities` reports `modelOverrides` and
  `secretGrants` — and refuses before sending, rather than letting the field
  die in a validator. Absence reads as a no, so a deployment too old to answer
  the probe at all is treated as not supporting either.

  The refusal opens with one of these two sentences:

  * **This MCPJam deployment does not support environment model overrides.**
  * **This MCPJam deployment does not support environment secret grants.**

  A second sentence then names what to drop, and it differs by surface —
  `--model` / `--clear-model` / `"modelId"` on the environment writes,
  `--compose-model` / `compose.models` on a composed launch, and the
  `--secret` / `--compose-secret` equivalents. Grep for the sentence above,
  not for the whole message.

  Every other flag on these commands works regardless; only the calls that
  actually carry a model or a grant are checked.
</Note>

***

## `cloud images` commands

Manage custom Computer sandbox images in your hosted MCPJam projects. An image is defined by a **blueprint** — YAML with a digest-pinned `base` image, `initialize` steps baked into the image at build time, and `maintenance` / `knowledge` text delivered to the agent at runtime (never executed automatically):

```yaml theme={"theme":"css-variables"}
base: debian:bookworm-slim@sha256:<64-hex-digest>
initialize:
  - name: Install tooling
    run: apt-get update && apt-get install -y git jq
maintenance:
  - name: Refresh deps
    run: cd ~/app && npm install
knowledge:
  - name: Test notes
    contents: Run `make test` before pushing.
```

All `images` commands require an `sk_` API key (via `--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior `mcpjam cloud login`.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `cloud images list`

| Flag                     | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| `--project <id-or-name>` | Project name or ID (defaults to the most recently updated project) |

### `cloud images get`

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

### `cloud images validate`

Lint a blueprint without saving it. Prints `ok` plus the resolved base digest, or structured errors with the YAML path of each violation.

| Flag                     | Required | Description                                    |
| ------------------------ | -------- | ---------------------------------------------- |
| `--file <path>`          | Yes      | Blueprint YAML path, or `-` to read from stdin |
| `--project <id-or-name>` | No       | Project name or ID                             |

### `cloud images create`

| Flag                     | Required | Description                                    |
| ------------------------ | -------- | ---------------------------------------------- |
| `--name <name>`          | Yes      | Display name for the new sandbox image         |
| `--file <path>`          | Yes      | Blueprint YAML path, or `-` to read from stdin |
| `--project <id-or-name>` | No       | Project name or ID                             |

### `cloud images edit`

| Flag                     | Required | Description                                        |
| ------------------------ | -------- | -------------------------------------------------- |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID                           |
| `--project <id-or-name>` | No       | Project name or ID                                 |
| `--name <name>`          | No       | New display name                                   |
| `--file <path>`          | No       | Replacement blueprint YAML path (or `-` for stdin) |

### `cloud images build`

Trigger an image build for the sandbox image (async). Poll `images logs` to check build status.

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

### `cloud images logs`

Show a sandbox image's builds (newest first) with their log preview.

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

### `cloud images use`

Boot your computer from this sandbox image. This rebuilds the computer — installed files are wiped.

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

### `cloud images reset`

Reset your computer to its current image, wiping mutable state.

| Flag                     | Required | Description        |
| ------------------------ | -------- | ------------------ |
| `--project <id-or-name>` | No       | Project name or ID |

### `cloud images promote`

Share a personal-draft sandbox image with the whole project. Requires project admin permissions.

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

### `cloud images delete`

Permanently delete a sandbox image from a project.

| Flag                     | Required | Description              |
| ------------------------ | -------- | ------------------------ |
| `--image <id-or-name>`   | Yes      | Sandbox image name or ID |
| `--project <id-or-name>` | No       | Project name or ID       |

***

## `cloud skills` commands

List and read the Cloud Skills in your hosted MCPJam projects. These commands exist to close a gap: three flags across the CLI take a skill ID (`eval run --compose-skill`, `eval cases run --compose-skill`, `environments ensure-adhoc --skill`) and until now there was no way to discover those IDs without opening the web app. Authoring stays in the app — these commands are read-only.

All `skills` commands require an `sk_` API key (via `--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior `mcpjam cloud login`.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `cloud skills list`

List the skills visible to you in a project — the project-shared ones plus your own personal drafts. Each row includes a `pinnability` field that says whether the skill's ID is usable with `--compose-skill` and environment `skillSelection`, and why not if it isn't.

| Flag                     | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| `--project <id-or-name>` | Project name or ID (defaults to the most recently updated project) |

### `cloud skills get`

Show one skill, including its SKILL.md body.

| Flag                     | Required | Description                        |
| ------------------------ | -------- | ---------------------------------- |
| `--skill <id>`           | Yes      | Skill ID, from `cloud skills list` |
| `--project <id-or-name>` | No       | Project name or ID                 |

***

## `cloud projects servers` commands

Manage the MCP servers saved in a project and connect new ones. All `projects servers` commands require an `sk_` API key (via `--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior `mcpjam cloud login`.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `cloud projects servers connect`

Start a connection request for an MCP server URL. Discovery, browser consent, and validation happen asynchronously. When the server requires OAuth, the command prints a private authorization link and (by default) opens it in a browser. The command then polls until the request reaches a terminal status.

| Flag                     | Required | Description                                                  |
| ------------------------ | -------- | ------------------------------------------------------------ |
| `--url <url>`            | Yes      | MCP server URL to connect                                    |
| `--project <id-or-name>` | No       | Project name or ID                                           |
| `--server <id>`          | No       | Existing server ID, when the project has several on this URL |
| `--name <name>`          | No       | Name for the server, if one is created                       |
| `--reauthorize`          | No       | Force a fresh authorization                                  |
| `--no-browser`           | No       | Print the authorization link instead of opening it           |
| `--no-wait`              | No       | Return as soon as the request is created                     |

With `--no-wait`, the command prints the request id and exits immediately. The request continues in the cloud; use `connect-status` to check it and `connect-cancel` to stop it.

Each pending request holds one of the account's five concurrent-connection slots for up to an hour. If `connect` reports `ACTIVE_REQUEST_LIMIT`, the error details include the ids of the open requests — cancel the abandoned ones with `connect-cancel` to free slots.

### `cloud projects servers connect-status`

Check the current state of a connection request started by `connect`.

| Flag             | Required | Description                     |
| ---------------- | -------- | ------------------------------- |
| `--request <id>` | Yes      | Connection request id (`scr_…`) |

### `cloud projects servers connect-cancel`

Cancel a pending connection request, releasing the concurrent-connection slot it holds. Cancelling an already-terminal request is a no-op.

| Flag             | Required | Description                     |
| ---------------- | -------- | ------------------------------- |
| `--request <id>` | Yes      | Connection request id (`scr_…`) |

Cancelling is idempotent — calling it twice on the same request is safe. Use this when an authorization was abandoned (link opened in the wrong browser, consent screen closed) and you want to free the slot before starting over.

***

## `inspector` commands

### `inspector open`

Start or attach to the local Inspector and open the UI.

| Flag                    | Required | Description                      |
| ----------------------- | -------- | -------------------------------- |
| `--inspector-url <url>` | No       | Local Inspector base URL         |
| `--tab <tab>`           | No       | Open Inspector on a specific tab |

### `inspector start`

Start the local Inspector in the background without opening a browser.

| Flag                    | Required | Description              |
| ----------------------- | -------- | ------------------------ |
| `--inspector-url <url>` | No       | Local Inspector base URL |

### `inspector stop`

Stop the local Inspector if it is running.

| Flag                    | Required | Description              |
| ----------------------- | -------- | ------------------------ |
| `--inspector-url <url>` | No       | Local Inspector base URL |

***

## `cloud tunnel`

Expose a local MCP server through an MCPJam relay tunnel and register it as a server in your hosted project, so evals and scenarios can target it. Requires an `sk_` API key or a prior `mcpjam cloud login`. The tunnel stays up until Ctrl-C; the server record outlives the session (calls fail fast at the edge until you re-run, which revives the same URL slug with a fresh secret).

```bash theme={"theme":"css-variables"}
# HTTP target
mcpjam cloud tunnel http://localhost:9090/mcp --server my-server --project acme

# stdio target (command goes after --)
mcpjam cloud tunnel --server everything --project acme -- npx -y @modelcontextprotocol/server-everything
```

| Flag                     | Required | Description                                                                                                                                                                                |
| ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--server <name>`        | Yes      | Server name to register in the project. An existing server with this name is pointed at the tunnel: its URL is overwritten, and stdio records are converted to HTTP. Hidden alias: `--id`. |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project)                                                                                                                         |
| `--api-key <key>`        | No       | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                          |
| `--api-url <url>`        | No       | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                          |
| `-e, --env <env...>`     | No       | Stdio environment assignment in `KEY=VALUE` format (stdio targets only)                                                                                                                    |
| `--cwd <path>`           | No       | Working directory for the stdio MCP server process (stdio targets only)                                                                                                                    |

With `--format json`, a single machine-readable startup object (public URL, server ID, slug, project) is written to stdout; ongoing status goes to stderr in both formats.

<Warning>
  The public tunnel URL embeds a bearer secret and is stored on the project
  server record so the platform can call it. Every re-run rotates the secret
  and disconnects any previous tunnel session for the same server.
</Warning>

***

## `readiness` hosted commands

Grade a saved server against a publisher's directory as the platform reaches it — through the saved server row and the authorize exchange. This is the hosted half of `readiness check`: it answers a different question (what the platform sees, not what your machine sees), can optionally spend credits for model observations, and leaves a persistent record.

All `readiness` hosted commands require an `sk_` API key (via `--api-key` or the `MCPJAM_API_KEY` environment variable) or a prior `mcpjam cloud login`.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `readiness start claude`

Start a hosted readiness run that grades a saved server against Anthropic's connector-directory rules. Returns immediately with a run id; poll with `readiness status`.

| Flag                      | Required | Description                                                             |
| ------------------------- | -------- | ----------------------------------------------------------------------- |
| `--server <idOrName>`     | Yes      | Saved server to grade                                                   |
| `--project <id-or-name>`  | No       | Project name or ID (defaults to the most recently updated project)      |
| `--ai-observations`       | No       | Add optional model observations. **Consumes MCPJam credits.**           |
| `--idempotency-key <key>` | No       | Replay guard: a retry with the same key returns the run already started |

### `readiness start openai`

Start a hosted readiness run that grades a saved server against OpenAI's app-directory rules. `--submission-mode` is required and never inferred — omitting it would silently report the package lane as not-applicable.

| Flag                       | Required | Description                                                                                                                                           |
| -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--server <idOrName>`      | Yes      | Saved server to grade                                                                                                                                 |
| `--submission-mode <mode>` | Yes      | Declared submission shape: `mcp-only` or `mcp-imported-skills`. The two package shapes need a local package — use `readiness check openai` for those. |
| `--project <id-or-name>`   | No       | Project name or ID                                                                                                                                    |
| `--ai-observations`        | No       | Add optional model observations. **Consumes MCPJam credits.**                                                                                         |
| `--idempotency-key <key>`  | No       | Replay guard                                                                                                                                          |

### `readiness status`

Read one hosted readiness run. The response carries three independent axes: `status` (whether the run finished), `overallStatus` (the grade — a `completed` run can be `not-ready`), and `llmObservations` (whether the optional model pass ran).

| Flag                     | Required | Description        |
| ------------------------ | -------- | ------------------ |
| `--run <id>`             | Yes      | Readiness run id   |
| `--project <id-or-name>` | No       | Project name or ID |

### `readiness list`

List hosted readiness runs, newest first.

| Flag                     | Required | Description                    |
| ------------------------ | -------- | ------------------------------ |
| `--project <id-or-name>` | No       | Project name or ID             |
| `--kind <publisher>`     | No       | Narrow to `claude` or `openai` |
| `--server <idOrName>`    | No       | Narrow to one saved server     |
| `--limit <n>`            | No       | Rows to return (1–100)         |

### `readiness cancel`

Stop a hosted readiness run that is still going. The run's real terminal state arrives on a later `readiness status` — this response reports the request, not the outcome.

| Flag                     | Required | Description        |
| ------------------------ | -------- | ------------------ |
| `--run <id>`             | Yes      | Readiness run id   |
| `--project <id-or-name>` | No       | Project name or ID |

### `readiness report`

Read a finished run's findings, ordered most-consequential-first and capped. `truncated` and `totalFindings` indicate when you are seeing a subset.

| Flag                     | Required | Description        |
| ------------------------ | -------- | ------------------ |
| `--run <id>`             | Yes      | Readiness run id   |
| `--project <id-or-name>` | No       | Project name or ID |

***

## `cloud eval` commands

All `eval` commands accept the shared platform flags below.

### Shared platform flags

| Flag                       | Description                                                                                                                                                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-key <key>`          | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`)                                                                                                                                                                                        |
| `--api-url <url>`          | MCPJam API base URL (defaults to `https://app.mcpjam.com/api/v1`)                                                                                                                                                                        |
| `--api-header <header...>` | Extra request header as `"Name: value"`, repeatable — for a deployment behind Cloudflare Access, a WAF, or a corporate proxy. Also `MCPJAM_API_HEADERS`, one per line. Cannot set `authorization`, `idempotency-key`, or `content-type`. |

### `cloud eval create`

Create a runnable eval suite from authored test cases (does not run it).

| Flag                       | Required | Description                                                                                                                               |
| -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `--project <id-or-name>`   | No       | Project name or ID (defaults to the most recently updated project)                                                                        |
| `--file <path>`            | No       | Path to a **create-API JSON** body (or `-` for stdin). A versioned suite file (`schemaVersion: "1"`) belongs on `eval run --file`         |
| `--json <json>`            | No       | Inline suite definition JSON (or `@file`, or `-` for stdin)                                                                               |
| `--name <name>`            | No       | Suite name (overrides the file)                                                                                                           |
| `--model <model>`          | No       | Suite-level default model (overrides the file)                                                                                            |
| `--provider <provider>`    | No       | Suite-level default provider (overrides the file)                                                                                         |
| `--server <id-or-name...>` | No       | Project HTTP server names or IDs (overrides the file)                                                                                     |
| `--client <id-or-name...>` | No       | Clients to attach the suite to (overrides the file). Without one the suite lists no client, and `eval run --client` has nothing to select |
| `--host <id-or-name...>`   | No       | **Deprecated** alias for `--client`. Passing both is an error.                                                                            |

### `cloud eval list`

List the eval suites saved in a project.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

### `cloud eval run`

Start an eval run of an existing suite, or upload a versioned suite file and run it. Provide **either** `--suite` **or** `--file`, not both.

By default the command prints a launch receipt and returns immediately. `--wait` polls every launched run to a terminal state.

**Exit codes (`--wait` only).** Without `--wait` this command's exit code is unchanged: `0` on any launch, `1` on a partial or wholly failed fan-out. `--wait` owns a separate six-code contract — a caller who asked this process to sit and watch a run to completion is asking it to say how the run went:

| Code | Meaning                                                                                                                                                                                                                                                                                                                  |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `0`  | Every waited run passed, unanimously, and (when a report was requested) every report was assembled                                                                                                                                                                                                                       |
| `1`  | A run **completed** with a **failed** verdict. The only condition that produces this code                                                                                                                                                                                                                                |
| `2`  | Usage error — bad flags, `--reporter`/`--out` without `--wait`, or an invalid suite file (after auth, before any server/model work)                                                                                                                                                                                      |
| `3`  | Auth failed — no credential, or the platform rejected it, at launch or mid-wait (a token that expired during a long poll). Zero credits spent if it happened before launch                                                                                                                                               |
| `4`  | Connection/setup failed before evaluation — a launch failure this CLI itself observed (network, timeout, rate limit, a partial or failed fan-out, an unrecognized launch error), or a local `--out` write failure — unless a more severe outcome (`1` or `3`) already applies to the same run, per the merge order above |
| `5`  | No valid verdict — `inconclusive`, a null/unrecognized result, a run whose own status is failed/cancelled/timed-out, a wait that hit its deadline, a mid-wait failure that was not auth-shaped, or a report that could not be assembled                                                                                  |

No infrastructure condition ever maps to `1` — see the same rule under `eval gate` below. A multi-target launch merges these worst-of across every waited run, in the order `1 > 3 > 4 > 5 > 0`: a real verdict failure is never masked by a sibling's infrastructure noise, and a credential failure (which may be *why* other targets show no verdict) outranks a plain connection failure.

This six-code contract is new and deliberately **different** from `eval gate`'s four codes below — see that command's own doc for why the two are not unified. An invalid suite file exits **2** either way — `eval validate` still exits 1 for the same contract failure, because it is a verdict on the file and this command is not.

| Flag                                         | Required              | Description                                                                                                                                                                                                          |
| -------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--suite <id-or-name>`                       | Yes, unless `--file`  | Eval suite name or ID                                                                                                                                                                                                |
| `--file <path>`                              | Yes, unless `--suite` | Versioned suite file to upload and run (`.yaml` or `.json`, or `-` for stdin)                                                                                                                                        |
| `--project <id-or-name>`                     | No                    | Project name or ID (defaults to the most recently updated project)                                                                                                                                                   |
| `--server <id-or-name...>`                   | No                    | Override the suite's saved server selection (HTTP servers only)                                                                                                                                                      |
| `--environment <id-or-name...>`              | No                    | Attached project environment(s) to run. Several values start one **paid run each**.                                                                                                                                  |
| `--client <id-or-name...>`                   | No                    | Attached client(s) to run, so the run is stamped with that client's config. Several values start one **paid run each**.                                                                                              |
| `--host <id-or-name...>`                     | No                    | **Deprecated** alias for `--client`. Passing both is an error. Not the host-compat catalog id `mcpjam compat --host` / `mcpjam tools --host` take — see [Two different `--host` flags](#two-different---host-flags). |
| `--all-targets`                              | No                    | Run **every** attached environment (or, if none, every attached host) — one **paid run per target**                                                                                                                  |
| `--repetitions <n>`                          | No                    | Run each case this many times (1-10), for this run only                                                                                                                                                              |
| `--iterations <n>`                           | No                    | **Deprecated** alias for `--repetitions`. Passing both is an error.                                                                                                                                                  |
| `--case <id-or-title...>`                    | No                    | Run only these cases instead of the whole suite                                                                                                                                                                      |
| `--exclude-skills`                           | No                    | Run the "without skills" A/B arm: nothing is pinned, and the run is labelled as excluded                                                                                                                             |
| `--refresh-snapshot`                         | No                    | **Persists a new host-config snapshot on the suite** (see below). Single-target runs only.                                                                                                                           |
| `--notes <text>`                             | No                    | Free-text note stored on the run                                                                                                                                                                                     |
| `--min-pass-rate <n>`                        | No                    | Pass threshold for this run (0-100), overriding the suite's own                                                                                                                                                      |
| `--match-options <json>`                     | No                    | Tool-call match options for this run, e.g. `'{"toolCallOrder":"exact"}'`                                                                                                                                             |
| `--idempotency-key <key>`                    | No                    | Retry-safety key: repeating the call returns the run it already started                                                                                                                                              |
| `--wait`                                     | No                    | Wait for every launched run to reach a terminal status                                                                                                                                                               |
| `--wait-timeout <ms>`                        | No                    | Maximum wait time (default 600000); requires `--wait`                                                                                                                                                                |
| `--reporter <json-summary\|junit-xml\|html>` | No                    | Write the completed structured report to stdout; requires `--wait`. `html` is a self-contained decision summary + failures page — traces/parity/history are not included.                                            |
| `--out <path>`                               | No                    | Atomically write the completed report to a file; defaults to `json-summary`, or uses `--reporter` when present; requires `--wait`                                                                                    |
| `--compose-client <id-or-name>`              | No                    | Compose a stack to run instead of naming a saved environment (see below). **Ephemeral by default** — pass `--save-targets` to attach it to the suite.                                                                |
| `--compose-host <id-or-name>`                | No                    | **Deprecated** alias for `--compose-client`                                                                                                                                                                          |
| `--compose-computer <id-or-name>`            | No                    | Sandbox image to pin on the composed stack                                                                                                                                                                           |
| `--compose-model <id...>`                    | No                    | Model(s) to run on the composed stack. Replaces the client's pinned model unless `--with-client-default` is set                                                                                                      |
| `--with-client-default`                      | No                    | Also launch an inherit cell using each client's pinned model, alongside `--compose-model`                                                                                                                            |
| `--compose-server <id-or-name...>`           | No                    | Server(s) to pin on the composed stack, snapshotted into a server group so a rerun keeps testing them. Mutually exclusive with `--compose-server-group`                                                              |
| `--compose-server-group <id>`                | No                    | Standalone server group to pin on the composed stack                                                                                                                                                                 |
| `--compose-host-servers`                     | No                    | Run against whatever servers the client points at **right now** instead of pinning a set — editing that client later changes what a rerun tests                                                                      |
| `--compose-skill <id...>`                    | No                    | Project-shared skill IDs to pin on the composed stack                                                                                                                                                                |
| `--compose-secret <id...>`                   | No                    | Project secret IDs the composed stack grants to its runs (`secrets list` shows them)                                                                                                                                 |
| `--save-targets`                             | No                    | Attach the composed environments to the suite (append, capped at 10). Without it the stack is ephemeral                                                                                                              |

#### Composing a target instead of naming one

`--compose-client <id-or-name>` builds an execution stack for this run instead of
naming a saved environment, refined by `--compose-computer`, `--compose-model`,
`--compose-server-group`, `--compose-skill` and `--compose-secret`. The stack becomes an unnamed,
content-addressed environment (the same one `environments ensure-adhoc`
produces), so composing the same stack twice reuses one environment.
`--compose-host` is the deprecated spelling of the same flag.

`--compose-model` and `--compose-secret` carry the deployment requirement
described under [Deployment support for model overrides and secret
grants](#deployment-support-for-model-overrides-and-secret-grants): a launch that names
either is refused before it composes anything if this deployment does not
publish the capability.

<Warning>
  **With `--save-targets`, this edits the suite.** The composed environment is
  appended to the suite's environment list — deliberately, because an
  environment the suite does not list is one nobody can re-run from the app
  afterwards. Without it the stack is ephemeral and the suite is untouched,
  except on a deployment with no ephemeral-launch support, where a
  single-target compose still attaches. The result reports both writes
  (`composed.environment.created`, `composed.attachment.attached`) even when
  the launch itself fails, so you always know what changed.
</Warning>

Any `--compose-*` flag alongside `--environment`, `--client`, `--server` or
`--all-targets` is a usage error: compose builds the stack the run uses, so
naming a target alongside it describes two different runs.

#### Two different `--host` flags

`--host` means two unrelated things in this CLI, and only one of them lives
under `cloud eval`.

| Command                                                       | `--host` names                                                                                                                                    |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mcpjam tools` / `resources` / `prompts` / `probe` / `compat` | A **host-compat catalog id** — which AI host to emulate. `claude`, `chatgpt`, `cursor`, `copilot`, `codex`, `goose`, and the rest of the catalog. |
| `mcpjam cloud eval run` / `cases run` / `update` / `create`   | A **client saved in your project** (`mcpjam cloud clients list`).                                                                                 |

The catalog sense keeps `--host`: it genuinely means "which AI host am I
emulating". Under `cloud eval`, **`--client` is canonical** and `--host` is a
deprecated alias for it — so `mcpjam cloud eval run --host claude` looks for a
saved client named `claude` and fails if you do not have one. When the value
you pass is a catalog id, the error says so and points at `--client`.

#### Which target runs

**Fan-out is explicit and never inferred.** Choosing for you would be choosing
how much to spend, so:

* **nothing attached** → the suite's saved server selection runs (unchanged);
* **exactly one** attached environment or client → that one runs
  automatically, and the run is stamped with its configuration;
* **several attached** → the run fails with `TARGET_REQUIRED`, listing every
  choice. Name one with `--environment` / `--client`, several by repeating the
  flag, or run all of them with `--all-targets`.

`--all-targets` expands **one axis**: attached environments if the suite has
any, otherwise attached clients — never a cross product, because an environment
already resolves a client. It starts **one paid run per target**, launched as a
single group that occupies one concurrency slot rather than N.

`--environment` / `--client` and `--server` are mutually exclusive: an
environment or client supplies a closed server set that an override cannot
change.

<Warning>
  `--refresh-snapshot` **changes the suite**, not just this run: it re-derives
  and stores the suite's host-config snapshot, so every future run uses the new
  one. It is rejected on any multi-target launch — several runs racing to write
  one snapshot would leave the suite pinned to whichever finished last.
</Warning>

#### Output and exit codes

With `--format json`, stdout is **exactly one JSON document** — the launch
receipt, with nothing appended — so CI can parse it directly. In human format
the CLI prints a `View:` link per started run, a `Started N/M runs (group …)`
summary when several launched, and a `Failed: <target> — <code>: <message>`
line per failure.

With `--wait`, the default output contains the launch receipt and completed
runs. `--reporter` replaces that output with the canonical structured report.
`--out` writes the same report atomically, including on failed-run and
post-completion reporting-error paths, before the command sets its exit code.

Without `--wait`, a partial or wholly failed fan-out **exits 1**; with
`--wait` it exits **4** (see the exit-code table above). A per-target failure
does not abort its siblings, so exiting 0 would let a pipeline read "1 of 3
runs never started" as a clean launch.

#### `--file`: upload and run a suite file

`eval run --file` reads the versioned suite file (`schemaVersion: "1"`),
authenticates, then validates. A contract-invalid file exits **2** after the
auth request — it does not start a run. `eval validate` on the same bytes
still exits 1: that command is a verdict on the file; this one is not.

The file's `suite.id` is the declared suite identity. The first run creates a
file-owned suite stamped with that id; later runs of the same id in the same
project update that suite rather than creating a second one. A UI-authored
suite has no declared id, so no file can claim it. Resolve is never by name.

File `passThreshold` is a fraction; the hosted suite grades on a percent. The
conversion refuses rather than approximates. File `repetitions` above **10**
are refused by name (`REPETITIONS_CAP`) and are not clamped.

Cases are synced in batches of at most 100 (`create_eval_cases`). Every
declared case is created or updated, including `disabled: true` rows. A case
the file **no longer declares** is deleted from the hosted suite before
launch. A case the file still declares but marks `disabled: true` is **kept,
with its history** — it is simply left out of the launch, so parking a flaky
test does not destroy its past results and re-enabling it later resumes the
same case.
The run is scoped to the enabled cases either way. A file with no enabled
cases is refused (`NO_ENABLED_CASES`) rather than launching the whole
persisted suite. `--case` may only name an enabled case. A later run updates
existing cases by declared id and creates new ones. An enabled case that
overrides `passThreshold` is refused: hosted grading is suite-wide. A
disabled case with the same override is ignored, matching how repetitions
are checked. Authored
`defaults.toolPolicy` and non-empty `defaults.validity` gates are refused at
hosted CLI launch (`TOOL_POLICY_UNSUPPORTED` / the corresponding validity
refusal). Hosted platform-authored suites do not currently have a backend
tool-policy field or enforcement path, so the CLI does not upload a suite that
would claim those guarantees. When supported by an eval runner, `toolPolicy`
is enforced at execution time with denied tools kept visible but blocked;
server annotations are advisory and **UNTRUSTED**. File `defaults.repetitions` is
inherited per case and is not uploaded as a suite `minIterations` floor.
Authored `defaults.systemPrompt` and `defaults.temperature` are uploaded as
execution defaults; omitted fields stay omitted so the platform applies its
own defaults. `target.hosts` and `target.environment` are attached before
launch. A file environment takes precedence over file hosts for that launch,
while an explicit CLI target takes precedence over both. When an explicit CLI
target is provided (for example `--host`), the file's `target.hosts` are
**not** synchronized to the suite's persisted host attachments — the suite
keeps whatever hosts it already has. Host attachment synchronization only
runs when no explicit CLI target overrides the file. Environment attachment
synchronization is unaffected and always runs.

Without `--idempotency-key`, the CLI derives one from the file's SHA-256,
declared suite id, project, target, and every run-affecting knob
(`--repetitions`, `--case`, `--min-pass-rate`, `--exclude-skills`,
`--match-options`, `--environment` / `--host` / `--server` / `--all-targets`,
`--refresh-snapshot`, compose flags, and the normalized
`--allow-approximated` set with its `--approval-reason`). Repeating the same
file with the same knobs returns the run it already started; changing a knob
starts a new one. The approval set is sorted and keyed by AUTHORED case id, so
`--allow-approximated a b` and `--allow-approximated b a` are the same run
while approving nothing is a different one. `--notes` is not part of the key.

##### Imported cases: live tool resolution, and per-run approval

Every file run performs the same live check `eval validate --project` performs,
**before it writes anything**. It is not opt-in and there is no flag to skip
it. A file with no deterministic `toolCall` steps asks no question and pays no
round trip.

What happens when a deterministic reference does not resolve depends on the
case:

| Case                                                     | Outcome                                                                                                                                                                                              |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Selected (enabled, and named by `--case` if you used it) | The launch **refuses before the suite is synced** — nothing is written and nothing is billed                                                                                                         |
| Imported, not selected                                   | Its claim is rewritten to `unresolved` (keeping `sourceCaseKey`) and it is **still persisted**, so the hosted record says what MCPJam found rather than still asserting a claim about a missing tool |
| Native, not selected                                     | Left exactly as authored. A hand-written case never acquires an `import` block from this — "somebody wrote this" and "something converted this" are different facts                                  |

Each target is checked on **its own** server set. `--host` is checked against
that host's configured servers rather than the file's, because that is the set
the run connects — validating `target.servers` would approve a tool the host
cannot reach. A host the FILE declares uses its own `servers` list when it
declares one (those are attached to the host before launch) and the host's
configured set when the field is omitted. An explicitly empty `servers: []`
is a real, empty set — it clears the attachment before launch — and is
validated as such, never as an omitted field. A step that pins `serverId` is resolved by id;
`serverName` is a display fallback and never overrides it.

If the run's target set cannot be enumerated before the suite exists
(`--all-targets`, `--compose-*`), or a host does not report a server set, or a
host pins a server the project no longer has, the launch refuses rather than
assuming the references are fine.

`--case` accepts a hosted row id, which does not exist yet when this check
runs. A selector this stage cannot map is therefore treated as possibly naming
any enabled case, so an unresolved reference still refuses; the launcher
resolves the selector itself a moment later.

A selected `approximated` case refuses unless this invocation approves it:

| Flag                             | Required       | Description                                                                                                                             |
| -------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `--allow-approximated <case...>` | No             | Approve an `approximated` imported case for **this run only**, by AUTHORED case id (`cases[].id`). Repeatable. `--file` runs only       |
| `--approval-reason <text>`       | With the above | Why the approximations are acceptable for this run. 1–500 characters after trimming; one reason covers every approval in the invocation |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval run --file .mcpjam/evals/s_billing.yaml \
  --allow-approximated c_refund_partial \
  --approval-reason "Reviewed against the upstream rubric; ENG-4821"
```

Checked before anything is billed, each with its own message: approving a
**native** case (nothing to approve), a **claimed-exact** one (needs no
approval), an **unsupported** or **unresolved** one (approval covers a case
whose behaviour was approximated, not one whose behaviour is missing), a
**disabled** or **unselected** one (the run will not execute it), an unknown
id, or the same id twice. The flags are refused outright with `--suite`: a
hosted suite's cases are not the ones this invocation authored, so an
authored-id selector has nothing to resolve against.

**The approval belongs to the run, never to the case.** You supply an id and a
reason and nothing else — the approver and the timestamp are derived by the
server and frozen into that run's own snapshot, because a caller-supplied
approver would file one person's approval under another's name and a
caller-supplied timestamp could be backdated past the edit that invalidated the
claim. Nothing persists: edit the suite, re-sync, or launch again, and the flags
are required again.

Scheduled runs and GitHub-App runs cannot approve anything. A scheduled suite
containing a selected approximation is refused by platform policy, which is the
intended behaviour — an unattended job is exactly the caller that must not be
able to wave one through.

`--file` pointing at create-API JSON (no `schemaVersion`) is a usage error
that names `eval create --file`. The converse is also true: `eval create --file`
on a versioned suite file names `eval run --file`.

Export of a file-owned suite writes `declaredId` as `suite.id`. Export of a
UI suite still writes the Convex id; running that file back is the ownership
refusal.

When the run targets an **attached project environment** (`eval environments
set`), it executes against that environment's resolved host config, closed
server set, and pinned plugin versions instead of the suite's saved selection.
Every run records the environment and the exact revision it used, so `eval
status` can answer "which configuration did this run actually execute against?"
long after the environment has been edited.

### `cloud eval server-facts`

Read the server a run was taken against.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--run <id>`             | Yes      | Eval run ID (from `eval run`)                                      |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

Per server: the tool count, the catalog's measured size, annotation and
output-schema coverage, and the deterministic tool-metadata prechecks — plus
what connect and discovery observed, and any conformance or readiness runs for
the same servers.

**None of it is a verdict.** A large tool surface is not a defect and a slow
connect is not a failure; only a precheck with `class: "spec_required"` names a
violation, and a row marked `protocolDependent` is a rule that may not apply to
this server's protocol version at all.

**Payload size is three different numbers** and this reports two of them, each
carrying its `payload.basis`: `aggregated_catalog_json` is the catalog as the
client assembled it (measured at capture), `normalized_snapshot` is what was
retained after redaction — smaller, and `payload.complete` says so. What the
model actually saw is a host fact and is not here. Token figures are estimated
as `json_chars_div_4` against a **reference** window, not measured against any
model's real context.

Server facts are computed on read, so there is no backfill window: a run that
finished long ago still answers. A run with nothing to describe answers inside
the document with `state: "unavailable"` and a reason — `snapshotMissing`,
`snapshotPartial` (the servers that answered are still listed, and their numbers
are real), or `setupNotObserved` (unmeasured, **not** failed).

Related conformance and readiness runs are joined by **server id alone**, so a
different server version, environment or auth context is not excluded by the
match. Each carries its own timestamp, and none of them is this run's verdict.

### `cloud eval status`

Get the status and summary of an eval run.

| Flag                            | Required | Description                                                                   |
| ------------------------------- | -------- | ----------------------------------------------------------------------------- |
| `--run <id>`                    | Yes      | Eval run ID (from `eval run`)                                                 |
| `--project <id-or-name>`        | No       | Project name or ID (defaults to the most recently updated project)            |
| `--diagnostics-limit <n>`       | No       | Failure diagnostics per page (1–200; default 20)                              |
| `--diagnostics-cursor <cursor>` | No       | Cursor from a previous response's `decisionSummary.diagnostics.nextCursor`    |
| `--stages`                      | No       | Print all six user-value chain rows for each failing trial (`--format human`) |

The response carries a `judges` block with each advisory grader's state and
per-case grades. `status: null` means that judge was never requested for the
run — different from a judge that ran and graded nothing. In `--format human`,
each judge that actually graded gets a one-line summary.

The failure diagnostics inside `decisionSummary` are **paginated**, and a page
that is not the whole set says so: `diagnostics.complete` is `false` and
`diagnostics.nextCursor` carries the next page. Without `--diagnostics-cursor`
a run with more failures than the page size reports only its first 20, so a
script that concludes "only these failed" from one page is reading a sample.
Walk the pages:

```bash theme={"theme":"css-variables"}
mcpjam cloud eval status --run "$RUN_ID" --diagnostics-limit 200 --format json
# then, while .decisionSummary.diagnostics.complete is false:
mcpjam cloud eval status --run "$RUN_ID" --diagnostics-limit 200 \
  --diagnostics-cursor "$NEXT_CURSOR" --format json
```

`--stages` expands each failing trial from one first-failed-stage line to all
six chain rows — connection, discovery, selection, tool call, response, user
value — with each row's state and reason in words. It is off by default because
six rows per trial is a lot of terminal on a run with twenty failures; the
first-break line above the per-trial detail is printed either way.

### `cloud eval judge`

Grade a finished eval run with LLM as Judge. **Spends your organization's model
budget.** Returns a pending receipt; read the grades from `eval status`.

| Flag                      | Required | Description                                                        |
| ------------------------- | -------- | ------------------------------------------------------------------ |
| `--run <id>`              | Yes      | Eval run ID (from `eval run`)                                      |
| `--project <id-or-name>`  | No       | Project name or ID (defaults to the most recently updated project) |
| `--force`                 | No       | Re-grade a run that already has a judge result                     |
| `--enable`                | No       | Grade this run even though the judge was off when it ran           |
| `--judge-model <id>`      | No       | Judge model for this run only                                      |
| `--judge-threshold <0-1>` | No       | Pass threshold for this run only                                   |

A run's grading config is pinned when the run starts, so turning the judge on
for the suite does not reach a run that was already recorded — `--enable` is
what grades it, and it changes nothing beyond that run. Passing neither
`--judge-model` nor `--judge-threshold` grades with the suite's own config,
clearing any override a previous request left on the run.

### `cloud eval stage-analytics`

Read the user-value chain funnel — how far value travelled through
`connection → discovery → selection → call → response → userValue`, and how
much of the run was measured at each stage.

Address it two ways, and exactly one of them per invocation:

| Flag                     | Required      | Description                                                                   |
| ------------------------ | ------------- | ----------------------------------------------------------------------------- |
| `--run <id>`             | XOR `--suite` | One run's funnel, as a single document                                        |
| `--suite <id-or-name>`   | XOR `--run`   | One page of that suite's runs, newest completion first — one document per run |
| `--project <id-or-name>` | No            | Project name or ID (defaults to the most recently updated project)            |
| `--cursor <cursor>`      | No            | Pagination cursor from a previous response (`--suite` only)                   |
| `--limit <n>`            | No            | Documents per page, 1–100 (`--suite` only)                                    |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval stage-analytics --run "$RUN_ID" --format json
mcpjam cloud eval stage-analytics --suite checkout --limit 10 --format json
```

**Counts are returned; rates are not.** Every field is a count of trials, and
the arithmetic is stated on the document: `passed + failed = measured`,
`measured + notMeasured = reached`, `reached + notReached + reachUnknown =
applicable`. Derive a rate only with its denominator in hand — **a zero
denominator means not measured**, never 0% and never 100%.

**Never sum across stages, and never merge across runs.** One trial is counted
in every stage's tally, so adding the six counts the same trial six times. Each
document describes one run's population; the `--suite` listing renders runs
side by side as a trend series and never aggregates them.

**A listing is only comparable within a partition.** Two funnels drawn beside
each other *is* a comparability claim, and it holds only across runs that share
`runGroupId`, `configRevision`, `caseSetFingerprint`, `stageAnalyzerVersion`
and `measurementsSchemaVersion`, and whose `materializationState` is `final`.
An **absent** `runGroupId`, `configRevision` or `caseSetFingerprint` blocks
comparability rather than being assumed compatible — two runs that both record
nothing compare equal while sharing nothing at all.

**Three absences, three different answers.** A run that does not exist fails as
a run-not-found error. A deployment that does not serve this route fails with a
message saying so — that is a fact about the deployment, not about the run. Only
a run that was retrieved and has no document reports `analyticsState:
"unmeasured"` with `analytics: null`, and that state is permanent: there is no
backfill, so a run that finished before stage measurement shipped will never
gain a funnel.

### `cloud eval iterations`

List per-iteration results for an eval run (pass/fail, tool calls, tokens, latency). Results are paginated.

`--project` is optional. Selection follows `--project` → input selector → `MCPJAM_PROJECT` → nearest project link → automatic newest accessible project.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--run <id>`             | Yes      | Eval run ID (from `eval run`)                                      |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |
| `--cursor <cursor>`      | No       | Pagination cursor from a previous response                         |
| `--limit <n>`            | No       | Max iterations per page (1–200)                                    |

### `cloud eval gate`

Apply a pass/fail policy to an eval run. Add `--wait` to poll an in-progress
run before evaluating it.

| Flag                                         | Required | Description                                                                                                                                                                                           |
| -------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--run <id>`                                 | Yes      | Eval run ID (from `eval run`)                                                                                                                                                                         |
| `--project <id-or-name>`                     | No       | Project name or ID (defaults to the most recently updated project)                                                                                                                                    |
| `--min-pass-rate-percent <0-100>`            | No       | Minimum share of iterations that must pass                                                                                                                                                            |
| `--no-gating-score-errors`                   | No       | Fail if any gating scorer errored                                                                                                                                                                     |
| `--min-scorer-pass-rate <scorerId=percent>`  | No       | Minimum pass rate for one scorer; repeatable                                                                                                                                                          |
| `--min-mean-score <scorerId=0..1>`           | No       | Minimum mean score for one scorer; repeatable                                                                                                                                                         |
| `--baseline <runId>`                         | No       | Baseline run ID to gate a regression delta against, in addition to any threshold flags above. Mutually exclusive with `--baseline-sha`                                                                |
| `--baseline-sha <sha>`                       | No       | Baseline **source commit SHA**, resolved server-side to the completed run in this suite recorded against it. Mutually exclusive with `--baseline`                                                     |
| `--min-sample-size <n>`                      | No       | Iterations required on **each** side before a pass-rate regression is decidable (default 5); requires `--baseline` or `--baseline-sha`                                                                |
| `--min-effect-size-percent <0-100>`          | No       | Smallest pass-rate drop worth failing on, as a percentage (default 1); requires `--baseline` or `--baseline-sha`                                                                                      |
| `--gate-deterministic-regressions`           | No       | Fail if a deterministic gating scorer flipped from passed to failed; requires `--baseline` or `--baseline-sha`                                                                                        |
| `--max-p95-latency-increase-ms <ms>`         | No       | Fail if p95 end-to-end latency rose by more than this many milliseconds vs the baseline; requires `--baseline` or `--baseline-sha`                                                                    |
| `--max-cost-usd <usd>`                       | No       | Fail if the run's MCPJam-billed cost exceeded this many dollars. Accepts decimals (`0.05`)                                                                                                            |
| `--max-cost-increase-percent <percent>`      | No       | Fail if cost rose by more than this percentage of the baseline's; requires `--baseline` or `--baseline-sha`. Values above 100 are allowed — a run costing three times its baseline is a 200% increase |
| `--wait`                                     | No       | Poll until the run reaches a terminal status                                                                                                                                                          |
| `--wait-timeout <ms>`                        | No       | Maximum wait time (default 600000)                                                                                                                                                                    |
| `--reporter <json-summary\|junit-xml\|html>` | No       | Write the structured eval and gate report to stdout. `html` is a self-contained decision summary + failures page — traces/parity/history are not included.                                            |
| `--out <path>`                               | No       | Atomically write the report to a file; defaults to `json-summary`, or uses `--reporter` when present                                                                                                  |

Both cost gates report **non-gateable** (exit 3) rather than passing when the
cost is unknown or only partly measured. Cost is stamped per iteration and is
absent whenever the model was not MCPJam-billed (your own API key), the run
used a harness (which mixes models within a turn), or the trial reported no
tokens — so a total built from some of the run is smaller than the truth, and
gating a ceiling against it would pass exactly the runs least understood. A
`--max-cost-increase-percent` gate is also non-gateable when the baseline cost
nothing, since a percentage of zero is undefined.

Report files are flushed before gate exit codes are set, including failed and
incomplete gate outcomes.

**`--baseline` / `--baseline-sha`.** Either flag fetches the same run
comparison `cloud eval compare` uses and folds a regression verdict into the
same report and exit code as the threshold flags above — either one alone
enables the pass-rate regression gate with its default tuning; the four tuning
flags need a baseline to mean anything and are a usage error without one. A
case set that changed, was added or removed, or ran an unequal iteration count
between the two runs makes the whole-run pass-rate and p95-latency gates
non-gateable (exit `3`) rather than a silent pass or a misread regression;
the deterministic per-case regression gate is exempt and can still fail
(exit `1`) on a matching case.

The two are **mutually exclusive** — passing both is a usage error (exit `2`).
There is deliberately no single flag that guesses which kind you meant: a
Convex run ID is an opaque string with no documented format, and an
abbreviated commit SHA is indistinguishable from one by construction, so the
kind is named by the flag rather than inferred. A commit SHA passed to
`--baseline` is rejected with a pointer to `--baseline-sha` instead of being
sent as a doomed run lookup.

A SHA that matches no completed run in the suite is **not** an error: it is the
ordinary "no baseline" outcome, `incomplete` (exit `3`), which means the
comparison established nothing — never a regression. When the SHA matched more
than one eligible run, or the lookup was truncated before it could prove
uniqueness, the gate report's baseline provenance records `baselineMatchCount`
alongside `baselineMatchCountTruncated`; the count is a **floor**, not a total,
so the two are always read together. An absent count means the match was
unambiguous.

Baseline provenance records both what was requested and what it resolved to —
`requestedBaselineKind`, `requestedBaselineRunId` or
`requestedBaselineCommitSha`, the backend-echoed `resolvedBaselineCommitSha`,
and the resolved `baseRunId` — so an archived report can answer "which run did
commit X compare against?" without the invocation that produced it. There is no
automatic baseline selection: a baseline is compared only when one is pinned.

**Gate waivers.** A run whose gate failed can be overridden by an authorized
user, on the record, until an expiry they name — see
[`cloud eval gate waive`](#cloud-eval-gate-waive) below. When a waiver is in
force, `eval gate` reports the outcome as **`waived`** and exits `0`.

`waived` is deliberately not the same as `passed`. The run keeps its own failed
result, the failing verdicts stay in the report, and the waiver — **who** granted
it, **why**, and **until when** — is named in every output format: the human
report, `json-summary`, `junit-xml` (as a `<skipped>` element on the gate
testcase, which does not fail the build and does not render as a clean green
row), and `html` (its own badge and section, neither green nor red).

Only a real verdict failure can be waived. An `incomplete` gate — a cancelled
run, a `--wait` timeout, a network failure, non-gateable score evidence — keeps
exit `3` no matter what waiver is on the run: a waiver granted because the evals
regressed is not consent to ship on an infrastructure failure. A `usage_error`
keeps exit `2` for the same reason.

The CLI also re-derives the waiver's expiry itself rather than trusting the
platform's `active` flag, so a lapsed waiver cannot keep a gate green.

**Exit codes.** `eval gate` is the command that fails a build, so it is the one
that maps a verdict onto an exit code. It keeps four:

| Code | Meaning                                                                                                                                                                             |
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`  | Every requested gate passed, **or** a failed gate was waived                                                                                                                        |
| `1`  | An eval **verdict** failed. Reserved for exactly that                                                                                                                               |
| `2`  | Usage error — an unknown scorer, or a threshold out of range                                                                                                                        |
| `3`  | Incomplete: the run was cancelled, the wait timed out, the network failed, or the run is non-gateable — its score evidence did not verify, or its **import evidence is incomplete** |

**Incomplete import evidence is exit `3`, decided first.** When the platform
reports the run's `importEligibility` as `incomplete` (or `gateable: false`
under any status), `eval gate` returns `3` **before** any verdict is computed
and before a waiver is considered. That ordering is the point: import
completeness is evidence ELIGIBILITY, not a measurement of the server, so it is
never exit `1` — the run has not said the server regressed, it has said its own
evidence cannot be relied on. A waiver cannot override it either, for the same
reason it cannot override a cancelled run: a waiver granted because the evals
regressed is not consent to ship on evidence nobody finished reviewing. Fix it
by re-running with the approvals the cases need, or with the unsupported cases
excluded.

**The baseline is checked too.** A `--baseline` / `--baseline-sha` gate rests
on two runs, so `eval gate` reads the resolved baseline run's own
`importEligibility` and returns `3` when the baseline cannot gate — the compare
response reports counters, not provenance, so the baseline is fetched by id
rather than assumed sound. Without that, a baseline whose own approximations
were never approved could still produce a confident "no regression". A baseline
run that cannot be read at all is also `3`: "we could not look" is not "it is
fine".

A run whose eligibility is `legacy` (no imported cases at all — every native
run) or `eligible` goes through the ordinary verdict logic unchanged. A server
that reports no eligibility at all behaves exactly as it did before the field
existed.

No infrastructure condition ever maps to `1`. A job that fails a release because
a network call flaked, and reports it as a regression, teaches people to ignore
the gate — so retry on `3`, and treat `1` as a real finding about the server.

These four are deliberately **not** the same set `eval run --wait` uses (see
that command above): `gate`'s exit `3` means "incomplete / non-gateable", and
remapping it to the six-code scheme would invert pipelines that already
retry on `3`. This is a deliberate compatibility exception for v1 — a
six-code migration for `gate` is deferred behind a future opt-in flag, kept
here so the exception stays documented rather than silently diverging.

### `cloud eval gate waive`

Override a **failing** run's gate until an expiry you name. Requires the manage
tier, enforced by the platform — whoever launched the run gets no exception for
having launched it.

This does **not** make the run pass. The run keeps its `result`, and the waiver
is reported by both things that read the gate: the GitHub Check Run and
`eval gate` itself.

| Flag                      | Required | Description                                                      |
| ------------------------- | -------- | ---------------------------------------------------------------- |
| `--run <id>`              | Yes      | Eval run ID (inherited from `eval gate`)                         |
| `--project <id-or-name>`  | No       | Project name or ID (inherited from `eval gate`)                  |
| `--reason <text>`         | Yes      | Why the gate is being overridden. Max 500 characters             |
| `--expires-in <duration>` | Yes      | How long the waiver lasts: `30m`, `12h`, `7d`. Capped at 30 days |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval gate waive --run run_123 --reason "hotfix ships today; regression tracked in ENG-4821" --expires-in 3d
```

<Warning>
  **The reason is stored unredacted, for the life of the suite.** It is readable
  by anyone who can see the suite and has no independent retention window. Never
  paste secrets, tokens, or customer data into it. The CLI prints this notice
  before it accepts a reason.
</Warning>

A bare number is rejected: `--expires-in 7` is ambiguous between seven minutes
and seven days, and the difference is a gate that reopens before lunch or three
weeks later. The 30-day cap is enforced by the platform, so the refusal names
the cap and what to do instead.

Waiving a run that already has a waiver in force is a **conflict**, not a second
waiver: the existing one is reported back and is left untouched, because two
active waivers would make "which reason is on the check" a race.

### `cloud eval gate unwaive`

Revoke a waiver, putting the gate and the GitHub Check Run back.

| Flag                     | Required | Description                                                         |
| ------------------------ | -------- | ------------------------------------------------------------------- |
| `--run <id>`             | Yes      | Eval run ID (inherited from `eval gate`)                            |
| `--project <id-or-name>` | No       | Project name or ID (inherited from `eval gate`)                     |
| `--waiver <id>`          | No       | Waiver to revoke. Omit to revoke whichever is in force over `--run` |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval gate unwaive --run run_123
```

Idempotent. A second revoke reports the **original** revocation rather than
restamping it — that is a success, not an error, and it preserves the record of
who actually ended the waiver. An already-expired waiver can still be revoked;
the audit trail distinguishes "this was wrong" from "this ran out".

### `cloud eval trace`

Fetch the full trace for one eval iteration (complete message history and spans). Can be large.

`--project` is optional. Selection follows `--project` → input selector → `MCPJAM_PROJECT` → nearest project link → automatic newest accessible project.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--run <id>`             | Yes      | Eval run ID (from `eval run`)                                      |
| `--iteration <id>`       | Yes      | Iteration ID (from `eval iterations`)                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

### `cloud eval screenshots`

Pull the rendered widget screenshots an eval iteration captured — printed
inline when the terminal supports images, otherwise as URLs, or saved to disk
with `--out`.

| Flag                     | Required | Description                                                                                                                                                            |
| ------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--run <id>`             | Yes      | Eval run ID (from `eval run`)                                                                                                                                          |
| `--iteration <id>`       | Yes      | Iteration ID (from `eval iterations`)                                                                                                                                  |
| `--out <path>`           | No       | Save the PNGs here instead of printing them. A directory — except without `--video`, where a single-screenshot iteration may name a file. Created if it does not exist |
| `--index <n>`            | No       | Only the Nth screenshot (1-based)                                                                                                                                      |
| `--video`                | No       | With `--out`, also save the iteration's replay recording beside the screenshots, as `<iteration>.mp4` or `<iteration>.webm`                                            |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project)                                                                                                     |

### `cloud eval video`

Get the replay recording an eval iteration made — prints the URL, or downloads
it with `--out`.

**Unattended runs on a hosted browser are recorded at 15 fps.** Idle time is
held rather than dropped, so the file's duration matches the run's wall clock
even when the page sat still for a minute. Recordings stop at 60 MiB and
**say so**: a capped take is a complete, playable *beginning* of the run rather
than a corrupt file, and both this command and the run page label it. Read
`videoMeta.truncated` before treating a duration as the length of the run.

A local widget-harness run records a `.webm` through Playwright and reports
nothing else about it; a hosted run records an `.mp4` and reports its duration,
rate, and distinct-frame count. `--format json` returns `videoMeta` when the
recording carried one.

Recording can be turned off for a deployment, in which case hosted runs simply
have no recording to fetch.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--run <id>`             | Yes      | Eval run ID (from `eval run`)                                      |
| `--iteration <id>`       | Yes      | Iteration ID (from `eval iterations`)                              |
| `--out <path>`           | No       | Download the recording to this file instead of printing the URL    |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

```bash theme={"theme":"css-variables"}
# The URL, plus what the recording says about itself
mcpjam cloud eval video --run "$RUN_ID" --iteration "$ITERATION_ID"

# Screenshots and the recording together, into one directory
mcpjam cloud eval screenshots --run "$RUN_ID" --iteration "$ITERATION_ID" \
  --out ./evidence/ --video
```

### `cloud eval get`

Show an eval suite's full settings.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

### `cloud eval validate`

Validate a local eval **suite file** — the versioned declarative document that
describes a suite, its defaults and its cases (`schemaVersion: "1"`, YAML
canonical, JSON accepted, conventionally `.mcpjam/evals/*.yaml`).

Offline by default: without `--project` this command builds no API client, reads
no API key and never touches the network. Passing `--project` is the only thing
that changes that, and it is keyed off the FLAG — a linked directory or an
`MCPJAM_PROJECT` in the environment never turns the offline command into a
networked one.

| Flag                     | Required | Description                                                                                                                              |
| ------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--file <path>`          | Yes      | Suite file to validate, `.yaml` or `.json` (or `-` for stdin)                                                                            |
| `--project <id-or-name>` | No       | Also resolve the file's deterministic tool references against this project's live servers. Opt-in; authenticates and makes network calls |

| Exit code | Meaning                                                                                                                                                                                      |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`       | Valid                                                                                                                                                                                        |
| `1`       | Parsed, but invalid against the suite-file contract. Every finding is reported, not just the first. `eval run --file` on the same bytes exits **2** — 1 is reserved there for a real verdict |
| `2`       | Nothing was validated: unreadable path, input over the 1 MiB limit, or malformed YAML                                                                                                        |

Input is capped at **1,048,576 bytes (1 MiB) of UTF-8** and is never truncated —
a file over the cap is rejected outright.

Under `--format json`, stdout carries this envelope whether the file is valid or
not:

```jsonc theme={"theme":"css-variables"}
// exit 0
{
  "valid": true,
  "file": ".mcpjam/evals/s_billing.yaml",
  "suite": { "id": "s_billing", "name": "Billing smoke", "cases": 12, "enabledCases": 11 },
  "findings": []
}

// exit 1 or 2
{
  "valid": false,
  "file": ".mcpjam/evals/s_billing.yaml",
  "stage": "contract",           // "contract" is exit 1; "input" and "parse" are exit 2
  "findings": [
    {
      "code": "SUITE_FILE_INVALID",   // or SUITE_FILE_TOO_LARGE / _EMPTY / _YAML_INVALID / _MULTIPLE_DOCUMENTS
      "path": ["cases", 1, "id"],      // field path as segments
      "pointer": "cases[1].id",        // the same path, rendered
      "message": "duplicate case id \"c_refund\""
      // `location` accompanies SUITE_FILE_YAML_INVALID findings only, where the
      // YAML parser reported a position: { "line": 24, "column": 9 }
    }
  ]
}
```

Findings are deterministically ordered and carry no timestamps, so two runs over
the same bytes produce byte-identical output.

An unreadable or missing path is a usage error and goes to stderr in the CLI's
standard error envelope rather than as findings — nothing was read, so there is
nothing to report about.

**What "valid" does not mean, without `--project`.** The offline check judges
the file against the suite-file contract. It does **not** re-resolve tool names,
server references or fixtures against a project's live discovery, so a file that
validates here can still fail to run.

#### `--project`: resolving the names against a real project

With `--project`, the command authenticates, resolves the named project, and
checks every deterministic `toolCall` step against the live tool inventory of
the targets the file would run on. Structural findings are unchanged and the
live half is additive, under its own key:

```jsonc theme={"theme":"css-variables"}
{
  "valid": false,
  "file": ".mcpjam/evals/s_billing.yaml",
  "suite": { "id": "s_billing", "name": "Billing smoke", "cases": 12, "enabledCases": 11 },
  "findings": [],
  "projectValidation": {
    "project": { "id": "proj_123", "name": "Alpha" },
    "targets": ["environment prod"],
    "valid": false,
    "findings": [
      {
        "code": "TOOL_REFERENCE_UNRESOLVED",   // or TOOL_DISCOVERY_UNAVAILABLE
        "path": ["cases", 1, "steps", 0, "toolName"],
        "pointer": "cases[1].steps[0].toolName",
        "caseId": "c_render_legacy",
        "caseTitle": "Renders the legacy receipt",
        "toolName": "render_gone",
        "serverName": "billing",
        "targetLabel": "environment prod",
        "disabled": true,
        "imported": true,
        "message": "Server \"billing\" in environment prod exposes no tool named \"render_gone\"."
      }
    ]
  }
}
```

Three properties of the live check are worth knowing before you rely on it:

* **Only `toolCall` steps count.** A tool named in prompt text is a hint the
  model may or may not act on, and an `assert` step is an expectation a case may
  legitimately fail at run time. Treating either as a deterministic reference
  would make a negative test unwritable.
* **Per target, never over their union.** A file that runs against several
  targets must resolve in all of them; a union would green-light a case that
  fails on two targets out of three.
* **Auth and network failures are command errors, not findings.** "Your file
  names a tool that does not exist" and "we could not look" send you to two
  different places, so they are never reported as the same thing.

A completed live check with unresolved references is a **verdict on the file**
and exits `1`, like any other contract failure. A failure to authenticate,
reach the project, or list a server's tools is a command error on stderr.

### `cloud eval export`

Write a hosted eval suite to a local suite file.

| Flag                     | Required | Description                                                                  |
| ------------------------ | -------- | ---------------------------------------------------------------------------- |
| `--suite <id-or-name>`   | Yes      | Eval suite to export (name or ID)                                            |
| `--project <id-or-name>` | No       | Project the suite belongs to (defaults to the most recently updated project) |
| `--out <path>`           | No       | Where to write (default `.mcpjam/evals/<suite-id>.yaml`)                     |
| `--force`                | No       | Replace an existing file at the output path                                  |

The default path uses the suite's **id**, not its display name: an id is stable
and path-safe, so a rename does not leave a second file behind.

**Fail-closed.** A suite file cannot express everything a hosted suite can, and
this command refuses rather than writing a file that describes a different test
than the dashboard runs. When any part of a suite is not representable it emits
`UNSUPPORTED_SUITE_EXPORT` findings, exits `1`, and writes **no file at all** —
not a partial one, not one with a warning comment. Every finding names the field
it is about, so the command's output is the authoritative list; the constructs
that refuse today include:

* several attached project environments, or legacy servers combined with an
  attached environment (the suite file can name one environment);
* a pinned sandbox image;
* automatic LLM-as-judge grading, non-default match options, and an iterations
  floor that raises a case above its own count;
* a compare-across-models case, a scenario-bound case, and cases that name
  **different model providers** (a suite file carries one suite-level provider);
* `replace`/`extend` check overrides, a case that inherits the suite's checks
  while also carrying its own, and any suite check the predicate contract does
  not recognise;
* a suite with no execution model, no minimum accuracy, no name, no runnable
  server or environment target, or no cases; a case with no steps; and a suite
  or case id that is not an opaque id;
* a suite that serializes past the 1 MiB suite-file limit — the size is a
  property of the suite, and this never truncates.

Case identity is never freshly minted: a case exports under its declared id, or
under its platform row id, or it refuses. Exporting the same suite twice
produces the same case ids.

| Exit code | Meaning                                                                                               |
| --------- | ----------------------------------------------------------------------------------------------------- |
| `0`       | Written                                                                                               |
| `1`       | The suite cannot be represented losslessly; nothing was written                                       |
| `2`       | Usage: the output path exists and `--force` was not passed, or the suite's cases did not fit one page |

The file is written through a sibling temp file and a rename, so an interrupted
write leaves the previous file exactly as it was.

### `cloud eval update`

Edit an eval suite's settings (only the flags you pass change).

| Flag                                       | Required | Description                                                                                                                                  |
| ------------------------------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `--suite <id-or-name>`                     | Yes      | Eval suite name or ID                                                                                                                        |
| `--project <id-or-name>`                   | No       | Project name or ID (defaults to the most recently updated project)                                                                           |
| `--file <path>`                            | No       | Suite-update JSON body (or `-` for stdin)                                                                                                    |
| `--json <json>`                            | No       | Inline suite-update JSON (or `@file`, or `-`)                                                                                                |
| `--name <name>`                            | No       | Rename the suite                                                                                                                             |
| `--description <text>`                     | No       | Suite description                                                                                                                            |
| `--server <id-or-name...>`                 | No       | Replace the suite's server selection (project server names)                                                                                  |
| `--computer-image <id-or-name\|off>`       | No       | Sandbox image eval runs boot a fresh computer from (list them with `mcpjam cloud images list`). `off` uses the provider's default base image |
| `--client <name...>`                       | No       | Replace client attachments (by name/ID)                                                                                                      |
| `--host <name...>`                         | No       | **Deprecated** alias for `--client`. Passing both is an error.                                                                               |
| `--model <id>`                             | No       | Execution model ID                                                                                                                           |
| `--system-prompt <text>`                   | No       | Execution system prompt                                                                                                                      |
| `--temperature <n>`                        | No       | Execution temperature                                                                                                                        |
| `--min-accuracy <pct>`                     | No       | Minimum accuracy, 0–100                                                                                                                      |
| `--min-iterations <1-10\|off>`             | No       | Floor on per-case iterations — every case runs at least this many times. `off` removes the floor                                             |
| `--tool-call-order <any\|in-order\|exact>` | No       | Tool call order                                                                                                                              |
| `--arguments <ignore\|partial\|exact>`     | No       | Argument matching                                                                                                                            |
| `--extra-tool-calls <unlimited\|N>`        | No       | Allowed extra tool calls                                                                                                                     |
| `--judge <on\|off>`                        | No       | Turn LLM-as-judge grading on/off. `on` both enables the judge and sets it to grade every run as it completes; `off` stops both               |
| `--judge-model <id>`                       | No       | Judge model ID                                                                                                                               |
| `--judge-threshold <0-1>`                  | No       | Judge pass threshold, 0–1 (a case passes when its score is at or above it)                                                                   |

### `cloud eval revisions`

List a suite's settings history, newest first: one entry per committed edit,
with who made it, which stored fields moved, the note they left, and how many
runs were launched against it. Rows carry no configuration snapshots.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |
| `--limit <n>`            | No       | Maximum revisions to return, 1–100 (defaults to 25)                |
| `--cursor <cursor>`      | No       | Pagination cursor from a previous page                             |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval revisions --suite "Asana MCP Evals" --format json
```

### `cloud eval github list`

List the repositories whose pull requests run an eval suite, plus the
repositories the MCPJam GitHub App can reach.

<Note>
  This subgroup was `cloud eval checks`, and `mcpjam cloud eval checks
      list|connect` still works. It was renamed because `checks` under `cloud eval`
  already meant a case's **grading rules** — the suite's checks, the app's
  Checks section, `create_eval_case`'s `checks` — while these two commands
  manage **GitHub** checks. Two resources under one noun.
</Note>

| Flag                     | Required | Description                                                                                             |
| ------------------------ | -------- | ------------------------------------------------------------------------------------------------------- |
| `--project <id-or-name>` | No       | Project name or ID — only used to pick the organization (defaults to the most recently updated project) |

`available: false` means GitHub Checks is not enabled for the organization at
all, so connecting a repository will not help. `connectable: null` means the
App could not be asked (GitHub unavailable, or no installation) — different
from an empty list, which would mean it reaches nothing.

### `cloud eval github connect`

Run this suite on every pull request to a repository. **Affects everyone who
opens a pull request on that repository**, and with `fail-closed` it can block
their merges.

| Flag                                       | Required | Description                                                                  |
| ------------------------------------------ | -------- | ---------------------------------------------------------------------------- |
| `--suite <id-or-name>`                     | Yes      | Eval suite name or ID                                                        |
| `--repo <owner/repo>`                      | Yes      | Repository to connect. Must be one `eval github list` reports as connectable |
| `--outage-policy <fail-open\|fail-closed>` | Yes      | What the check reports when MCPJam cannot conclude                           |
| `--project <id-or-name>`                   | No       | Project name or ID (defaults to the most recently updated project)           |

`--outage-policy` has no default on purpose: it decides what happens to other
people's pull requests during an MCPJam outage, and it is not editable from
this command afterwards. Retargeting a repository at a different suite, pausing
it, and disconnecting all live in the app's Settings → Integrations, where every
connected repository is visible at once.

Connecting also turns on pull-request comments: MCPJam posts one comment on each
pull request to the repository and keeps that same comment updated on every later
push. Comments are on for every connected repository, and can be turned off per
repository in the app's Settings → Integrations → GitHub.

### `cloud eval delete`

Permanently delete an eval suite (and its cases and runs).

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

### `cloud eval schedule`

Enable or disable scheduled runs for a suite.

| Flag                         | Required | Description                                                          |
| ---------------------------- | -------- | -------------------------------------------------------------------- |
| `--suite <id-or-name>`       | Yes      | Eval suite name or ID                                                |
| `--project <id-or-name>`     | No       | Project name or ID (defaults to the most recently updated project)   |
| `--enable`                   | No       | Enable scheduled runs                                                |
| `--disable`                  | No       | Disable scheduled runs                                               |
| `--interval <minutes>`       | No       | Run interval in minutes (5–10080)                                    |
| `--environment <id-or-name>` | No       | Project environment the scheduled runs launch (only with `--enable`) |

Pass either `--enable` or `--disable` (not both).

A scheduled run launches exactly one run, so a suite with several attached environments
must pin one with `--environment`; a suite with exactly one defaults to it. `--environment`
is rejected with `--disable`: disabling preserves the existing pin, so accepting one there
would silently do nothing.

### `cloud eval environments` subcommands

Attach or detach the project environments an eval suite runs against. Attaching is what
makes `eval run --environment` available for the suite.

#### `cloud eval environments set`

Replace the suite's attached environments (this sets the whole list, in order).

| Flag                            | Required | Description                                                        |
| ------------------------------- | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`          | Yes      | Eval suite name or ID                                              |
| `--environment <id-or-name...>` | Yes      | Project environments to attach, in order                           |
| `--project <id-or-name>`        | No       | Project name or ID (defaults to the most recently updated project) |

```bash theme={"theme":"css-variables"}
mcpjam cloud eval environments set --suite "Checkout smoke" --environment Staging Prod
```

Rejected if it would strand an enabled schedule pinned to an environment being removed —
repoint the schedule (`eval schedule --enable --environment ...`) or disable it first.

#### `cloud eval environments clear`

Detach every environment, reverting the suite to its saved server selection.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

Subject to the same schedule guard as `set`: rejected if the suite has an
enabled schedule pinned to one of the environments being removed. Repoint the
schedule or disable it first.

### `cloud eval cases` subcommands

#### `cloud eval cases list`

List a suite's test cases.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

#### `cloud eval cases get`

Show one test case.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--case <id-or-title>`   | Yes      | Eval case title or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

#### `cloud eval cases run`

Run a single case as a persisted, fully-queryable run — inspect it with `eval iterations` /
`eval steps` like any other run.

| Flag                              | Required | Description                                                                                                                               |
| --------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `--suite <id-or-name>`            | Yes      | Eval suite name or ID                                                                                                                     |
| `--case <id-or-title>`            | Yes      | Eval case title or ID                                                                                                                     |
| `--project <id-or-name>`          | No       | Project name or ID (defaults to the most recently updated project)                                                                        |
| `--server <id-or-name...>`        | No       | Override the suite's saved servers for this run                                                                                           |
| `--environment <id-or-name>`      | No       | Project environment to run against (must be attached to the suite)                                                                        |
| `--client <id-or-name>`           | No       | Attached client to run against, so the run is stamped with that client's config                                                           |
| `--host <id-or-name>`             | No       | **Deprecated** alias for `--client`. Passing both is an error.                                                                            |
| `--compose-client <id-or-name>`   | No       | Compose a stack to run instead of naming a saved environment: the client it runs as. **Ephemeral** — this command has no `--save-targets` |
| `--compose-host <id-or-name>`     | No       | **Deprecated** alias for `--compose-client`                                                                                               |
| `--compose-computer <id-or-name>` | No       | Sandbox image to pin on the composed stack                                                                                                |
| `--compose-model <id>`            | No       | Model to run on the composed stack, instead of the host's                                                                                 |
| `--compose-server-group <id>`     | No       | Standalone server group to pin on the composed stack                                                                                      |
| `--compose-skill <id...>`         | No       | Project-shared skill IDs to pin on the composed stack                                                                                     |
| `--compose-secret <id...>`        | No       | Project secret IDs the composed stack grants to its runs (`secrets list` shows them)                                                      |
| `--repetitions <n>`               | No       | Run the case this many times (1-10)                                                                                                       |
| `--iterations <n>`                | No       | **Deprecated** alias for `--repetitions`. Passing both is an error.                                                                       |
| `--idempotency-key <key>`         | No       | Retry-safety key: repeating the call returns the run it already started                                                                   |

Every `--compose-*` flag behaves exactly as it does for `eval run` — see
[Composing a target instead of naming one](#composing-a-target-instead-of-naming-one).
Target selection follows the same rules ([Which target runs](#which-target-runs)),
except that a single case run targets ONE thing — there is no `--all-targets` here.

#### `cloud eval cases create`

Add a test case to a suite (definition via `--file`/`--json`).

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |
| `--file <path>`          | No       | Case JSON body (or `-` for stdin)                                  |
| `--json <json>`          | No       | Inline case JSON (or `@file`, or `-`)                              |
| `--title <title>`        | No       | Case title (overrides the body)                                    |

#### `cloud eval cases update`

Edit a test case (definition via `--file`/`--json`).

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--case <id-or-title>`   | Yes      | Eval case title or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |
| `--file <path>`          | No       | Case JSON body (or `-` for stdin)                                  |
| `--json <json>`          | No       | Inline case JSON (or `@file`, or `-`)                              |
| `--title <title>`        | No       | Rename the case                                                    |

#### `cloud eval cases delete`

Permanently delete a test case.

| Flag                     | Required | Description                                                        |
| ------------------------ | -------- | ------------------------------------------------------------------ |
| `--suite <id-or-name>`   | Yes      | Eval suite name or ID                                              |
| `--case <id-or-title>`   | Yes      | Eval case title or ID                                              |
| `--project <id-or-name>` | No       | Project name or ID (defaults to the most recently updated project) |

#### `cloud eval cases generate`

AI-generate test cases from the suite's tools (spends credits).

| Flag                         | Required | Description                                                                                                       |
| ---------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------- |
| `--suite <id-or-name>`       | Yes      | Eval suite name or ID                                                                                             |
| `--project <id-or-name>`     | No       | Project name or ID (defaults to the most recently updated project)                                                |
| `--mode <normal\|negative>`  | No       | Generation mode (default `normal`)                                                                                |
| `--server <id-or-name...>`   | No       | Servers to discover tools from (default: suite's)                                                                 |
| `--environment <id-or-name>` | No       | Discover tools from this attached environment's server set                                                        |
| `--case-model <id...>`       | No       | Execution model(s) for the generated cases                                                                        |
| `--simple <n>`               | No       | How many easy, single-tool cases                                                                                  |
| `--multi-tool <n>`           | No       | How many medium, 2+ tool cases                                                                                    |
| `--multi-turn <n>`           | No       | How many multi-turn follow-up cases                                                                               |
| `--complex <n>`              | No       | How many hard / cross-server cases                                                                                |
| `--negative <n>`             | No       | How many negative (no-tool) cases                                                                                 |
| `--vary-user-styles`         | No       | Vary query phrasing across a realistic range of user styles                                                       |
| `--idempotency-key <key>`    | No       | Retry-safety key: repeating the call replays the first attempt's drafts instead of generating (and billing) again |

***

## `mcp` command

### `mcp`

Run MCPJam as an MCP server over stdio so MCP clients (Claude Desktop, Claude Code, Cursor, ...) can connect to, exercise, and debug other MCP servers. See [MCPJam as an MCP server](/cli/mcp-server) for the exposed tools and client setup.

```bash theme={"theme":"css-variables"}
npx -y @mcpjam/cli@latest mcp
```

The command takes no flags of its own. The global `--timeout <ms>` flag sets the default per-request timeout against target servers, and `--quiet` suppresses the startup notice on stderr. Stdout carries only JSON-RPC; never pipe other output into it.

***

## `telemetry` commands

Telemetry commands inspect and configure anonymous CLI telemetry. They never emit telemetry events themselves.

### `telemetry status`

Shows the effective telemetry state, install ID state, state file path, debug mode, and disable reason when disabled. This command does not create an install ID.

### `telemetry disable`

Persistently disables anonymous CLI telemetry by writing `enabled: false` to the telemetry state file. If no install ID exists yet, this command does not create one.

### `telemetry enable`

Persistently enables anonymous CLI telemetry. If no install ID exists yet, this command creates a random install UUID.

***

## `cloud` workspace

`mcpjam cloud` is the account-bound namespace. Log in, link a project, then run evals, tunnels, and the rest of the Cloud groups.

### Cloud account session

See [`cloud login` / `logout` / `whoami`](#cloud-login--logout--whoami) below.

### `cloud link`

Pin the current Git worktree (or `--here` cwd) to a Cloud project by writing `.mcpjam/project.json`. No secrets. Bare `link` ignores an existing file when choosing what to pin.

| Flag        | Description                                                                                                         |
| ----------- | ------------------------------------------------------------------------------------------------------------------- |
| `[project]` | Optional project name or ID to pin. Omit to pick the most recently updated project.                                 |
| `--here`    | Write the link in the current working directory instead of the Git worktree root.                                   |
| `--remove`  | Remove the nearest project link (or the current-directory link with `--here`). Cannot be combined with `[project]`. |

### `cloud status`

Zero-network. Prints credential source, deployment URL, and the project selector that Cloud commands would use (flag / env / link / automatic). API keys are redacted.

The report includes `credential.valid` (`true` / `false` / `null`) and `deployment.valid`. `null` means no credential is configured and is a successful informational status. An invalid explicit `--api-key` or `--api-url` still emits the complete JSON report, sets `ok: false`, includes an `error` string, and exits `1`. Other Cloud commands reject those same values with exit `2` and no report.

### `cloud organizations list`

List organizations and their ids. An `sk_` key sees only its own. Use the id with `mcpjam cloud projects list --org <id>`.

### `cloud projects list`

| Flag         | Description                                                                  |
| ------------ | ---------------------------------------------------------------------------- |
| `--org <id>` | Restrict the listing to one organization (ID only; see `organizations list`) |

### `cloud sessions list`

List Playground chat sessions in the selected project. Same project-selection rule as other Cloud commands.

| Flag                     | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| `--project <id-or-name>` | Project name or ID (defaults to the most recently updated project) |
| `--all-projects`         | List across every accessible project. Conflicts with `--project`.  |
| `--status <status>`      | Filter by session status                                           |
| `--limit <n>`            | Maximum sessions to return (1–200)                                 |

### `cloud sessions send`

Send one message to a project's MCP servers and print the reply plus the raw tool calls, per-call latency and token usage — the telemetry a participant in the conversation cannot see.

**This spends model credits on every call.**

`--idempotency-key` is required and must be **stable for the intent**, not regenerated per attempt. It is what makes a retry safe: with the same key, a turn that already completed is replayed rather than re-run and re-billed. A fresh key per attempt deduplicates nothing.

Omit `--session` to start a conversation; pass the `sessionId` the command returns to continue one. A session created by `cloud sessions browser open` is initially unconfigured and accepts its model configuration with its first send. Configuration is fixed after the first model turn, so `--model`, `--environment`, `--server`, `--system-prompt` and `--tool-mode` are refused alongside `--session` rather than silently repinning the session.

| Flag                            | Description                                                                                                                                                                                           |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--message <text>`              | The message to send, as the user (required)                                                                                                                                                           |
| `--idempotency-key <key>`       | Stable key for this turn's intent (required). Reuse it when retrying.                                                                                                                                 |
| `--session <sessionId>`         | Continue this session instead of starting one                                                                                                                                                         |
| `--model <modelId>`             | Provider-prefixed model id, e.g. `anthropic/claude-sonnet-5`. Required to start a session. A bare id is rejected rather than guessed — unprefixed ids are indistinguishable from local Ollama models. |
| `--environment <environmentId>` | Target this environment's servers. Conflicts with `--server`.                                                                                                                                         |
| `--server <serverId...>`        | Target these project servers. Conflicts with `--environment`.                                                                                                                                         |
| `--system-prompt <text>`        | System prompt for the session                                                                                                                                                                         |
| `--tool-mode <mode>`            | `read_only` (default) advertises only tools annotated `readOnlyHint: true`. `auto` advertises everything and **may cause real side effects**.                                                         |
| `--max-steps <n>`               | Maximum engine steps this turn (1–16)                                                                                                                                                                 |
| `--max-tool-calls <n>`          | Cap the tool calls this turn may make. `0` answers without tools.                                                                                                                                     |
| `--temperature <n>`             | Sampling temperature (0–2)                                                                                                                                                                            |
| `--project <id-or-name>`        | Project name or ID (required to start a session)                                                                                                                                                      |

`read_only` filters on a hint the SERVER asserts, so it is a policy this host applies rather than a guarantee it can verify. A server is free to annotate a mutating tool as read-only.

<Note>
  Hosted session browsers require browser availability for your account and deployment. An unavailable browser reports a refusal or `browser.attached: false` with its reason. The live pane opens in the Playground when the browser workspace is enabled.
</Note>

Add `--browser` to advertise browser tools for this turn. The first browser turn requires `--browser-mode allow_all|read_only|allowlist` unless the selected client supplies a browser policy. Optional `--browser-origins <origin...>` and `--browser-tools <name...>` restrict that grant; `--browser-profile <profileId>` selects its initial saved profile. The grant and profile are fixed thereafter. A client's current browser policy and the session's `read_only` tool mode may further restrict execution.

A continuation must opt in again with `--browser`; a plain send does not wake the desktop. Screenshots are returned as URLs and trace pointers. Use the printed **Open in Playground** link to view the transcript, take over the browser, and select **Resume agent** when finished. API session transcripts are view-only there.

### `cloud sessions browser`

Drive the session browser without a model turn. All verbs except a new `open` require `--session <sessionId>`; use the public session ID, never its read-only conversation UUID or logical browser ID.

| Verb                                          | Description                                                                                                                                                            |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `open`                                        | Open or wake an existing session browser. Omit `--session` and provide `--project`, a browser policy, and a stable `--idempotency-key` to create an empty API session. |
| `navigate <url>`                              | Navigate to an allowed URL. Use a stable `--command-id` for retries.                                                                                                   |
| `act --verb <verb>`                           | Act on the page; pass `--target` as JSON and optional `--value` and `--expected-state`.                                                                                |
| `invoke <toolKey> --input <json>`             | Invoke a permitted page tool.                                                                                                                                          |
| `observe --mode <mode>`                       | Read the page; `--mode screenshot --download <dir>` saves screenshot evidence.                                                                                         |
| `note <text>`                                 | Append a note to the command ledger.                                                                                                                                   |
| `trace`                                       | Read command receipts and screenshot URLs.                                                                                                                             |
| `artifact --command-id <id> --download <dir>` | Fetch screenshot evidence for a command.                                                                                                                               |
| `close`                                       | Release the desktop; a later open or browser turn can revive the logical browser.                                                                                      |

Desktop time is metered. The default cap is two live conversation desktops per user across projects; close one before retrying a capacity refusal. Sleeping browsers wake on the next browser command. Human takeover blocks agent actions; after `browser_in_use`, wait for the person to hand control back and observe the current page. Never repeat an `unknown` command under a new ID without inspecting its trace and page state.

### `cloud sessions show`

Print a session's metadata and a window of its raw messages. Message `index` values are **absolute transcript positions** — the same indices `cloud sessions trace` spans reference, so the two outputs join on that number.

A transcript that could not be read reports `transcriptUnavailable: true` and a `null` `messageCount` — never `0`, which would claim the conversation is empty.

| Flag                      | Description                                                            |
| ------------------------- | ---------------------------------------------------------------------- |
| `--session <sessionId>`   | The session to read (required)                                         |
| `--after-message <index>` | Start the window at this absolute transcript index                     |
| `--limit <n>`             | Messages to return (1–200)                                             |
| `--project <id-or-name>`  | Optional project scope; a session in another project answers not-found |

### `cloud sessions trace`

Print a session's per-turn execution spans: per-tool-call latency, token usage, and indices into the transcript.

Returns the **latest turn** by default rather than the whole session — a session's spans are the largest thing the API emits, and the turn you just took is the one you are debugging.

A turn whose spans could not be read reports `spansUnavailable: true` rather than an empty `spans` array: "made no calls" and "could not fetch" are opposite conclusions.

| Flag                         | Description                                              |
| ---------------------------- | -------------------------------------------------------- |
| `--session <sessionId>`      | The session to trace (required)                          |
| `--turn <turnId>`            | Return exactly this turn. Conflicts with `--after-turn`. |
| `--after-turn <promptIndex>` | Page forward from this turn index                        |
| `--limit <n>`                | Turns to return (1–20). Defaults to 1.                   |
| `--no-spans`                 | Per-turn summaries without span payloads                 |
| `--project <id-or-name>`     | Optional project scope                                   |

***

## `cloud login` / `logout` / `whoami`

These commands manage your MCPJam platform session. `login` opens a browser for OAuth and stores the session locally; `logout` removes it; `whoami` shows the account behind the current credentials.

### `cloud login`

| Flag              | Default                         | Description                                                                                                                                                                                                                                           |
| ----------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-url <url>` | `https://app.mcpjam.com/api/v1` | MCPJam API base URL. The value is persisted with the session so subsequent cloud commands target the same deployment without needing `--api-url` again. Must be a valid `http(s)` URL — an invalid value exits with code `2` before any network call. |
| `--no-browser`    |                                 | Print the login URL to stderr instead of opening a browser.                                                                                                                                                                                           |

The `MCPJAM_API_URL` environment variable is equivalent to `--api-url`. An invalid value in either source is a hard error (exit 2).

**URL resolution order for cloud commands after login:**

1. Explicit `--api-url` flag
2. `MCPJAM_API_URL` environment variable
3. The `apiUrl` stored with the OAuth session (only when the stored login is the credential; `sk_` API keys do not inherit the stored URL)
4. Production default (`https://app.mcpjam.com/api/v1`)

**Login output fields:**

After a successful login, the result includes:

| Field       | Description                                                             |
| ----------- | ----------------------------------------------------------------------- |
| `status`    | `"logged_in"`                                                           |
| `issuer`    | OAuth issuer URL                                                        |
| `apiUrl`    | MCPJam API base URL persisted with the session                          |
| `authFile`  | Path to the stored session file                                         |
| `email`     | Account email address (present when account lookup succeeded)           |
| `plan`      | Account plan (present when account lookup succeeded)                    |
| `expiresAt` | ISO 8601 access-token expiry (present when the token carries an expiry) |

Account details (`email` and `plan`) are fetched from the platform after the OAuth exchange completes. If the lookup fails, login still succeeds and a warning is printed to stderr (suppressed with `--quiet`).

### `cloud logout`

No additional flags. Removes the stored MCPJam session.

When `MCPJAM_API_KEY` is set to an active `sk_` key, `logout` prints a warning to stderr (human format only) noting that the CLI remains authenticated via the environment variable even after the stored session is cleared. JSON output is unaffected.

### `cloud whoami`

| Flag              | Default                         | Description                                        |
| ----------------- | ------------------------------- | -------------------------------------------------- |
| `--api-key <key>` |                                 | MCPJam `sk_` API key (overrides `MCPJAM_API_KEY`). |
| `--api-url <url>` | `https://app.mcpjam.com/api/v1` | MCPJam API base URL.                               |

***

## Exit codes

| Code  | Meaning                                                                                                                                    |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `0`   | Success / all checks passed                                                                                                                |
| `1`   | Command ran but reported a failure                                                                                                         |
| `2`   | Invalid arguments or configuration                                                                                                         |
| `3`   | `tasks conformance` only: the run was incomplete — its task-dependent checks never ran, so it is neither a pass nor a spec violation       |
| `6`   | `tasks watch` only: the task needs input this run could not answer. Answer it with `tasks update`, or re-run with `--interactive` in a TTY |
| `7`   | `tasks watch` only: `--duration-ms` elapsed before the task reached a terminal status                                                      |
| `130` | Interrupted by a signal (`tasks watch` aborted, or a second Ctrl-C during `tunnel` shutdown)                                               |

Codes above `2` are reserved for outcomes a script should act on *differently* — not merely for naming a failure. See [MCP Tasks](/cli/tasks#watch-exit-codes) for the full `tasks watch` mapping.


# Server Inspection
Source: https://docs.mcpjam.com/cli/server-inspection

Probe, diagnose, and export MCP server connectivity and capabilities

The `server` command group gives you a breadth-first view of any MCP server — connectivity, OAuth discovery, capabilities, tools, resources, and prompts — without writing code.

## Quick start

```bash theme={"theme":"css-variables"}
# One command does everything: probe, connect, sweep tools/resources/prompts
mcpjam server doctor --url https://your-server.com/mcp
```

If the server requires OAuth, doctor reports `oauth_required` with the discovery metadata. See [OAuth login](/cli/oauth-login) to obtain a token, then re-run with `--oauth-access-token`.

## Commands

### `server probe`

Stateless HTTP probe — no full client connection. Tests transport selection, OAuth discovery, and `WWW-Authenticate` parsing.

```bash theme={"theme":"css-variables"}
mcpjam server probe --url https://your-server.com/mcp
```

Use this when you want to check reachability and OAuth metadata without triggering a full MCP initialize handshake. The probe sends an unauthenticated `initialize` request and inspects the response.

**What it returns:**

* Transport type (streamable-http, SSE, or failed)
* OAuth metadata (resource metadata URL, authorization server metadata, registration strategies, scopes)
* `WWW-Authenticate` header parsing

Sensitive fields such as `Authorization` headers are automatically redacted to `[REDACTED]` in the printed result.

### `server doctor`

Combined triage — runs probe, then attempts a full client connection, then sweeps tools, resources, resource templates, and prompts. Returns a single JSON artifact.

```bash theme={"theme":"css-variables"}
# Human-readable summary
mcpjam server doctor --url https://your-server.com/mcp

# Full JSON artifact with RPC logs
mcpjam server doctor --url https://your-server.com/mcp --rpc --out doctor.json

# Stdio server
mcpjam server doctor --command node --args server.js --cwd /path/to/project
```

**What it returns:**

* `status`: `ready`, `oauth_required`, or `error`
* Probe results (transport, OAuth discovery)
* Initialization info and negotiated capabilities
* Counts: tools, resources, resource templates, prompts

The `--out <path>` flag writes the full JSON artifact to a file — useful for handoff to another engineer or agent.
For stdio targets, those artifacts only record the explicit env keys you passed
via `-e/--env`; inherited shell variables are not enumerated.

Sensitive fields such as `Authorization` headers are automatically redacted to `[REDACTED]` in all printed output and written artifacts, including RPC logs attached via `--rpc`.

### `server info`

Get initialization info for a connected server.

```bash theme={"theme":"css-variables"}
mcpjam server info --url https://your-server.com/mcp --access-token $TOKEN
```

Returns the server's `serverInfo` (name, version) and `capabilities` from the MCP initialize response.

### `server capabilities`

Get resolved server capabilities.

```bash theme={"theme":"css-variables"}
mcpjam server capabilities --url https://your-server.com/mcp --access-token $TOKEN
```

### `server validate`

Connect to a server and verify the debugger surface works.

```bash theme={"theme":"css-variables"}
mcpjam server validate --url https://your-server.com/mcp --access-token $TOKEN
```

### `server ping`

Simple connectivity check.

```bash theme={"theme":"css-variables"}
mcpjam server ping --url https://your-server.com/mcp
```

### `server export`

Export a full snapshot of the server's tools, resources, prompts, and capabilities as JSON.

```bash theme={"theme":"css-variables"}
mcpjam server export --url https://your-server.com/mcp --access-token $TOKEN > snapshot.json
```

## Shared server flags

Every `server` subcommand accepts these flags for specifying how to connect:

| Flag                           | Description                                                                                                            |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `--transport <transport>`      | Explicit transport type (`http` or `stdio`)                                                                            |
| `--url <url>`                  | HTTP MCP server URL                                                                                                    |
| `--access-token <token>`       | Bearer access token                                                                                                    |
| `--oauth-access-token <token>` | OAuth bearer access token                                                                                              |
| `--refresh-token <token>`      | OAuth refresh token                                                                                                    |
| `--client-id <id>`             | OAuth client ID (used with `--refresh-token`)                                                                          |
| `--client-secret <secret>`     | OAuth client secret (used with `--refresh-token`)                                                                      |
| `--credentials-file <path>`    | Load OAuth credentials from a file created by `oauth login --credentials-out` or `oauth conformance --credentials-out` |
| `--header <header>`            | HTTP header in `Key: Value` format (repeatable)                                                                        |
| `--client-capabilities <json>` | Client capabilities as inline JSON, `@path`, or `-` for stdin                                                          |
| `--command <command>`          | Command for a stdio server                                                                                             |
| `--args <arg...>`              | Preferred stdio command arguments                                                                                      |
| `--command-args <arg>`         | Legacy stdio command argument (repeatable)                                                                             |
| `-e, --env <env...>`           | Stdio environment `KEY=VALUE` values                                                                                   |
| `--cwd <path>`                 | Working directory for the stdio child process                                                                          |

Stdio child processes inherit the parent shell environment by default. Use
`-e/--env` to add values or override inherited ones, and `--cwd` when the
command must run from a project directory.
If you pass `--transport`, the CLI validates that it matches the target flags
you provided instead of silently inferring the transport.

## Common patterns

### Triage an unknown server

```bash theme={"theme":"css-variables"}
# Start with doctor
mcpjam server doctor --url https://unknown-server.com/mcp

# If it returns oauth_required, login first
mcpjam oauth login --url https://unknown-server.com/mcp \
  --protocol-version 2025-11-25 --registration dcr

# Then re-run with the token
mcpjam server doctor --url https://unknown-server.com/mcp --oauth-access-token $TOKEN
```

For multi-command sessions, save credentials to a file to avoid re-extracting the token:

```bash theme={"theme":"css-variables"}
mcpjam oauth login --url https://unknown-server.com/mcp --credentials-out creds.json
mcpjam server doctor --url https://unknown-server.com/mcp --credentials-file creds.json
mcpjam tools list --url https://unknown-server.com/mcp --credentials-file creds.json
```

### Compare before/after a deploy

```bash theme={"theme":"css-variables"}
# Capture a snapshot before
mcpjam server export --url https://your-server.com/mcp --access-token $TOKEN > before.json

# Deploy...

# Capture after
mcpjam server export --url https://your-server.com/mcp --access-token $TOKEN > after.json

# Diff
diff <(jq -S . before.json) <(jq -S . after.json)
```

### CI health check

```bash theme={"theme":"css-variables"}
mcpjam server doctor --url $MCP_SERVER_URL --access-token $TOKEN --format json | jq '.status'
# exits 0 if ready, 1 if not
```


# MCP Tasks
Source: https://docs.mcpjam.com/cli/tasks

Create, inspect, answer and watch MCP Tasks from the CLI — legacy 2025-11-25 and the SEP-2663 extension

The `tasks` commands give the CLI the manual-testing loop the Inspector's Tasks tab has: create a task from a tool call, read it, answer its input, cancel it, and watch it run to a terminal status — on whichever tasks wire the server resolves to.

For judging a server's *conformance* to the tasks spec, see [MCP Tasks Conformance](/cli/tasks-conformance). These commands do the opposite job: they assume the server works and let you drive it.

<Note>
  Every task command connects directly to the server you name, exactly like `tools call`. Tasks are not available in hosted mode.
</Note>

## The two wires

A connection resolves to exactly one tasks wire, and the verbs differ between them. Ask before you assume:

```bash theme={"theme":"css-variables"}
mcpjam tasks capabilities --url https://your-server.com/mcp
```

```json theme={"theme":"css-variables"}
{ "wire": "extension", "toolCalls": true, "list": false, "cancel": true, "update": true, "inlineResult": true }
```

`tasks capabilities` always exits `0` — `"wire": "none"` is a valid answer, not a failure. Every other task command exits `1` on a server with no tasks wire.

|                  | `legacy` (2025-11-25)         | `extension` (SEP-2663, 2026-07-28)                         |
| ---------------- | ----------------------------- | ---------------------------------------------------------- |
| Created by       | `params.task` on `tools/call` | a per-call `CreateTaskResult` opt-in                       |
| Client-set TTL   | yes (`--task-ttl`)            | no — the server owns it                                    |
| `tasks list`     | yes                           | no (removed by SEP-2663)                                   |
| Result           | separate `tasks result` call  | inline on `tasks get`                                      |
| `input_required` | out-of-band elicitation       | `inputRequests` on `tasks get`, answered by `tasks update` |

Pass `--wire legacy` or `--wire extension` to any command to assert the wire you expect; a mismatch exits `1` rather than quietly running against the other one.

<Note>
  On the legacy wire the sub-capabilities are independent: a server that declares only `tasks.cancel` still resolves to `wire: "legacy"`. Commands check the specific capability they need and exit `1` with `TASKS_UNSUPPORTED` if the server never declared it, rather than issuing a request it did not advertise. `tasks list` is the one that would otherwise mislead — the SDK turns a method-unavailable response into an empty list, which reads as "no tasks running" when the truth is "this server cannot tell you".
</Note>

## Creating a task

Add `--task` to a normal `tools call`. The CLI picks the right opt-in for the resolved wire:

```bash theme={"theme":"css-variables"}
mcpjam tools call \
  --url https://your-server.com/mcp \
  --tool-name long_job \
  --task
```

```json theme={"theme":"css-variables"}
{
  "status": "task_created",
  "wire": "extension",
  "task": { "resultType": "task", "taskId": "ext-task-1", "status": "working", "ttlMs": 60000 }
}
```

If the server decides the call is cheap enough to answer synchronously — legal on the extension wire — you get the ordinary tool result instead, plus a note on stderr that no task materialized. A synchronous answer keeps the ordinary `tools call` exit contract: a result with `isError: true` exits `1`, exactly as it would without `--task`. The 0-on-`isError` rule below applies only to a task that completed.

`--task-ttl <ms>` requests a TTL. It applies only to the legacy wire; on the extension wire it is a usage error (exit `2`), because SEP-2663 gives the client no say in the TTL.

`--task` cannot be combined with `--ui`, `--reporter`, `--validate-response`, or `--expect-success`: a task-creation envelope is not a `CallToolResult`, and those flags all assume one.

## Inspecting a task

```bash theme={"theme":"css-variables"}
mcpjam tasks get --task-id ext-task-1 --url https://your-server.com/mcp
mcpjam tasks list --url https://your-server.com/mcp          # legacy wire only
mcpjam tasks result --task-id legacy-task-1 --url …          # legacy wire only
mcpjam tasks cancel --task-id ext-task-1 --url …
```

A task id the server does not know — never created, or expired past its TTL — comes back as `TASK_UNKNOWN_OR_EXPIRED` with exit `1`.

`tasks result` exists only on the legacy wire. On the extension wire a completed task carries its result inline, so `tasks result` exits `1` and points you at `tasks get`.

`tasks cancel` on the extension wire is **cooperative**: the ack is empty, the task may still be running, and it may still reach `completed`. Re-poll with `tasks get`.

## Answering `input_required`

On the extension wire a task that needs input exposes `inputRequests` on `tasks get`. Answer them with `tasks update`:

```bash theme={"theme":"css-variables"}
mcpjam tasks update \
  --url https://your-server.com/mcp \
  --task-id ext-task-1 \
  --input-responses '{"name": {"action": "accept", "content": {"name": "Luca"}}}'
```

`--input-responses` accepts inline JSON, `@path`, or `-` for stdin, like `--tool-args`. The ack is empty by design — the task's status does not move until a later `tasks get`.

Only `elicitation/create` requests are answerable. A task asking for `roots/list` or `sampling/createMessage` is reported as an unanswered rejection; the Inspector UI cannot answer those either.

## Watching a task

`tasks watch` polls a task until it reaches a terminal status, printing transitions to stderr and a single envelope to stdout:

```bash theme={"theme":"css-variables"}
mcpjam tasks watch --url https://your-server.com/mcp --task-id ext-task-1
```

```json theme={"theme":"css-variables"}
{
  "outcome": "completed",
  "wire": "extension",
  "taskId": "ext-task-1",
  "task": { "status": "completed", "result": { "content": [{ "type": "text", "text": "done" }] } }
}
```

Or create and watch in one invocation, over one connection:

```bash theme={"theme":"css-variables"}
mcpjam tools call --url … --tool-name long_job --task --task-watch
```

which emits `{ "created": …, "watch": … }` and exits with the watch's code.

Interrupting before the task exists — Ctrl-C while the `tools/call` is still in flight — emits `{ "status": "aborted", "phase": "create", "wire": … }` and exits `130`, the same code an interrupted watch uses. There is no task id to report yet, so `phase` says where the interrupt landed instead.

### Pacing

`--poll-interval-ms` sets **your** minimum interval. The server's advertised `pollIntervalMs`, any `Retry-After` the transport surfaces on a failed read, and error backoff still apply as floors on top of it, so the CLI cannot poll a server faster than that server asked for.

`--duration-ms` bounds the whole watch (default 5 minutes). `--max-input-rounds` and `--max-consecutive-errors` bound the input loop and transient read failures.

### Interactivity

`--interactive` answers the task's elicitations from the terminal. Like every other interactive verb, it needs a real TTY on stdin: piped or redirected stdin, `--yes`, `CI`, or `MCPJAM_NON_INTERACTIVE` all make it decline cleanly rather than prompt into the void. A declined or unanswered input ends the watch with exit `6`, and you can answer out of band with `tasks update` and watch again.

<Warning>
  **`--interactive` answers task input on the extension wire only.** The legacy wire has no `tasks/update`, and its `Task` carries no `inputRequests` — the request arrives out of band as an `elicitation/create` stamped with `relatedTaskId`, which a polling loop cannot correlate with the task it is watching. `tasks watch` therefore reports `input_required` on that wire and exits `6` rather than polling an unchanging status until `--duration-ms` runs out. Drive a legacy task that needs input from the Inspector UI, which reads the elicitation off a live stream.
</Warning>

Ctrl-C (or `SIGTERM`) aborts the watch cleanly: you still get an envelope, with `"outcome": "aborted"` and exit `130`.

<Note>
  `tasks watch` does not consult the host tasks policy. Like the Inspector's Tools tab, it is a debugging affordance — the policy is a modelling concern for hosts, not a boundary on manual testing.
</Note>

### Watch exit codes

| Code  | Outcome                                                           |
| ----- | ----------------------------------------------------------------- |
| `0`   | `completed`                                                       |
| `1`   | `failed`, `cancelled`, `expired`, `unreachable`                   |
| `6`   | `input-required` — the task needs input this run could not answer |
| `7`   | `timeout` — `--duration-ms` elapsed first                         |
| `130` | `aborted` — interrupted by a signal                               |

A completed task whose result carries `isError: true` still exits `0`: the task finished, and the tool's error is in the envelope. Branch on `task.result.isError` if you need to fail on it.

Outcomes that collapse onto `1` are still distinguishable — `outcome` is always in the envelope.


# MCP Tasks Conformance
Source: https://docs.mcpjam.com/cli/tasks-conformance

Validate MCP Tasks wire behavior — legacy and SEP-2663 extension — from the CLI

The `tasks conformance` command validates the MCP Tasks wire your server exposes: which wire the connection resolves to, whether declaration hygiene holds for that wire, and whether the server honours the observable parts of the contract (result-type discipline, `-32021` on undeclared requests, TTL shapes, inline results, and `Mcp-Name` routing for HTTP transports).

<Note>
  Tasks conformance provokes and then polls a real task, so it requires a
  persistent connection. It is not available in hosted mode.
</Note>

## Quick start

```bash theme={"theme":"css-variables"}
mcpjam tasks conformance --url https://your-server.com/mcp
```

For a local stdio server:

```bash theme={"theme":"css-variables"}
mcpjam tasks conformance --command node --args server.js --cwd /path/to/project
```

On the extension wire you must name the probe tool. Auto-selection reads `execution.taskSupport`, which the 2026-07-28 `ToolSchema` strips, so a tasks-extension server cannot advertise which tool creates a task. Without `--tool-name` the six task-dependent checks cannot run and the command reports `"outcome": "incomplete"` (exit code `3`) rather than a green run:

```bash theme={"theme":"css-variables"}
mcpjam tasks conformance \
  --url https://your-server.com/mcp \
  --tool-name long_job
```

A `--tool-name` the server does not list is treated the same way: the run is incomplete, and the message names both the tool you asked for and the tools the server actually lists.

Emit CI-friendly JUnit XML:

```bash theme={"theme":"css-variables"}
mcpjam tasks conformance \
  --url https://your-server.com/mcp \
  --reporter junit-xml > tasks-report.xml
```

## What it checks

| Check id                               | Category    | What it asserts                                                                                                                                                                                                                                                               |
| -------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tasks-wire-resolvable`                | `dispatch`  | The negotiated protocol version and capabilities resolve to exactly one tasks wire.                                                                                                                                                                                           |
| `tasks-declaration-hygiene`            | `dispatch`  | Outbound requests carry `params.task` only on the legacy wire and the extension declaration only on the extension wire.                                                                                                                                                       |
| `tasks-result-type-discipline`         | `creation`  | A task-eligible `tools/call` returns either a normal tool result or a flat `CreateTaskResult` with `resultType: "task"` and a non-empty `taskId`. The discriminator is required whether absent or wrong — it is the only signal that separates a task from a standard result. |
| `tasks-undeclared-creation-refused`    | `creation`  | On the extension wire, a `tools/call` that did not carry the extension declaration must not come back as a `CreateTaskResult`; the server must answer normally or reject with `-32021`.                                                                                       |
| `tasks-ttl-shape`                      | `lifecycle` | TTL and poll interval use the era-native shapes: `ttlMs`/`pollIntervalMs` on the extension, `ttl`/`pollInterval` on the legacy wire.                                                                                                                                          |
| `tasks-inline-result`                  | `lifecycle` | A completed extension task carries its result inline on `tasks/get`; a legacy task exposes it via `tasks/result`.                                                                                                                                                             |
| `tasks-mcp-name-routing`               | `lifecycle` | Over HTTP, `tasks/get` is sent with `Mcp-Name` set to the task id.                                                                                                                                                                                                            |
| `tasks-undeclared-capability-rejected` | `lifecycle` | `tasks/get`, `tasks/update`, `tasks/cancel` and a task-filtered `subscriptions/listen` sent WITHOUT the extension declaration must each be rejected with `-32021` (Missing Required Client Capability).                                                                       |

## Categories and check ids

Three categories are available:

* `dispatch`
* `creation`
* `lifecycle`

Use `--category` to run a subset by category, or `--check-id` to run specific checks:

```bash theme={"theme":"css-variables"}
# Dispatch checks only
mcpjam tasks conformance \
  --url https://your-server.com/mcp \
  --category dispatch

# Specific checks
mcpjam tasks conformance \
  --url https://your-server.com/mcp \
  --check-id tasks-ttl-shape \
  --check-id tasks-inline-result
```

<Note>
  If you pass both `--category` and `--check-id`, the explicit `--check-id`
  selection wins.
</Note>

## Options

| Flag                    | Description                                                                                                |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| `--category <category>` | Check category to run. Repeat for multiple. Default: all.                                                  |
| `--check-id <id>`       | Specific check id to run. Repeat for multiple. Default: all.                                               |
| `--tool-name <tool>`    | Tool used to provoke a task. Required for servers whose tools carry no task metadata (the extension wire). |
| `--tool-args <json>`    | Tool arguments as a JSON object.                                                                           |
| `--poll-timeout <ms>`   | How long to poll a created task for a terminal status. Default: `30000`.                                   |
| `--reporter <reporter>` | Structured reporter output: `json-summary` or `junit-xml`.                                                 |

## Example output

```bash theme={"theme":"css-variables"}
mcpjam tasks conformance \
  --url https://your-server.com/mcp \
  --tool-name long_job \
  --format json
```

Typical success summary:

```json theme={"theme":"css-variables"}
{
  "passed": true,
  "outcome": "passed",
  "summary": "8/8 checks passed, 0 failed, 0 could not run, 0 not applicable",
  "discovery": {
    "protocolVersion": "2026-07-28",
    "wire": "extension",
    "toolCount": 3,
    "taskCapableToolCount": 1,
    "probedTool": "long_job",
    "createdTaskId": "task-abc123"
  }
}
```

## Outcomes: passed, failed, incomplete

A run reports one of three outcomes, and `passed` is `true` only for the first:

| `outcome`    | Exit code | Meaning                                                                                                      |
| ------------ | --------- | ------------------------------------------------------------------------------------------------------------ |
| `passed`     | `0`       | Every selected check either ran and passed, or does not apply to this server.                                |
| `failed`     | `1`       | At least one check found a violation.                                                                        |
| `incomplete` | `3`       | Nothing failed, but at least one selected check could not be run, so the run does not establish conformance. |

Every skipped check says which kind of skip it is in `skipReason`:

* `not-applicable` — the check cannot apply here (an extension-only check on a legacy connection, `Mcp-Name` routing over stdio, any task check on a connection with no tasks wire). These never hold a run back.
* `could-not-run` — the check applies but was never exercised (no probe tool resolved, the named tool is not listed, the tool produced no task, the task never became readable). These make the run `incomplete`.

An incomplete run carries a root `incompleteReason` naming the checks that did not run and what to change; non-quiet runs also print it to stderr. A skipped check can never add up to a passing verdict.

## Shared connection flags

`tasks conformance` uses the same shared target flags as the rest of the CLI:

| Flag                           | Description                                                   |
| ------------------------------ | ------------------------------------------------------------- |
| `--transport <transport>`      | Explicit transport type (`http` or `stdio`)                   |
| `--url <url>`                  | HTTP MCP server URL                                           |
| `--access-token <token>`       | Bearer access token                                           |
| `--oauth-access-token <token>` | OAuth bearer access token                                     |
| `--refresh-token <token>`      | OAuth refresh token                                           |
| `--client-id <id>`             | OAuth client ID (with `--refresh-token`)                      |
| `--client-secret <secret>`     | OAuth client secret (with `--refresh-token`)                  |
| `--header <header>`            | HTTP header in `Key: Value` format (repeatable)               |
| `--client-capabilities <json>` | Client capabilities as inline JSON, `@path`, or `-` for stdin |
| `--command <command>`          | Command for a stdio server                                    |
| `--args <arg...>`              | Preferred stdio command arguments                             |
| `--command-args <arg>`         | Legacy stdio command argument (repeatable)                    |
| `-e, --env <env...>`           | Stdio environment `KEY=VALUE` values                          |
| `--cwd <path>`                 | Working directory for the stdio child process                 |

`--transport` is optional; without it, `--url` implies HTTP and `--command` implies stdio.

## Notes

* Exit codes are CI-friendly: `0` when all selected checks pass, `1` when any check fails, `2` for invalid command usage, and `3` when the run is incomplete because a selected check could not be run.
* Declaration hygiene is asserted against captured outbound JSON-RPC bytes, not re-derived from intent.
* The undeclared-request checks (`tasks-undeclared-creation-refused`, `tasks-undeclared-capability-rejected`) apply to the extension wire only and are skipped on the legacy wire.
* `tasks-undeclared-capability-rejected` needs a live task to probe with, so it is reported as `could-not-run` (and the run as incomplete) when no task could be provoked. Its probes run last, after every check that reads the task, because a server that wrongly accepts an undeclared `tasks/update` or `tasks/cancel` would otherwise mutate the very task the other checks inspect. If the server does not implement `subscriptions/listen` at all (`-32601`), that sub-probe is reported as a warning instead of a failure.
* The `tasks-mcp-name-routing` check applies to HTTP transports only and is skipped for stdio servers.
* Checks that require a created task cannot run when no probe tool resolves. That is a `could-not-run` skip, so the command exits `3` instead of reporting a pass — pass `--tool-name` to fix it.

## Related commands

* [Apps conformance](/cli/apps-conformance) for MCP Apps surface validation
* [Server inspection](/cli/server-inspection) for breadth-first connectivity and capability triage
* [Command reference](/cli/reference) for the full flag list


# CLI Telemetry
Source: https://docs.mcpjam.com/cli/telemetry

Anonymous command-level telemetry, opt-outs, and debug mode

`mcpjam` collects anonymous command-level telemetry so we can understand CLI usage and reliability. Telemetry is designed for aggregate product insight, not user or server inspection.

Telemetry is enabled by default. The first command invocation that is not opted out writes `telemetry.json` with `enabled: true` and a random install UUID.

## What's collected

The CLI emits one `cli_command` event after a command action starts. Event properties are limited to:

| Property       | Description                                                        |
| -------------- | ------------------------------------------------------------------ |
| `platform`     | Always `cli`                                                       |
| `command`      | Command and subcommand names only, such as `server probe`          |
| `success`      | Whether the command exited successfully                            |
| `exit_code`    | CLI exit code                                                      |
| `duration_ms`  | Command duration in milliseconds                                   |
| `error_code`   | Normalized error code, not the error message                       |
| `cli_version`  | Installed CLI version                                              |
| `os`           | Operating system, such as `darwin`, `linux`, or `win32`            |
| `arch`         | CPU architecture                                                   |
| `node_version` | Node.js version                                                    |
| `transport`    | `http` or `stdio`, inferred from flag presence                     |
| `is_ci`        | Whether the command appears to be running in CI                    |
| `ci_name`      | Coarse CI provider enum, such as `github_actions`, only when in CI |

## What's not collected

Telemetry does not collect raw argv, URLs, hostnames, ports, tokens, headers, environment values, working directories, file paths, tool names, resource names, prompt names, error messages, stack traces, repository names, branch names, workflow names, or CI job IDs.

## Install ID

Telemetry uses a random install UUID as the event `distinct_id`. The UUID is stored in the same platform cache directory used by update checks, in a `telemetry.json` file:

| Platform | Cache directory                               |
| -------- | --------------------------------------------- |
| macOS    | `~/Library/Caches/mcpjam`                     |
| Linux    | `$XDG_CACHE_HOME/mcpjam` or `~/.cache/mcpjam` |
| Windows  | `%LOCALAPPDATA%\\mcpjam\\Cache`               |

The install ID is created on the first real telemetry capture or when you run `mcpjam telemetry enable`. Running `mcpjam telemetry disable` before an ID exists does not create one.

## Opt out

Disable telemetry for one invocation:

```bash theme={"theme":"css-variables"}
mcpjam --no-telemetry server doctor --url https://your-server.com/mcp
```

Disable telemetry persistently:

```bash theme={"theme":"css-variables"}
mcpjam telemetry disable
```

Check or re-enable telemetry:

```bash theme={"theme":"css-variables"}
mcpjam telemetry status
mcpjam telemetry enable
```

Environment opt-outs:

| Variable                      | Effect            |
| ----------------------------- | ----------------- |
| `DO_NOT_TRACK=1`              | Disable telemetry |
| `MCPJAM_TELEMETRY_DISABLED=1` | Disable telemetry |

`--no-telemetry` overrides environment and persisted state for the current invocation. Disable environment variables override persisted state.

## Debug mode

Set `MCPJAM_TELEMETRY_DEBUG=1` to print the exact sanitized payload to stderr instead of sending it:

```bash theme={"theme":"css-variables"}
MCPJAM_TELEMETRY_DEBUG=1 mcpjam server probe --url https://your-server.com/mcp
```

Debug mode only logs when telemetry is otherwise enabled. If `--no-telemetry`, `DO_NOT_TRACK=1`, `MCPJAM_TELEMETRY_DISABLED=1`, or persisted disabled state applies, no payload is logged or sent.


# Tools, Resources & Prompts
Source: https://docs.mcpjam.com/cli/tools-resources-prompts

List, read, and call MCP server tools, resources, and prompts

Once connected (with or without auth), the `tools`, `resources`, and `prompts` command groups let you explore and exercise what the server exposes.

## Running commands as a host

Add `--host <id>` to any `tools`, `resources`, or `prompts` command to connect the way a real host would — sending that host's `clientInfo`, advertised `clientCapabilities`, and protocol version in the MCP `initialize` handshake, exactly as the Inspector Playground does.

```bash theme={"theme":"css-variables"}
mcpjam tools list --url https://your-server.com/mcp --host claude
```

Valid host IDs match the presets in the Inspector: `claude`, `chatgpt`, `cursor`, `copilot`, `codex`, `mcpjam`.

**Tool visibility with `--host`:** `tools list --host` hides tools whose `_meta.ui.visibility` is `["app"]` — those are app-only tools the host's model cannot see. The output includes a `host` field and a `toolsDroppedVisibility` count. Hosts that opt out of visibility filtering (such as `cursor`) keep all tools. `tools call --host` rejects app-only tools with a usage error; omit `--host` to call them as an operator.

**Conflict with `--client-capabilities`:** `--host` and `--client-capabilities` both set the exact advertised capabilities and cannot be combined. The CLI exits with a usage error if both are passed.

## Interactive multi-round-trip (`input_required`)

Add `--interactive` to `tools call`, `prompts get`, or `resources read` to handle servers that return an `input_required` result. When the server embeds an elicitation request, the CLI renders it to the terminal and collects your response from stdin, then retries the original operation — repeating for as many rounds as the server needs.

```bash theme={"theme":"css-variables"}
mcpjam tools call --url https://your-server.com/mcp \
  --tool-name book_flight \
  --tool-args '{}' \
  --interactive
```

**How prompts appear:** The server's message is printed to stderr, followed by a consent prompt (`[a]ccept / [d]ecline / [c]ancel`). For form fields, each field is prompted in turn with its type shown in brackets. For URL-mode elicitations, the URL is printed as plain text and you are asked for consent — the CLI never opens a browser automatically.

**Non-interactive fallback (`--yes`):** Pass `--yes` alongside `--interactive` to decline every embedded input request without prompting. This is also the automatic behavior when stdin is not a TTY or the `CI` environment variable is set — the CLI declines cleanly rather than blocking.

**Stdin conflict:** `--interactive` reads answers from stdin, so it cannot be combined with `--tool-args -`, `--tool-args-stdin`, or `--prompt-args -`. Pass arguments as inline JSON or `@path` instead.

**Capability advertisement:** `--interactive` advertises `elicitation: { form, url }` in the MCP `initialize` handshake. If you also pass `--client-capabilities` or `--host` and that exact set omits `elicitation`, the CLI exits with a usage error — add `"elicitation": { "form": {}, "url": {} }` to the pinned set or drop `--interactive`.

## Tools

### List tools

```bash theme={"theme":"css-variables"}
mcpjam tools list --url https://your-server.com/mcp --access-token $TOKEN
```

Returns every tool the server exposes, including names, descriptions, and input schemas.

### Call a tool

```bash theme={"theme":"css-variables"}
mcpjam tools call --url https://your-server.com/mcp --access-token $TOKEN \
  --tool-name search_docs \
  --tool-args @params.json --quiet --format json
```

Tool arguments can be inline JSON, `@path`, `-` for stdin, or `--tool-args-stdin` as a shorthand for stdin. The result includes the tool's response content.

<Note>
  `tools call` supports `--debug-out <path>` to capture the full request/response
  trace for debugging. For stdio targets, the artifact records only the
  explicit env keys passed through `-e/--env`; inherited shell variables are
  not enumerated.
</Note>

### Calling as a less-capable client

Real MCP clients differ from each other, and from the spec. Two flags let you
call a server as one of those clients, without authoring a host first — the
same behaviors a `--host` carries via its `mcpProfile`:

```bash theme={"theme":"css-variables"}
# Read only the first page of tools/list, like the hosts that never follow
# nextCursor. Tools past page one are invisible to the call.
mcpjam tools list --url https://your-server.com/mcp --first-page-only

# Call as a client that never implemented MRTR. On 2026-07-28 this stops
# advertising `elicitation`, so a server that would have elicited answers
# -32021 instead of starting a round.
mcpjam tools call --url https://your-server.com/mcp \
  --tool-name confirm_delete --tool-args '{"id":"42"}' --no-mrtr
```

Unlike the header flags below, neither is HTTP-only — pagination truncation and
the MRTR knob both work over stdio, so they mean the same thing with
`--command`.

`--first-page-only` has a second-order effect worth knowing on `2026-07-28`:
the SEP-2243 mirroring source is the aggregated tool list the client cached, so
a tool from page two is called with **no** `Mcp-Param-*` headers and a strict
server answers `-32020`. That is not a bug in the flag — it is exactly how a
real first-page-only client fails, and it is often the reason a tool "works in
the inspector but not in that host".

### Debugging `Mcp-Param-*` headers (SEP-2243)

On a `2026-07-28` HTTP connection, a tool argument annotated with `x-mcp-header`
is mirrored into an `Mcp-Param-{Name}` request header, and a conforming server
cross-checks the two. Two flags let you exercise the failure paths on purpose:

```bash theme={"theme":"css-variables"}
# Call as a client that has NOT implemented the mirroring. A conforming server
# should answer -32020 HeaderMismatch; the CLI surfaces that as-is.
mcpjam tools call --url https://your-server.com/mcp   --tool-name execute_sql --tool-args '{"region":"us-east"}'   --no-param-headers

# Send a deliberately WRONG value, reproducing a header/body mismatch exactly.
mcpjam tools call --url https://your-server.com/mcp   --tool-name execute_sql --tool-args '{"region":"us-east"}'   --mcp-header Mcp-Param-Region=eu-west
```

`--mcp-header` uses `Name=Value` (not `--header`'s `Key: Value`) because a
mirrored value routinely contains `:`. Repeat it for several headers; the last
occurrence of a name wins.

Supplying any `Mcp-Param-*` header turns the automatic mirroring **off** for
that call, so the headers you pass are the only `Mcp-Param-*` ones sent. This
is not a convenience — it is what makes the override work at all: the MCP
client merges the mirrored values *over* caller-supplied headers, so with
mirroring left on your deliberately wrong value would be silently replaced by
the correct one and no `-32020` would ever occur. Headers outside the
`Mcp-Param-*` family (`Mcp-Method`, `Mcp-Name`, …) are unaffected and do not
change mirroring.

When either flag is in play — or with `--rpc` — the CLI prints the headers that
actually went out to **stderr**, including on the failure path, so stdout stays
a clean machine-readable result:

```
Mcp-Param-*: Mcp-Param-Region=us-east
```

Both flags are HTTP-only: SEP-2243 mirroring is a Streamable HTTP concern, so
using them with `--command` is a usage error rather than a silent no-op.

### Render a tool result in Inspector

If `tools list` shows `_meta.ui.resourceUri`, deprecated `_meta["ui/resourceUri"]`, or `openai/outputTemplate` in `toolsMetadata`, the tool has interactive UI. Add `--ui` to execute it once and render the completed result in Inspector's Playground:

```bash theme={"theme":"css-variables"}
mcpjam tools call --url https://your-server.com/mcp --access-token $TOKEN \
  --tool-name create_view \
  --tool-args @params.json \
  --ui \
  --quiet --format json
```

Without `--ui`, `tools call` returns the raw tool result. With `--ui`, it opens Inspector by default in a TTY and returns an envelope containing the raw `result`, `inspectorBrowserUrl`, and compact `inspectorRender` evidence. `inspectorRender.status` is the UI signal: `rendered` means Inspector accepted the render, `skipped` means the tool succeeded but the active browser client, a render precondition, or the render wait was missing, and `error` means a non-recoverable Inspector render command failed. `inspectorRender.remediation` is always present and is one of `open_browser`, `retry`, `reconnect_server`, or `none`. Skipped renders emit a stable root `warning` plus `inspectorRender.warning` with `code`, `message`, `remediation`, and optional `browserUrl`, `hasActiveClient`, and `inspectorStarted` fields. Stable skipped-render codes are `no_active_client`, `timeout`, `disconnected_server`, and `unsupported_in_mode`. Pass `--require-render` when a skipped render should become a hard error instead of a warning. Browser automation can open `inspectorBrowserUrl` itself and pass `--no-open`; `--inspector-url` is the local Inspector backend/API base URL. If you already know the Inspector browser/client URL, pass `--frontend-url <url>` to use it directly and skip health-advertised frontend checks and local dev port discovery.

For agent/browser automation, either let `--ui` open Inspector automatically in a TTY, or open `http://127.0.0.1:6274/#playground` in the automation browser first and run `tools call --ui --no-open --quiet --format json`. Add `--attach-only` when startup, browser opening, and discovery should be disallowed. Agents should confirm `inspectorRender.status === "rendered"` before assuming the UI is visible, and use `inspectorRender.remediation` to recover from skipped renders. Default non-TTY `--ui` runs do not open a browser unless `--open` is passed. TTY stderr runs print the Playground URL and initial wait message unless `--quiet` is set; the elapsed-seconds heartbeat only appears when stderr is a TTY. The normal JSON output is compact; pass `--debug-out <path>` for the full render envelope.

For a local stdio server:

```bash theme={"theme":"css-variables"}
mcpjam tools list \
  --command npx \
  --args -y @modelcontextprotocol/server-everything \
  --cwd $PWD \
  -e DEBUG=1
```

## Resources

### List resources

```bash theme={"theme":"css-variables"}
mcpjam resources list --url https://your-server.com/mcp --access-token $TOKEN
```

### Read a resource

```bash theme={"theme":"css-variables"}
mcpjam resources read --url https://your-server.com/mcp --access-token $TOKEN \
  --resource-uri "file:///docs/readme.md"
```

Use `resources read --resource-uri ui://...` when you need to inspect raw widget HTML or other UI resources directly.

### List resource templates

```bash theme={"theme":"css-variables"}
mcpjam resources templates --url https://your-server.com/mcp --access-token $TOKEN
```

## Prompts

### List prompts

```bash theme={"theme":"css-variables"}
mcpjam prompts list --url https://your-server.com/mcp --access-token $TOKEN
```

### Get a prompt

```bash theme={"theme":"css-variables"}
mcpjam prompts get --url https://your-server.com/mcp --access-token $TOKEN \
  --prompt-name summarize \
  --prompt-args '{"text": "Hello world"}'
```

## Common patterns

### Enumerate the full server surface

```bash theme={"theme":"css-variables"}
# One-shot: doctor already sweeps everything
mcpjam server doctor --url https://your-server.com/mcp --access-token $TOKEN

# Or individually for focused output
mcpjam tools list --url $URL --access-token $TOKEN --format json | jq '.tools[].name'
mcpjam resources list --url $URL --access-token $TOKEN --format json | jq '.resources[].uri'
mcpjam prompts list --url $URL --access-token $TOKEN --format json | jq '.prompts[].name'
```

### Test a specific tool end-to-end

```bash theme={"theme":"css-variables"}
# List to find the tool
mcpjam tools list --url $URL --access-token $TOKEN --format json \
  | jq '.tools[] | select(.name == "search_docs")'

# Call it
mcpjam tools call --url $URL --access-token $TOKEN \
  --tool-name search_docs \
  --tool-args @params.json \
  --quiet \
  --format json
```

For generated payloads, pipe JSON through stdin:

```bash theme={"theme":"css-variables"}
echo '{"query":"setup guide"}' | mcpjam tools call --url $URL --access-token $TOKEN \
  --tool-name search_docs --tool-args - --quiet --format json
```

`--tool-args-stdin` is equivalent to `--tool-args -` and cannot be combined with `--tool-args` or `--params` in the same command:

```bash theme={"theme":"css-variables"}
generate-params | mcpjam tools call --url $URL --access-token $TOKEN \
  --tool-name search_docs --tool-args-stdin --quiet --format json
```

### Using a credentials file

Instead of passing `--access-token` to every command, save credentials once and reuse the file:

```bash theme={"theme":"css-variables"}
# Login and save
mcpjam oauth login --url https://your-server.com/mcp --credentials-out creds.json

# Reuse everywhere
mcpjam tools list --url https://your-server.com/mcp --credentials-file creds.json
mcpjam resources list --url https://your-server.com/mcp --credentials-file creds.json
mcpjam tools call --url https://your-server.com/mcp --credentials-file creds.json \
  --tool-name search_docs --tool-args @params.json --quiet --format json
```

### No-auth servers

For servers that don't require authentication (like `mcp.excalidraw.com/mcp`), omit the auth flags:

```bash theme={"theme":"css-variables"}
mcpjam tools list --url https://mcp.excalidraw.com/mcp
mcpjam tools call --url https://mcp.excalidraw.com/mcp \
  --tool-name read_me
```

Stdio child processes inherit the parent shell environment by default. Use
`-e/--env` to add values or override inherited ones for local subprocesses.


# XAA Debugger (CLI)
Source: https://docs.mcpjam.com/cli/xaa

Run the full Cross-App Access (ID-JAG) grant chain against your authorization server and MCP server, headlessly, from one command

`mcpjam xaa run` drives a complete Cross-App Access flow — the
[Identity Assertion Authorization Grant](https://datatracker.ietf.org/doc/draft-ietf-oauth-identity-assertion-authz-grant/)
(ID-JAG, an active IETF Internet-Draft) — against your authorization server and
MCP server. MCPJam plays the **enterprise identity provider** (it mints and
signs the ID-JAG) and the **client/agent** (it redeems the ID-JAG and calls
your MCP server with the resulting access token). Your authorization server
and MCP server are the system under test.

```bash theme={"theme":"css-variables"}
mcpjam xaa run \
  --url http://localhost:8788/mcp \
  --issuer-base-url http://localhost:6274/api/mcp \
  --sub alice@example.com \
  --client-id my-registered-client
```

## Recommended: use with Agent Skills

The easiest way to use the CLI is through the
[MCPJam skill](https://github.com/MCPJam/inspector/tree/main/skills/mcp-inspector),
which gives your agent full context on every command and workflow.

```bash theme={"theme":"css-variables"}
npx skills add mcpjam/inspector --skill mcp-inspector
```

Once installed, your agent will know how and when to invoke `mcpjam` commands
automatically — and how to read `xaa run` output conservatively (advertisement
is evidence, redemption is the verdict; an issuer it can't reach is your setup,
not a server bug).

## The three parties

Cross-App Access separates trust into three relationships that are easy to
conflate. The CLI covers two of them and simulates the third:

| Relationship | Who configures it                                        | In this test                                                    |
| ------------ | -------------------------------------------------------- | --------------------------------------------------------------- |
| Client ↔ IdP | Enterprise SSO registration                              | Simulated — the CLI **is** the IdP, so this leg always succeeds |
| RAS ↔ IdP    | Your authorization server trusts the IdP's issuer + JWKS | **You set this up once** (see below)                            |
| Client ↔ RAS | OAuth client registration at your authorization server   | Exercised per run via `--registration`                          |

## Before you run: make the issuer reachable

The CLI signs ID-JAGs with a local key pair (`~/.mcpjam/xaa-idp-private.pem`;
override the directory with `XAA_IDP_KEY_DIR`). Your authorization server must
be able to fetch the matching public key, so `--issuer-base-url` must be an
origin that **publishes that same key** — not your real IdP's URL.

The local MCPJam inspector serves exactly this: it publishes the issuer
metadata and JWKS from the same key directory at
`/api/mcp/xaa/.well-known/openid-configuration` and
`/api/mcp/xaa/.well-known/jwks.json`, no authentication required.

```bash theme={"theme":"css-variables"}
mcpjam inspector start    # serves the issuer at http://localhost:6274/api/mcp/xaa
```

* **Local authorization server:** pass
  `--issuer-base-url http://localhost:6274/api/mcp`.
* **Cloud authorization server:** it cannot reach your localhost. Expose the
  inspector origin through a tunnel and pass the public origin instead.

Then register the issuer in your authorization server: trust
`<issuer-base-url>/xaa` as an ID-JAG issuer and point it at the JWKS URL above.

The first flow step, `verify_issuer_publication`, fails fast if the issuer
origin is unreachable or publishes a different key — nothing is sent to your
servers until it passes.

## What a run checks

Each run walks the grant chain and reports every step:

1. **`verify_issuer_publication`** — the configured issuer publishes the CLI's
   local signing key
2. **`discover_resource_metadata`** — protected-resource metadata
   ([RFC 9728](https://www.rfc-editor.org/rfc/rfc9728)) names the
   authorization server protecting `--url`
3. **`discover_authz_metadata`** — authorization-server metadata
   ([RFC 8414](https://www.rfc-editor.org/rfc/rfc8414)) provides the token
   endpoint and capability advertisements
4. **`mint_id_jag`** — the mock IdP exchanges the simulated user's identity
   assertion for an ID-JAG (`typ: oauth-id-jag+jwt`, audience = your
   authorization server, `resource` = your MCP server)
5. **`redeem_id_jag`** — the ID-JAG is presented at your token endpoint via
   the JWT bearer grant
   ([RFC 7523](https://www.rfc-editor.org/rfc/rfc7523)); your server
   validates it and issues its own access token
6. **`authenticated_mcp_request`** — an MCP `initialize` with the issued
   token, advertising the
   [Enterprise-Managed Authorization extension](https://github.com/modelcontextprotocol/ext-auth/blob/main/specification/stable/enterprise-managed-authorization.mdx)

The result also records **capability evidence** separately from operational
outcomes: whether your authorization server advertises the ID-JAG grant
profile and the JWT bearer grant, which token-endpoint auth method was
selected, and whether your MCP server advertises the enterprise-managed
authorization extension back. Missing advertisements are reported as findings
but never fail a flow that operationally succeeds — redemption is the verdict,
advertisement is evidence.

Capability evidence fields are three-valued: `advertised`, `not_advertised`
(the metadata key was present but did not include the expected value), or
`unknown` (the metadata key was absent entirely). `unknown` is weaker evidence
than `not_advertised` — the two are not equivalent.

The decoded ID-JAG claims and a local signature-verification verdict
(`idJag.verified`) are included in the result so you can inspect exactly what
your authorization server received. `idJag.verified` confirms the CLI's own
mint was well-formed and correctly signed; it is not evidence that your
authorization server validated the assertion — the AS's behavior is in the
`redemption` block. Raw tokens, assertions, and secrets are always
`[REDACTED]` in the output, including when a server reflects them back inside
an error body.

## Registration strategies

`--registration` selects how the CLI identifies itself to your authorization
server, mirroring the client-registration models MCP clients use:

| Strategy                                 | Credentials needed                           | Posture notes                                                                                                                                                                                                                                                                     |
| ---------------------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `preregistered` (default)                | `--client-id` (+ optional `--client-secret`) | Matches the draft's recommended deployment                                                                                                                                                                                                                                        |
| `dcr`                                    | none                                         | A diagnostic: the CLI performs open [RFC 7591](https://www.rfc-editor.org/rfc/rfc7591) registration and supplies the IdP→RAS client mapping itself. Each run may leave a registration behind. RFC 7591 conformance findings are reported as warnings in `registration.warnings[]` |
| `cimd`                                   | none                                         | Public client by default — the run completes but is flagged: the draft recommends confidential clients                                                                                                                                                                            |
| `cimd` + `--client-auth private-key-jwt` | none                                         | Confidential CIMD: the CLI generates a local EC P-256 key, publishes it through the hosted reflector, and signs a `client_assertion`. The private key never leaves your machine                                                                                                   |

The `registration.warnings[]` array in the result uses these codes:

| Code                          | Meaning                                                       | Weight                                                                                           |
| ----------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `public_client`               | Client authenticates with no key or secret                    | Posture note — the draft recommends confidential clients, but this is not itself a vulnerability |
| `profile_metadata_not_echoed` | DCR response omitted `authorization_grant_profiles_supported` | Informational — RFC 7591 allows a server to ignore unknown metadata                              |
| `grant_types_not_echoed`      | DCR response omitted `grant_types`                            | Informational — same rationale; redemption is the real signal                                    |
| `missing_no_store`            | DCR credential response lacked `Cache-Control: no-store`      | Low compliance finding — RFC 7591 requires this header for credential-bearing responses          |

For confidential CIMD, **the key is the identity**: the reflector URL derived
from the public key becomes the `client_id`. Deleting
`~/.mcpjam/xaa-client-private.pem` (or changing `XAA_CLIENT_PRIVATE_KEY`)
mints a new identity, and any allowlisting on your authorization server must
be updated.

## Identity assertion formats

`--assertion-format` selects the identity rail the simulated enterprise uses:

* `oidc` (default) — the mock IdP mints an OIDC ID token as the
  token-exchange subject token.
* `saml` — the mock IdP mints a **SAML 2.0 assertion** as the subject token,
  and the ID-JAG carries a `saml-nameid` `sub_id` claim so a SAML-federated
  authorization server can resolve the user.

The ID-JAG itself is always a JWT; the format flag changes the SSO leg and the
subject identifier, not the grant. Either format composes with **any**
registration strategy.

Use `--sub` (and optionally `--email`) to control the simulated user, and
`--scopes` for the permissions requested in both the ID-JAG and the token
request.

## Scenarios

### First run against a local authorization server

```bash theme={"theme":"css-variables"}
mcpjam inspector start   # serves the issuer

mcpjam xaa run \
  --url http://localhost:8788/mcp \
  --issuer-base-url http://localhost:6274/api/mcp \
  --sub alice@example.com \
  --client-id my-registered-client \
  --scopes "mcp.access"
```

### Confidential client without pre-registration

```bash theme={"theme":"css-variables"}
mcpjam xaa run \
  --url http://localhost:8788/mcp \
  --issuer-base-url http://localhost:6274/api/mcp \
  --sub alice@example.com \
  --registration cimd \
  --client-auth private-key-jwt
```

### SAML-federated enterprise

```bash theme={"theme":"css-variables"}
mcpjam xaa run \
  --url http://localhost:8788/mcp \
  --issuer-base-url http://localhost:6274/api/mcp \
  --sub alice@example.com \
  --client-id my-registered-client \
  --assertion-format saml
```

### CI

The JSON result goes to stdout; progress and advisory notes go to stderr. The
exit code is `0` when the flow completes and `1` otherwise, so a run can gate
a pipeline directly. Pass `--quiet` to suppress the advisory notes.

```bash theme={"theme":"css-variables"}
mcpjam xaa run \
  --url "$MCP_SERVER_URL" \
  --issuer-base-url "$ISSUER_ORIGIN" \
  --sub ci-probe@example.com \
  --client-id "$XAA_CLIENT_ID" \
  --quiet > xaa-result.json
```

## Skipping discovery

* `--authz-server-issuer` pins the authorization server and skips
  protected-resource discovery (useful while your MCP server's RFC 9728
  metadata is still in progress).
* `--token-endpoint` pins the token endpoint and skips authorization-server
  discovery entirely. Not valid with `dcr` or `cimd`, which need the metadata
  document for the registration endpoint / CIMD advertisement.
* `--https-only` rejects non-HTTPS and private targets; by default the CLI
  allows `http://localhost` for local development.

## What this is not

These are targeted debugging checks for the current ID-JAG draft, not a
conformance suite. The negative-test scorecard — sending deliberately broken
ID-JAGs (bad signature, wrong audience, expired, and six more) and verifying
your authorization server rejects each — lives in the
[inspector's XAA Debugger](/inspector/xaa-debugger), which also visualizes the
flow and inspects the ID-JAG interactively before it is sent.

## Related standards

* [Identity Assertion Authorization Grant (ID-JAG)](https://datatracker.ietf.org/doc/draft-ietf-oauth-identity-assertion-authz-grant/)
* [OAuth 2.0 Token Exchange (RFC 8693)](https://www.rfc-editor.org/rfc/rfc8693)
* [JWT Bearer Grant (RFC 7523)](https://www.rfc-editor.org/rfc/rfc7523)
* [Protected Resource Metadata (RFC 9728)](https://www.rfc-editor.org/rfc/rfc9728)
* [MCP Enterprise-Managed Authorization](https://github.com/modelcontextprotocol/ext-auth/blob/main/specification/stable/enterprise-managed-authorization.mdx)


# Getting Started
Source: https://docs.mcpjam.com/getting-started

Start testing your MCP server in minutes

## 1. Open MCPJam

**Web**: HTTPS only, runs in the browser with no install. Share server links with teammates. **Terminal** and **Desktop**: HTTP/S and local STDIO.

<CardGroup>
  <Card title="Web App" icon="globe" href="https://app.mcpjam.com">
    HTTPS only. No install. Share with your team.
  </Card>

  <Card title="Terminal" icon="terminal">
    `npx @mcpjam/inspector@latest`. HTTP/S and local STDIO.
  </Card>

  <Card title="Desktop App" icon="monitor" href="/installation">
    Mac or Windows. HTTP/S and local STDIO.
  </Card>
</CardGroup>

## 2. Draw something

On first launch, MCPJam connects the Excalidraw sample server (diagramming MCP app) and opens the **Playground** with this prompt:

```
Draw me an MCP architecture diagram
```

Press **Send**. The diagram renders in the app: tool call, then widget, then inspect. No API key, ngrok, or ChatGPT/Claude subscription needed.

<Check>
  You should see a tool call to `create_view` followed by a rendered diagram inline in the chat. If you don't, check that the Excalidraw server shows **Connected** in the **Connect** tab.
</Check>

<Tip>
  Use the widget debug icons to open tool input/output, CSP activity, widget state, and JSON-RPC for that result.
</Tip>

## 3. Connect your own server

Open the **Connect** tab in the left sidebar (labelled **Servers** in some builds). Click **Add server**.

* **HTTP**: Paste a URL ending in `/mcp`. The web app accepts **HTTPS** URLs only. Desktop and Terminal accept **HTTP** or **HTTPS**. Add a bearer token or use the [OAuth Debugger](/inspector/guided-oauth) if the server requires auth.
* **STDIO** (Desktop and Terminal only): Paste a command such as `npx -y @modelcontextprotocol/server-everything`. Not available in the web app. See [Hosted App](/hosted/overview).

Each **Connected** server is available in the Playground, Tools, Prompts, and Resources. Connect more from the **Connect** tab anytime.

<Check>
  You should see your server show **Connected** with a green dot in the **Connect** tab, and its tools should appear in the Playground's left rail.
</Check>

## 4. Where to go next

<CardGroup>
  <Card title="Playground" icon="joystick" href="/inspector/playground">
    IDE-style workspace for developing against your MCP servers. Chat with frontier models, invoke tools by hand, render OpenAI Apps SDK and MCP app widgets, and inspect every step via Chat / Trace / Raw. Compare up to 3 models side by side; debug widgets with the emulator (iframe, `window.openai`, CSP, device frames, locale).
  </Card>

  <Card title="OAuth Debugger" icon="shield-check" href="/inspector/guided-oauth">
    Step through your MCP authorization flow and inspect each stage. Check conformance across protocol versions (03-26, 06-18, 11-25) and client registration paths: Dynamic Client Registration (DCR), client pre-registration, and CIMD.
  </Card>

  <Card title="Projects" icon="folder" href="/inspector/projects">
    Group servers; share configuration with your team
  </Card>

  <Card title="API keys" icon="key" href="/reference/api-keys">
    The three things called "API key" in MCPJam (LLM provider key, `MCPJAM_API_KEY`, per-server bearer) and when you need each one.
  </Card>
</CardGroup>

## FAQ

<AccordionGroup>
  <Accordion title="Does MCPJam need a database?" icon="database">
    No. MCPJam Inspector runs as a hosted web app, a desktop app, or via `npx` — none of them require you to install or configure a database. Accounts, sharing, and OAuth token storage on [app.mcpjam.com](https://app.mcpjam.com) are handled by the hosted backend; the Terminal and Desktop apps store config locally on disk.
  </Accordion>

  <Accordion title="How do I check which version of Inspector I'm running?" icon="tag">
    All three installs (Web, Desktop, Terminal) show the running version in the **Settings** tab under **About**.

    To see the latest published version on npm, run this. It's useful for confirming an `@latest` pull actually moved you forward:

    ```bash theme={"theme":"css-variables"}
    npm view @mcpjam/inspector version
    ```

    The Desktop and Web apps update on their own. For the Terminal install, the `@latest` tag in `npx @mcpjam/inspector@latest` pulls the most recent release every run.
  </Accordion>

  <Accordion title="Can MCPJam auto-install MCP servers into my repo?" icon="package">
    Not directly. MCPJam doesn't reach into your repo. The closest workflow is **Skills**: install the MCPJam CLI skill into a project so an agent working in that repo knows how to use MCP tools and call MCPJam against your servers. See [Skills](/inspector/skills).
  </Accordion>

  <Accordion title="Where are my files saved?" icon="folder">
    It depends what you mean. The word covers three different things:

    * **OAuth credentials** from `mcpjam oauth login`: `~/.mcpjam/config.json`. Override per command with `--credentials-out`. See [CLI OAuth login](/cli/oauth-login).
    * **Eval results**: uploaded to your MCPJam project when `MCPJAM_API_KEY` is set to an MCPJam API key (`sk_…`). See [Saving Results](/sdk/concepts/saving-results).
    * **Inspector UI changes** (server config, client edits): persisted via the **Save** button in the project view.
  </Accordion>

  <Accordion title="How do I sign out of the hosted app?" icon="log-out">
    Open the **account menu** at the bottom of the sidebar and pick **Sign out**. You'll be returned to the public app as a guest. See [Hosted App → Signing out](/hosted/overview#signing-out).
  </Accordion>
</AccordionGroup>


# GitHub Checks
Source: https://docs.mcpjam.com/github-checks

Run an eval suite against the MCP server built from every pull request, and read the result as a check run, a pull-request comment, and a machine-readable JSON block.

Connect a repository and MCPJam runs the eval suite bound to it against the MCP
server built from each pull request's **head commit**, then reports back as a
check run named **MCPJam Evals**.

Nothing about the check reads your default branch. It builds the pull request,
starts the server it produces, runs the suite against that server, and reports
what it measured.

<Note>
  GitHub Checks is configured in the hosted app, at **Settings → Integrations →
  GitHub**. Every write on this page — connecting, installing, disconnecting, and
  changing per-repository settings — requires an organization owner or admin.
  Members can open the page to see the current setup, but all controls are
  disabled and a note explains why.

  **Connect a GitHub account** starts it. You sign in to GitHub so we can confirm
  which accounts you administer, then pick one from the list that comes back —
  an organization, or your own account. Installing the app is not on its own
  proof that an installation is yours to connect, which is why the sign-in step
  is there and not skippable.

  Accounts that already have the app appear in that list ready to connect.

  **Install on another account**, in that same list, is **not generally available
  yet.** During the beta the MCPJam app can only be installed on the MCPJam
  organization, so there is no other account for it to reach — following it sends
  you to the existing installation rather than offering a choice. Even once that
  opens up, GitHub sends an administrator who already has the app somewhere to
  that installation instead of asking which account to use. So for now, connect
  an account that already appears in the list.

  A GitHub account connects to **one MCPJam organization at a time**, for the
  whole account rather than per repository. Two organizations cannot split the
  repositories of one GitHub account between them. If the account is already
  connected elsewhere, connecting it again is refused — free it up in the
  organization holding it, or use a different account.

  Which organization is holding it is **rolling out**: until it reaches your
  deployment the refusal names nothing, so ask an owner of the GitHub account,
  who can see the installation. After it reaches you the account picker marks
  the account before you click, and names the holding organization when you are
  a member of it — when you are not, it still cannot name it, because that is a
  fact about an organization you cannot see.

  The **pull-request comment** described below is rolling out: it is enabled per
  deployment by an operator, and until it is on for yours a connected repository
  gets the check run and no comment. The per-repository toggle is visible before
  then and records your choice; nothing is posted until the rollout reaches you.
</Note>

## Suite credentials in approved forks

Each connected repository has **Allow suite credentials in approved forks** in
Settings and in its suite's connected-repository section. It is off by default;
only organization admins can change it.

* **Off:** approved fork runs require a credential-free suite. Checks needing
  saved credentials report **Action required**.
* **On:** approved fork runs can use that suite's existing credentials and
  connected services. Only approve code you trust. This does not inject saved
  secrets into the PR server.

Every new commit needs maintainer approval. Enabling the setting does not upgrade
old approvals or running jobs: approve again. Changes to selected credentials
also require fresh approval. Disabling blocks further credential access and
stops affected runs; credentials already delivered cannot be taken back.
Changing the connected suite resets the setting off. Environments with plugin
credential grants that cannot be verified remain blocked.

## `mcpjam.yaml`

MCPJam can usually work out how to build and start a server on its own. When it
cannot — or when you would rather say it explicitly than let detection guess —
commit an `mcpjam.yaml` at the **root of the repository**.

A committed file is authoritative: MCPJam runs what it says and never falls back
to detection. A file that is present but invalid fails the check rather than
being ignored, because ignoring it would run a server the author never declared.

```yaml mcpjam.yaml theme={"theme":"css-variables"}
version: 1

checks:
  build: npm ci && npm run build
  start: node dist/server.js
  port: 8080
  path: /mcp
  env:
    LOG_LEVEL: debug
```

### Fields

| Field              | Required | What it is                                                                                                           |
| ------------------ | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `version`          | Yes      | Must be `1`.                                                                                                         |
| `checks.build`     | Yes      | The command that installs and compiles. An install-only ecosystem still declares one — `npm ci`, `uv sync --frozen`. |
| `checks.start`     | Yes      | The command that starts the server. It must keep running.                                                            |
| `checks.port`      | Yes      | An integer, 1–65535. The port your server listens on.                                                                |
| `checks.path`      | Yes      | The MCP endpoint path. Must start with `/`.                                                                          |
| `checks.transport` | No       | Only `streamable-http`, which is also the default. Version 1 of this file is HTTP-only.                              |
| `checks.env`       | No       | Environment variables to set before `start`.                                                                         |

Unknown keys **inside `checks:`** are rejected — inside the authoritative
section a stray key is a typo that would otherwise silently change what runs.
Unknown keys at the top level are ignored, so the file can grow new sections
without breaking older checks. The file itself is capped at 32 KB.

### `env`

`checks.env` is a flat mapping of at most 20 entries. Names must be uppercase
environment-variable names (`^[A-Z][A-Z0-9_]*$`, up to 64 characters) and values
must be **strings** of up to 1024 characters — quote anything that looks like a
number, a boolean, or an empty value, or the file is rejected rather than
coerced.

<Warning>
  **Never put a credential in `checks.env`.** This file is committed to the
  repository, so it is readable by anyone who can read the repository, and MCPJam
  stores its values verbatim alongside the check. It is for the flags a server
  needs in order to boot — `LOG_LEVEL`, `FIXTURE_MODE` — and for nothing that
  would matter if it were printed in public.
</Warning>

### Binding and ports

**The port is what has to match; the bind address is not.** MCPJam reaches your
server through the sandbox's own host bridge, from inside the box, so a server
bound to loopback is reachable. What fails is listening on a port other than the
one `checks.port` declares, or a process that exits after startup — both are
reported as an unhealthy server, and the check body says so.

If your framework needs a `HOST` (or equivalent) variable set to bind the way you
want, set it in `checks.env` like any other boot flag. Only add it if your
framework actually needs it — it is not something MCPJam requires.

## The pull-request comment

Alongside the check run, MCPJam posts **one comment per pull request** and
updates that same comment in place on every later push. It is posted for every
conclusion, including a pass — so the comment is always current, and there is
never a thread of stale verdicts to scroll past.

The comment carries:

* **The verdict line** — what the check concluded on this head commit.
* **What the result is not.** A check that could not produce a verdict says so
  in the comment rather than leaving you to infer it from a colour.
* **"What to fix"** — one line per failing case, saying what to go and change.
  Each line names the case and the one thing to look at; it does not claim to
  know why the case failed.
* **A collapsed JSON block**, described below, for coding agents.

Comments are posted for every connected repository by default, and an admin can
turn them off per repository — see [Turning comments
off](#turning-comments-off).

## The machine-readable JSON block

The last section of the comment is a collapsed, fenced JSON block. It exists so a
coding agent working on the pull request can read the result without parsing
prose, and it is stable enough to be worth parsing.

It is a **bounded projection** of MCPJam's eval decision contract — not the whole
contract, and not a dump of the run. It carries:

| Key                         | What it says                                                                                                        |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `schemaVersion`             | The version of this projection. Read it before anything else.                                                       |
| The head SHA                | The commit this result describes. A result whose SHA is not the tip is out of date.                                 |
| The outcome and conclusion  | What MCPJam measured, and what it reported to GitHub.                                                               |
| Trial counts                | How many trials ran, passed, and failed.                                                                            |
| One entry per failing trial | Its **first failed stage**, its failure category, and the tool names the case expected versus the ones it observed. |
| `omittedFields`             | The fields of the contract this projection deliberately leaves out.                                                 |

Two things to know when reading it:

* **It is a projection, and it says so.** Anything the projection drops is named
  in `omittedFields`, so a consumer can tell "MCPJam did not include this" from
  "MCPJam measured nothing here". Do not infer an absent field's value.
* **The first failed stage is a location, not a diagnosis.** It says where the
  chain stopped — connection, discovery, selection, the tool call, the response,
  or the user's request being satisfied — and nothing about why. Treating it as a
  cause is how a change lands in the wrong file.

## Review comments on removed tool names

When a failing case asserts a tool that was never called, MCPJam looks for that
tool name in the pull request's own diff. If the diff **removes** a line
containing the name in quotes, and the name does not come back on any added line,
MCPJam leaves a single review comment on that removed line.

That comment is a **location, not a cause**. It says: the case expected this
tool, this line is where the name stopped existing, start here. It does not claim
the removal is a mistake — renaming or deleting a tool on purpose is an ordinary
thing for a pull request to do, and when it is, the fix is to update the case in
MCPJam rather than to put the line back.

MCPJam never requests changes. Review comments are posted as plain comments, so
they never sit as a blocking review on your pull request.

## What appears in a comment, and what never does

Comments and review comments are **public on a public repository**. What MCPJam
writes there is deliberately bounded.

**What can appear:**

* fixed MCPJam copy — the same sentences on every repository;
* the head SHA, the check-run id, and the pull-request number;
* links back into MCPJam;
* eval case titles, as you wrote them;
* tool names;
* stage reasons and counts;
* the same clamped build and server output the check run already shows.

**What never appears:**

* organization, project, or installation identifiers;
* model or provider names;
* prompts;
* tool-call arguments;
* credentials — evidence strings are credential-redacted before they are
  rendered.

If a case title or a tool name is itself sensitive, treat it as published: those
are the two fields that carry your own words into a public comment.

## Per-repository switches

Each connected repository has three labeled switches:

| Switch          | What it controls                                                                                                                                                                                      |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Checks**      | Whether the eval suite runs on pull requests to this repository.                                                                                                                                      |
| **Conformance** | Whether a conformance check runs alongside the eval suite. Disabled and dimmed while **Checks** is off, because it is a sub-setting of checks.                                                        |
| **Comments**    | Whether MCPJam posts a pull-request comment with the result. Independent of **Checks** — you can pause checks while still allowing MCPJam to comment, or turn comments off while checks keep running. |

## Turning comments off

Comments are **on for every connected repository by default**, including
repositories connected before the feature existed. An admin opts a repository
out; there is nothing to opt in to.

Go to **Settings → Integrations → GitHub**, find the repository in **Connected
repositories**, and turn off the **Comments** switch.

This setting changes nothing about the check itself — whether it runs, what it
measures, and how it reports are all decided elsewhere. Only the comment stops.


# My First ChatGPT App
Source: https://docs.mcpjam.com/guides/first-chatgpt-app-react

Build a simple Coffee Shop ChatGPT App with React

We're going to create a simple Coffee Shop ChatGPT App where users can order and drink coffees!

## Getting Started

Clone the repo and navigate to the Coffee Shop example:

```bash theme={"theme":"css-variables"}
git clone https://github.com/MCPJam/inspector.git
cd inspector/examples/chatgpt-apps/CoffeeShop
```

## Setting up the MCP Server

### Creating the MCP Server

```typescript server.ts theme={"theme":"css-variables"}
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { createServer, IncomingMessage, ServerResponse } from "http";

const server = new McpServer({
  name: "coffee-shop",
  version: "1.0.0"
});
```

### Registering Resources and Tools

Widget resources and tools work together. The resource provides the widget HTML, and tools reference it to display your UI:

```typescript server.ts theme={"theme":"css-variables"}
// Register the widget resource
server.registerResource(
  "coffee-widget",
  "ui://widget/coffee.html",                // Identifier that tools reference via openai/outputTemplate
  { description: "Coffee Shop widget" },
  async () => ({
    contents: [{
      uri: "ui://widget/coffee.html",
      mimeType: "text/html+skybridge",      // Marks this as a widget that receives window.openai
      text: WIDGET_HTML,
      _meta: {
        "openai/widgetPrefersBorder": true,  // Adds a border around your widget in the chat
      }
    }]
  })
);

// Register a tool that uses this widget
server.registerTool(
  "orderCoffee",
  {
    title: "Order Coffee",
    description: "Order a coffee to add to your collection.",
    _meta: {
      "openai/outputTemplate": "ui://widget/coffee.html",    // Which widget to display
      "openai/widgetAccessible": true,                       // Allow widget buttons to call this tool
      "openai/toolInvocation/invoking": "Brewing coffee...", // Loading message while tool runs
      "openai/toolInvocation/invoked": "Coffee ready!"       // Success message when tool completes
    }
  },
  async () => {
    // ...
    return {
      structuredContent: {                                   // Data for widget + model reasoning
        coffeeCount: coffeeCount,
        message: "Here's your coffee! ☕️"
      },
      content: [{                                            // Text the model uses to craft its response
        type: "text" as const,
        text: `Ordered a coffee! You now have ${coffeeCount} coffees.`
      }]
    };
  }
);
```

## Understanding the Widget

ChatGPT Apps can display interactive widgets inside the chat. Widget resources are registered by your app and become available when the client connects to your MCP server. Your widget can be built using vanilla JavaScript or a framework like React (optionally with TypeScript), and is bundled into a self-contained HTML file. When your tool is called, the client renders this HTML inside a sandboxed iframe and injects `window.openai` into it, which is how your widget communicates with the client and invokes tools on your MCP server.

### The `window.openai` API

`window.openai` provides globals and methods for calling tools, sending follow-ups, and managing layout.

Here's a basic React widget structure using the `window.openai` API:

```tsx src/CoffeeShopWidget.tsx theme={"theme":"css-variables"}
import { StrictMode, useState, useCallback, useRef, useEffect } from "react";
import { createRoot } from "react-dom/client";
import { useToolOutput } from "./hooks/useToolOutput"; // Listens for toolOutput changes via the openai:set_globals event
import type { CoffeeToolOutput } from "./types"; // TypeScript types for window.openai and our tool's output

function CoffeeShopWidget() {
  const toolOutput = useToolOutput();
  const [state, setState] = useState<CoffeeToolOutput>({
    // ... fallback values for TypeScript
  });

  // Sync state when toolOutput changes (e.g., from chat commands)
  const prevToolOutputRef = useRef<CoffeeToolOutput | undefined>(undefined);
  useEffect(() => {
    if (toolOutput && toolOutput !== prevToolOutputRef.current) {
      prevToolOutputRef.current = toolOutput;
      setState(toolOutput);
    }
  }, [toolOutput]);

  // Call tools directly from button clicks (requires openai/widgetAccessible: true)
  const handleOrder = useCallback(async () => {
    const result = await window.openai?.callTool("orderCoffee", {});
    if (result?.structuredContent) {
      setState(result.structuredContent);
    }
  }, []);

  // ... more handlers and JSX in the full example
}
```

* **`window.openai.toolOutput`** - The `structuredContent` your MCP server returned. It's the data your tool sends to both the widget and the model for context
* **`window.openai.callTool()`** - Lets widget buttons trigger server tools directly (requires `openai/widgetAccessible: true` in the tool's metadata)
* **`openai:set_globals`** - Event that fires when users trigger tools via chat (e.g., "order me a coffee"), keeping everything in sync

<Note>
  Our Coffee Shop stores state on the server, so the widget just reads `toolOutput`. If you need to persist state in the widget and expose it to the client, use `window.openai.widgetState` and `window.openai.setWidgetState()`.
</Note>

For more on the `window.openai` component bridge (file uploads, modals, follow-up messages, and more), see the [OpenAI Apps SDK docs](https://developers.openai.com/apps-sdk/reference#windowopenai-component-bridge).

### Display Modes

Widgets can request different display modes to optimize their presentation:

* **Inline** (default) - Widget renders within the chat message flow
* **Picture-in-Picture (PiP)** - Widget floats at the top of the screen, staying visible while scrolling
* **Fullscreen** - Widget expands to fill the entire viewport

Widgets start in inline mode. To request a different mode:

```javascript theme={"theme":"css-variables"}
window.openai.requestDisplayMode({ mode: "pip" });
window.openai.requestDisplayMode({ mode: "fullscreen" });
```

Users can exit PiP or fullscreen by clicking the close button, returning to inline. Our Coffee Shop uses the default inline mode, but you can test all three in MCPJam Inspector's Playground.

When your widget is in fullscreen, you can add an **"Open in \<server name>"** button to the header chrome by calling `setOpenInAppUrl`. This lets users open the widget's underlying web app in a new tab:

```javascript theme={"theme":"css-variables"}
window.openai.setOpenInAppUrl({ href: "https://app.example.com/item/42" });
```

Only `http` and `https` URLs are accepted. The link opens in a new tab with `noopener,noreferrer`.

This capability is optional — hosts can turn it off, in which case `window.openai.setOpenInAppUrl` won't be defined. Guard for it before calling, for example `window.openai.setOpenInAppUrl?.({ href: "https://app.example.com/item/42" })`.

### Content Security Policy (CSP)

Widgets run in a sandboxed iframe, so you need to declare which external domains your widget can interact with. Configure these permissions in your resource's `_meta` using one of two formats:

* **`_meta.ui.csp`** (preferred) — uses camelCase field names and is the standard MCP Apps format
* **`_meta["openai/widgetCSP"]`** (legacy) — the original ChatGPT Apps format, still supported

When both are present, `ui.csp` values take precedence field-by-field; any field absent from `ui.csp` falls back to `openai/widgetCSP`.

Both formats support the same domain categories:

| `ui.csp` field    | `openai/widgetCSP` field | Purpose                                            |
| ----------------- | ------------------------ | -------------------------------------------------- |
| `connectDomains`  | `connect_domains`        | Domains your widget can fetch from (API calls)     |
| `resourceDomains` | `resource_domains`       | Domains for static assets (images, fonts, scripts) |
| `frameDomains`    | `frame_domains`          | Domains allowed in nested iframes                  |

<Note>
  Without declaring a domain in your CSP, the sandbox will block the request. Only declare the domains you actually need.
</Note>

#### Third-party APIs with referrer or origin restrictions

Some APIs (such as Google Maps with a referrer-restricted key) validate the page URL before serving a response. Widgets run at a real sandbox URL, so you can allowlist it in your API provider's dashboard. Locally that is `http://localhost:*` **and** `http://127.0.0.1:*` — the sandbox uses whichever of the two you did not open the Inspector with — and `https://sandbox.mcpjam.com` when hosted. See [View origins](/inspector/view-origins) for how to read the exact value.

You still need to declare the API's domains in `_meta.ui.csp`:

* Add script and asset CDN domains to `resourceDomains`
* Add any XHR/fetch endpoints to `connectDomains`

## A note on authentication and monetization

To keep this guide focused on building your first ChatGPT App, we've skipped authentication and monetization. For production apps, check out [OpenAI's authentication docs](https://developers.openai.com/apps-sdk/build/auth) and [monetization docs](https://developers.openai.com/apps-sdk/build/monetization).

## Running Your App

### Build and start the server

```bash theme={"theme":"css-variables"}
npm install
npm start
```

This builds the React widget with Vite and starts the server at `http://localhost:8787/mcp`.

### Testing with MCPJam Inspector

The easiest way to test your app:

1. Run the inspector: `npx @mcpjam/inspector@latest`
2. Enter URL: `http://localhost:8787/mcp`
3. Try your app in our Playground!

### Connecting to ChatGPT

To connect your app to ChatGPT:

1. Sign in to MCPJam Inspector (tunnels require an account)
2. Click **Create Tunnel** on the server card with your server connected
3. Use the tunnel URL as your connector endpoint in ChatGPT

For more information, see [Tunneling](/inspector/connecting-servers#tunneling).

## What's next?

Now that your Coffee Shop is running, you can:

1. **Test the flow** - Call the `orderCoffee` tool to see your widget
2. **Try the buttons** - Click "Order" and "Drink" to interact with your server
3. **Chat naturally** - Say "order me 3 coffees" and watch the widget update
4. **Iterate and expand** - Add more tools, improve the UI, or build something completely new!

Congratulations! You've built your first ChatGPT App! 🎉


# My First MCP App
Source: https://docs.mcpjam.com/guides/first-mcp-app

Build a simple Reservation MCP App from scratch

We're going to create a simple Reservation MCP App for a restaurant called Jammy Wammy!

## Setting up your environment

### Initialize the project

```bash theme={"theme":"css-variables"}
mkdir reservation-app
cd reservation-app
npm init -y
```

### Install dependencies

```bash theme={"theme":"css-variables"}
npm install @modelcontextprotocol/sdk @modelcontextprotocol/ext-apps react react-dom zod
npm install -D @types/node @types/react @types/react-dom typescript vite @vitejs/plugin-react vite-plugin-singlefile cross-env
```

### Configure package.json

Add the following to your `package.json`:

```json theme={"theme":"css-variables"}
{
  "type": "module",
  "main": "server.ts",
  "scripts": {
    "build": "cross-env INPUT=reservation-app.html vite build",
    "start": "node server.ts"
  }
}
```

### Create tsconfig.json

```json theme={"theme":"css-variables"}
{
  "compilerOptions": {
    "target": "ESNext",
    "lib": ["ESNext", "DOM", "DOM.Iterable"],
    "module": "ESNext",
    "moduleResolution": "bundler",
    "allowImportingTsExtensions": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "verbatimModuleSyntax": true,
    "noEmit": true,
    "jsx": "react-jsx",
    "strict": true,
    "skipLibCheck": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "noFallthroughCasesInSwitch": true
  },
  "include": ["src"]
}
```

### Create vite.config.ts

```ts theme={"theme":"css-variables"}
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import { viteSingleFile } from "vite-plugin-singlefile";

const INPUT = process.env.INPUT;
if (!INPUT) {
  throw new Error("INPUT environment variable is not set");
}

const isDevelopment = process.env.NODE_ENV === "development";

export default defineConfig({
  plugins: [react(), viteSingleFile()],
  build: {
    sourcemap: isDevelopment ? "inline" : undefined,
    cssMinify: !isDevelopment,
    minify: !isDevelopment,

    rollupOptions: {
      input: INPUT,
    },
    outDir: "dist",
    emptyOutDir: false,
  },
});
```

## Creating your first View

Now let's create the HTML file and React component that will be displayed when the MCP tool is called.

### Create reservation-app.html

First, create the HTML file that will serve as the entry point for your app:

```html theme={"theme":"css-variables"}
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta name="color-scheme" content="light dark">
  <title>Reservation App</title>
</head>
<body>
  <div id="root"></div>
  <script type="module" src="/src/reservation.tsx"></script>
</body>
</html>
```

This is a standard HTML file with a few important elements:

* A `root` div where React will mount your app
* A module script that imports your React component
* Color scheme meta tag for light/dark mode support

### Create src/reservation.tsx

```tsx theme={"theme":"css-variables"}
import { useApp } from "@modelcontextprotocol/ext-apps/react";
import { StrictMode, useCallback } from "react";
import { createRoot } from "react-dom/client";

const IMPLEMENTATION = { name: "Reservation App", version: "1.0.0" };

function ReservationApp() {
  const { app, error } = useApp({
    appInfo: IMPLEMENTATION,
    capabilities: {},
  });

  const handleMenuRequest = useCallback(async () => {
    if (!app) return;
    try {
      await app.sendMessage({
        role: "user",
        content: [{ type: "text", text: `What's on the menu at Jammy Wammy?` }],
      });
    } catch (e) {
      console.error("Failed to send message:", e);
    }
  }, [app]);

  if (error) return <div><strong>ERROR:</strong> {error.message}</div>;
  if (!app) return <div>Loading...</div>;

  return (
    <main style={{
      padding: "20px",
      fontFamily: "system-ui, -apple-system, sans-serif",
      maxWidth: "600px",
      margin: "0 auto"
    }}>
      <h1 style={{ fontSize: "24px", marginBottom: "16px" }}>
        🎉 Reservation Confirmed!
      </h1>
      
      <p style={{ fontSize: "16px", marginBottom: "24px" }}>
        Your table at <strong>Jammy Wammy</strong> is ready!
      </p>

      <button
        onClick={handleMenuRequest}
        style={{
          padding: "12px 24px",
          fontSize: "16px",
          backgroundColor: "#007bff",
          color: "white",
          border: "none",
          borderRadius: "6px",
          cursor: "pointer",
          fontWeight: "500"
        }}
      >
        What's on the menu?
      </button>
    </main>
  );
}

createRoot(document.getElementById("root")!).render(
  <StrictMode>
    <ReservationApp />
  </StrictMode>,
);
```

### Understanding the MCP App features

#### The `useApp` hook

The `useApp` hook is the foundation of your MCP App. It establishes the connection between your UI and the MCP host.

```tsx theme={"theme":"css-variables"}
const { app, error } = useApp({
  appInfo: IMPLEMENTATION,
  capabilities: {},
});
```

* **`appInfo`**: Identifies your app with a name and version
* **`capabilities`**: Declares what features your app supports (empty for this simple example)
* **`app`**: The app instance you'll use to interact with the MCP host
* **`error`**: Contains any connection errors

#### Sending messages with `app.sendMessage()`

The `sendMessage` method lets your app send follow-up messages to the LLM, enabling interactive conversations:

```tsx theme={"theme":"css-variables"}
await app.sendMessage({
  role: "user",
  content: [{ type: "text", text: `What's on the menu at Jammy Wammy?` }],
});
```

This lets your UI trigger additional LLM interactions. When the user clicks "What's on the menu?", the message is sent back to the LLM, which can then call another MCP tool (like `get-menu`) to respond.

#### What else can your app do?

The `app` object provides several methods to interact with the MCP host:

* **`app.sendMessage()`** - Send messages to the LLM (as shown above)
* **`app.callServerTool()`** - Call other MCP tools on your server directly
* **`app.sendLog()`** - Send log messages to the host for debugging
* **`app.openLink()`** - Request the host to open a URL (only `http` and `https` URLs are allowed; other schemes resolve `{ isError: true }`)
* **`app.getHostContext()`** - Get information about the host environment (safe area insets, theme, etc.)

For a complete reference of all available methods and capabilities, see the [MCP Apps API documentation](#).

#### Mounting the React app

Don't forget to mount your app to the DOM:

```tsx theme={"theme":"css-variables"}
createRoot(document.getElementById("root")!).render(
  <StrictMode>
    <ReservationApp />
  </StrictMode>,
);
```

## Setting up the MCP Server

Now let's create the server that will register your tools and serve the UI.

### Create server.ts

```ts theme={"theme":"css-variables"}
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import type { CallToolResult, ReadResourceResult } from "@modelcontextprotocol/sdk/types.js";
import fs from "node:fs/promises";
import path from "node:path";
import { registerAppTool, registerAppResource, RESOURCE_MIME_TYPE } from "@modelcontextprotocol/ext-apps/server";

const DIST_DIR = path.join(import.meta.dirname, "dist");

export function createServer(): McpServer {
  const server = new McpServer({
    name: "Jammy Wammy Reservation Server",
    version: "1.0.0",
  });

  const resourceUri = "ui://reservation/reservation-app.html";

  // Resource: The built HTML file
  registerAppResource(server,
    resourceUri,
    resourceUri,
    { mimeType: RESOURCE_MIME_TYPE },
    async (): Promise<ReadResourceResult> => {
      const html = await fs.readFile(path.join(DIST_DIR, "reservation-app.html"), "utf-8");
      return {
        contents: [{ uri: resourceUri, mimeType: RESOURCE_MIME_TYPE, text: html }],
      };
    },
  );

  // Tool: get-reservation - Shows the reservation UI
  registerAppTool(server,
    "get-reservation",
    {
      title: "Get Reservation",
      description: "Make a reservation at Jammy Wammy restaurant.",
      inputSchema: {},
      _meta: { ui: { resourceUri } },
    },
    async (): Promise<CallToolResult> => {
      return { 
        content: [{ 
          type: "text", 
          text: "Reservation confirmed at Jammy Wammy! 🎉" 
        }] 
      };
    },
  );

  // Tool: get-menu - Returns the menu (no UI)
  server.registerTool(
    "get-menu",
    {
      title: "Get Menu",
      description: "Get the menu for Jammy Wammy restaurant.",
      inputSchema: {},
    },
    async (): Promise<CallToolResult> => {
      const menu = `
🍽️ Jammy Wammy Menu

Appetizers:
- Bruschetta - $8
- Calamari - $12

Main Courses:
- Margherita Pizza - $16
- Spaghetti Carbonara - $18
- Grilled Salmon - $24
- Chicken Parmesan - $20

Desserts:
- Tiramisu - $9
- Panna Cotta - $8

Beverages:
- House Wine (glass) - $10
- Craft Beer - $7
- Fresh Lemonade - $4
      `.trim();

      return { 
        content: [{ 
          type: "text", 
          text: menu 
        }] 
      };
    },
  );

  return server;
}

async function main() {
  const server = createServer();
  const transport = new StdioServerTransport();
  await server.connect(transport);
}

main().catch((e) => {
  console.error(e);
  process.exit(1);
});
```

### Understanding the server components

#### Creating the MCP Server

```ts theme={"theme":"css-variables"}
const server = new McpServer({
  name: "Jammy Wammy Reservation Server",
  version: "1.0.0",
});
```

This initializes your MCP server with a name and version.

#### Registering the App Resource

```ts theme={"theme":"css-variables"}
const resourceUri = "ui://reservation/reservation-app.html";

registerAppResource(server,
  resourceUri,
  resourceUri,
  { mimeType: RESOURCE_MIME_TYPE },
  async (): Promise<ReadResourceResult> => {
    const html = await fs.readFile(path.join(DIST_DIR, "reservation-app.html"), "utf-8");
    return {
      contents: [{ uri: resourceUri, mimeType: RESOURCE_MIME_TYPE, text: html }],
    };
  },
);
```

`registerAppResource` registers a resource with UI metadata so the host knows where the assets are.

#### Registering an App Tool (with UI)

```ts theme={"theme":"css-variables"}
registerAppTool(server,
  "get-reservation",
  {
    title: "Get Reservation",
    description: "Make a reservation at Jammy Wammy restaurant.",
    inputSchema: {},
    _meta: { ui: { resourceUri } },
  },
  async (): Promise<CallToolResult> => {
    return { 
      content: [{ 
        type: "text", 
        text: "Reservation confirmed at Jammy Wammy! 🎉" 
      }] 
    };
  },
);
```

This is the tool that will be directly called by the LLM to render your tool.
We give it our `resourceUri` in the `_meta` field allowing us to have multiple tools to render different UI.
We can also return text content to the LLM as well to give it more context about our View.

#### Registering a Regular Tool (without UI)

```ts theme={"theme":"css-variables"}
server.registerTool(
  "get-menu",
  {
    title: "Get Menu",
    description: "Get the menu for Jammy Wammy restaurant.",
    inputSchema: {},
  },
  async (): Promise<CallToolResult> => {
    return { 
      content: [{ type: "text", text: menu }] 
    };
  },
);
```

Regular tools work just like standard MCP tools - they return text content without any UI.

#### Starting the Server

```ts theme={"theme":"css-variables"}
async function main() {
  const server = createServer();
  const transport = new StdioServerTransport();
  await server.connect(transport);
}
```

This connects your server to STDIO transport, allowing it to communicate with MCP clients.

## Building and Running Your App

Now that everything is set up, let's build and run your MCP App!

### Build the app

The build process uses Vite to bundle your React component and the HTML file into a single, self-contained HTML file:

```bash theme={"theme":"css-variables"}
npm run build
```

This will create a `dist/reservation-app.html` file with all your JavaScript and CSS inlined - perfect for serving through MCP.

### Run the server

Start your MCP server:

```bash theme={"theme":"css-variables"}
npm start
```

Your server is now running and communicating via STDIO. It's ready to be connected to an MCP client like MCPJam Inspector!

### Testing with MCPJam Inspector

The easiest way to test your app is using MCPJam Inspector:

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector
```

On first launch MCPJam auto-connects a sample Excalidraw server. Open the **Servers** tab and add your own, then head to the **Playground** to render widgets, debug CSP/device behavior, and drive your app with an LLM.

### What's next?

Now that your server is running, you can:

1. **Test the flow** - Call the `get-reservation` tool to see your UI
2. **Try the menu button** - Click "What's on the menu?" and watch the LLM call the `get-menu` tool
3. **Iterate and expand** - Add more tools, improve the UI, or build something completely new!

Congratulations! You've built your first MCP App! 🎉


# Hosted App
Source: https://docs.mcpjam.com/hosted/overview

Use MCPJam Inspector in the browser — no installation required

MCPJam Inspector is available as a hosted web app at [app.mcpjam.com](https://app.mcpjam.com). Build, test, debug, and evaluate MCP servers and apps with frontier models — all from the web.

<Card title="Open MCPJam Inspector" icon="arrow-up-right-from-square" href="https://app.mcpjam.com">
  Launch the hosted inspector in your browser
</Card>

## Why use the hosted app?

* **No installation** — Open a browser and start testing.
* **Always up to date** — You're always on the latest version without running `npx @mcpjam/inspector@latest`.
* **Shareable** — Share server URLs with your team. Everyone can inspect the same server without local setup.

## Share your MCP server

Send a link to your MCP server or app the same way you'd share a Google Doc. Anyone with a MCPJam account who opens it can immediately start chatting with the server right in their browser.

Try it yourself with the [Excalidraw MCP server](https://app.mcpjam.com/shared/excalidraw/mN4lJGfKanjpnBdZvE-lL):

<iframe title="MCPJam Server Sharing" />

## Quick start

1. Go to [app.mcpjam.com](https://app.mcpjam.com)
2. Click **Add server** and enter your MCP server's URL (must be HTTPS)
3. Choose your authentication method (**Auto** is the default — connects without credentials and escalates to OAuth if the server requires it)
4. Start inspecting tools, resources, and prompts

## Usage limits

The hosted app uses a daily credit system to power AI features like chat.

* **Guests** get a free daily usage allowance and can use a subset of the free hosted models. The sidebar shows a progress bar with how much of your daily limit you've used and when it resets. When you reach the limit, you'll be prompted to sign in.
* **Signed-in users** get **10×** the daily usage of a guest. Your daily usage and any paid credit balance are visible in the account menu at the bottom of the sidebar.
* **Paid credits** can be topped up from the billing page and are shown alongside your daily limit in the account menu.
* **When credits run out**, only the free hosted models are locked. Models you access with your own API key (BYOK) remain available regardless of your credit balance.

When your credit balance reaches zero, free models are grayed out in the model picker so you can see the limit before sending a message. Models from your own API keys remain available. If you have paid top-up credits, free models stay enabled even after your daily free allowance is exhausted.

## Signing out

Open the **account menu** at the bottom of the sidebar (click your avatar or email) and choose **Sign out**. You'll be returned to the public app as a guest, and any signed-in usage allowance is replaced by the guest allowance until you sign back in.

OAuth tokens you connected while signed in remain in your hosted workspace and are restored the next time you sign in on the same account.

## OAuth token reveal

When you connect an OAuth server in the hosted app, your OAuth credentials are stored securely in Vault. You can inspect those tokens on demand from the server's **Overview** tab without them ever being written to `localStorage` or shared with other users.

To reveal tokens:

1. Open the **Servers** tab and click the server name to open its detail modal.
2. Go to the **Overview** tab.
3. Under **OAuth Tokens**, click **Reveal tokens**.
4. Each token (access token, refresh token, ID token) is masked by default. Click the eye icon next to a token to reveal its value, or click the copy icon to copy it directly without revealing it.

Expired tokens are automatically refreshed before being returned. Revealed values are kept component-local and are cleared when you close the modal.

<Note>
  The **Reveal tokens** button only appears after the server has been synced to
  your hosted workspace. If the button is not visible, disconnect and reconnect
  the server to trigger a sync.
</Note>

## What's different

The following features are not available in the hosted app. They are available when running MCPJam locally via [npx, Docker, or the desktop app](/installation).

### HTTPS only

The hosted app requires all MCP server connections to use **HTTPS**. If your server only supports HTTP, use the [local inspector](/installation) instead.

### No STDIO connections

STDIO servers are not available in the hosted version of MCPJam. Use a Streamable HTTP endpoint instead, or switch to the [local inspector](/installation).

### OAuth tokens can't auto-refresh for local authorization servers

In hosted mode, the cloud backend handles token refresh on your behalf. If your MCP server's OAuth authorization server runs on a private address (localhost, a LAN IP, or a `.local`/`.internal` hostname), the cloud cannot reach it to refresh tokens after they expire.

When you connect to such a server, the app shows a warning toast. You can continue using the server until the access token expires, but you'll need to re-run the OAuth flow manually each time. For fully-local setups, use the [local inspector](/installation) instead.

### No tunneling

Tunneling exposes a local MCP server over HTTPS for remote access. Since the hosted app connects to already-remote servers, tunneling is not applicable.

### No Tracing tab

The **Tracing** tab is not available in the hosted app. It is a live log of the JSON-RPC traffic between the Inspector and a server you are connected to, and it streams from the local Inspector process's RPC bus — which a hosted deployment does not run. Use the [local inspector](/installation) for the full log. In the hosted app, the JSON-RPC frames for a chat turn are captured and shown with that turn in the Playground instead.

Getting eval traces OUT of the hosted app is a different thing, and it does work: open a completed run and use **Export traces** for an OTLP JSON file (see [Evals](/inspector/evals#exporting-traces)), or fetch the same projection from `GET /api/v1/trace-exports/otlp` in the [public API](/reference/public-api). Both use OpenInference semantic conventions and land in Arize Phoenix, Datadog, or any other OTLP-compatible backend.

## Billing

### Payment history

The **Payment history** card on the org billing page lists your recent credit top-ups. Each row shows:

| Column  | Details                                                |
| ------- | ------------------------------------------------------ |
| Date    | When the top-up was processed                          |
| Amount  | Amount charged in USD                                  |
| Status  | `Succeeded`, `Pending`, or `Failed`                    |
| Receipt | Link to the Stripe-hosted receipt (opens in a new tab) |

If a receipt isn't available yet the cell shows **Processing** (for pending charges) or **Not available** (for succeeded charges without a receipt link).

To add credits, click **Top up** — either from the empty state when you have no history, or from the credit balance card above.

### Spend budget

An owner or admin can cap what the whole organization spends on MCPJam-billed
work — chat, evals, judges and computer time — for each billing period, from
**Organization → Budget**.

* Set the cap in dollars, and choose the percentages to be warned at. The
  default warns at 80%; reaching the cap always notifies, whether or not you
  list it.
* Warnings go to the organization's owners and admins in the notifications
  bell, and to the owner by email.
* **When the budget is reached**, new chat turns, eval runs and computer starts
  are refused for the whole organization. The refusal says the budget is
  reached and that an owner or admin can raise it — buying credits does not
  clear it. Work already in flight finishes, and running computers keep going
  until they hibernate.
* A scheduled eval that hits the budget pauses its schedule rather than
  retrying into the same cap every window; it resumes once you raise the cap or
  the period rolls over.
* Runs that use **your own API keys** are never counted against the budget and
  never blocked by it. Neither is the free daily allowance, which is not
  credits.
* The window is your organization's own billing period: the subscription cycle
  for team and enterprise plans, and the calendar month otherwise. Raising or
  lowering the cap mid-period does not reset what has already been spent.

Any member can see the budget and how much of it is left; only owners and
admins can change it. Every change is recorded in the organization's audit log
with who made it and what it was before.

The same budget is readable and writable from the API at
`/v1/organizations/{organizationId}/spend-budget`.

### Seat payment failures

When someone you invited to a paid-seat organization signs up and the automatic seat charge fails or is cancelled, the org billing page shows a **Seat payment didn't go through** notice. As the org owner you can:

* Click **Retry payment** to reopen the charge and complete payment (including any 3DS challenge).
* Click **Remove invite** to remove the invite entirely if you no longer want to add that person.

The invited member's organization appears greyed out in their sidebar until the seat is paid.


# Slack
Source: https://docs.mcpjam.com/hosted/slack

Run MCPJam from Slack — mention the bot to ask about your projects, and approve the actions that spend credits

<Note>
  The Slack app is rolling out, and its **settings** — connected workspaces, the
  org-wide default project, and channel bindings — are enabled per organization.
  If yours does not have them yet, the Slack card under **Settings →
  Integrations** opens a page with nothing on it. Everything else on this page
  describes the bot itself, which behaves the same once your workspace is
  connected. Ask us to enable it for your organization.
</Note>

The MCPJam Slack app puts the agent where your team already is. Mention it in a
channel or thread and it answers about your projects, servers, and eval runs.
Anything that would **spend credits or change state** comes back as a button
for a human to approve first.

## Install

The app is installed **per workspace**, by someone who can approve apps for it,
from the install link we give you. Installing does not give the bot access to
anyone's MCPJam data — individual accounts are linked separately, below.

Once a workspace is connected — and once the settings above are enabled for
your organization — **Settings → Integrations → Slack** lists it, along with
the org-wide default project and any channel bindings. The bot works in Slack
either way; without those settings there is just nothing to configure here.

## Connect your account

Open the app's **Home** tab in Slack and click **Connect MCPJam**. This links
your Slack user to your MCPJam account, and it is what the bot acts as.

Two things follow from that, and they are the whole security model:

* The bot's own credential names the **bot**, not you. Every request also
  carries your Slack identity, and the server resolves *your* MCPJam account
  and re-checks your org membership.
* If you are not linked, the bot cannot act for you at all — it replies with a
  connect prompt instead. The same is true if you are removed from the
  organization later.

Set a **default project** in the Home tab too. Without one the bot knows who
you are but not where to act, and will say so.

## Ask it something

Mention the bot in any channel it can see:

```
@MCPJam how did the checkout suite do this week?
```

It replies in a thread. Follow-ups in that thread do not need the mention — the
thread is bound to the conversation, so context carries.

Useful things to ask for:

* the status of a recent eval run, or why it failed
* which servers a project has, and whether they are healthy
* a summary of a suite's recent pass rates

## Approvals

When the answer requires doing something — starting an eval run, generating
cases, changing a schedule — the bot does not just do it. It posts what it
proposes with **Approve** and **Dismiss** buttons.

Three things are true of that button, each on purpose:

* **The person who clicks is the person it runs as.** Not whoever asked. The
  server re-resolves the clicker and re-checks their access, so a button left
  in a channel cannot be used by someone without it.
* **The button carries only an id.** What the action does comes from the
  proposal MCPJam stored, never from the button itself.
* **Double-clicking is safe.** The proposal transitions once, server-side. The
  second click is told it lost, not billed.

Approved runs post a status message that updates in place as the run
progresses. If a run fails, the bot attaches screenshots of what broke.

## Removing it

Uninstall the app from Slack's **Apps** settings to end the workspace install.
To unlink just your own account, use **Disconnect** in the app's Home tab —
that leaves the workspace install alone and only stops the bot acting as you.


# Okta SSO & SCIM
Source: https://docs.mcpjam.com/hosted/sso

Sign in to MCPJam with Okta via SAML 2.0, and automate user provisioning and deprovisioning with SCIM

<Note>
  Single sign-on and SCIM provisioning are enterprise features, configured per
  organization. Ask us to enable them for yours and we'll send you a setup link —
  or, where it's enabled, use **Settings → Organization → Configure SSO**.
</Note>

MCPJam supports **SAML 2.0** single sign-on and **SCIM 2.0** user provisioning
with Okta. Your Okta admin does the setup once, in a guided flow, and after
that your directory is the source of truth: assigning someone to MCPJam in Okta
gives them access, and unassigning them takes it away.

Setup takes about 10–15 minutes.

## Before you start

* **An MCPJam organization** on a plan that includes SSO, and a setup link for
  it. Your MCPJam contact provides one; each link is specific to your
  organization.
* **Okta admin access**, enough to add an app integration and configure
  provisioning.
* Your email domain **verified** on your MCPJam organization. We do this for
  you when you're onboarded — it's what lets us route your users to Okta and
  manage their accounts.

## Part 1 — Single sign-on

<Steps>
  <Step title="Open the setup link and choose Okta">
    The setup portal shows two values for your organization: an **ACS URL** and an
    **SP Entity ID**. Keep this tab open — you'll come back to it in step 4.
  </Step>

  <Step title="Add MCPJam in Okta">
    In the Okta Admin Console, go to **Applications → Browse App Catalog**, search
    for **MCPJam**, and add it.

    In the app's SAML setup, paste the values from step 1:

    | MCPJam setup portal | Okta field                  |
    | ------------------- | --------------------------- |
    | ACS URL             | Single Sign-On URL          |
    | SP Entity ID        | Audience URI (SP Entity ID) |
  </Step>

  <Step title="Check the attribute mappings">
    The catalog integration preconfigures the attributes MCPJam needs. Confirm they
    are present:

    | Name        | Value            |
    | ----------- | ---------------- |
    | `email`     | `user.email`     |
    | `firstName` | `user.firstName` |
    | `lastName`  | `user.lastName`  |

    Optionally add a `groups` attribute if you want Okta groups to drive MCPJam
    roles.
  </Step>

  <Step title="Send Okta's metadata back to MCPJam">
    On the app's **Sign On** tab, copy the **Metadata URL**. Paste it into the
    MCPJam setup portal from step 1 and finish. The connection goes active
    immediately.
  </Step>

  <Step title="Assign users and test">
    Assign users or groups to the MCPJam app in Okta. Assigned users can now sign
    in — from the Okta dashboard, or by entering their work email on the MCPJam
    sign-in page, which redirects them to Okta.

    Accounts are created automatically on first sign-in, with membership in your
    MCPJam organization. Nobody needs to be invited by hand.
  </Step>
</Steps>

## Part 2 — Provisioning (SCIM)

SCIM is optional but recommended: it keeps MCPJam in sync with your directory
and, importantly, **removes access immediately** when someone is offboarded
rather than when their session happens to expire.

<Steps>
  <Step title="Open the provisioning setup link">
    Choose Okta. The portal shows a **SCIM endpoint** and a **bearer token**.
  </Step>

  <Step title="Enable provisioning in Okta">
    In the MCPJam app, open **Provisioning → Configure API Integration**. Paste the
    endpoint and token, then enable:

    * **Create Users**
    * **Update User Attributes**
    * **Deactivate Users**
  </Step>

  <Step title="Assign users or groups">
    Okta now manages the lifecycle:

    * assigning someone provisions their MCPJam account and organization
      membership;
    * profile changes sync automatically;
    * unassigning or deactivating someone deactivates their MCPJam membership and
      **revokes their active sessions immediately**.
  </Step>
</Steps>

## Roles

MCPJam organizations separate **owner**, **admin**, and **member** access.
Admins manage members and organization settings; members work in the product
without administrative access. Owners additionally control ownership and
billing.

Users signing in through Okta are members by default. Map an Okta group to the
admin role if you want your Okta groups to decide who administers MCPJam.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Sign-in fails with an audience or recipient mismatch">
    The **Single Sign-On URL** or **Audience URI** in Okta doesn't match the values
    from the MCPJam setup portal. Re-copy both from the portal — they are unique to
    your organization.
  </Accordion>

  <Accordion title="Users sign in but land somewhere unexpected">
    Their email domain may not be verified on your MCPJam organization, so we can't
    tell which organization they belong to. Contact us and we'll verify it.
  </Accordion>

  <Accordion title="Provisioning shows errors in Okta">
    Re-check the SCIM endpoint and bearer token — tokens are specific to one
    directory connection. If you removed and re-created the connection, the old
    token stops working.
  </Accordion>

  <Accordion title="A deprovisioned user still appears in MCPJam">
    Deactivated members are retained for audit purposes, without access — their
    sessions are revoked and they cannot sign in.
  </Accordion>
</AccordionGroup>

## Support

Email [support@mcpjam.com](mailto:support@mcpjam.com) or your MCPJam contact.


# MCPJam
Source: https://docs.mcpjam.com/index

Test, debug, and evaluate MCP servers, MCP apps, and ChatGPT apps.

MCPJam is the development platform for **MCP servers, MCP apps, and ChatGPT apps**. Debug every JSON-RPC message and OAuth exchange, run evaluations across multiple LLMs, and track accuracy over time, so you catch regressions before they ship.

<Frame>
  <img alt="MCPJam Inspector" />
</Frame>

## Start in one command

<CardGroup>
  <Card title="Web" icon="globe" href="https://app.mcpjam.com">
    Open [app.mcpjam.com](https://app.mcpjam.com). HTTPS servers, no install, shareable links.
  </Card>

  <Card title="Terminal" icon="terminal">
    ```bash theme={"theme":"css-variables"}
    npx @mcpjam/inspector@latest
    ```

    HTTP/S and local STDIO.
  </Card>

  <Card title="CLI" icon="square-terminal" href="/cli/overview">
    ```bash theme={"theme":"css-variables"}
    npm i -g @mcpjam/cli
    ```

    Probe, doctor, OAuth, evals — in CI or your shell.
  </Card>
</CardGroup>

## Four surfaces, one product

<CardGroup>
  <Card title="Inspector" icon="binoculars" href="/inspector/playground">
    IDE-style workspace for MCP development. Chat with frontier models, invoke tools by hand, render widgets, compare hosts and models side by side.
  </Card>

  <Card title="CLI" icon="terminal" href="/cli/overview">
    Stateless tool for connectivity, OAuth, conformance, and tool execution. JUnit/JSON reporters built for CI.
  </Card>

  <Card title="SDK" icon="code" href="/sdk">
    `@mcpjam/sdk` — TypeScript toolkit for unit tests, e2e tests, and evals. Works inside Jest or Vitest.
  </Card>

  <Card title="Hosted" icon="cloud" href="/hosted/overview">
    Run MCPJam in the browser at [app.mcpjam.com](https://app.mcpjam.com). Share servers and sessions with your team.
  </Card>
</CardGroup>

## What you can build

<AccordionGroup>
  <Accordion title="Debug an MCP server end-to-end" icon="bug">
    Connect any HTTP, HTTPS, or STDIO server, watch every JSON-RPC message live in the Playground, and inspect tool input/output, widget state, and CSP activity per call.

    [Open the Playground →](/inspector/playground) · [JSON-RPC logger →](/inspector/playground#json-rpc-logger)
  </Accordion>

  <Accordion title="Build an MCP App or ChatGPT App widget" icon="layout-dashboard">
    The unified widget renderer handles both `_meta.ui.resourceUri` (MCP Apps) and `openai/outputTemplate` (ChatGPT Apps). The `window.openai` shim is always injected, so you can iterate without a real host.

    [First MCP App →](/guides/first-mcp-app) · [First ChatGPT App in React →](/guides/first-chatgpt-app-react)
  </Accordion>

  <Accordion title="Validate OAuth conformance across protocol versions" icon="shield-check">
    Step through every stage of the MCP authorization flow against protocol versions 03-26, 06-18, 11-25, and 2026-07-28 (Latest), with DCR, pre-registration, and CIMD coverage.

    [Guided OAuth →](/inspector/guided-oauth) · [CLI OAuth conformance →](/cli/oauth-conformance)
  </Accordion>

  <Accordion title="Run evals across LLMs and gate CI on regressions" icon="flask-conical">
    Author cases with expected tool calls, optional expected outputs, and deterministic checks. Run across Claude, GPT, Gemini, and your own providers — or move the same suite into Jest/Vitest with `@mcpjam/sdk`.

    [Evaluate in Inspector →](/inspector/evals) · [Run evals from the SDK →](/sdk/concepts/running-evals) · [CI integration →](/cli/ci)
  </Accordion>

  <Accordion title="Auto-launch Inspector from your dev workflow" icon="code">
    Launch the Inspector with a prefilled server URL, bearer token, and starting tab via CLI flags — wire it into your dev script or a framework integration.

    [Launch from code →](/inspector/launch-from-code)
  </Accordion>

  <Accordion title="Share a server with your team" icon="users">
    In the hosted app, share a server URL the same way you'd share a Google Doc. Teammates connect with one click and pick up where you left off.

    [Hosted overview →](/hosted/overview)
  </Accordion>
</AccordionGroup>

## Capabilities at a glance

| Capability           | Description                                                                                                                                                                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Playground           | IDE-style workspace combining chat, manual tool calls, widget rendering, Chat/Trace/Raw views, and multi-model and multi-host compare. OpenAI Apps SDK and MCP App UIs, text tools, Chrome DevTools-style widget emulator. [Read more](/inspector/playground) |
| OAuth Debugger       | Guided MCP OAuth conformance checks: protocol versions 03-26, 06-18, 11-25, 2026-07-28 (Latest); DCR, client pre-registration, CIMD. [Read more](/inspector/guided-oauth)                                                                                     |
| MCP Server Debugging | Manually run tools, resources, templates, and elicitation; full JSON-RPC logs.                                                                                                                                                                                |
| Skills               | Skills in the Playground, read from your local filesystem or from a project. [Read more](/inspector/skills)                                                                                                                                                   |
| Projects             | Shared server groups with real-time team sync. [Read more](/inspector/projects)                                                                                                                                                                               |
| Evals                | Test cases with expected tool calls, run across LLMs, metrics. [Read more](/inspector/evals)                                                                                                                                                                  |
| CLI                  | Run MCPJam checks, probes, and evals from the terminal. Perfect for local dev loops and CI. [Read more](/cli/overview)                                                                                                                                        |
| SDK                  | Programmatic access to MCPJam for custom tooling, scripting, and integrations. [Read more](/sdk)                                                                                                                                                              |
| CI/CD                | Run MCPJam checks and evals in GitHub Actions and other CI systems to gate PRs on regressions. [Read more](/cli/ci)                                                                                                                                           |

## Get help

<CardGroup>
  <Card title="Discord" icon="discord" href="https://discord.gg/JEnDtz8X6z">
    Ask the team and the community.
  </Card>

  <Card title="GitHub" icon="github" href="https://github.com/MCPJam/inspector">
    File an issue or open a PR.
  </Card>

  <Card title="Changelog" icon="newspaper" href="/changelog/overview">
    What shipped, and when.
  </Card>
</CardGroup>


# Harness hosts: Claude Code, Codex, and Cursor
Source: https://docs.mcpjam.com/inspector/claude-code-host

Run a real agent runtime — Claude Code, Codex, or the Cursor CLI — inside your project's Computer, and observe native tools, MCP client, and execution behavior directly

A harness host runs a real agent runtime inside your project's Computer (a cloud Linux sandbox) instead of MCPJam's emulated chat loop. You are observing the actual runtime — its native tools, its own agent loop, its real execution behavior — not a simulation of it.

Two of them run on **MCPJam's model access**, and everything below — turn mechanics, host settings, billing, failure modes — describes those two:

* **Claude Code** — runs the real Claude Code agent with Anthropic models. Supports selected MCP servers and native tool approval.
* **Codex** — runs the real Codex CLI agent with OpenAI gpt-5 family models. Supports skills and selected MCP servers (through a host-executed relay — see the caveat below).

The third runs on **your own vendor account**:

* **Cursor CLI** — runs the real Cursor agent, signed in with your own `CURSOR_API_KEY` and billed to that Cursor account. MCPJam mints no model lease for it, so it needs a credential the other two do not. This page documents that credential in [Harness credentials on your own vendor account](#harness-credentials-on-your-own-vendor-account); its turn mechanics and host settings are not covered here yet. The Playground model picker shows **Cursor Auto** locked — the Cursor runtime picks its own model on your account, so the picker has nothing to select.

<Note>
  The Claude Code and Codex hosts are available to organizations where the
  respective feature has been enabled. Contact your account team if you don't
  see them in the host template catalog.
</Note>

## What happens when you run a turn

1. **Pre-flight check.** Before the stream opens, MCPJam verifies that the host can run: the model is supported by the selected harness, the Computer data plane is configured, and the host settings are ones the runtime can honor. Any failure returns a clear error before the turn starts — a turn never silently falls back to the emulated engine.
2. **Where it runs.** By default the host's project Computer is reserved or woken (provisioned on first use). In the desktop app and `npx @mcpjam/inspector`, a Claude Code host can instead run **natively on your own machine** — see [Running Claude Code on your machine](#running-claude-code-on-your-machine) below. A turn never moves between the two on its own: if the target you chose cannot run, you are told why, and you decide.
3. **Credential delivery.** MCPJam installs a short-lived model credential into the sandbox's egress layer outside the VM. Neither the sandbox nor your browser ever holds a real model key. The agent runs pointed at MCPJam's model proxy, which verifies the credential and meters every generation. There is no raw-key fallback.
4. **MCP delivery.** The two harnesses deliver MCP servers differently:
   * **Claude Code (native)** — selected MCP servers are written into the session's MCP config, each pointed at MCPJam's per-server proxy tunnel. The runtime's own MCP client connects to them and the model calls the tools through native function calling. No upstream credentials enter the sandbox.
   * **Codex (host-executed)** — the Codex CLI does not make MCP server tools model-callable in the mode MCPJam drives ([openai/codex#19425](https://github.com/openai/codex/issues/19425)), so MCPJam instead enumerates each selected server's tools at turn start and hands them to the agent as *host-executed* tools. The bridge injects their descriptions into the prompt, the model invokes them through a CLI shim, and **MCPJam executes them on its own server** against the already-authorized connection. Nothing about your servers — no URL, no token — enters the sandbox.
5. **The turn runs.** The agent's own loop executes; native tools run inside the sandbox; file changes land on the Computer's disk; the transcript and trace persist like any other chat.

## Running Claude Code on your machine

<Note>
  Available in the MCPJam desktop app and `npx @mcpjam/inspector` — never in the
  hosted product — and only where MCPJam has enabled it for your account.
</Note>

A Claude Code host can run the real agent as a process on your own computer
instead of in a cloud sandbox. Same agent, same tools, same MCP servers, same
metered model access through MCPJam's proxy — but the files it works on are
your actual checkout, and the commands it runs happen on your machine.

**This is not a sandbox.** The agent runs as your user account. The folder you
grant is where it *starts*, not a boundary it cannot leave: anything your user
can read or change, it can read or change. Edits inside the folder run freely;
commands still ask for approval in chat. If that is not what you want, run the
host hosted, which is the default.

Three things happen once, in this order, before your first local turn:

1. **Install the local runtime.** About 200 MB, downloaded once and shared by
   every project: Claude Code, its supporting runtime, and the Node it runs on.
   MCPJam verifies the download's signature and then verifies the contents
   again after extraction, and refuses anything that does not match what this
   Inspector build expects.
2. **Choose the project folder.** In the desktop app this opens your OS folder
   picker. This is the directory the agent starts in.
3. **Authorize.** The authorization is scoped to this project and this machine,
   and it expires on its own. Changing the runtime, the folder, or the
   permission profile means authorizing again — the old authorization does not
   carry over, by design.

While a local turn runs, MCPJam keeps the model credential in the Inspector
process and hands the agent only a per-session token that works with nothing
else. Stopping a session stops the whole process tree it started, including the
agent binary. **Stop all local sessions** in the local-execution panel ends
everything immediately.

**When it is not offered.** Your platform is not supported,
the runtime is not installed, you are signed in as a guest, the session is a
shared scenario or swarm run, or the feature is not enabled for your account.
The selector says which.

## Host settings

### Claude Code

| Setting                                 | Behavior                                                                                                                                                                                                                                                                                                                                    |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model                                   | Honored — must be an MCPJam-provided Anthropic model. Bring-your-own-key models are not supported and fail closed.                                                                                                                                                                                                                          |
| System prompt                           | Honored — passed to the runtime.                                                                                                                                                                                                                                                                                                            |
| Require tool approval                   | Honored — the toggle is enabled for Claude Code hosts. When on, the turn pauses for your decision before each side-effecting tool call — the runtime's own write-class built-ins (Bash, Write, Edit, and the like) and the tools of any selected MCP server. Read-only tools run freely. Approval and selected MCP servers can be combined. |
| Selected MCP servers                    | Honored — delivered via the session's MCP config through MCPJam's proxy.                                                                                                                                                                                                                                                                    |
| Skills                                  | Honored — runtime skills are materialized into the sandbox.                                                                                                                                                                                                                                                                                 |
| Temperature and other sampling controls | Not honored — the Claude Code runtime owns its own sampling. These controls are grayed out in the UI.                                                                                                                                                                                                                                       |
| Progressive tool disclosure             | Not applied — the real runtime owns tool discovery.                                                                                                                                                                                                                                                                                         |

### Codex

| Setting                                 | Behavior                                                                                                                                                                                                                               |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model                                   | Honored — must be an MCPJam-provided OpenAI gpt-5 family model. Other models are not supported and fail closed.                                                                                                                        |
| System prompt                           | Honored — passed to the runtime.                                                                                                                                                                                                       |
| Require tool approval                   | Not supported — the Codex runtime does not pause for tool approval. The toggle is disabled.                                                                                                                                            |
| Selected MCP servers                    | Honored, via a host-executed relay rather than a native MCP client. Tools are named identically to Claude Code (`mcp__<server>__<tool>`), so traces and eval assertions match across the two harnesses. See the fidelity caveat below. |
| Skills                                  | Honored — runtime skills are materialized into the sandbox.                                                                                                                                                                            |
| Temperature and other sampling controls | Not honored — the Codex runtime owns its own sampling. These controls are grayed out in the UI.                                                                                                                                        |
| Progressive tool disclosure             | Not applied — the real runtime owns tool discovery.                                                                                                                                                                                    |

<Note>
  Host-page tool-level toggles (such as tool visibility) are not re-enforced
  for harness runs. Controls the runtime cannot honor are disabled in the UI
  rather than silently ignored.
</Note>

## Codex MCP fidelity caveat

Codex's MCP support is **not** equivalent to Claude Code's, and the difference matters if you are using it to benchmark a connector:

* The model does **not** call your tools through native function calling. The harness bridge injects each tool's name, description, and input schema into the prompt, and the model runs `node <shim> <toolName> '<json>'` through its built-in `bash` tool. Tool-selection behavior therefore reflects how well the model follows those shim instructions, not how a native MCP client would behave.
* Tools execute **host-side**, in MCPJam's server process, not in the sandbox. For a remote HTTP connector the difference is small (the same outbound request, from a different origin). For a stdio/local server it is larger — the server runs where MCPJam runs, not next to the agent.
* Tool schemas are enumerated **once, at turn start**. There is no `tools/list_changed` subscription, so a server that changes its tool list mid-turn is only picked up on the next turn.
* Every projected tool's description is part of the prompt, so a server with many tools inflates each turn of the conversation.

Claude Code hosts are unaffected by all of the above — they use the runtime's real MCP client.

## Requirements

* A project **Computer** (the E2B data plane must be configured for your deployment) — **or**, for a Claude Code host in the desktop app or `npx`, the local runtime installed and authorized (see [Running Claude Code on your machine](#running-claude-code-on-your-machine)).
* For **Claude Code**: an MCPJam-provided Anthropic model selected on the host.
* For **Codex**: an MCPJam-provided OpenAI gpt-5 family model selected on the host.
* A **signed-in project member** — guest sessions cannot run harness hosts.

No credential configuration is needed **for Claude Code and Codex**. Broker delivery is on by default and the model proxy is always-on.

## Harness credentials on your own vendor account

Not every harness runs on MCPJam's model access. A harness that authenticates
against the **customer's own account with the runtime vendor** — the Cursor CLI
host, which signs in with a `CURSOR_API_KEY` and bills every generation to that
Cursor account — takes no MCPJam model lease at all. Its credential comes from a
**project secret**, and the delivery you choose decides which surfaces can run it.

| Delivery                   | Where the value lives                                                                                                                             | Works in playground chat                                                                                                                                                      | Works in hosted evals and swarms |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| **Brokered** (recommended) | Never leaves MCPJam's backend. The egress proxy injects it as a header on the way out of the sandbox; the sandbox holds a non-secret placeholder. | **No** — playground chat runs on your persistent project Computer, and brokered secrets are composed only onto the disposable boxes an eval, swarm or scenario run provisions | **Yes**                          |
| **Materialized**           | Resolved by MCPJam's server and exported as an environment variable inside the sandbox.                                                           | Yes                                                                                                                                                                           | **No**                           |

Hosted eval and swarm launches **refuse** an environment that selects a
materialized secret. That is deliberate: only the chat path resolves and injects
materialized values, so on a runner-claimed attempt the credential would simply
be missing and the run would report a healthy connector as broken. A brokered
secret has no such problem — the backend composes it into the box's network
policy and the runner does not participate.

So for a Cursor CLI host that you intend to run in an eval suite or a swarm,
**create the secret as brokered**, under **Project Settings → Secrets**, bound to:

| Field    | Value            |
| -------- | ---------------- |
| Name     | `CURSOR_API_KEY` |
| Delivery | Brokered         |
| Hosts    | `api2.cursor.sh` |
| Header   | `Authorization`  |
| Template | `Bearer {}`      |

Then **select it into the environment the host runs in**. That selection is the
grant, not a formality: brokered secrets are composed onto a box from its
environment's selection, so a secret the environment does not grant is never
delivered no matter how it is bound.

Every way that can go wrong is refused **before the sandbox is touched**, with a
message that names the fix:

| What is wrong                                                 | What the turn says                                                                                                                           |
| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Neither delivery configured                                   | Names the variable and both deliveries, and points at Project Settings → Secrets                                                             |
| Brokered secret exists but the environment does not select it | Says so, and asks you to select it into that environment — not to create a second one                                                        |
| The host is not running from a Project Environment at all     | Says there is no grant boundary, so no brokered secret can reach the box                                                                     |
| Brokered secret bound to the wrong host                       | Names the host it found and the host the runtime needs                                                                                       |
| The run is a **replay** of an earlier eval run                | Says MCPJam cannot confirm the credential for a replay, and that nothing about the secret needs changing — launch the suite directly instead |

<Note>
  **Replaying** an eval run cannot verify a brokered credential. A replay keeps
  the original run's environment internally, but MCPJam's runner is not told
  which environment that was, so it cannot check the grant before the sandbox
  starts and refuses rather than guess. Launch the suite directly to run a
  Cursor CLI host; nothing about the secret or the environment needs to change.
</Note>

<Note>
  A **persistent project Computer** (the ordinary playground chat box) does not
  receive brokered secrets today — they are composed onto the disposable boxes
  that scenario, eval and swarm runs provision. Chat on a project Computer
  therefore needs a **materialized** `CURSOR_API_KEY`. It is fine for one
  environment to hold both.
</Note>

### Project secrets require KMS envelope encryption

Storing **any** project secret requires the deployment to have envelope
encryption turned on. Without it MCPJam refuses the write outright, with:

> This deployment cannot store project secrets yet: envelope encryption
> (SECRETS\_KMS\_WRITE) is not enabled.

This is a deployment prerequisite, not a toggle: without envelope writes the
stored value is not cryptographically bound to its project and its owner, and a
credential store whose isolation rests on row checks alone is not the design.

MCPJam Cloud has this configured. A **self-hosted or local development**
deployment must set, on the Convex backend:

| Variable                 | Purpose                                                                           |
| ------------------------ | --------------------------------------------------------------------------------- |
| `SECRETS_KMS_WRITE=1`    | Turns envelope writes on. Nothing stores a project secret without it.             |
| `GCP_KMS_SA_EMAIL`       | Service account that can use the key.                                             |
| `GCP_KMS_SA_PRIVATE_KEY` | That account's PKCS#8 PEM private key.                                            |
| `GCP_KMS_KEY_RESOURCE`   | Full crypto key resource name (`projects/*/locations/*/keyRings/*/cryptoKeys/*`). |
| `KMS_AAD_ENVIRONMENT`    | The environment label bound into the additional authenticated data.               |

Set the four key variables **before** turning `SECRETS_KMS_WRITE` on, and prove
the key actually works first — four correctly spelled variables are not the same
as a usable key, and every way that can fail (missing IAM permission, wrong
region, a private key truncated by a dashboard field) looks like success until
the first real write. Reads are unaffected: a deployment holding older secrets
keeps opening them.

Until KMS is configured, a self-hosted deployment cannot store a
`CURSOR_API_KEY` in either delivery, and so cannot run the Cursor CLI host.

## Billing

* **Computer time** — the harness runtime keeps the Computer awake, so it meters into your org's monthly computer-time allowance exactly like terminal use. A turn that runs **on your own machine** uses no Computer and meters no computer time.
* **Model tokens** — every generation is priced and settled by the model proxy against your org, the same accounting as regular chat, **whether the turn ran hosted or on your own machine**. Spend caps and empty-wallet rejections apply before the stream starts either way.

## Failure modes

None of these fall back to the emulated engine — a turn that says it ran the real runtime did. All fail closed; a failed start spends nothing.

| Condition                                                                                 | What you see                                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Enterprise-managed authorization policy on the host                                       | Pre-flight error — the combination is rejected rather than silently bypassed.                                                                                                                                                                                                                                                                                                                          |
| Require tool approval on a Codex host                                                     | Pre-flight error — the Codex runtime doesn't pause for tool approval, so the turn is refused rather than run unapproved, whether or not any MCP server is selected. The toggle is disabled on Codex hosts, so a host that carried approval in from another harness keeps the stored value: clear `requireToolApproval` through the API or CLI, or switch the host off Codex to reach the toggle again. |
| Computer data plane not configured                                                        | Pre-flight error naming the data plane requirement.                                                                                                                                                                                                                                                                                                                                                    |
| Model not supported by the selected harness                                               | Pre-flight error asking you to pick an eligible model.                                                                                                                                                                                                                                                                                                                                                 |
| Cursor CLI host configured with an ordinary model id (e.g. `anthropic/claude-sonnet-4.5`) | Pre-flight error naming the id it found — the Cursor runtime ignores any pinned model and picks its own, so the host must carry `cursor/auto`. Reset the host's model field.                                                                                                                                                                                                                           |
| Computer at daily start cap                                                               | Start-limit dialog with upgrade option.                                                                                                                                                                                                                                                                                                                                                                |
| Org out of compute allowance and credits                                                  | Computer pauses with a billing notice.                                                                                                                                                                                                                                                                                                                                                                 |
| Org spending limit reached                                                                | Clean rejection before any model call.                                                                                                                                                                                                                                                                                                                                                                 |
| Sandbox dies mid-run                                                                      | Turn errors; the next turn starts a fresh session.                                                                                                                                                                                                                                                                                                                                                     |

**Running on your own machine** (Claude Code, desktop app and `npx` only):

| Condition                                                                | What you see                                                                                                         |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| The local runtime is not installed                                       | The Native option is offered but not selectable, with an Install step in the authorization sheet.                    |
| The runtime on disk does not match what this Inspector expects           | Refused — reinstall it. MCPJam will not launch a runtime it cannot verify.                                           |
| Your platform cannot prove whole-tree cleanup                            | Native is not offered, and says why: MCPJam could not guarantee that stopping a session stops everything it started. |
| The authorization expired, or the runtime, folder or permissions changed | Refused, with the authorization sheet — the old authorization does not cover the new terms.                          |
| Guest, shared scenario session, or swarm run                             | Native is not offered; the turn runs hosted.                                                                         |
| The agent's control channel is reachable from your network               | The session stops rather than proceeding. A local agent's control channel is never published to the LAN.             |

**Eval-specific pre-flight refusals** (harness eval runs only):

| Condition                                                                                        | What you see                                                                                                                  |
| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Host grants `bash` but no computer environment is pinned on the suite                            | Pre-flight error — pin a computer environment in suite settings, or remove the computer-backed built-in from the host config. |
| Suite is scoped to the organization (no project)                                                 | Pre-flight error — the harness provisions and bills the computer against a project; move the suite into a project and retry.  |
| A case pins a short model id (e.g. `claude-sonnet-4-6` instead of `anthropic/claude-sonnet-4-6`) | Pre-flight error naming both spellings — re-save the case with the full canonical id.                                         |

## Related

* [Computer](/inspector/computer) — Manage your project's Computer, sandbox images, and lifecycle.
* [Playground](/inspector/playground) — The workspace where you select hosts and run turns.


# Clients
Source: https://docs.mcpjam.com/inspector/clients

Reusable MCP client configurations for testing servers and apps under realistic conditions

<Note>
  Open the **Connect** tab in the left sidebar to manage Clients (labelled
  **Servers** in some builds). The page name "Clients" refers to the
  configuration object, not the sidebar label.
</Note>

A **Client** is a named, reusable configuration that defines how MCPJam connects to and talks to your MCP servers when you test them in the Playground, Scenarios, and Evals. It bundles everything that would normally be scattered across one-off settings — model, system prompt, attached servers, protocol-level knobs, and MCP App sandbox controls — behind a single picker.

Real MCP hosts (Claude Desktop, ChatGPT, Cursor, your own product) each advertise slightly different capabilities and apply different sandbox policies to MCP Apps. Clients let you reproduce those environments in MCPJam so you can verify your server behaves correctly *before* shipping to a specific host.

## Why Clients exist

Without Clients, every Chat or Eval would be tied to whatever ambient settings were active when you ran it. That makes results hard to reproduce and hard to share. Clients fix this by giving you:

* **Reproducibility** — A snapshot of every setting that affects a run, so a teammate hitting the same Client gets the same conditions.
* **Host emulation** — Configure the inspector to look like Claude Desktop, ChatGPT, or your own host so you can verify your MCP server or MCP App degrades gracefully across them.
* **Faster iteration** — Switch between configurations with one click instead of editing fields in five places.

## What you can configure

A Client groups settings into two layers: the **protocol layer** (how the MCP connection itself behaves) and the **app layer** (what an MCP App running inside the client is allowed to do).

### Protocol-level settings

These map directly to fields the inspector sends during the standard MCP `initialize` handshake and on every subsequent request.

* **Supported protocol versions** — The ordered list of MCP protocol versions the client advertises. The first entry is sent as `protocolVersion`; the rest form the accept-set. Useful for testing how your server handles older or newer clients.
* **`clientInfo`** — The `name`, `version`, and any additional fields sent in `initialize.params.clientInfo`. Set this to match a real host's identifier when you want your server to take a host-specific code path.
* **Client capabilities** — The capabilities object sent in `initialize` (sampling, roots, elicitation, experimental, etc.). Enable or disable each one to test what your server does when a capability is missing.
* **Advertise EMA support by default** — Toggle in the **MCP Protocol** settings card that adds the `io.modelcontextprotocol/enterprise-managed-authorization` extension to the stored `clientCapabilities`. When on, the extension appears in the MCP Protocol JSON editor and host-card capability chips. This controls the stored baseline only — XAA-configured connections always advertise the extension at connect time regardless of this setting.
* **Elicitation** — Toggle in the **MCP Protocol** settings card that controls the top-level `clientCapabilities.elicitation` key. When off, the key is absent. When on, the inspector advertises `{ form: {} }`, signalling that the client can handle elicitation requests from the server. An optional **URL mode** sub-checkbox (visible only when Elicitation is on) adds `{ url: {} }` to the elicitation object, indicating the client can also handle requests to open a URL. Enabling this before server-side elicitation support is active is safe — it is a no-op rather than a break.
* **Mirror tool params into `Mcp-Param-*` headers (SEP-2243)** — Dropdown in the **MCP Protocol** settings card. **Mirror** (the default) copies a tool argument annotated with `x-mcp-header` into an `Mcp-Param-{Name}` request header on `tools/call`, as MCP `2026-07-28` requires. **Omit** deliberately simulates a client that has *not* implemented the mirroring — real-world support is uneven, and browser clients never send these headers — so you can see what your server does when they don't arrive. A conforming server should answer `-32020 HeaderMismatch`; MCPJam surfaces that failure as-is rather than silently retrying. The setting applies to the whole client, not per server, because conformance is a property of the client.
* **Request timeout** — Per-request timeout in milliseconds applied to all MCP calls (`tools/call`, `resources/read`, etc.). Lower it to verify your server handles cancellation correctly; raise it for long-running tools.
* **Default headers** — Headers attached to every HTTP-transport request. Useful for auth tokens, tenant IDs, or anything else your server expects upstream.

### Per-server connection overrides

You can override the request timeout and headers **per server** from the project-wide server list. This lets a single project connect to one server with a long timeout and an auth header while keeping defaults for the others.

### Agent behavior

These settings control how the model interacts with tools during a chat session. They appear in the **Agent** tab of the Client focus panel.

* **Require tool approval** — When on, the model pauses before tool calls so you can approve or deny each one. The setting **raises** the bar and never lowers it, so it is not the whole story in either direction. Browser and WebMCP page tools follow the chat’s Tool Approval setting. With it off, agents can act in signed-in websites without a per-call prompt. Browser device permission is a separate prerequisite for local browsing and inspection. Others never pause even when it is on: read-only lookups of your own project, the progressive-discovery meta-tools, and read-only UI actions. Web search follows the switch — its query is your text leaving for a third party, and the call spends credits.
* **Respect tool visibility** — Controls whether the inspector honors the SEP-1865 `_meta.ui.visibility` field. When on (the default for all templates except Cursor), tools whose visibility is `["app"]` are hidden from the model's tool list. When off, every tool flows to the model regardless of visibility, matching hosts that don't yet implement visibility filtering (for example, real Cursor as of version 3.4.20).
* **Progressive tools** — Three-state toggle (`On` / `Off` / `Auto`). When on, the inspector injects `search_mcp_tools` and `load_mcp_tools` meta-tools instead of sending the full tool catalog upfront, reducing context usage for large tool sets. `Auto` lets the backend decide based on catalog size.
* **Built-in tools** — A checkbox list of host-managed built-in tools (for example, `web_search`) that the model can use on each turn. Selected tool IDs are forwarded with the chat request so the server can resolve them alongside your MCP server tools. Browser is not listed here — it is configured in the **Browser** tab. This section only appears when the deployment has a populated built-in tool catalog; it is hidden otherwise.

### Browser settings

The **Browser** tab in the Client focus panel separates saved browser configuration from your personal browser preferences. It is visible when Browser is enabled for your account or when the client already has a browser configured.

**This client** — Saved configuration that applies to every chat and environment using this client:

* **Browser** — Enable or disable the browser tool for this client. When the feature is not yet rolled out to your account, the toggle is disabled unless the client already has Browser enabled (so existing configurations remain inspectable).
* **Browser profile** — Optionally pin a saved browser profile for this client. Without a selection, new chats use your personal default profile. This field appears only when Browser is enabled or a profile is already saved.

**Your browser** — Personal preferences that do not change the client configuration:

* **Browser location** — Choose between local and cloud browser for new Playground chats. Changing this affects new chats only; any chat already bound to a browser keeps its existing location. Environments always use Cloud regardless of this preference.
* **Device permission** — Grant or revoke permission for the local browser on this machine. Appears only when local browser is available and selected.
* **Saved profiles** — Manage your personal browser profiles.

#### Browser canvas island

The client canvas shows a **Browser** island (or **+ Browser** when not yet enabled) alongside the Computer island. The island displays the active profile label. Clicking it opens the Browser tab.

In environment canvases (read-only views), clicking the Browser island navigates to the referenced client's Browser tab rather than editing the environment inline.

### MCP App settings (SEP-1865)

When your server exposes [MCP Apps](https://modelcontextprotocol.io/docs/extensions/apps) — interactive UIs delivered via `ui://` resources — the Client controls how the inspector renders and sandboxes them. These map to the fields a real host advertises in `HostCapabilities` and `HostContext`.

* **Host capabilities** — The set of `HostCapabilities` keys the inspector advertises in the `ui/initialize` handshake. Each host preset reflects that host's published capability subset: for example, the **Copilot** preset advertises only `openLinks`, `serverTools`, `updateModelContext`, and `message` — matching Microsoft 365 Copilot's published Component-bridge table — and omits `serverResources` and `logging`. Use this to verify your app degrades gracefully when a capability it relies on is absent.
* **Sandbox CSP mode** — `host-default`, `declared`, or `relaxed`. Controls the baseline Content Security Policy the inspector applies to your app's iframe.
* **CSP restrictTo** — Per-directive intersections (`connectDomains`, `resourceDomains`, `frameDomains`, `baseUriDomains`) that further narrow what the app is allowed to fetch, embed, or set as a base URI. Restrict-only — the inspector never adds an undeclared domain.
* **Custom CSP directives** — Verbatim source-expression overrides (e.g. `script-src: 'unsafe-eval' 'wasm-unsafe-eval'`) for cases where you need to model exactly what a particular host emits at the browser layer.
* **Permission Policy** — Per-feature toggles for `camera`, `microphone`, `geolocation`, and `clipboard-write`. Choose `resource-declared` (grant what the app's resource metadata asks for), `deny-all`, or `custom` (your own allow map).
* **Display modes** — Which of `inline`, `fullscreen`, and `pip` the inspector advertises in `HostContext.availableDisplayModes`. Test how your app responds when a requested mode isn't available.
* **Theme and styles** — The CSS variables and font block the inspector injects into `HostContext.styles` so you can verify your app picks up host theming.
* **`window.openai` injection** — Toggle whether the inspector injects the OpenAI Apps SDK `window.openai` shim into widget HTML before sandboxing. ChatGPT, Copilot, and MCPJam host presets enable this by default; Claude, Cursor, and Codex presets leave it off. You can override the preset per host using the toggle in the **Apps Extension** tab. The chip in the host capability matrix shows the current state and whether it comes from the preset or a manual override.

#### Apps Extension tab

The Apps Extension tab contains two independent capability matrices that never cross-gate — toggling a row in one has no effect on the other.

**`window.openai` matrix (ChatGPT compat shim)** — Controls the `window.openai.*` surface advertised to apps using the ChatGPT compatibility shim. A master **Inject window\.openai** toggle enables or disables injection entirely; a collapsed per-method disclosure lets you activate individual methods to build an exact subset from scratch. Switching on any single method turns injection on with only that method enabled; turning the last method off disables injection again.

**`app.*` matrix (SEP-1865 spec bridge)** — Controls the `app.*` surface advertised to apps using the primary MCP Apps protocol. A master **MCP App support** switch advertises the MCP UI client extension (`io.modelcontextprotocol/ui`). The matrix rows include:

* **`availableDisplayModes`** — multi-checkbox cluster (`inline`, `fullscreen`, `pip`); unchecking the last mode force-enables `inline`.
* **`widgetDisplayModeRequests`** — tri-state host policy (`accept`, `user-initiated-only`, `decline`) for widget-initiated display-mode requests.
* **Boolean dimensions** — a flat list of all `app.*` capability flags (e.g. `toolInputPartial`, `toolCancelled`, `serverResources`, `logging`, `toolInfo`, `openLinks`, `serverTools`, `updateModelContext`, `message`). Rows that diverge from the selected host preset are stored as sparse overrides.

<Note>
  Widget iframes are only rendered when the effective `clientCapabilities` advertise
  `io.modelcontextprotocol/ui` with a `mimeTypes` array that includes
  `text/html;profile=mcp-app`. Custom capability configs with a bare
  `{ extensions: { "io.modelcontextprotocol/ui": {} } }` entry (no `mimeTypes`
  array) will not render widgets. SDK-default host shapes (Claude, ChatGPT,
  Copilot, MCPJam) include the correct `mimeTypes` and are unaffected.
</Note>

<Note>
  All MCP App settings only take effect when the connected server actually
  delivers a `ui://` resource. For plain tool-only servers, the protocol-level
  settings are what matter.
</Note>

## Creating a Client

1. Open the **Clients** tab in the inspector.
2. Click **New Client**.
3. Give it a name (e.g. `Claude Desktop`, `ChatGPT`, `Strict CSP`).
4. Open the focus panel to drill into protocol or MCP App settings.
5. Click **Save**.

<Note>
  Server selection is project-wide, not per-client. Use the **Auto-connect**
  toggle in the Servers tab header to enroll all project servers for
  auto-connection. Per-server overrides (headers, timeout) are also configured
  there.
</Note>

The canvas on the left shows a live diagram of the Client — the model, attached servers, and any per-server overrides. Cards flash briefly when you switch between Clients so you can see what changed.

While the Connect canvas is open, a **client selector** is pinned to the top-left of the canvas. It shows two controls:

* **Add client** pill — opens the Create client dialog. Quick-add buttons for Claude, ChatGPT, and Copilot appear alongside it so you can spin up a template client without filling in the form.
* **Switcher pill** — shows the active client's logo, name, and its position in the list (e.g. `2 / 3`). Click to open a dropdown listing all clients; select one to switch the canvas to it. A delete button appears on hover for each client (disabled on the last remaining client).

The header client bar hides itself while the canvas is open to avoid duplicating the control.

## Using a Client

Once saved, a Client shows up in the picker at the top of every consumer:

* **Chat** — Drives the model, system prompt, attached servers, and connection settings for the conversation.
* **Scenarios** — Same as Chat, but persisted as a saved scenario configuration.
* **Evals** — Every test run inside an Eval uses the selected Client, so results are reproducible across teammates.
* **Swarms** — Journeys in the Swarms surface can target any project client, including standalone ones.

Switching the active Client in any of these views swaps the entire configuration in place. There's no separate "apply" step.

### Standalone clients (Swarms-managed)

Some clients are **standalone** — they are owned by the Swarms surface and have no Scenario or publish surface. If you select a standalone client in the Scenarios tab, the tab shows a **"Managed by Swarms"** notice instead of a scenario. The client picker stays visible so you can switch to a publishable client.

Standalone clients do not appear as options in the Scenarios client picker dropdown. If you need a client for both Scenarios and Swarms journeys, create a regular client from the Connect tab instead of from Swarms → Clients.

## Server attachments

A **server attachment** is a named, reusable set of servers drawn from the project pool. Server attachments let you decouple *which servers are active* from the Client configuration itself, so you can swap the server set without touching the model or prompt settings.

### Eval suites

When you create an eval suite, you can pick a server attachment in the **Servers** section of the create dialog. All hosts attached to the suite run against that same server set. You can also change the attachment later from the suite header.

* If no attachment is selected, the suite header shows an amber indicator. Runs will be rejected at start until an attachment is picked.
* Click the attachment pill in the suite header to open the picker and switch to a different attachment or create a new one inline.

**Creating an attachment inline:**

1. Open the server attachment picker (suite create dialog or suite header).
2. Click **Create new attachment…** in the dropdown.
3. Enter a name (e.g. `Production servers`) and check the servers to include.
4. Click **Create**. The new attachment is selected immediately.

### Scenarios

The **Servers** section on the scenario Publish tab shows which project servers the scenario connects to. Click **Edit servers** to open the attachment editor and update the selection.

* If no servers are picked, the scenario displays an amber warning. Chat sessions started from that scenario will have no tools until at least one server is selected.
* The server pick is sticky — editing the client's model or system prompt does not change it.

## Managing Clients

From the Clients index page:

* **Duplicate** — Copy a Client when you want a small variant (e.g. `Claude Desktop — strict CSP`) without rebuilding from scratch.
* **Delete** — Removes the Client. The inspector blocks deletion if any scenario or eval still depends on it; resolve those first, or use force delete if you're sure.

<Tip>
  Keep one Client per host you intend to ship to. Duplicate it to make scoped
  variants for individual debugging sessions instead of editing the original —
  that way your "known-good" baselines stay stable.
</Tip>

## Sharing across a project

Clients are scoped to the project, like servers and views. Every member of a shared project sees the same Client list and any saves sync automatically. This means a teammate can reproduce your exact test conditions by selecting the same Client from the picker — no copy-pasting settings.

## Comparing hosts

The **Compare** view lets you see how multiple host configurations stack up side by side — useful for spotting differences in capabilities, sandbox policies, or protocol settings before you ship to a specific host.

### Opening Compare

Compare is the third tab in the Connect section of the sidebar (`Servers | Client | Compare`). Click **Compare** from any Connect view to open it. The `/host-compare` route is preserved, so any existing bookmarks or links continue to work.

### Deep-linking to a specific selection

Append `?hosts=<id1>,<id2>` to the `/host-compare` URL to pre-select a specific set of hosts on load:

```
/host-compare?hosts=host-abc,host-def
```

The URL updates automatically as you toggle hosts on or off. When the default selection (ChatGPT and Claude presets, in that order) is active, the `?hosts` parameter is omitted so shared links stay clean.

## Best practices

* **Mirror your target hosts.** Create one Client per real host you care about (`Claude Desktop`, `ChatGPT`, your own product) and keep its `clientInfo`, capabilities, and sandbox settings aligned with what that host actually advertises.
* **Test the negative paths.** Build a "minimal" Client with most capabilities disabled and a strict CSP — running your server against it surfaces assumptions that won't hold on more restrictive hosts.
* **Pin Evals to a Client.** Always select a Client before running an eval suite so the results are reproducible. Evals run against ambient settings drift the moment someone toggles something elsewhere.
* **Duplicate before debugging.** When you're chasing a bug, duplicate the Client first and tweak the copy. Your baseline stays intact for comparison.
* **Share Compare links.** Toggle down to the hosts you care about and copy the URL — the `?hosts=` parameter preserves your selection for teammates.


# Compatibility
Source: https://docs.mcpjam.com/inspector/compatibility

Check whether your MCP server works on each AI host — conformance gates, per-host findings, and live widget rendering.

The Compatibility destination answers one question: **does my server work on this host?** It evaluates your connected server against a catalog of real AI hosts (Claude, ChatGPT, Cursor, Copilot, Codex, and others) and surfaces blockers, degraded experiences, and protocol gaps — without requiring you to deploy to each host first.

## How to open it

Connect a server, then select **Compatibility** from the left sidebar. With multiple connected servers, a matrix appears at the top; click any row to drill into that server's full report.

## What the report shows

Each host row has a verdict and, when findings exist, an expandable detail section.

### Verdicts

| Verdict      | Meaning                                                                                                                |
| ------------ | ---------------------------------------------------------------------------------------------------------------------- |
| **Works**    | No blockers or degraded findings detected.                                                                             |
| **Degraded** | The server functions but some widgets fall back to text, or a capability the widget uses isn't supported by this host. |
| **Blocked**  | One or more app-only tools are unusable on this host — they have no text fallback and the host can't render their UI.  |
| **Unknown**  | Widget HTML hasn't been analyzed yet, or a required dimension couldn't be determined.                                  |

Verdicts are derived from static connect-time data (tools list + `_meta` bags + the protocol version negotiated). They update as tools load.

### Conformance gate

Above the per-host rows, a **Conformance gate** shows whether the server passes the MCP protocol conformance checks. A server that fails conformance may work on some hosts but is likely to behave inconsistently. Fix conformance issues first — they affect every host.

### Findings

Expand a host row to see the full findings list, split into two lanes:

* **Apps lane** — widget render failures and capability gaps. Examples: an app-only tool whose UI the host can't render, a widget that calls `ui/open-link` on a host that doesn't support it.
* **Server lane** — protocol-version compatibility. If the server negotiated a version the host doesn't advertise, this lane surfaces it as an informational note (the host may still negotiate a shared version).

Each finding includes a title, a detail sentence, and where applicable a **remediation** hint — for example, "Declare an MCP Apps template alongside the OpenAI one."

### Provenance badges

Findings carry a provenance label that tells you how confident the data is:

| Provenance             | Meaning                                                                           |
| ---------------------- | --------------------------------------------------------------------------------- |
| **Probe-captured**     | Values captured from a real host via DevTools or the inspector's JSON-RPC logger. |
| **Vendor docs**        | Verified from the host vendor's published documentation.                          |
| **Observed**           | Stamped by a live widget render run (see below).                                  |
| **Best-effort preset** | Unverified — the preset was authored without a live capture.                      |

## Live widget rendering

For hosts that render MCP Apps widgets, a **Run live** button appears next to the host row. Clicking it invokes the server's widget tool inside an emulated version of that host and captures the result inline. A live run upgrades the finding provenance from `assumed` or `probe` to `observed`, giving you the highest-confidence verdict available.

Live rendering requires:

* A connected server with at least one widget tool.
* The host to support MCP Apps rendering (Claude, ChatGPT, Cursor, Copilot, Codex all do).

## Test in host

From the server detail modal's Compatibility tab (not the standalone page), a **Test →** button appears next to each host. Clicking it creates a new host configuration seeded from that host's preset with your server attached, then navigates to the Playground so you can interact with the server under that host's emulation.

## Multi-server matrix

When two or more servers are connected, the page leads with a **servers × hosts matrix**. Each cell shows the worst verdict across that server's findings for that host. Click a server row to load its full report below the matrix.

## Relationship to host templates

The host profiles used for evaluation are the same presets available in the [host template picker](/inspector/host-templates). If a host you care about isn't in the catalog, you can [contribute a preset](/inspector/host-templates) — the compatibility engine will pick it up automatically.

## Relationship to conformance testing

The Compatibility destination is a **static analysis** surface: it reads the tools list and widget metadata without running the server. For dynamic conformance testing (protocol checks, OAuth flows, MCP Apps spec checks), use the [CLI conformance commands](/cli/apps-conformance) or the [SDK conformance suites](/sdk/reference/protocol-conformance).


# Computer
Source: https://docs.mcpjam.com/inspector/computer

A personal cloud Linux workstation for your project — open a terminal, run tools, and give your agents a real bash environment

<Note>
  The Computer is available on accounts where the feature has been enabled. The **Computer** tab only appears when it is active for your account.
</Note>

Each project gives every member an opt-in **Computer**: a personal cloud Linux
workstation with a web terminal and a `bash` tool your agents can call. It boots
from a base image (Debian + Node + Python) or a custom [sandbox image](#sandbox-images)
you define, sleeps automatically after about 30 minutes idle or shortly after
you close the terminal, and wakes on next use.

Open it from the **Computer** tab. The first time you open the terminal, MCPJam
provisions the machine; after that it resumes in about a second.

## Chat attachments

When the previewed host has a computer attached and you are signed in, files you
attach in the chat composer are uploaded into the computer's filesystem at
**`/home/user/attachments/`** in addition to being sent inline to the model.
A short note is appended to your message so the model — and any MCP server
running on the computer — can reference the files by path:

```
[Attachments uploaded to the computer — use the bash tool to read them]
- report.csv: /home/user/attachments/ab12cd34-report.csv
```

If the upload fails (for example, because the daily start cap is reached or the
storage quota is exceeded), the send is **aborted** and the composer is left
intact — your message and attachments are never sent without the files landing
on the computer first.

<Note>
  This only applies when the host config attaches a personal computer and you
  are signed in. Guest sessions and hosts without a computer attached keep the
  existing inline-only behavior.
</Note>

## Agent tools

When the **Computer** tab is open, agents have access to four tools for managing
the computer's lifecycle. These tools are only available while you are on the
Computer screen and require a signed-in project.

| Tool                    | What it does                                                                                                                                                              |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ui_start_computer`     | Provisions the computer on first use, or wakes it if it is asleep. Counts against the daily start cap — a cap rejection is returned as an error and the start is refused. |
| `ui_hibernate_computer` | Puts the computer to sleep immediately. Non-destructive: files and state are preserved.                                                                                   |
| `ui_reset_computer`     | Resets the computer to its image, **wiping all files**. Requires confirmation.                                                                                            |
| `ui_delete_computer`    | Tears the computer down entirely, **deleting all files**. Requires confirmation.                                                                                          |

<Note>
  Opening the interactive terminal is a human action and is not available as an
  agent tool. The terminal token never appears in the agent transcript.
</Note>

Agents can check the computer's current state (lifecycle status, environment,
whether a terminal is open) via the app snapshot (`ui_snapshot_app`) while on
the Computer screen.

## Sleep and hibernate

Your computer sleeps automatically when it has been idle for about 30 minutes,
or shortly after you close the terminal. Sleeping is non-destructive — the disk
and memory are snapshotted and restored on wake, so everything you left running
is still there.

To put the computer to sleep immediately, click **Hibernate now** in the
Computer tab toolbar (only visible when the computer is running). You'll be
asked to confirm, then the terminal closes and the machine hibernates. It wakes
automatically the next time you open the terminal or an agent calls the `bash`
tool.

## Sandbox images

A sandbox image is a custom image your computer boots from, defined by a
**blueprint** — a small YAML file:

```yaml theme={"theme":"css-variables"}
# base must be an allowlisted official image (debian, ubuntu, node, python)
# pinned by an immutable @sha256 digest.
base: debian:bookworm-slim@sha256:<64-hex-digest>

# Baked into the image at build time.
initialize:
  - name: Install tooling
    run: apt-get update && apt-get install -y git jq

# Handed to the agent as reference — NEVER run automatically. If deps look
# stale, the agent can run these itself via bash.
maintenance:
  - name: Refresh deps
    run: cd ~/app && npm install

# Model-facing notes, injected into the agent's context whenever a chat
# turn runs with the bash tool on a computer booted from this image.
knowledge:
  - name: Test notes
    contents: Run `make test` before pushing.
```

The editor lints as you type (the backend is the single validator; the CLI has
`mcpjam cloud images validate` for the same check). Build the image, then **Use on
computer** to boot from it. `base` / `initialize` edits need a re-build;
`maintenance` / `knowledge` edits take effect at the next chat turn without
one. Images can be kept private to you or shared with the whole project.
Manage them from **Change** on the image strip.

<Note>
  Image definitions (their blueprints) live in the project database, not on
  any one computer's disk — see [Data & persistence](#data-and-persistence).
</Note>

## Scenario sandbox

When a scenario is backed by an environment that has a pinned computer image,
the `bash` tool runs on a **per-conversation ephemeral sandbox** booted from
that image — not on your personal computer.

### How it works

* **Per conversation.** Each new conversation gets its own isolated machine.
  Files you write and packages you install in one conversation are not visible
  to any other conversation.
* **Persistent across turns.** Within a single conversation, the same sandbox
  is reused on every turn — you can write a file on turn 1 and read it on
  turn 5.
* **Reaped after idle.** If a conversation goes idle for a long time, the
  sandbox is deleted (not paused). When that happens, you'll see a notification
  in the chat: *"This conversation's sandbox was reset after being idle — files
  and shell state from earlier turns are gone."*
* **Image from the environment.** All members on the same scenario boot from the
  same environment image, so the machine is consistent regardless of who is
  chatting.
* **No bash if the image is unavailable.** If the environment's image can't
  boot right now, the scenario loads normally without a `bash` tool — no error,
  and no fallback to your personal computer.

### What is unchanged

* **Your personal computer** is unaffected. Nothing a scenario conversation
  writes lands on your personal computer, and nothing on your personal computer
  is visible inside a scenario sandbox.
* **Host-backed scenarios** (those not tied to an environment) continue to use
  your personal computer as before.
* **Environments with no pinned image** also continue to use your personal
  computer as before.

## Browser panel in the Playground

When a host configuration includes a browser tool, the Playground shows a browser panel alongside the chat. Each conversation owns its own browser — switching between chats reattaches the existing browser for that conversation rather than opening a new one or navigating away.

### Switching conversations

When you switch from one conversation to another, the Playground reconnects to that conversation's existing browser. The live tabs, page content, unsaved form text, and scroll position are preserved exactly as you left them. No navigation happens on reattach.

If the conversation's browser is not yet running when you open it, the panel shows an **Open a browser for this conversation** button. The panel also polls in the background so it can attach automatically if the agent starts a browser after the pane is already open.

### Closing the browser pane

Closing the browser panel stops the viewer — it does not hand control back to the agent. If you held the browser (took manual control), that hold stays parked after you close the pane. The agent remains blocked until you explicitly click **Hand back** to release control. A parked hold does not keep a metered cloud browser running indefinitely, but it does keep the agent waiting.

### Browser panel layout

The open and expanded state of the browser panel is tracked per conversation. Opening or closing the panel in one chat does not affect any other chat's panel state.

## Data & persistence

Your computer is **scratch space, not durable storage.** Treat it like a fresh
dev box you can rebuild at any time — convenient and fast, but not a place to
keep the only copy of anything.

<Warning>
  Files persist when your computer sleeps, but they aren't backed up — keep
  anything important in git or elsewhere.
</Warning>

### What persists

* **The filesystem.** When your computer sleeps, its entire disk and memory are
  captured in a pause snapshot and restored on wake, so files you created and
  tools you installed are still there when you come back.

### What wipes it

Each of these rebuilds the machine from its image and **deletes every file on
the disk**:

* **Reset** — restores the computer to a clean copy of its current image. Reset
  sits on the image strip (next to **Change**) and is available once the computer
  is ready or asleep.
* **Changing the image** — switching to a different environment (or back to the
  base image) rebuilds from that image.
* **Delete** — tears the computer down entirely.
* **Guest inactivity** — computers owned by signed-out (guest) sessions are
  deleted when idle instead of paused, so their files don't carry over between
  sessions. Sign in for a computer that sleeps and resumes with its disk intact.

### What survives independently

These live in the project database (Convex), not on the computer's disk, so they
are unaffected by reset, image changes, or deletion:

* **Cloud skills** attached to the project.
* **Image blueprints** — the definitions of your custom images. Deleting a
  computer never deletes your sandbox images; you can boot a fresh computer
  from the same image again.

### The no-backup caveat

There is no backup and no volume behind the computer — the pause snapshot *is*
the storage. Once the disk is wiped by any of the actions above, its contents
are gone for good. Anything you want to keep should live in version control, a
remote, or another durable store.


# Multi-server connections
Source: https://docs.mcpjam.com/inspector/connecting-servers

Connect to multiple MCP servers with different transport methods

You can connect to MCP servers over **STDIO** or **Streamable HTTP**, with OAuth 2 and Dynamic Client Registration (DCR) support. Multiple servers can be connected at once within a project.

## Manage connections

You can connect to multiple MCP servers simultaneously. To manage connections, go to the **Servers** tab in the MCPJam sidebar.

<Frame>
  <img alt="Multiple" />
</Frame>

* Toggle MCP servers to enable or temporarily deactivate a server.
* Hit the 3 dots. You can reconnect, edit, or disconnect to a server.
* Click "View server info" to open a detailed modal showing server metadata, protocol version, capabilities, and instructions.

### Server info modal

View server details in a dedicated modal dialog:

* **Server metadata**: Name, title, version, protocol version, and transport type
* **Capabilities**: Tools, prompts, and resources supported by the server
* **Instructions**: Server-provided instructions for AI assistants
* **Raw capabilities**: Full JSON of server and client capabilities (with copy button)
* **OpenAI Apps SDK**: For servers using the OpenAI Apps SDK, view widget metadata in a separate tab

### History tab

The **History** tab in the server detail modal shows a timeline of every revision recorded for that server. Each row displays the revision number, a tool-count delta relative to the previous revision, and a timestamp.

Click any revision row to expand it. The expanded panel shows:

* **Diff view** (default for revisions 2 and later) — highlights tools added, removed, or updated compared to the previous revision.
* **Snapshot view** — lists all tools captured at that revision, with each tool's name and description.

A **Diff / Snapshot** toggle appears at the top of the expanded panel when a previous revision exists. Revision 1 opens directly in Snapshot view since there is nothing to diff against.

### Server information

Each connected server card displays:

* **Server icon and version**: If provided by the server, the icon and version number appear in the card header
* **Connection status**: Real-time indicator showing connected, connecting, or failed status with retry count
* **Transport type**: Shows whether the server uses STDIO or Streamable HTTP

Click "Show details" to expand additional server information:

* **Server capabilities**: Lists available features (Tools, Prompts, Resources)
* **MCP protocol version**: The protocol version the server implements
* **Instructions**: Server-provided instructions for AI assistants
* **Documentation link**: Direct link to the server's documentation if available

### OpenAI app metadata

Servers that provide OpenAI-compatible tool metadata display an OpenAI logo indicator. Expanding these servers shows two tabs:

* **Info tab**: Standard server information and capabilities
* **OpenAI Tool Metadata tab**: Detailed metadata for each tool including read/write permissions and widget configuration

<Note>
  Using the [hosted app](https://app.mcpjam.com)? Only HTTPS connections are available. STDIO and plain HTTP connections require the [local inspector](/installation). See [Hosted App](/hosted/overview) for details.
</Note>

## STDIO servers

Connect to STDIO with commands. Commands vary based on what language your MCP server is built with. For example, connecting to an MCP server built with Node might look like:

```bash theme={"theme":"css-variables"}
node /path/to/your/server.js
```

<Info>
  For local development, always use absolute paths (e.g.,
  `/Users/yourname/project/server.py`) to avoid path resolution issues.
</Info>

<Frame>
  <img alt="STDIO" />
</Frame>

### Environment variables

You can configure environment variables with STDIO. Click "Add Variable" in the Environment Variables section. This is often used to pass dynamic values like API tokens to a STDIO server.

## HTTP servers

Connect to any Streamable HTTP MCP server by selecting **HTTP** in the transport dropdown and pasting in the URL — typically ending in `/mcp`.

<Frame>
  <img alt="Streamable HTTP" />
</Frame>

### Custom headers

Expand **Custom Headers** to add arbitrary request headers (for example, proxy auth or tenant identifiers). Headers are sent on every request to the server.

### Authentication

MCPJam is compliant with the MCP authorization spec and supports Dynamic Client Registration (DCR), client pre-registration, and Client ID Metadata Documents (CIMD). You can:

1. Choose **No Authentication**
2. Choose **Bearer Token** — use this if you already have an API token and don't need to register an OAuth client.
3. Choose **OAuth** — follow [MCP Authorization](https://modelcontextprotocol.io/specification/draft/basic/authorization) to obtain a bearer token. You can optionally set scopes or a custom client ID and secret. For step-by-step debugging, see the [OAuth Debugger](/inspector/guided-oauth).

### Request timeout

Expand **Additional Configuration** to override the request timeout (default `10000` ms). Increase this when talking to slow or cold-start servers.

## Tunneling

Expose your local MCP servers over HTTPS for remote access. This lets you share your servers with external clients and applications without deploying them.

Each tunnel is scoped to a single server and secured at the network edge: the tunnel URL contains a secret key (`?k=…`) that is enforced before any traffic reaches your machine. Requests without the correct key are rejected with a 401 at the edge.

### Create a tunnel

1. Go to the **MCP Servers** tab.
2. Click **Create Tunnel** on the server card you want to expose.
3. Copy the tunnel URL from the server card and paste it into your MCP client (for example, ChatGPT's custom connector settings).

The tunnel URL is stable — the base domain stays the same across sessions. If you close and recreate a tunnel for the same server, you get the same domain with a new secret key.

### Rotate the secret

Click the **Rotate** button (refresh icon) on the tunnel pill to issue a new secret key. The base domain does not change, but the old URL stops working immediately. Use this if you shared the URL with someone who should no longer have access.

### Recent requests

Click the **log icon** on the tunnel pill to open the recent-requests panel. It shows the last 20 JSON-RPC method calls that arrived through the tunnel, refreshing every 4 seconds. The panel closes automatically when you close the tunnel.

### Close a tunnel

Click the **close (×) button** on the tunnel pill to stop remote access. The tunnel closes immediately and all remote connections are terminated. The domain is reserved so a future tunnel for the same server reuses it.

<Warning>
  Anyone with the full tunnel URL — including its secret key — can access this
  server. Share it carefully, rotate it if it leaks, and close the tunnel when
  you are done.
</Warning>


# Tracing & debugging
Source: https://docs.mcpjam.com/inspector/debugging

See all messages sent to and from your MCP server

MCPJam Inspector gives you full visibility into every exchange between the inspector and your MCP server, across the Playground, Tools, Prompts, and Resources tabs.

## JSON-RPC logging

Every tab that talks to a connected server renders a live JSON-RPC logger. Each entry records the direction (client → server or server → client), the method name, a timestamp, and the full request/response payload. Click an entry to expand and inspect.

See [Playground → JSON-RPC Logger](/inspector/playground#json-rpc-logger) and [MCP Inspector](/inspector/tools-prompts-resources#json-rpc-logging) for how it surfaces in each tab.

## Widget debugging

When using ChatGPT Apps or MCP Apps in the Playground, you can debug widget state and globals directly from tool call results.

### Debugging in the chat

Click on any tool call in the chat to expand its debug panel. For widgets, you'll see three tabs:

* **Data** - The tool's input arguments and output result
* **Widget State** - The current state persisted by the widget (updated in real-time as the widget calls `setWidgetState`)
* **Globals** - Environment information passed to the widget including theme, display mode, locale, and device capabilities

A "Click to debug" hint appears on tool calls that have widget debug information available.

### Debugging in the Tools tab

When testing OpenAI Apps or MCP Apps from the Tools tab, the results panel includes a **State/Globals** button alongside the Component and Raw JSON views. This shows the same widget state and globals information in a dedicated panel.

<Tip>
  Widget state persists across follow-up turns in a conversation. Keep state
  under 4k tokens for optimal performance.
</Tip>


# MCP Docs Server
Source: https://docs.mcpjam.com/inspector/docs-server

Connect AI applications to MCPJam documentation

## Overview

MCPJam provides an MCP (Model Context Protocol) server that gives AI applications direct access to our documentation. Get contextual help about MCPJam features while working in your AI-powered development tools.

## MCP Server URL

Use the following URL to connect AI applications to the MCPJam documentation:

```
https://docs.mcpjam.com/mcp
```

## Supported Applications

You can use this MCP server with any application that supports the Model Context Protocol, including:

* Claude Desktop
* Claude Code
* Cursor
* VS Code with MCP extensions
* Other MCP-compatible tools

## Getting Started

1. Copy the MCP server URL above
2. In your AI application, add a new MCP server connection
3. Paste the URL: `https://docs.mcpjam.com/mcp`
4. Start asking questions about MCPJam Inspector, MCP Evals, or contributing!

## What You Can Ask

Once connected, you can ask questions like:

* "How do I connect to an MCP server in Inspector?"
* "What are the available transport protocols?"
* "How do I set up OAuth testing?"
* "Show me how to run evals"
* "What LLM providers are supported?"

## Benefits

* **Always Up-to-Date**: Documentation is served in real-time from our latest docs
* **Contextual Help**: Get answers specific to your current task
* **Fits Your Workflow**: Works inside the tools you already develop in
* **No Context Switching**: Stay in your editor while accessing documentation

## Troubleshooting

If you're having trouble connecting:

1. Verify your AI application supports MCP
2. Check that the URL is entered correctly
3. Ensure you have an active internet connection
4. Refer to your application's MCP setup documentation

## Contributing

Want to help improve the docs server or contribute to MCPJam? Check out our [GitHub repository](https://github.com/MCPJam/inspector) or join our [Discord community](https://discord.gg/JEnDtz8X6z).


# Evaluate
Source: https://docs.mcpjam.com/inspector/evals

Author test cases for your MCP server, run them across models, and gate pass/fail on deterministic checks plus optional LLM-as-judge.

A benchmark score doesn't tell you whether your server holds up in production. Evaluate lets you pin down the behaviors you care about — which tools fire, with what arguments, what the final answer looks like — and run them across the models your users actually use.

## Requirements

Evals are available to both signed-in users and guests. Guests authenticate automatically using their guest session, so no sign-in is required to run or generate tests.

## How it's organized

<Steps>
  <Step title="Project">
    Holds your servers and your suites. Everything below lives in one project.
  </Step>

  <Step title="Suite">
    A group of cases plus the defaults they share: attached servers, models to run against, default checks, judge config, and argument-matching mode.
  </Step>

  <Step title="Case">
    One scenario you want to verify — a prompt (or a sequence of prompts), the tools you expect to fire, an optional expected output, and the checks that decide pass/fail.
  </Step>

  <Step title="Run">
    One execution of a suite. Produces **iterations** — one per case × model. Each iteration has its own transcript, tool calls, tokens, duration, and verdict.
  </Step>
</Steps>

## Authoring a case

Click **New case** in the suite header (or in the empty-state screen when no cases exist yet) to open the case workspace. The case is a draft until you press **Save** — backing out without saving leaves your suite unchanged.

### Case workspace

The case editor is a two-pane workspace. The left column is where you write the case; the right column shows the selected trial — a live run in progress, a persisted iteration from history, a recording session, or a spec preview.

**Left column — the case form**

Every case opens in this form, however it was authored — in the app, through the CLI, or from an SDK suite file. It has four sections:

* **User asks** — the prompt the model receives.
* **In the app** — interact steps (clicks, typing) recorded against a live widget, shown in the order they run.
* **Scorers** — everything that grades this case, grouped by the link of the user-value chain it measures, in the same vocabulary the suite's Grading tab uses.
* **Judge · Goal completion** — the goal sentence, and what the judge will actually do with it.

**Scorers**

One list, one Add menu. Each row says what it checks, who wrote it, and what a miss does:

* **Route** — the first row under Selection. It answers "which tool should handle it?" and carries that answer's strictness: *Reach the tool* allows extra calls, *Exact route* requires the exact order with no extras. The route is always a Gate.
* **Step N** — a check written as a step (what the CLI and the SDK produce). It is graded in place, at its position in the run, and is labelled by that position: a step-level "no tool errors" reads **No tool errors so far**, because it only sees the transcript up to that point. Editing one rewrites that step; it never becomes a case-level check.
* **This case** — a check you added here. It is graded once, over the finished transcript.
* **Suite** — a default the suite applies to every case. Read-only here, with a link to the suite settings.

A row's **role** says what a miss does: **Gate** fails the trial, **Warn** highlights the miss without failing it, **Report** only records it. The control appears when your deployment supports check policy; otherwise the role still shows, read-only.

Because a failing gate halts the run, a check written as a step can leave later checks unrun. The trial column marks those **Skipped**.

**The route question**

The answer here is what marks a case negative — never an inference from its steps:

* **A named tool** — the case must take that route.
* **No tool should be called** — a negative case. It passes only if the model called nothing. A check that requires a tool call contradicts this, and the form says so.
* **Any route** — the case names no route and is graded by its scorers. This is a normal positive case, and it is what most CLI- and SDK-authored cases are.

A brand-new case that asserts nothing at all must answer this question before it can be saved or run: without a check, a goal sentence, or a named tool, it would pass vacuously.

**Judge · Goal completion**

The card shows which model grades this case, against what threshold, how many suite criteria also apply, and whether the judge runs automatically or on request. All of that is the suite's; **Edit in suite settings** goes there.

What the case owns is the goal sentence, and the hint under it says what the judge will actually grade against:

* A **goal sentence** — the judge grades against it, and the suite's criteria still apply.
* **No sentence, but a route** — the judge grades against the expected route.
* **Neither, but suite criteria** — the judge grades against those.
* **None of the above** — the judge grades against the request itself, with its score capped at 0.85.

**Skip the judge for this case** turns the judge off for this case only. It applies to the next saved run.

**Also in this case**

A case can hold steps this form has no section for — a second prompt, or a pinned tool call. Those are listed at the bottom of the column with the turn they belong to. Click **Edit in Steps** to open the full step list.

**Steps**

**Steps** (top right of the column) swaps the form for the flat, ordered step list, in the same workspace with the same trial on the right. **Back to form** returns.

**Right column — the selected trial**

The trial opens on its **Scorecard**: the same scorers the left column lists, in the same order, each with what happened to it on this trial. A row shows its result, its provenance, its role, and — where there is one — the reason, which for a step is the message the runner recorded.

The summary line counts **gates only**: "2 of 2 gates passed · 1 warn". A Warn miss is shown on its row and never counted as a failure, because it did not fail the trial. A scorer with no result reads **Not measured** rather than passing by default. The verdict word itself stays in the header above.

The judge's row hosts the judge panel, so labelling a trial works where the judge sits. Under the rows, **Score rows** expands the raw score contract, including any row the backend could not join.

The other tabs are unchanged: **Chat**, **Tool Calls** and **Trace** show the transcript, and **Steps** shows execution order — now with the same check names the rest of the page uses and the reason under a failed step.

The right column resolves to one of these views, in precedence order:

1. **Explicit history pick** — a trial you selected from History.
2. **Recording** — a live recording session is active.
3. **Live run** — a quick run is in progress.
4. **Latest** — the most recent persisted iteration with a trace.
5. **Spec preview** — the authored steps rendered as a conversation preview.
6. **Empty** — no trial yet; start typing a prompt.

<Note>
  A **negative case** is one you marked **No tool should be called**. Meta questions ("what params does `search` take?"), conversational drift, and ambiguous prompts are the usual shape. A case that simply names no tool is *not* negative — it is graded by its checks.
</Note>

### History inspect

Click **History** in the trial header to open a list of past iterations. Selecting one freezes the left form on that trial's snapshot — the exact steps, checks, and settings that were in effect when it ran — so you can compare what you see in the trace against what was authored at the time.

While a trial is frozen, an **Inspecting** banner appears at the top of the left column. If the current draft differs from the frozen snapshot, a **Changed** indicator appears. Click **Edit case** to return to the live draft; the selected trial stays visible on the right.

### Next run sheet

Click the **Model**, **Trials**, or **Host** chip above the form to open the **Next run** sheet. It separates two kinds of settings:

* **Saved with this case** — *Trials* and *Model* are written to the case before every quick run and on Save. Changes here persist.
* **For this run** — *Host* applies only to the next run and is not saved to the case.

Click **Suite settings** in the sheet to configure hosts and other suite-level defaults.

## Case attachments

<Note>
  Case attachments require a **pinned computer environment** on the suite. The attachments panel is hidden for draft cases and appears only after a case is saved.
</Note>

Each saved test case has an **Attachments** section at the bottom of the case editor. Files you upload here are seeded into the iteration's fresh sandbox at `/home/user/attachments/<filename>` before the model's first turn runs — so the model finds the same files on every iteration, model, and re-run.

**Limits:** up to 20 files and 30 MB total per case.

To add files, click **Add file** and pick one or more files. To remove a file, click the **×** next to its name. Changes take effect on the next run.

When attachments are present, a note is automatically prepended to the case's first user turn listing each file's path, so the model knows where to find them without you having to mention it in the prompt.

**Fail-honest behavior:** if a file cannot be seeded — because its stored content is no longer available, the download fails, or a cap is exceeded — the iteration fails immediately rather than running silently without the file. This prevents a case that relies on a file from producing a misleading passing result.

## Recording widget interactions

If your MCP server uses MCP Apps (widgets rendered in a sandboxed iframe), you can record real in-widget interactions — clicks and typing — directly into a test case as `interact` steps.

Click **Start recording** in the *In the app* section of the case form. The button is disabled until the prompt is non-empty — type a prompt first, then start recording. Once active, a recorder shim is injected into the widget sandbox. Every click and text input you make in the widget is captured as a locator-stable `interact` step and appended to the case. The locator is built from stable identifiers in priority order: `data-testid` → ARIA role + accessible name → visible text → CSS path. This means the recorded step resolves to the same element when the case runs headlessly.

Recorded interact steps and widget assertions appear in the **In the app** section in the order they were captured. This order is also the execution order — the runner processes steps sequentially and stops at the first failure.

### Widget `ui/message` follow-ups

Some widget interactions send a `ui/message` back to the host (for example, a cart button that says "Show my cart") rather than calling a tool directly. During eval execution, these follow-up messages automatically drive a continuation model turn — the same way Playground handles them live. The resulting tool call (e.g. `view-cart`) is attributed to the same `interact` step's turn, so a **Tool was called** check on that turn sees the call.

This means an eval case can faithfully verify a full widget interaction sequence: prompt → widget renders → user clicks → model reacts → tool fires → assertion passes.

### What the trace shows

The Chat/Trace tab renders widget tool calls as the same app-attributed cards you see in Playground, so you can inspect exactly what the widget returned and what the model did with it.

* **JSON tool results** — Tool outputs that are JSON (including JSON wrapped in a text block) render as structured data in the card's **Data** tab rather than as a raw text block. There is no duplicate result row below the card.
* **Frozen widget diagnostics** — When a widget was captured as a screenshot during the run, the card's **Sandbox** tab shows the recorded policy: resource URI, CSP declaration, permissions, permissive mode, and any console errors or blocked requests logged during that run. Live-only fields (sandbox attributes, lifecycle, view origin) show as **not recorded** because they are not available after the run completes.

### The screen recording

The **Replay** tab plays a recording of the run beside the screenshot
filmstrip, and every step that captured an offset seeks the video to it.

Unattended runs on a hosted browser are recorded at **15 fps**. Idle time is
held rather than dropped, so the file's duration matches the run's wall clock
even when the page sat still for a minute — the frame count above the player
is the number of *distinct* frames, which is small on a page that did not move
and is not a sign of a fault.

Recordings stop at **60 MiB** and say so. A capped take is a complete, playable
*beginning* of the run, not a corrupt file and not the whole run: the player
shows a **Stopped at the size limit** badge, so a twelve-minute recording of a
forty-minute run is never mistaken for the run itself. Long runs that routinely
cap are the signal to shorten the case rather than to distrust the video.

A run that never uses a browser tool records nothing, and costs nothing.

Checks are what actually decide pass/fail. They're pure functions of the iteration transcript, so the verdict is the same every time you replay it — which is the property you want if you're using Evaluate as a regression gate.

Set defaults on the suite; override per case with **inherit** (use suite defaults), **replace** (use only the case's list), or **extend** (suite defaults, then the case's list).

| Check                             | Passes when                                                                                            | Scope        |
| --------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------ |
| **Tool was called with…**         | The named tool was called with arguments matching your spec (per the suite's `argumentMatching` mode). | Turn or case |
| **Tool was called at least once** | The named tool fired at least once across the turn.                                                    | Turn or case |
| **Tool was never called**         | The named tool did not fire. The core of negative cases.                                               | Turn or case |
| **First tool called was…**        | The first tool call in the turn matched the named tool.                                                | Turn or case |
| **Response contains…**            | The final assistant message contains a substring.                                                      | Turn or case |
| **Response matches regex…**       | The final assistant message matches a regex.                                                           | Turn or case |
| **No tool errors**                | No tool call returned an error.                                                                        | Turn or case |
| **Final message non-empty**       | The model produced a final assistant message.                                                          | Turn or case |
| **Widget rendered**               | The named widget was rendered during the turn.                                                         | Turn or case |
| **Widget render latency under N** | The widget rendered within the specified milliseconds.                                                 | Turn or case |
| **Widget no console errors**      | The widget rendered without console errors.                                                            | Turn or case |
| **Token budget under N**          | Total token usage stayed below the cap.                                                                | Case only    |

When a check is authored on an individual prompt turn it is evaluated against **that turn's slice** of the transcript — tool calls, assistant message, tool errors, and widget observations for that turn only. `Token budget under N` is case-only because per-turn token usage is not reliably captured; all other checks are turn-scopable.

<Note>
  **A per-trial token ceiling is not a spend control.** It is set without seeing
  what the suite normally uses, it is in a unit that is not comparable across
  models, and it goes stale on every prompt or model change. `Token budget under
    N` remains a valid check and keeps working wherever it is already authored,
  but for spend, two things replace it: [cost on every run](#cost), which shows
  what a suite actually costs, and `--max-cost-increase-percent`, which fails CI
  when a change makes the suite more expensive relative to its own baseline. For
  a ceiling on what the whole organization may spend, see the [spend
  budget](/hosted/overview#spend-budget).
</Note>

### Harness system tools in assertion dropdowns

When a suite is configured to run under a harness host (such as Claude Code or Codex), the tool-name pickers in the case editor also offer the harness's native **system tools** — for example `bash`, `read`, `webSearch`, `WebFetch` — alongside your MCP server tools. This lets you write assertions like **Tool was never called** for `bash` or **Tool was called with…** for `read` without having to type the wire name by hand.

System tools appear only in assertion pickers. They are not available in pinned tool-call steps or widget-assertion "View (tool)" selects, because those require a tool that MCPJam can invoke directly.

If an MCP tool and a system tool share the same name, the MCP tool takes precedence and the system tool is not added to the list.

### Argument matching

Tool-call argument comparison runs in one of three modes, configured at the suite:

* **partial** *(default)* — every expected key must be present and match; extra keys in the actual call are ignored. Best for "I care about `query` and `limit`, not what else the model put in."
* **exact** — actual args must equal expected args, key-for-key.
* **ignore** — only the tool name is checked.

Type placeholders (`"string"`, `"number"`, `"boolean"`) match any value of that type, which lets you assert shape without locking in a literal.

### Validator settings

Each suite has default validator settings controlling how tool calls are matched. Override them at three levels:

| Level         | Where                                    | Persists?    |
| ------------- | ---------------------------------------- | ------------ |
| Suite default | Suite settings → Default validators      | Yes          |
| Case override | Test case editor → Validators            | Yes          |
| Run override  | Suite header → validators (sliders) icon | No — one run |

A run override shows an **override** badge; click **Reset** in the popover to clear it.

## LLM as judge

For cases where "did the right tools fire" isn't enough — anything graded on the quality of the final answer — the judge grades the run against your **expected output** if you set one, and against the user prompt otherwise. It's advisory: it produces a score and a rationale, but doesn't gate the run unless you ask it to.

* **On by default** at the suite level. The cost is gated by an explicit **Run judge** click on the run-detail page — it won't run for every iteration unless you turn auto-run on.
* **Calibrate per suite.** Judge scores aren't comparable across domains; a 0.7 on one suite isn't a 0.7 on another.
* When grading against the prompt rather than an expected output, scores are capped at 0.85 — you can't get a "perfect" without saying what perfect means.

Open a completed run and find the **LLM as Judge** panel (labelled **Goal judge** when shown inside the suite results split). Pick a judge model (default: `openai/gpt-5.4-mini`) and a threshold (default 0.7), then click **Run judge**. Each case gets a score, an advisory verdict, a one-line reason, and rubric hits. The judge never runs automatically unless you enable **Auto-run** in suite settings.

Judge results are also accessible outside the app. `mcpjam cloud eval status` returns a `judges` block in its JSON output with each grader's status, summary, model used, threshold, and per-case grades. To request grading from the CLI, use `mcpjam cloud eval judge --run <id> --project <name>`. See the [`cloud eval judge` reference](/cli/reference#cloud-eval-judge) for the full flag list, including `--force` to re-grade and `--enable` to grade a run recorded before the judge was turned on.

The same judge config and defaults are shared with **Swarm journeys**. When creating a new journey, expand **Advanced → Judge** to set the judge model and enable auto-grade for that journey. Session score badges (showing the verdict and score) appear in the Sessions list and run matrix cells whenever a session has been graded. Sessions with no transcript still expose the on-demand judge entry point so failed or empty sessions can be graded against the journey goal.

### MCP App widgets in swarm sessions

When a swarm session uses MCP App tools that return widgets, the session viewer replays those widgets as full interactive components — the same rendering you see in the Playground. Widgets are captured per turn as the run executes and are available once the run completes.

<Note>
  During a live run, MCP App tool calls appear as plain tool pills in the session pane. The interactive widget renders once the run finishes and the snapshots are persisted.
</Note>

## Running

A run needs three things, all picked from the suite header:

1. **Servers** — one or more attached to the suite. Cases can attach their own subsets if they only need part of the surface.
2. **Models** — the multi-model picker is the whole point. Each model produces its own iteration per case, so you can see where Claude passes and ChatGPT trips.
3. **Run all** — kicks off every case × every model. **Run one** from a case row runs just that case.

<Tip>
  If **Run all** is disabled, a connected server is missing, no models are selected, or a run for that suite is already in progress. The header pickers will tell you which condition applies.
</Tip>

### Run review

When you click **Run** on a suite from the **Suites** landing list, a review sheet opens before the run launches. It shows the suite's current configuration — clients, iteration count, and settings — so you can confirm or adjust before committing. The run only starts when you confirm in the sheet; closing it leaves the suite unchanged.

### Pre-run credit estimate

When enabled on your account, an **ⓘ** icon appears beside run controls before you launch. Hovering it shows an estimated credit cost for the run about to start — priced against the exact models, cases, and iteration count the button will execute.

The estimate appears on:

* **Run all** in the suite header — prices every case × model × iterations × run plans.
* **Per-case Run** buttons in the cases list and case sidebar — prices that case's configured models for one quick run.
* **Run / Run compare** in the template editor — prices the currently selected models and the current (possibly unsaved) prompt.
* **Swarm journey cards** — prices the journey's next run across its live targets.

The tooltip shows a headline figure and a per-model or per-component breakdown. Copy varies by how well-grounded the estimate is:

| Copy                                                   | Meaning                                         |
| ------------------------------------------------------ | ----------------------------------------------- |
| Estimated: \~N credits — based on your recent runs     | Full coverage, grounded in your own run history |
| Estimated: \~N credits — partly based on recent runs   | Full coverage, partially grounded in history    |
| Rough estimate: \~N credits — improves after first run | Full coverage, heuristic only (no history yet)  |
| At least \~N credits — part of this run is unpriced    | Some models or run slices could not be priced   |
| Estimate unavailable for these models                  | No pricing data available                       |

1 credit = \$0.01. Your free daily allowance and any BYOK keys change what is actually deducted. The estimate is informational only — it never blocks or gates a run.

### Frozen execution snapshots

The first run of a suite saves the set of MCP servers used as a frozen snapshot; reruns reuse it, so connecting new servers can't silently change what a suite runs against. Click **Update snapshot** in the suite header to re-save the current servers and start a new run.

For harness runs, the run's pinned skills are also frozen at run start and delivered to each iteration from that snapshot — not re-fetched from the live project pool. This means editing a project skill mid-run does not affect iterations already in progress, and the "without skills" A/B arm (configured via `skillsOverride: "exclude"`) runs with no skills regardless of what the project currently holds.

## Computer environment

<Note>
  The computer environment picker is only visible when the **computers** feature flag is enabled on your account and a project is selected for the suite.
</Note>

In suite settings, the **Computer environment** select lets you pin a built Docker environment so every eval iteration boots a fresh sandbox from the same image. This keeps results comparable across runs and edits — each iteration gets an identical starting state.

* Choose an environment from the list. Environments that have not finished building are labelled **(not built)**; selecting one will cause the run to fail immediately.
* Choose **None (default image)** to run without a pinned environment.
* The agent receives a `bash` tool inside the sandbox.

Once a run completes, the **Run detail** page shows an **Environment** row with the name of the environment used. If the environment has since been deleted, the row shows its raw identifier instead so the record remains durable. Hovering the label shows the image template, digest, and provider from the frozen snapshot taken at run time.

## Suite execution config

Each eval suite has a **Default Execution Config** section that controls the model, system prompt, temperature, tool approval, connection settings, capabilities, and host context used when running the suite.

* **Model and prompt** — Set the default model ID, system prompt, and temperature for all runs in the suite. When running evals against MCP Apps that render widgets, the eval harness automatically enables browser interaction (screenshot, click, type) for any model that supports both vision and tool calling — not just Claude. Models that lack vision or tool calling run without browser interaction.
* **Tool approval** — Toggle `requireToolApproval` to pause before each tool call during a run.
* **Minimum iterations** — Set a floor on how many times each case runs. Every case runs at least this many times, regardless of its own iteration count. Use `mcpjam cloud eval update --suite <name> --min-iterations <1-10>` to set this from the CLI; `off` removes the floor.
* **Server selection** — Servers are not configured here. They come from the suite's environment. The server picker is intentionally hidden in this editor.
* **Save / Reset** — Click **Save config** to persist changes. Click **Reset** to revert to the last saved state. Unsaved edits are preserved if the page refreshes the config from the server, but are discarded when you switch to a different suite.

Changes to the suite execution config apply to future runs only. Existing run snapshots are not affected.

<Note>
  **Personal computers are not available for eval suites.** The personal computer toggle and computer-backed tools (such as Bash) are hidden and blocked in the suite execution config editor. If the project's default host config has a computer attached, resetting a suite to that default automatically strips the computer and any computer-backed tool ids.
</Note>

<Note>
  **A host that grants `bash` requires a pinned computer environment.** If the suite's host config includes `bash` (or another computer-backed built-in) but no computer environment is pinned, the run is refused at pre-flight rather than executing with the tool silently absent. Pin a computer environment in suite settings, or remove the computer-backed built-in from the host config.

  This applies to every eval run — harness and emulated alike. A single-case run (the **Run** button on an individual case outside a suite) is always refused for computer-backed built-ins because that surface never provisions a computer; run the case as part of a suite with a pinned environment instead.
</Note>

### How suite defaults apply to test cases

The suite-level **system prompt** and **temperature** are runtime defaults: when a test case does not set its own system prompt or temperature, the suite values are used for that iteration. A per-case override always wins — the suite default only fills the gap.

<Note>
  If you have existing suites where test cases do not specify a system prompt, those cases will now run with the suite's system prompt applied. Cases that already set their own system prompt are unaffected.
</Note>

## CI tab

The **CI** tab (when enabled for your project) shows only CI-active suites — suites created by the SDK or suites that CI has reported into at least once. Playground-only suites stay in the **Evaluate** tab. If a project has no CI evals yet, the CI tab shows a quickstart prompt.

The commit rail in the CI tab groups runs by commit SHA and only includes CI (SDK-sourced) runs. Playground runs on mixed suites are excluded from the rail so they don't appear as spurious manual-commit groups.

### Pass Rate vs Accuracy

The metric label shown in run headers, the runs table, charts, and hero stats follows the **run's source**, not the suite's creation source:

* **Pass Rate** — shown for runs submitted by the SDK (CI runs). Reflects per-case pass/fail.
* **Accuracy** — shown for runs triggered manually from the UI, via the API, or on a schedule.

On a mixed suite (one that has received both CI and playground runs), the label updates to match the most recent run. Legacy runs without a recorded source fall back to the suite's creation source.

### CI-managed suites

A suite marked **CI** keeps its configuration somewhere other than this app —
either a suite file committed to a repository (`mcpjam cloud eval run --file`), or a
suite the MCPJam SDK created when it reported a run. Those suites are
**read-only here**: name, settings, environments, schedule, models, skills,
execution config and cases all refuse edits, in the app and through the API
alike, with `409` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`.

One caller is exempt, and only one: the suite file itself. A write that sends
the suite's own `suite.id` as `declaredSuiteId` is the file syncing what it
already declares, so it is allowed through — which is how `mcpjam cloud eval
run --file` keeps working. Naming any other id refuses exactly as loudly as
naming none, so it is not a way in.

That is not a permission problem, and no role change lifts it. The reason is
that an edit made here would not survive: the file sync deletes any case the
file does not name, and an SDK suite's cases are regenerated from the next
report. Before this, such an edit lasted until the next CI run and then vanished
with nothing recording why.

Two ways forward, both offered on the suite:

* **Edit the test file** in your repository and run it again. The suite keeps its
  history and its CI wiring.
* **Duplicate to edit.** The copy is app-owned and fully editable, and CI can
  never target it.

Everything that is not a configuration edit is unchanged: run it, replay a run,
compare runs, waive a gate, read every setting.

### CI-synced test cases

Test cases written by a CI report carry a **CI** chip next to their name in the cases list. Opening such a case in the editor shows a notice: "Synced from CI — the next CI report may overwrite manual edits." Manual edits to these cases are preserved until the next CI ingest, which may overwrite them.

### Suite deletion

The delete button in the suite switcher is hidden for any suite that is CI-active — either created by the SDK or one that CI has reported into. This protects CI run history; the next CI report would recreate the suite anyway.

## Reading results

### Suite navigation

Opening the Evaluate tab lands on the **Runs** view by default, showing run history across all suites with grouped runs, pass-rate changes, and filters. Switch to **Suites** using the header tabs to see the suites overview. To drill into a suite, click its name; the **suite switcher** in the breadcrumb lets you jump between suites, create a new one, or delete the current one. If no suites exist yet, the same empty-state hero appears on both the Runs and Suites tabs — click **Eval my server: \<server name>** to open the create-suite form with that server and a name already filled in, or click **Create suite** to start blank.

### Suite view

* **Suite accuracy** — pass rate of the most recent run, with the last three runs' trend so you can see whether you're improving or regressing.
* **Run insights** — an AI-written diff against your previous completed run: which cases moved, which tools changed behavior, which models diverged. Skim this first; it usually points at the right rabbit hole.
* **Runs tab** — every run with its summary metrics. Click in for the iteration-level breakdown. The table shows **Date** first, then **Run**; suite group rows show a date range and a chevron on the left. Client/model pairings are shown inline per run row; when there are more than two, a **+N** badge appears — hover it to see all pairings in a tooltip.
* **Cases tab** — every case with its latest verdict and a quick replay button.
* **Executions tab** — a flat, filterable list of every individual test execution across all cases, sorted most-recent-first. Each row shows the case name, result (passed/failed/pending/cancelled), and timestamp. Click any row to open it in the compare view.

When you run a suite against multiple host configurations at once, the Runs list groups them into a single collapsible **run group** row showing mean accuracy and longest duration across all hosts. Expand it to inspect each host's individual run.

The **Runs / Cases** segmented control switches between the run history list and the test-cases overview. **Performance by Model** in the run summary appears only when more than one model was used. To compare two runs, check their boxes in the Runs list and click **Compare**.

Client and server filters on the **Suites** tab live in the column headers rather than as separate filter pills.

#### Where a run came from

Every run carries an origin badge — **UI**, **SDK**, **API**, **CLI**, **MCP**,
**GitHub**, **Scheduled** — and the Source chips above the Runs table filter on
it. The filter runs on the server, so an empty result means the project has no
such runs, not that they were further down the list.

Two different things go into that badge, and the distinction matters if you are
using it as evidence:

* **Stamped.** The platform decides a run's `source` itself, and a caller cannot
  set it. That is what makes it trustworthy — and also why everything launched
  through the public API is `API`: from the server's side, a CLI run, a GitHub
  Actions job and an agent's MCP call are all API calls.
* **Declared.** The launching process says which of those it is. The CLI declares
  `CLI` (or `GitHub` inside an Actions job, with the commit and job filling the
  CI column); the hosted MCP worker declares `MCP` and names the calling agent.
  Hovering a declared badge says so.

A declared origin is a label, not proof, so nothing is authorized on the
strength of it. Alongside both, the platform records the credential the request
actually authenticated with — which is what lets the **Run by** column say *via
API key ····3f9a* under the key owner's name, rather than making an automated
launch look like that person clicking Run.

#### Cross-host matrix

When two or more host configurations are attached to a suite, a **By case / By host** toggle appears. **By host** shows a matrix — one column per host, one row per case. Each cell shows pass/fail dots, pass rate, median latency, and token usage. A host detached after runs were recorded stays visible, labelled **historical**.

### Cost

Every iteration carries the MCPJam-billed cost of the model work it did,
priced when the trial finished and stamped with the rates used — so a run's
cost does not move later when prices change. It rolls up onto cases, runs and
the metric strip.

A dash (`—`) means **no cost was observed**, never that the trial was free:

| Why                                        | What you see                                                                                                                                                                                                    |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Your own API key (BYOK)                    | MCPJam did not bill it, so there is no cost to report.                                                                                                                                                          |
| A harness run (Claude Code, Codex, Cursor) | A harness turn mixes models — sub-agents on cheaper models under a main one — and the trial's totals carry no per-model split, so no honest single figure exists. Harness cost arrives with billed attribution. |
| No token usage reported                    | Nothing to price.                                                                                                                                                                                               |

A cost your own SDK runner reported is shown but labelled **runner** — it is
real, but MCPJam neither computed nor verified it.

When only some of a run's trials were priced, the total says so. That
distinction matters most in CI: a run that looks cheaper may simply be a run
we priced less of, which is why the compare API reports coverage counts
alongside the cost and a cost gate treats partial coverage as non-gateable
rather than as a pass.

### Run view

The run header shows a **HOLD** or **SHIP** pill for each client/model pairing in the launch. All pairings that share the same decision are grouped into one pill — a single **HOLD** pill lists every client that did not pass, and a **SHIP** pill lists every client that did. Intermediate states (**Running**, **Grading**, **Inconclusive**) each get their own pill. The **Compare runs** and **Run again** buttons are visible directly in the header.

Below the header, the hero section shows a row per client/model pairing with a pass/fail bar and counts. When a previous launch exists, each pairing row also shows a delta — how many more or fewer cases passed compared to the equivalent pairing in the prior launch. The first launch of a suite shows no deltas.

The **Test cases** section (formerly the matrix) lists every test case with a search box and **Status**, **Client**, and **Model** filters on one row. There is no master pass/fail bar or "RUN RESULTS" eyebrow label in this section.

* **Per-iteration row** with case, model, pass/fail, tokens, cost, duration, and the tool calls that actually happened. Each row also shows a one-line summary of where that trial's user-value chain stopped — for example `broke at Tool response`, `Request satisfied`, or `chain withheld` — so you can scan the table without opening each row.
* **Expected vs actual** tool calls side-by-side when an iteration fails. For failed iterations, a **Categorized diff** above the raw Expected/Actual grids groups discrepancies into four categories:
  * **Missing** — expected tool calls that were never made
  * **Extra** — actual calls that weren't expected (reported but non-fatal by default)
  * **Out of order** — calls that happened in the wrong sequence (when order checking is enabled)
  * **Arg mismatch** — right tool name, wrong arguments (shown side-by-side)
* **User-value chain** — when you expand a trial, six numbered stage cards show how far value travelled: Session connected → Tools discovered → Tool selected → Tool called → Response returned → Request satisfied. The card for the first stage that failed opens automatically. Click any card to read what happened at that stage and why; click it again to close. This is shown for every trial — passing and failing alike. A trial that delivered end to end reads as the full delivery story; a failing trial opens on the break.
* **Full trace** — every turn, every tool call, every token. This is the thing you couldn't see before; spend time here. When a turn ran through a harness (e.g. Claude Code), the trace detail pane shows an **Engine** badge — "Claude Code" for harness turns, "Emulated" for turns that ran through MCPJam's built-in chat engine — so you can tell at a glance which runtime handled each turn. The user-value chain also appears above the transcript in the trace pane, so the answer to "where did this trial stop" is on the same screen as the evidence.
* **Per-model breakdown** to compare how the same case behaves across models.
* **Suggested fixes** — after a run completes, a **Suggested fixes** panel ranks tool-quality and workflow issues by impact. Each issue has a **Copy** button that copies a ready-to-use fix prompt (tool description + input schema) for a coding agent; **Copy top N** copies the top issues combined.
* **Predicate Gate** — when `successPredicates` are configured on a case, an expandable Predicate Gate section in the iteration detail lists each predicate with a PASS/FAIL verdict, a one-line summary, and the evaluator's reason. Hidden when no predicates are configured.

#### Comparing two runs

Select any two completed runs and click **Compare**. The diff view shows per-case status changes: **Passed**, **Still failing**, **Regressed** (pass→fail), **Fixed** (fail→pass), **New**, **Removed**, and **Changed** (config differed). Summary metrics show deltas for tokens, cost, and duration.

### Case view

* **Pass rate across runs** — is this case stable, flaky, or trending down?
* **Pass rate by model** — does this case only work on one model?
* Every past iteration with its trace, so you can A/B a regression against a working run.

## Promoting sessions to cases

Any completed chat session can be turned into a test case without rewriting it from scratch. The full conversation is compiled into multi-turn prompt turns, and the dialog lets you pick a destination suite or create a new one.

**From the Playground** — Open the **Sessions** rail, hover a session, and click **Promote to test case**.

**From the Swarms tab** — Open a completed run, click a session row to select it, then click **Promote to test case** above the session viewer. The session must belong to a completed run attempt; sessions from in-progress runs show an error in the dialog instead of blocking at submit.

In both cases the dialog shows the servers recorded for that session, lets you set a case title and suite name, and (when attachment pickers are enabled) lets you configure the server attachment and client host for the new suite.

## Generating cases from your tools

The **Generate** button reads your attached servers' tool catalog and drafts realistic cases — a mix of positive ("call `search` with a query") and negative ("don't call `delete_user` on a meta-question"). Treat it as a draft: skim, edit the prompts to match how your users actually talk, tighten the checks, then save.

When the suite has a saved server group spanning two or more servers, the generator produces coverage for each server individually plus at least one cross-server case. Suites without a saved server group treat all available servers as a single pool.

## Plugin provenance on runs

When a run executed against an environment that pins a plugin, the run detail page shows a **Plugins** row directly below the environment label. Each chip displays the plugin name and a short bundle hash — the content-addressed identity of the exact bundle the run used.

The bundle hash is what makes two runs of the "same" plugin comparable: a re-import mints a new hash even when the name is unchanged, and an edit changes the hash under an unchanged name. The row is read-only provenance — it records what ran and offers no way to restore or re-pin those versions.

When a run is the "without skills" arm of an A/B compare, a **skills excluded** badge appears alongside the plugin chips. This distinguishes a deliberately skill-free run from one whose skills failed to load; the plugin's MCP servers were still connected for that run.

## OpenAI submission report

For runs that pinned a plugin, a **Submission report** button appears in the run header alongside **Export traces**. Clicking it downloads a Markdown document formatted for OpenAI's plugin directory review, which asks for five prompts where the plugin should be invoked and three where it must not.

The report is built from the run's own frozen case records — not from the live suite — so it names the exact bundle hash the run executed. It selects up to five passing positive cases and up to three passing negative cases. When there are not enough passing cases to fill the quota, failing cases are included to reach the count and listed explicitly at the top of the document under **Not ready to submit**.

The report also flags these conditions as not ready to submit:

| Condition                   | What it means                                                                                      |
| --------------------------- | -------------------------------------------------------------------------------------------------- |
| Fewer than 5 positive cases | The run did not execute enough "should invoke" cases. Add cases to the suite and re-run.           |
| Fewer than 3 negative cases | The run did not execute enough "should NOT invoke" cases. Mark cases as negative tests and re-run. |
| Failing cases included      | One or more selected cases failed; they are named at the top of the document.                      |
| No plugin pinned            | The run pinned no plugin version, so the report cannot state which bundle was evaluated.           |
| Skills excluded             | The run is the "without skills" A/B arm. Submit the arm that includes skills.                      |

The button only appears on runs that pinned a plugin. Plugin-free runs produce no submission document.

## Exporting traces

Open a completed run and click the **Export traces** button (download icon in the run header) to download the run's traces as an OTLP JSON file. The export uses OpenInference semantic conventions plus `mcp.app.*` extensions, making it compatible with Arize Phoenix, Datadog, and any other OTLP-compatible observability backend.

### Export options

| Option                          | Description                                                                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **This run**                    | Exports only the iterations from the selected run.                                                                                                                  |
| **Whole project**               | Exports all recorded sessions across every run in the project.                                                                                                      |
| **Include content & artifacts** | Off by default. When enabled, the export includes prompts, outputs, tool arguments and results, and screenshots. Disable this when the content may contain secrets. |

The downloaded file is named `mcpjam-traces-<scope>-<timestamp>.json`.

<Note>
  If the project is too large to export in a single file, use **This run** scope and export individual runs instead.
</Note>

## What to author first

If you're new to the surface, the shortest useful loop is:

1. Connect a server in the Inspector.
2. On the **Suites** or **Runs** landing, click **Eval my server: \<server name>** to open the create-suite form with that server and a name already filled in.
3. Review the generated cases, confirm the clients and iteration count, then click **Run** to launch.
4. Open the failures, and decide whether the bug is in your server, your prompt, or the model.

New suites are named **Suite 1**, **Suite 2**, and so on — the form picks the next unused number automatically. You can rename the suite at any time.

Alternatively, click **Create suite** to start from a blank suite and author cases manually:

1. Attach the server you're shipping.
2. **Generate** a starting set of cases.
3. Delete the ones that don't match real usage; tighten checks on the rest.
4. Add the two or three models your users will hit.
5. **Run all**, open the failures, and decide whether the bug is in your server, your prompt, or the model.

That last step is the one that benefits most from spending time in the trace view.


# OAuth Debugger
Source: https://docs.mcpjam.com/inspector/guided-oauth

Debug your MCP server's OAuth implementation at every step of the handshake

The OAuth Debugger provides a visual, step-by-step interface for testing and debugging your MCP server's OAuth 2.1 implementation. View every step of the OAuth handshake in detail with guided explanations, inspect network messages, and validate your implementation against multiple protocol versions. The debugger is fully compliant with [MCP OAuth specification](https://modelcontextprotocol.io/specification/draft/basic/authorization).

<Frame>
  <img alt="MCPJam OAuth Debugger Interface" />
</Frame>

## Key Features

The OAuth Debugger includes:

* **Visual Step-by-Step Guide** - Interactive flow guide with detailed explanations for each OAuth step
* **Multi-Protocol Support** - Test against OAuth spec versions 03-26, 06-18, 11-25, and 2026-07-28 (Latest)
* **Full Registration Methods** - Support for client pre-registration, Dynamic Client Registration (DCR), and Client ID Metadata Documents (CIMD)
* **Network Inspection** - View all HTTP requests and responses with headers and body content
* **Educational Context** - Built-in teachable moments and tips for common OAuth issues
* **Sequence Diagram** - Visual representation of the OAuth flow synchronized with your progress, showing the exact resource value the flow sends
* **Export Capabilities** - Copy logs in guide or raw format for debugging and documentation

### Resource indicator warnings

When the server's Protected Resource Metadata (PRM) advertises a `resource` identifier that does not fully satisfy RFC 9728 — for example, a same-origin resource on a different path than the transport endpoint — the debugger logs a warning at the `received_resource_metadata` step and continues the flow. The warning includes the advertised value, the server URL, and whether strict MCP clients (such as Quick OAuth or the official MCP SDK) will refuse to connect.

The debugger intentionally proceeds rather than failing so you can observe real server behavior. Use the [OAuth Conformance](/cli/oauth-conformance) tool if you need strict RFC 9728 validation.

### Path-scoped authorization servers

Some multi-tenant authorization servers (such as Scalekit) scope their OAuth endpoints under a path — for example `https://<env>.example.com/resources/res_x` — while their metadata advertises the origin root as the `issuer`. By default, Inspector enforces the RFC 8414 §3.3 requirement that the advertised `issuer` exactly matches the URL discovery started from, so those servers fail discovery with an error that names the mismatch.

To connect to a path-scoped authorization server, enable the **Path-scoped authorization server** switch in **Advanced settings**:

* In the **OAuth Debugger**, open the **Configure** dialog and expand **Advanced settings**.
* In the **Connect** server form, open **Authentication → Advanced settings**.

The toggle is **off by default**. When off, the strict exact-match check applies and the error message names the toggle so you can find the fix from the error itself.

When enabled, the following rules still apply:

* The advertised issuer must be a same-origin, segment-aware path-prefix ancestor of the discovery URL. For example, `/resources` is accepted as an ancestor of `/resources/res_x`, but `/resources-evil` is not.
* The `token_endpoint` and `registration_endpoint` in the server's metadata must stay on the advertised issuer's own origin. A present endpoint that points off-origin is rejected even with the toggle on.
* A query string or fragment on either side of the comparison disqualifies the relaxation (RFC 8414 §2 forbids both in an issuer identifier).
* Cross-origin issuers are always rejected.

When the toggle accepts a path-scoped issuer, a **warning** is recorded in the OAuth flow log because strict MCP clients may still refuse to connect to the server being debugged.

The setting is stored per server and applies to both the OAuth Debugger and the Connect flow, so the behavior is consistent across both surfaces.

### Authorization-response `iss` mismatch warnings

When the authorization server returns an `iss` parameter on the callback that does not match the issuer recorded from its metadata, Inspector's behavior depends on the protocol version selected for the flow:

* **2026-07-28 (Latest)** — The flow is blocked before the authorization code is redeemed. The error names both the recorded issuer and the one returned on the callback so you can compare them directly.
* **2025-11-25 and earlier** — The flow continues and a warning toast is shown naming both issuers. These protocol versions do not mention `iss` or RFC 9207 issuer validation, so blocking would enforce a rule the selected version does not contain.

In both cases the mismatch is recorded on the OAuth trace at the `received_authorization_code` step, visible in the **Last OAuth Trace** panel.

## Inline OAuth tracing

When you connect a server using OAuth 2.0, Inspector automatically records a trace of every step in the flow and surfaces it in the connection UI — no separate tool required.

**On the connection card**, if OAuth fails, a banner identifies the exact step where the failure occurred (for example, "OAuth failed during Token Request").

**In the server detail panel**, the **Last OAuth Trace** section shows:

* Each OAuth step (discovery, client registration, PKCE, authorization redirect, token exchange) with a status indicator and message
* The error text for any failed step
* An **HTTP History** log of every request and response, with tokens and secrets automatically redacted

The trace persists across page reloads so you can inspect it after the browser returns from the authorization redirect. It is cleared when you disconnect or reset OAuth data for the server.

Use the inline trace for a quick diagnosis of real connection attempts, and switch to the OAuth Debugger when you need a guided, step-by-step walkthrough against a specific protocol version.

## Getting Started

To begin debugging OAuth flows:

1. **Navigate to the OAuth Debugger** - Click the OAuth Flow tab in MCPJam Inspector
2. **Configure a Target Server** - Click **Configure** to open the OAuth Profile Modal prefilled with the currently selected server, or click **Add** to open a blank form and save a new target without changing the current one
3. **Enter Server Details** - Provide your server's base URL and select your protocol version
4. **Choose Registration Method** - Select CIMD, DCR, or pre-registered based on your implementation
5. **Start Testing** - Click **Continue** to begin the OAuth flow

The debugger will guide you through each step of the OAuth handshake, providing real-time feedback and educational context.

<Note>
  The header **Add Server** button also opens a blank form so you can add a new target without affecting the server currently selected in the debugger.
</Note>

## OAuth trace in server overview

When you connect a server using OAuth, Inspector records a structured trace of the entire flow. You can view this trace at any time — even after a failed connection — by opening the server's detail panel and scrolling to the **Last OAuth Trace** section.

The trace shows:

* **Source** - Which part of the flow produced the trace (`interactive_connect`, `callback`, `refresh`, or `hosted_callback`)
* **Current step** - The step the flow reached before completing or failing
* **Step-by-step breakdown** - Each OAuth step with its status (`success`, `pending`, or `error`), a summary message, and optional HTTP request/response details
* **HTTP history** - A bounded list of all HTTP calls made during the flow, with headers and bodies sanitized to redact sensitive values such as tokens and secrets
* **Failure highlight** - If a step failed, the error panel on the server card and the overview both identify the specific step that caused the failure

The trace persists across page reloads so you can inspect a failed OAuth flow after returning to the inspector.

### Reading OAuth log entries

OAuth trace steps also appear as filterable entries in the traffic log. Select **OAuth** from the **Source** filter in the log panel to isolate them. See [OAuth log entries](/inspector/tools-prompts-resources#oauth-log-entries) for details on the status indicators.

## Bearer token authentication

When you configure an HTTP server with a bearer token, the **Auth** tab shows a **Bearer Token Authentication** panel instead of the OAuth panel. The panel confirms that the server is using bearer auth and directs you to the server configuration to manage or rotate the token.

The **Auth** tab also displays an **Authentication** label that reflects the current auth type for the selected server: **Bearer Token**, **OAuth**, **Process-based** (STDIO), or **No Authentication**.

### Saved bearer tokens

When a bearer token has been saved for a server, the token value is hidden in the edit form. The **Bearer Token** field shows the placeholder text **"Saved — enter a new value to replace"** to indicate that a token is stored without revealing it.

* **Leave the field blank** to keep the existing saved token when you save other changes to the server.
* **Enter a new value** to replace the saved token on the next save.

In hosted mode, a **Reveal** button appears next to the **Bearer Token** label. Clicking it fetches and displays the stored token value so you can inspect or copy it. The revealed value is cleared when you close the panel or navigate away.

<Note>
  Bearer tokens are stored securely and are never written to your browser's local storage in hosted mode. In local mode, the token is stored in your browser's local storage alongside the rest of your server configuration.
</Note>

## Pre-registered clients

When you select **Pre-registered** as the registration strategy, a **Client ID** is required before you can save the configuration. Because the pre-registered flow skips Dynamic Client Registration entirely, there is no way to obtain a client ID at runtime — the flow will fail without one.

If you try to save without a Client ID, the form blocks the save and shows an inline error. The **Advanced settings** section expands automatically so you can enter the Client ID without having to find and open it yourself. Switching the **Registration** dropdown away from **Pre-registered** (for example, to **Dynamic (DCR)**) clears the error.

### Pre-registered client secrets

When you configure a pre-registered OAuth client with a client secret, the secret is stored securely and never exposed in the UI after saving. The client secret field shows the following controls:

* **Reveal** (hosted mode only) — Fetches and displays the stored secret in an editable field. You can copy it or type directly in the field to replace it. Click **Hide** to collapse the field without saving changes.
* **Clear** — Marks the secret for removal. The secret is deleted from storage when you save. Click **Undo** to cancel before saving.
* **Replace** — After clicking **Reveal**, type a new value into the revealed field to replace the stored secret on the next save.

When a secret is saved but not yet revealed, the field shows a "A client secret is saved" message in place of an input box. No empty input is shown until you click **Reveal**.

<Note>
  In hosted mode, client secrets are stored in a secure vault and are never written to your browser's local storage. In local mode, the secret is stored in your browser's local storage alongside the rest of your server configuration.
</Note>

### Authorization header conflict warning

If you enable OAuth and also add a custom `Authorization` header under **Advanced settings**, a warning appears. OAuth automatically sets the `Authorization` header when exchanging tokens, which may override or conflict with your custom value. Remove the custom header or switch to a different authentication method to avoid unexpected behavior.

## Navigating to the OAuth Debugger from the logger

When an OAuth error appears in the request log, a **Continue in OAuth Debugger** button is shown on hover for that log row. Clicking it switches the selected server to the one that produced the error and deep-links directly to the OAuth Debugger (`#oauth-flow`), so you can investigate the failure without manually locating the right server.

## Reconnecting OAuth servers

The enable/connect toggle on a server card or detail modal reuses existing credentials only — it will not launch a new OAuth consent screen. If the existing tokens are expired or missing and a fresh consent flow is required, the toggle reports a re-authentication error instead of redirecting you to the authorization server.

To start a full OAuth flow, use the explicit **Reconnect** button on the server card or detail modal. This is the only path that will open the OAuth consent screen when new credentials are needed.

## Desktop app behavior

When using the OAuth Debugger in the MCPJam Desktop app, the authorization step opens in your system browser instead of inside the app. After you approve access in the browser, you will see a **"Continue in MCPJam Desktop"** page that automatically hands control back to the desktop app. If the handoff does not happen automatically, click the link on that page to return.

Once the browser callback is received, the desktop app resumes the OAuth flow and displays the result in the OAuth Flow tab.


# Home
Source: https://docs.mcpjam.com/inspector/home

Org-scoped dashboard with activity stats, recommended servers, and quick-start clients

The Home dashboard is the default landing page for authenticated users with an active organization. It gives you a snapshot of your org's activity and lets you connect recommended MCP servers or spin up sandbox clients without leaving the page.

<Note>
  The Home dashboard is available in the hosted app at [app.mcpjam.com](https://app.mcpjam.com) and requires you to be signed in with an active organization.
</Note>

## Overview

When you open MCPJam, the Home tab loads at `/home` (also the root `/`). If you don't have an active organization yet, the page shows a **Get started** prompt that takes you to the Organizations page.

<Note>
  If you are signing in for the first time, MCPJam routes you to the [Playground](/inspector/playground) instead of Home so you can start exploring right away. After your first visit to the Playground, subsequent logins land on Home as normal.
</Note>

With an organization selected, the dashboard shows:

* **Greeting** — a time-aware greeting with your first name.
* **Org stats strip** — live counts for teammates, projects, servers, eval suites, tool executions (last 30 days), and messages sent (last 30 days).
* **MCPJam Agent composer** — a text input to start or resume an agent conversation directly from the home page.
* **What's new** — a compact list of recent platform releases and changes, with per-item dismiss and a "Clear all" action.
* **Recommended servers** and **Recommended clients** — displayed side by side in a two-column grid.

## Org stats strip

The stats strip appears directly below the greeting and shows six metrics for your organization:

| Stat                  | Description                            |
| --------------------- | -------------------------------------- |
| Teammates             | Number of members in your organization |
| Projects              | Total projects in your organization    |
| Servers               | Total MCP servers across all projects  |
| Eval suites           | Number of evaluation suites            |
| Tool executions · 30d | Tool calls made in the last 30 days    |
| Messages · 30d        | Messages sent in the last 30 days      |

Click **projects** or **servers** to jump to the Connect tab. Click **eval suites** to jump to the Evals tab. The 30-day metrics refresh automatically in the background.

## Recommended servers

The **Recommended servers** section shows a curated list of MCP servers. To connect one:

1. Make sure you have an active project selected.
2. Click **Connect** next to the server you want.
3. MCPJam creates the server on your active project and navigates to the Connect tab.

If the server is already connected to your project, you'll see an info toast and be taken to the Connect tab instead.

## Recommended clients

The **Recommended clients** section lets you create a pre-configured sandbox client from a template. Available templates include Claude, ChatGPT, and Cursor.

To create a client:

1. Make sure you have an active project selected.
2. Click the template you want.
3. MCPJam creates the client on your active project and opens it in the Clients tab.

## What's new

The **What's new** section appears above the recommended servers and clients grid and lists recent platform updates.

* **Hover** a row to open a popover with a muted looping video preview (or a YouTube thumbnail if no preview clip is available).
* **Click** a row to open a panel with the full video, a "Watch on YouTube" link, an optional Docs link, and the full release description. Click **Got it** or press **Escape** to close.
* **Dismiss** a single update with the **✕** button that appears on hover.
* **Clear all** dismisses every pending update at once.
* If there are more than three updates, a **View more** button expands the list inline.

## MCPJam Agent panel

The MCPJam Agent is a persistent chat panel that lets you ask questions about docs, evals, and tools without leaving your current tab. It is available on every tab — Playground, Evals, Clients, and others — so you can keep a conversation going while you work.

### Opening the panel

* Click the **Ask MCPJam** button in the top-right header bar.
* Press **⌘\\** (Mac) or **Ctrl+\\** (Windows / Linux) from anywhere in the app. The shortcut is ignored when focus is inside a text input, textarea, or content-editable field.

### What the agent can do

The agent acts by driving the inspector UI directly — every action it takes is visible on your screen as it happens. It uses the documentation and web search to answer knowledge questions, and navigates or operates the app on your behalf through UI actions.

| Capability                     | What it does                                                                                                                |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| **Navigate**                   | Opens tabs, pages, and panels in the inspector                                                                              |
| **Playground**                 | Opens the playground, selects tools, and prefills or runs tool calls                                                        |
| **Server management**          | Adds, connects, disconnects, and removes MCP servers on the Connect screen by chat                                          |
| **App snapshot**               | Reads the current state of the UI to observe before acting                                                                  |
| **Documentation & web search** | Answers questions about MCPJam features and MCP concepts                                                                    |
| **Clarifying questions**       | Asks a multiple-choice question inline when a request is genuinely ambiguous, then continues the same turn with your answer |

For server management, adding and connecting are always two separate steps — the agent saves the server first, then connects it, so you can see each change as it happens. If a server requires OAuth authorization, the agent reports that and waits for you to click **Authorize** yourself; it never redirects the page on your behalf. Removing a server is irreversible from chat, so it always pauses for approval even when the **Tool approval** toggle is off.

When the agent needs to clarify your intent before acting, it renders a question card inline in the thread with 2–4 choices. A **Something else** option is always present so you can answer in your own words if none of the choices fit. Once you pick an option or type a free-text answer, the agent continues the same turn with your response. If you send a new message or press **Stop** instead of answering, the question is dismissed and the agent does not attempt to answer the original request — it responds to whatever you say next.

Actions that mutate the UI (such as running a tool call) pause for your explicit approval before they run when the **Tool approval** toggle is on. Destructive actions — those that can't be undone by looking at the screen — always pause for approval regardless of the toggle setting.

### Using the panel

* **New chat** — Click **New chat** in the panel header to start a fresh conversation.
* **Resume a session** — Previously started sessions are listed in the panel's home view. Click one to continue.
* **Resize** — Drag the left edge of the panel to adjust its width. The minimum width is 360 px; the maximum is 50% of the viewport. Dragging below the minimum snaps the panel closed.
* **Close** — Click the **✕** button in the panel header, or press the keyboard shortcut again.

### Tool approval

The composer footer includes a **Tool approval** toggle (shield icon).

* **Toggle off (default)** — Additive UI actions apply immediately. Destructive actions (those that can't be undone by looking at the screen) still pause for approval.
* **Toggle on (strict)** — Every mutating UI action pauses before it runs.

When the agent pauses, an **Approve / Deny** pill appears in the chat. Denying a call is final — the agent will not retry it automatically. The preference is saved to your browser's local storage and shared across the Home takeover and the side panel.

### Widget rendering

The agent supports MCP Apps widget rendering. When a workspace tool returns an interactive widget — for example, asking "show my servers" — the widget renders inline in the chat, just like widgets in the LLM Playground. You can interact with the widget directly without leaving the panel.

### Persistence

The panel's open state, width, and active session are saved to your browser's local storage and restored on reload. Conversations persist server-side, so you can close and reopen the panel without losing your chat history.

On mobile viewports (below 768 px) the panel opens as a full-width drawer instead of an inline side panel.

<Note>
  The MCPJam Agent panel is available to all signed-in users.
</Note>

## Notifications

Notifications are accessible from the sidebar. Click **Notifications** next to the **Support** entry at the bottom of the sidebar to open the notifications panel. Unread notifications show a badge count that is visible in both expanded and collapsed sidebar states.

## What's next

* [Projects](/inspector/projects) — create and manage projects for your servers
* [Clients](/inspector/clients) — configure and test MCP clients
* [Servers](/getting-started#3-connect-your-own-server) — connect your own MCP servers


# Host Compatibility
Source: https://docs.mcpjam.com/inspector/host-compat

See which hosts your MCP server will work in — Claude, ChatGPT, Cursor, Copilot, Codex — the moment you connect

When you connect an MCP server, MCPJam automatically evaluates how well it will work in each major host and shows the result directly on the server card. No extra steps required — the data comes from what the inspector already captures at connect time.

## The support pill

Once a server is connected, a **support pill** appears at the bottom of its card. It shows a supporter count ("Supported by N") alongside up to three client logos — ChatGPT and Claude lead the stack when they support the server, since they're what most people are checking for first.

* **Supported** — the pill shows the count and up to three client logos.
* **No supporters** — once analysis has settled, the pill reads "No clients support this server" in a red-tinted style. Mid-analysis ("Checking compatibility…") and failed checks ("Compatibility checks unavailable") are shown instead when the result isn't known yet.

A client counts as a supporter when its compat status is green — meaning the host profile covers all of the server's relevant requirements. A `works` verdict that carries a blocking or protocol-mismatch finding is not counted.

Click the pill to open the **Clients** tab in the server detail modal for the full per-host breakdown.

## The Clients tab

The server detail modal has a **Clients** tab (alongside Config, Overview, Tools, and History). It shows a per-host breakdown:

* **Verdict** — Works / Degraded / Blocked / Unknown, with a colored dot.
* **Provenance** — How trustworthy the host profile is. Hover the verdict dot to see one of:
  * *Verified from vendor docs* — the profile is based on a published vendor capability table.
  * *Probe-captured from a real client* — the profile was captured from a live client session.
  * *Best-effort preset — unverified* — the profile is a best-effort estimate.
* **Findings** — Click the summary line to expand the full list of findings for that host. Each finding shows:
  * A severity icon (blocker, degraded, or info).
  * A title and detail explaining what's affected and why.
  * A **remediation** hint where one exists (for example, "Declare an MCP Apps template alongside the OpenAI one").
* **Test button** — Creates a new host pre-configured for that host's template with your server already attached, then opens the Playground. This is the one-click path from "degraded in Copilot" to testing the fix.

## What the report checks

The report is computed from data already on the client at connect time — no extra network calls beyond the normal tool list fetch.

### Widget rendering

The most common source of findings. The report checks whether each host can render the widget bridges your server's tools declare:

* **MCP Apps** (`_meta.ui.resourceUri`) — rendered by Claude, ChatGPT, Cursor, and Copilot. Not rendered by Codex (a CLI with no widget surface).
* **OpenAI Apps** (`openai/outputTemplate`) — rendered by ChatGPT and Copilot via the `window.openai` shim. Not rendered by Claude, Cursor, or Codex.
* **Dual-bridge** (both keys declared) — renderable wherever either bridge is supported.

When a host can't render a widget, the verdict is **Degraded** (the tool falls back to its plain-text result). If the tool is declared app-only (`_meta.ui.visibility` excludes `"model"`), there is no text fallback and the verdict is **Blocked**.

### Widget capability gaps (L1 scan)

For each widget, the inspector reads the widget's HTML via `resources/read` and scans it for the host APIs it actually calls — for example, `ui/message`, `tools/call`, `window.openai.sendFollowUpMessage`. A capability finding fires only when:

1. A widget in your server actually uses a capability, **and**
2. The host's capability matrix doesn't support it.

This means findings are server-specific, not generic host knowledge. If your widget doesn't call `ui/message`, you won't see a "follow-up messages won't work" finding even on a host that lacks it.

If the widget HTML can't be read (the `resources/read` call fails), the capability dimension stays **Unknown** rather than showing a false Works verdict.

## Verdicts in detail

### Works

All checks passed. The server's tools, widgets, and declared capabilities are fully supported by this host.

### Degraded

One or more features will be lost, but the server still connects and the model can still call tools. Common degraded findings:

* A widget falls back to plain text because the host doesn't render that widget bridge.
* A widget calls `ui/message` (follow-up messages) but the host doesn't support it.
* A widget calls `tools/call` (server tool calls from within the widget) but the host doesn't support `serverTools`.

### Blocked

A hard blocker makes the server unusable in this host. Currently this only fires when a widget tool is declared app-only and the host can't render it — the tool is hidden from the model and has no text fallback, so it's effectively dead.

<Note>
  Transport and auth blockers (for example, a local stdio server that a cloud host can't reach) are intentionally omitted from the report. A local dev server is expected not to be reachable from ChatGPT or Copilot — flagging it would be noise during development.
</Note>

### Unknown

The report doesn't have enough information. This happens when:

* The tool list hasn't loaded yet (the pill shows "Checking compatibility…" briefly after connect).
* Every `resources/read` call for the server's widgets failed, so the capability scan couldn't run.

Unknown is not the same as Works — the report withholds a verdict rather than guess.

## Provenance and confidence

Host profiles are best-effort. The provenance label on each verdict tells you how much to trust it:

| Provenance | Source                                                               | Confidence                        |
| ---------- | -------------------------------------------------------------------- | --------------------------------- |
| Vendor doc | Published capability table (e.g. Microsoft's Copilot MCP Apps table) | High                              |
| Probe      | Captured from a live client session (e.g. Cursor 3.14.27)            | High, but versioned               |
| Assumed    | Best-effort preset judgment                                          | Lower — treat as a starting point |

The copy in the Clients tab always says "prototype · static checks" to make clear this is not a live test against the real host. Use the **Test** button to open the Playground and verify behavior under the host's emulated environment.

## Acting on findings

### Fix a degraded widget

If a widget is degraded because the host doesn't render its bridge, the finding includes a remediation hint:

* **MCP Apps widget on an OpenAI-only host** — add an `openai/outputTemplate` key alongside `_meta.ui.resourceUri` so the tool has both bridges.
* **OpenAI Apps widget on an MCP-Apps-only host** — add a `_meta.ui.resourceUri` key alongside `openai/outputTemplate`.

After updating your server, reconnect and the support pill will re-evaluate.

### Test in an emulated host

Click **Test** next to any host in the Clients tab to:

1. Create a new host pre-configured from that host's template (Claude, ChatGPT, Cursor, Copilot, or Codex).
2. Attach your server to it automatically.
3. Open the Playground with that host selected.

From there you can chat with your server under the host's emulated capabilities and sandbox policy to verify the fix.

## Related

* [Clients](/inspector/clients) — Configure host emulation settings, capability matrices, and sandbox policies.
* [Playground](/inspector/playground) — Test your server under emulated host environments.
* [Contribute a Host](/inspector/host-templates) — Add a new host preset to the registry.


# Contribute a Host
Source: https://docs.mcpjam.com/inspector/host-templates

Add a new host preset to MCPJam so widgets can render under that host's identity, capabilities, and sandbox policy.

MCPJam ships built-in host presets for Claude, ChatGPT, Cursor, Copilot, Codex, and MCPJam itself. Each preset captures what a real host publishes during MCP `initialize` and `ui/initialize` — `clientInfo`, `hostInfo`, capability advertise, `hostContext` (theme, viewport, device, locale), and sandbox policy (CSP directives, permissions, allow-features). With a preset selected, a widget rendered in the [Playground](/inspector/playground) behaves the way it would in production under that host.

If your host isn't in the list yet — or if Microsoft, Anthropic, OpenAI, etc. ship a new capability and the inspector hasn't caught up — adding a preset is a small, self-contained contribution. This page walks through it.

<Note>
  Host presets are values, not code. There's no runtime to write. A new preset
  is a TypeScript object that conforms to two interfaces, plus a logo file.
</Note>

## What a preset includes

A host preset has two pieces:

1. **Host style** — branding, label, scenario chrome, font CSS, and the capability **surface** the host supports (which MCP Apps and OpenAI Apps SDK features it implements). Defined in `client/src/lib/client-styles/built-ins.ts`.
2. **Catalog host** — the backend-owned host row used for creation, compare, caniuse, and "update to latest". It includes label/provenance facts plus the full `HostConfigInputV2` template: `clientCapabilities`, `hostContext`, `mcpProfile`, image policy, sandbox policy, model defaults, and other host fields. Defined in `mcpjam-backend/convex/marketHostCatalog/templates.ts` and `mcpjam-backend/convex/marketHostCatalog/seed.ts`.

The backend catalog is canonical at runtime. The SDK carries a generated fallback snapshot for CLI/offline/dev paths; do not hand-edit SDK fallback data when changing host facts.

## The probe-driven philosophy

The single most important rule when adding a host: **capture values from a live probe, don't invent them.**

Every existing preset has inline comments citing a real source — a captured `ui/initialize` response, a DevTools Network capture of the host's response Content-Security-Policy header, or an official vendor doc. Examples from the codebase:

* `CLAUDE_HOST_STYLE_VARIABLES` — verbatim from Claude's published design tokens.
* `CHATGPT` template `cspDirectives` — "captured 2026-05-18 via DevTools → Network → oaiusercontent.com response".
* `COPILOT` template — links to Microsoft Learn's "Supported MCP Apps capabilities in Copilot" table.

Inventing values silently misrepresents the host. A widget that works in MCPJam-as-YourHost but fails in production YourHost is the failure mode we're avoiding.

<Info>
  When a field can't be probed (no live capture, no public doc), say so in a
  code comment. Several existing presets use phrases like *"Undocumented; chosen
  to match the ChatGPT convention"* — that's honest. Don't hide guesses behind
  clean values.
</Info>

## Steps

<Steps>
  <Step title="Probe the real host">
    Capture as much as you can from the live host before writing any code:

    * **MCP `initialize` response** — `serverInfo`, `clientInfo`, advertised `capabilities` (including the `experimental` block and the MCP UI extension).
    * **`ui/initialize` response** — `hostInfo`, `hostCapabilities`, `hostContext` (theme, displayMode, containerDimensions, locale, timeZone, userAgent, platform, deviceCapabilities, safeAreaInsets, styles).
    * **Outer iframe** — `sandbox=` attribute, `allow=` attribute, and the response `Content-Security-Policy` header.
    * **Inner iframe** (if the host nests) — same three things.

    DevTools → Network → Response Headers + the inspector's own JSON-RPC logger ([Playground](/inspector/playground)) are the easiest capture surfaces. Save these somewhere referenceable — you'll cite them in code comments.
  </Step>

  <Step title="Add the host style">
    Open `client/src/lib/client-styles/built-ins.ts` and add a `HostStyleDefinition` for your host alongside `CLAUDE_HOST_STYLE`, `CHATGPT_HOST_STYLE`, etc.

    ```ts theme={"theme":"css-variables"}
    export const YOURHOST_HOST_STYLE: HostStyleDefinition = {
      id: "yourhost",
      mcp: {
        // Which Apps SDK protocol the host implements. UIType.MCP_APPS for
        // MCP Apps spec, UIType.OPENAI_SDK for the OpenAI Apps SDK surface.
        protocolOverride: UIType.MCP_APPS,
        // window.navigator-like fields exposed to widget HTML.
        platform: YOURHOST_PLATFORM,
        // Optional @font-face block injected into the widget iframe.
        fontCss: YOURHOST_FONT_CSS,
        // Which capabilities your host implements. Use MCP_APPS_FULL_SURFACE,
        // MCP_APPS_NO_CLAIMS_SURFACE, or a custom subset.
        mcpAppsCapabilities: MCP_APPS_FULL_SURFACE,
        resolveStyleVariables: getYourHostStyleVariables,
      },
      chatUi: {
        label: "YourHost",
        shortLabel: "YourHost-style host",
        pickerDescription: "YourHost scenario chrome",
        logoSrc: yourHostLogo,
        family: "yourhost",
        resolveChatBackground: (theme) => YOURHOST_CHAT_BACKGROUND[theme],
        loadingIndicator: YourHostMarkIndicator,
      },
    };
    ```

    Then register it in `BUILT_IN_HOST_STYLES` (same file, bottom):

    ```ts theme={"theme":"css-variables"}
    export const BUILT_IN_HOST_STYLES: readonly HostStyleDefinition[] = [
      MCPJAM_HOST_STYLE,
      CLAUDE_HOST_STYLE,
      CHATGPT_HOST_STYLE,
      CURSOR_HOST_STYLE,
      COPILOT_HOST_STYLE,
      CODEX_HOST_STYLE,
      YOURHOST_HOST_STYLE,
    ];
    ```
  </Step>

  <Step title="Drop in the logo">
    Add the host's logo to `client/public/` as `yourhost_logo.png` (or `.svg`). Existing presets use 256-512px square assets.

    Then register it in the inspector's host UI metadata so picker/compare surfaces can show the right icon. Logos are presentation-only; they do not live in the backend catalog template.
  </Step>

  <Step title="Add the backend catalog host">
    In `mcpjam-backend/convex/marketHostCatalog/templates.ts`, add the full `HostConfigInputV2` template keyed by host id. In `mcpjam-backend/convex/marketHostCatalog/seed.ts`, add the catalog metadata for that same id: label, provenance, MCP Apps rendering, protocol versions, verification date, and image support.

    The template should include every host field that can change — model, colors/styles, host context, sandbox, visibility, progressive tool discovery, image policy, and MCP Apps capability matrix. Fill in what you probed, delete what doesn't apply (see [Field guide](#field-guide) for what each one does):

    ```ts theme={"theme":"css-variables"}
    // templates.ts
    yourhost: {
      hostStyle: "yourhost",
      modelId: "anthropic/claude-haiku-4.5",
      systemPrompt: "",
      temperature: 0.7,
      requireToolApproval: false,
      respectToolVisibility: true,
      progressiveToolDiscovery: false,
      serverIds: [],
      optionalServerIds: [],
      builtInToolIds: [],
      connectionDefaults: { headers: {}, requestTimeout: 10000 },
      clientCapabilities: {
        extensions: {
          "io.modelcontextprotocol/ui": {
            mimeTypes: ["text/html;profile=mcp-app"],
          },
        },
      },
      hostContext: {
        theme: "dark",
        displayMode: "inline",
        availableDisplayModes: ["inline", "fullscreen"],
        containerDimensions: { width: 720, maxHeight: 5000 },
        locale: "en-US",
        timeZone: "America/Los_Angeles",
        userAgent: "yourhost",
        platform: "desktop",
        deviceCapabilities: { touch: false, hover: true },
        safeAreaInsets: { top: 0, right: 0, bottom: 0, left: 0 },
      },
      mcpProfile: {
        profileVersion: 1,
        initialize: {
          clientInfo: { name: "yourhost-mcp", version: "1.0.0" },
        },
        apps: {
          uiInitialize: {
            hostInfo: { name: "YourHost", version: "1.0.0" },
          },
          compatRuntime: { openaiApps: false },
          mcpAppsOverrides: {
            availableDisplayModes: ["inline", "fullscreen"],
            openLinks: true,
            serverTools: true,
            serverResources: true,
            logging: true,
            widgetDisplayModeRequests: "accept",
          },
          sandbox: {
            csp: { mode: "declared" },
            permissions: {
              mode: "custom",
              allow: { clipboardWrite: true },
            },
            sandboxAttrs: ["allow-forms"],
            allowFeatures: { fullscreen: "*" },
          },
        },
      },
    },
    ```
  </Step>

  <Step title="Regenerate the SDK fallback">
    After backend catalog/template changes, regenerate the SDK fallback snapshot from the backend seed:

    ```bash theme={"theme":"css-variables"}
    MCPJAM_BACKEND_DIR=/path/to/mcpjam-backend npm run generate:host-catalog-fallback -w @mcpjam/sdk
    ```

    The generated SDK catalog is fallback data only. Product UI and API template creation should fetch the live backend catalog first.
  </Step>

  <Step title="Test it">
    Run the inspector locally, create a new host using your preset, and try a widget under it:

    * Does the brand pill render with your logo?
    * Does `app.getHostContext()` (or `window.openai.theme`, etc.) report what you'd expect from the real host?
    * Does a widget that probes `hostInfo.name === "YourHost"` take that branch?
    * Are CSP violations reported in DevTools the same ones the real host would emit?

    The Playground's "Compare hosts" mode lets you render the same tool call under YourHost alongside Claude or ChatGPT side-by-side — useful for catching capability gaps.
  </Step>
</Steps>

## Field guide

A non-exhaustive reference for the fields most often gotten wrong:

| Field                              | What to capture                                                                                     | Common mistake                                                                                                          |
| ---------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `clientInfo.name`                  | Verbatim from the host's MCP `initialize` request.                                                  | Inventing a "friendly" name. Apps that branch on this string break under your preset.                                   |
| `hostInfo.name`                    | Verbatim from `ui/initialize`. **Different protocol layer from `clientInfo`.**                      | Reusing `clientInfo.name`. ChatGPT uses `openai-mcp` and `chatgpt` respectively.                                        |
| `mcpProfile.apps.mcpAppsOverrides` | Only what the host's MCP Apps surface actually supports.                                            | Advertising capabilities the renderer can't honor. If `listChanged` notifications aren't forwarded, omit the sub-field. |
| `cspDirectives`                    | Verbatim from the host's response CSP header.                                                       | Hand-curating a "reasonable" set. Real hosts ship surprising values (e.g. real ChatGPT emits only `frame-src`).         |
| `sandboxAttrs`                     | From the iframe `sandbox=` attribute, **minus** `allow-scripts` + `allow-same-origin` (baseline).   | Including the baseline. The schema layers your entries on top of the renderer's required-by-spec set.                   |
| `permissions.allow`                | The intersection of the outer iframe `allow=` and what `ui/initialize.hostCapabilities` advertises. | Granting permissions the host advertises but doesn't actually attach.                                                   |
| `containerDimensions`              | The host's intent (e.g. 720px chat column), not your inspector's iframe size.                       | Treating it as a literal viewport. Views interpret it as policy intent.                                                 |

## Gotchas

<Warning>
  **Almost never set `restrictTo` on `sandbox.csp`.** SEP-1865 makes
  `restrictTo` an *intersection* with the view's declared `_meta.ui.csp`, not a
  union. Adding the host's production allowlist here can only narrow widgets —
  never help them. A view declaring `connect-src https://api.example.com`
  silently goes to zero if your `restrictTo.connectDomains` doesn't list
  `api.example.com`. Use `mode: "declared"` and trust the view's CSP. The only
  legitimate use of `restrictTo` is to express a *deny* intent via empty arrays
  (see the Copilot preset's `frameDomains: []`).
</Warning>

<Warning>
  **`mcpProfile.apps.mcpAppsOverrides` is what the host advertises for MCP
  Apps.** Listing capabilities the renderer doesn't actually implement misleads
  widget authors — they'll gate on the advertised flag, hit a dead path at
  runtime, and blame their widget. If a capability's `listChanged` notifications
  aren't forwarded by the renderer yet, omit the sub-field. Several existing
  presets call this out explicitly in comments — match that discipline.
</Warning>

<Tip>
  When mirroring a host that doesn't render widgets at all (e.g. Codex CLI),
  replace `clientCapabilities` instead of spreading — a spread leaks the
  SDK-default MCP UI extension back in and misrepresents the host as UI-capable.
  See the Codex preset for the pattern.
</Tip>

## Open a PR

When the preset is working locally:

1. Add a screenshot of a widget rendered under your preset to the PR description.
2. Cite the sources for each non-default value (capture date for probes, doc URL for vendor specs).
3. If any field is a guess, mark it as such in a comment.

The maintainers will eyeball the values against the cited captures, run a couple of widgets under the preset, and merge. If your host evolves later — a new capability, a CSP change — open a follow-up PR with a fresh probe.

<Info>
  Stuck on a probe? Open a thread in the [MCPJam
  Discord](https://discord.gg/JEnDtz8X6z) — the team can usually help with
  capture techniques or sanity-check what a vendor's docs actually mean.
</Info>


# Launch from Code (Beta)
Source: https://docs.mcpjam.com/inspector/launch-from-code

Programmatically launch and configure MCPJam Inspector from your code or build tools

<Warning>
  **Beta Feature** - This API is in active development. We'd love your feedback!

  [Share feedback on Discord](https://discord.gg/JEnDtz8X6z) or [open a GitHub issue](https://github.com/MCPJam/inspector/issues)
</Warning>

Launch MCPJam Inspector programmatically from your code. This is useful for:

* **MCP server developers** - Auto-launch the inspector when running your dev server
* **Framework integrations** - Embed inspector into your Vite dev workflow

## CLI Flags

Configure the inspector via CLI flags when running `npx @mcpjam/inspector@latest`:

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest \
  --url http://localhost:3000/mcp \
  --name "My Server" \
  --tab playground
```

### Available Flags

| Flag                   | Description                           | Example                           |
| ---------------------- | ------------------------------------- | --------------------------------- |
| `--url <url>`          | HTTP transport URL to connect to      | `--url http://localhost:3000/mcp` |
| `--name <name>`        | Display name for the server           | `--name "My Server"`              |
| `--tab <tab>`          | Initial tab to navigate to            | `--tab playground`                |
| `--bearer <token>`     | Bearer token for Authorization header | `--bearer sk-xxx`                 |
| `--oauth`              | Trigger OAuth flow on connect         | `--oauth`                         |
| `--header <key=value>` | Custom header (repeatable)            | `-H "X-Custom=value"`             |

### Example: OAuth Server

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest \
  --url https://my-oauth-server.com/mcp \
  --name "OAuth Server" \
  --oauth \
  --tab servers
```

## Programmatic API

Use `launchInspector()` to spawn the inspector from any Node.js script:

```bash theme={"theme":"css-variables"}
npm install @mcpjam/inspector
```

```typescript theme={"theme":"css-variables"}
import { launchInspector } from "@mcpjam/inspector";

const inspector = await launchInspector({
  server: {
    name: "My MCP Server",
    url: "http://localhost:3000/mcp",
  },
  defaultTab: "tools",
});

console.log(`Inspector running at ${inspector.url}`);

// When done, stop the inspector
await inspector.stop();
```

### LaunchOptions

| Option            | Type                     | Default | Description                                                                     |
| ----------------- | ------------------------ | ------- | ------------------------------------------------------------------------------- |
| `server.name`     | `string`                 | -       | Display name for the server in the UI                                           |
| `server.url`      | `string`                 | -       | HTTP URL of the MCP server                                                      |
| `server.headers`  | `Record<string, string>` | -       | Custom headers to send with requests                                            |
| `server.useOAuth` | `boolean`                | `false` | Trigger OAuth flow on connect                                                   |
| `defaultTab`      | `string`                 | -       | Initial tab: `"servers"`, `"tools"`, `"resources"`, `"prompts"`, `"playground"` |

### InspectorInstance

The returned instance provides:

| Property  | Type                  | Description                         |
| --------- | --------------------- | ----------------------------------- |
| `url`     | `string`              | Full URL where inspector is running |
| `process` | `ChildProcess`        | Underlying Node.js child process    |
| `stop()`  | `() => Promise<void>` | Gracefully stop the inspector       |

### Example: Auto-launch with your MCP server

```typescript theme={"theme":"css-variables"}
// dev.ts - Run with: npx tsx dev.ts
import { launchInspector } from "@mcpjam/inspector";
import { startMyMCPServer } from "./server";

async function main() {
  // Start your MCP server
  const server = await startMyMCPServer({ port: 3000 });
  console.log("MCP server running on http://localhost:3000/mcp");

  // Launch inspector connected to it
  const inspector = await launchInspector({
    server: {
      name: "Dev Server",
      url: "http://localhost:3000/mcp",
    },
    defaultTab: "playground",
  });

  console.log(`Inspector: ${inspector.url}`);

  // Handle cleanup
  process.on("SIGINT", async () => {
    await inspector.stop();
    await server.close();
    process.exit(0);
  });
}

main();
```

## Vite Plugin

For Vite-based projects, use the plugin to auto-launch the inspector alongside your dev server:

```typescript theme={"theme":"css-variables"}
// vite.config.ts
import { defineConfig } from "vite";
import { mcpInspector } from "@mcpjam/inspector/vite";

export default defineConfig({
  plugins: [
    mcpInspector({
      server: {
        name: "My MCP Server",
        url: "http://localhost:3000/mcp",
      },
      defaultTab: "playground",
    }),
  ],
});
```

When you run `npm run dev`, the inspector automatically launches and connects to your server.

### Plugin Options

| Option       | Type      | Default        | Description                                     |
| ------------ | --------- | -------------- | ----------------------------------------------- |
| `server`     | `object`  | -              | Server configuration (same as programmatic API) |
| `defaultTab` | `string`  | `"playground"` | Initial tab to navigate to                      |
| `autoLaunch` | `boolean` | `true`         | Launch inspector when Vite starts               |

## Feedback Wanted

This is a beta feature and we're actively iterating on the API. We'd love to hear:

* What use cases are you building?
* What's missing from the API?
* Any bugs or rough edges?

<Card title="Share Feedback" icon="message-circle" href="https://discord.gg/JEnDtz8X6z">
  Join our Discord to share feedback and connect with other MCP developers
</Card>


# Playground
Source: https://docs.mcpjam.com/inspector/playground

Chat with your MCP servers, invoke tools by hand, render widgets, and inspect every step — all in one IDE-style workspace

The Playground is the main workspace for developing against your MCP servers. It combines chat, manual tool invocation, widget rendering, traces, and a live JSON-RPC logger into a single IDE-style surface so you can build, test, and debug an MCP server (or compare several side by side) without switching tabs.

<Frame>
  <img alt="MCPJam Playground" />
</Frame>

<Note>
  The Playground replaces the standalone **Chat** and **App Builder** tabs.
  Everything those tabs did is here, plus multi-host compare and a dockable
  Sessions rail.
</Note>

## Key Features

* **Chat with frontier models** — Talk to your server using Claude, GPT, Gemini, DeepSeek, and more at no cost, no API key required. Organizations can configure additional model providers in their organization settings.
* **Manual tool invocation** — Run any tool from the Tools rail with your own inputs to iterate on a widget or response without prompting a model.
* **Widget emulator** — Render widgets exposed via `openai/outputTemplate` (ChatGPT Apps) or `ui.resourceUri` (MCP Apps) the same way a real host would, with Inline, Picture-in-Picture, and Fullscreen display modes.
* **Chat / Trace / Raw views** — Switch between the conversation, a timestamped timeline of every step (with per-step latency and tokens), and the exact JSON payload sent to the model.
* **Multi-model compare** — Send a single prompt to up to 3 models at once and watch the responses side by side.
* **Multi-host compare** — Render the same tool call under multiple host configurations (ChatGPT vs Claude, different device sizes, different capability sets) at the same time. New sessions start with ChatGPT, Claude, and Cursor pre-selected so the comparison is immediate.
* **Display context controls** — Device viewport, light/dark theme, host style, locale, timezone, CSP mode, device capabilities, and safe area insets — all live in the toolbar.
* **Per-widget debugging** — Inspect raw tool input/output, `widgetState`, model context, and CSP violations from icons above each widget.
* **JSON-RPC logger** — Real-time, filterable feed of every message between the inspector and your servers.
* **Sessions rail** — Resume, archive, and search past conversations from the left rail.
* **Skills and prompts** — Type `/` to inject skills or MCP prompts directly into the chat.
* **Edit tool input/output live** — Click **Edit** on any tool-result card to tweak the input or output JSON and watch the widget re-render instantly; **Run** re-executes the tool with your edited input.
* **Voice input** — Click the mic button in the chat input to record your voice; the audio is transcribed and the text is inserted into the composer.

## Getting Started

1. **Connect a server** — Add and connect an MCP server from the [Clients](/inspector/clients) tab (or the **+** button in the chat input).
2. **Open the Playground** — Switch to the Playground tab.
3. **Send a message or run a tool** — Either chat with your server from the input bar and let the model call tools, or pick a tool from the left rail and execute it with your own input.

When a tool returns a widget, it renders immediately in the chat thread with a row of debug icons above it. When the model calls a tool, you see the call, the result, and any widget output inline.

## Layout

The Playground is a three-column IDE shell. Each side rail is collapsible — drag the divider or click the peek strip to hide or restore it.

**Left rail — Tools / Sessions**

* **Tools** (default) — Every tool exposed by your connected server(s). Pick one to fill in inputs and run it manually. When more than one server is selected, tools are aggregated across all of them and routed automatically on execution.
* **Sessions** — Browse, resume, rename, archive, and search past chat threads in the current project.

**Center — Chat surface**

* Chat composer, conversation thread with inline tool calls and widgets, and the host context toolbar.
* Top of the pane has a segmented control to switch between **Chat**, **Trace**, and **Raw** views.

**Right rail — Logs / Shell**

* **Logs** (always available) — Real-time feed of MCP protocol messages. Search, filter by source server, filter by log level, copy, download, or clear the buffer.
* **Shell** (shown when the previewed host has a Project Computer) — A live terminal on the computer the harness runs on. Click **Open terminal** to provision or wake the computer and connect. Switching between Logs and Shell keeps both the terminal WebSocket and the log stream alive so neither is dropped.

## Chat, Trace, and Raw Views

Switch between three perspectives on the same conversation from the segmented control at the top of the center pane:

* **Chat** — The normal thread with inline messages, tool calls, and rendered widgets.
* **Trace** — A timestamped timeline of every step (user turns, agent turns, tool calls) with per-step and total latency and token counts. Expand any step to inspect its raw input and output.
* **Raw** — The full JSON payload sent to the model (system prompt, tool definitions, conversation history). Available for both live sessions and previously-saved sessions you reopen.

<Frame>
  <img alt="Trace view showing a timestamped timeline with latency and token counts" />
</Frame>

## Tools Rail (Manual Invocation)

Open the **Tools** tab in the left rail to invoke any tool by hand:

* Select a tool to load its input form generated from the tool's JSON Schema.
* Fill in fields, toggle optional fields on or off, and click **Run** to execute.
* The result appears in the center pane just like a model-issued tool call, with the same row of debugging icons and widget rendering.
* Click **Refresh** to re-fetch the tool list from the server (useful after a server reload).
* Click **Save** to persist the current inputs as a saved request you can rerun, rename, duplicate, or delete from the same panel.

When multiple servers are selected at the same time, the Tools rail aggregates every server's tools into one list and routes each execution to the right server automatically.

## Model Picker

The model picker in the chat input lets you choose which LLM to use. Models are organized into two tabs:

* **Free models** — Frontier models (Claude, GPT, Gemini, DeepSeek, and more) available at no cost with no API key required.
* **Your providers** — Models from providers configured in your organization's settings on mcpjam.com.

The picker opens on the tab that matches your currently selected model. When you search, results from both tabs appear together.

### Out-of-credits state

When your MCPJam credit balance reaches zero, free models are grayed out in the picker with a tooltip explaining the limit. Models from your own providers remain enabled — switching to one is the fastest way to keep working. Paid top-up credits (if you have any) keep free models available even after your daily free allowance is exhausted.

If the out-of-credits dialog appears, clicking **Bring your own key** closes the dialog and opens the model picker directly on the **Your providers** tab so you can switch to an own-key model without leaving the chat.

## Multi-Model Compare

Send one prompt to up to 3 models simultaneously. Open the model picker, toggle **Multiple models** on, and select the models you want to compare. Each model's response — including tool calls and rendered widgets — appears in its own column.

<Note>
  The **Multiple models** toggle is hidden when the active session is shared
  with the project (project-visible). Enabling it on a shared session would
  mutate experiment state for every collaborator. Switch the session back to
  private to use multi-model compare.
</Note>

<Frame>
  <img alt="Playground running the same prompt across multiple models in split-screen" />
</Frame>

## Multi-Host Compare

Open the host picker (top-left of the center header) to render the same tool calls under different host configurations side by side — for example, ChatGPT vs Claude host style, or desktop vs mobile viewports.

New sessions start with **ChatGPT, Claude, and Cursor** pre-selected so you land directly in a 3-way compare. To change the lineup, open the host picker:

* **Click a row** to switch to that client immediately. The picker closes and any comparison lineup collapses to a single column.
* **Click the checkbox** on a row to add or remove that client from the comparison grid without changing the active client. Ticking a second row immediately opens a side-by-side view; unticking removes that column.

The comparison grid is capped at 3 clients. When the cap is reached, the checkboxes for remaining clients are disabled, but you can still click their rows to switch to them. Your selection is saved per project, so returning users keep whatever lineup they last configured.

One column is designated the **lead host**; chip tooltips in the toolbar say "Editing lead host: \<name>" so it's clear which column the host controls (theme, device, locale, etc.) currently bind to. Click another column's chip to promote it to lead.

This is the fastest way to verify that a widget renders correctly across the host environments you intend to ship to.

<Note>
  The host picker is hidden when the active session is shared with the project
  (project-visible), for the same reason as multi-model compare. Switch the
  session back to private to use multi-host compare.
</Note>

## Sessions

The **Sessions** tab in the left rail lists every saved conversation in the current project, grouped by recency. From here you can:

* Open a session to resume it in the center pane.
* Rename, archive, or delete individual sessions.
* Archive all sessions at once.
* Search across session titles.

Sessions are project-scoped, so teammates in the same project see (and can resume) shared sessions.

## Host Context Toolbar

The toolbar above the chat thread controls how the active host renders your widgets. Every change applies to the next tool call and to already-rendered widgets where applicable.

### Device Viewports

* **Desktop** (1280×800)
* **Tablet** (820×1180)
* **Mobile** (430×932)
* **Custom** — Any width and height from 100 to 2560 pixels

### Theme

Toggle light and dark mode to test both appearances.

### Host Style

Switch between Claude and ChatGPT host styles. For MCP Apps that use the MCP Apps style variables, the ChatGPT toggle translates the widget's styles to ChatGPT's design tokens.

Each host style ships with a distinct `hostCapabilities` preset. Switching from one preset to another changes the capabilities advertised in the `ui/initialize` handshake, so widgets that gate on optional fields (such as `serverResources` or `logging`) will behave differently under each style.

| Preset  | `HostContext.toolInfo` | `availableDisplayModes`           |
| ------- | ---------------------- | --------------------------------- |
| Claude  | Included               | `["inline", "fullscreen", "pip"]` |
| Copilot | Omitted (`undefined`)  | `["inline", "fullscreen"]`        |
| ChatGPT | Included               | `["inline", "fullscreen", "pip"]` |

**Copilot** follows Microsoft 365 Copilot's published bridge: `toolInfo` is not delivered and `pip` is not advertised, so a widget that calls `app.getHostContext()?.toolInfo` on the Copilot preset correctly sees `undefined`.

### Host Capabilities

The **Host Capabilities** button in the toolbar (next to Host Context) lets you override the `hostCapabilities` blob advertised to MCP App widgets in the `ui/initialize` response.

By default, the active host style's preset is used. Click **Host Capabilities** to open the editor and customize which capabilities the mock host claims to support. This is useful for testing how your widget behaves when specific capabilities are absent or present — for example, verifying that your widget degrades gracefully when `serverResources` is not advertised.

* **No override** — the active host style's preset is used (shown as the placeholder in the editor)
* **Custom override** — your JSON object is advertised verbatim; an "Override" badge appears on the button
* **Clear override** — resets back to the host style preset

<Note>
  Host Capabilities configures what the mock host *advertises*. Enforcement —
  rejecting widget requests for capabilities not in the blob — is a planned
  follow-up. Until then, the inspector will still service all widget requests
  regardless of what the handshake advertises.
</Note>

### Locale

Pick a BCP 47 locale (e.g. `en-US`, `es-ES`, `ja-JP`, `fr-FR`) to verify internationalization.

### Timezone

Choose from 19 IANA timezones plus `UTC` to test time-aware widgets.

<Note>The timezone control is only available for MCP Apps (SEP-1865).</Note>

### CSP Mode

* **Permissive** (default) — Allows all HTTP and HTTPS resources. Recommended for development.
* **Strict** — Only allows domains declared in your widget's CSP metadata (`openai/widgetCSP` or `_meta.ui.csp` for ChatGPT Apps, `ui.csp` for MCP Apps).

### Device Capabilities

* **Hover** — Enable or disable hover support to test mouse vs touch-only behavior.
* **Touch** — Toggle touch input to simulate mobile and tablet devices.

### Safe Area Insets

Simulate notches, rounded corners, and gesture areas. Pick a preset (None, iPhone with Notch, iPhone with Dynamic Island, Android gesture navigation) or set custom top/bottom/left/right values.

## Widget Controls

### Download confirmation

When an MCP App widget triggers a `ui/download-file` request, the inspector shows a confirmation dialog before the download runs. The dialog displays the filename, MIME type, and URI so you can make an informed decision. Click **Download** to allow it or **Cancel** to deny it — a denial is surfaced to the widget as a `{ isError: true }` result so the widget can handle it gracefully.

Only one confirmation prompt is shown at a time. If a second download request arrives while a prompt is open, it is automatically denied.

### Display Modes

All three display modes are supported for both ChatGPT Apps and MCP Apps:

* **Inline** (default) — Widget renders within the chat message flow.
* **Picture-in-Picture** — Widget floats at the top of the screen, staying visible while scrolling.
* **Fullscreen** — Widget expands to fill the entire viewport.

Widgets can request mode changes from their own code; users can exit PiP or Fullscreen via the close button.

### Widget rendering

All widgets — whether they declare `openai/outputTemplate` (ChatGPT Apps), `_meta.ui.resourceUri` (MCP Apps), or both — render through a single unified renderer. The `window.openai` compatibility shim is always injected, so both `window.openai.*` calls and `ui/*` JSON-RPC messages work regardless of which metadata the tool declares.

Widget rendering is gated on the active Client's `clientCapabilities`. Hosts that advertise the MCP UI extension (Claude, ChatGPT, MCPJam, Copilot, Cursor) render widgets as normal. Hosts that strip the extension — such as Codex, an elicitation-only CLI — show the plain tool result row instead. Per-server capability overrides are honored.

## Per-Widget Debugging

Every tool result in the chat thread has a row of icons in its header. Click an icon to open the matching debug panel below the widget.

### Sign-in links

When a computer-backed tool (such as Bash) runs a command that triggers a device-flow authentication prompt — for example, `gh auth login` — any sign-in URLs surfaced by the tool appear as clickable links directly in the chat thread under a **Sign-in link** heading. This saves you from hunting through scrollback for the URL. Only `https:` URLs are rendered as links; plain `http:` and non-web schemes are never shown.

### Data

Raw tool input, output, and error details for the call.

### Widget State (ChatGPT Apps)

The current `widgetState` object and when it was last updated. This is what your widget sets via `window.openai.setWidgetState` and what gets passed back to the model on the next turn.

### Model Context (MCP Apps)

The context your widget has sent back to the model via `ui/update-model-context`. Only appears when the widget has set model context.

### CSP

In Strict mode, a badge shows the number of blocked requests. The panel includes:

* **Suggested fix** — Copyable JSON snippet to add to your `openai/widgetCSP` or `ui.csp` field.
* **Blocked requests** — Each violation with the directive and source that was blocked.
* **Declared domains** — The connect, resource, frame, and base URI domains your widget currently declares.

#### Mounts

The debug panel includes a **Mounts** section that appears only when a widget iframe has been torn down and rebuilt more than once during a session. Each entry shows the mount index, the reason it fired, and a timestamp. A healthy widget has exactly one mount; multiple entries indicate an unexpected re-mount that may have wiped widget state.

### Edit input & output

Click **Edit** on a tool-result card to make the **Input** and **Result** JSON editable in place. Edits re-render the widget live with no reload; **Run** re-executes the tool with the edited input, and **Revert** restores the original. Edits are local to the session — nothing is persisted.

## Right Rail

The right rail has two tabs. The **Logs** tab is always present; the **Shell** tab appears when the previewed host has a Project Computer attached.

### Logs tab

Streams every message between the inspector and your connected server(s) in real time. Each entry shows the direction (client→server or server→client), the method name, and a timestamp. Click an entry to expand its full JSON payload.

Logged messages include:

* Tool invocation requests and responses
* `window.openai` API calls from your widget
* Widget state updates
* MCP App JSON-RPC messages (`tools/call`, `ui/initialize`, `ui/message`, etc.)

Controls in the logger header:

* **Search** the buffer
* **Filter** by source server
* **Log level** filter
* **Copy** the visible entries to your clipboard
* **Download** the buffer as a file
* **Clear** the buffer
* **Collapse** the rail

When a search is active and the source filter is hiding matching entries, a banner appears below the toolbar showing how many matches are hidden. Click **Clear filter** in the banner to reset the source filter to **All** and reveal those entries. The empty state reads **No matches in this view** (rather than **No logs yet**) whenever logs exist but are all filtered out.

### Shell tab

When the previewed host has a Project Computer, a **Shell** tab appears next to **Logs**. Click **Open terminal** to provision or wake the computer and open a live terminal session on the same machine the harness runs on. The computer status chip in the Shell header shows the current state (starting, ready, error, etc.).

Both tabs stay mounted while you switch between them, so the terminal WebSocket and the log stream are never dropped.

## Input Bar

Click the **+** button on the left of the chat input to manage servers, attach files, and tune model behavior.

**Servers**
A list of every server in your project, sorted by status. Each row shows the server name and transport type (`HTTP` or `stdio`, npx/desktop only). Connect, disconnect, or toggle a server on or off for the current conversation. **+ Add server** opens the add-server modal without leaving the Playground.

**Attach files**
Add files to the conversation in any of these ways:

* **File picker** — Click the **+** button and choose **Attach files** to browse and select files.
* **Paste** — Copy an image to your clipboard and paste it directly into the chat input (`Ctrl+V` / `Cmd+V`). Pasted images without a filename are saved automatically as `pasted-image-1.png`, `pasted-image-2.png`, and so on.
* **Drag and drop** — Drag one or more files from your desktop and drop them onto the chat input. A "Drop image or file to attach" overlay appears while you're dragging to confirm the drop target.

Supported file types include images, PDFs, text, JSON, CSV, XLSX, and other common formats. Invalid file types show an error message below the input.

**System Prompt & Temperature**
Edit the system prompt the model sees before your messages, and adjust the temperature slider (0 = deterministic, higher = more creative).

**Tool Approval**
When on, pauses the model before tool calls so you can approve or deny each one.

The switch **raises** the bar and never lowers it, so it is not the whole story in either direction.

Some actions pause whether it is on or off:

* local shell commands (`bash`);
* destructive UI actions.

Local Browser (`browser_*`) and WebMCP page tools follow the chat’s Tool Approval setting. With it off, calls run without per-call approval. Browser device permission remains a separate prerequisite.

Others never pause even when it is on: read-only lookups of your own project, the progressive-discovery meta-tools, read-only UI actions, and tools an MCP App provided. Everything in between follows the switch, including web search — its query is your text leaving for a third party, and the call spends credits.

Two things sit just outside those lists. An **unattended run** (an eval, a swarm) has nobody to ask, so its browser tools follow the `toolPolicy` it declared up front instead of pausing. And **loading a skill that an MCP server provided** is always tagged as third-party content and always bound to the manifest digest you were shown — but whether it pauses follows the switch.

**Voice input**
Click the mic button (right side of the input) to record your voice. The audio is transcribed using Whisper and the resulting text is appended to whatever you've already typed in the composer. Recording stops automatically after 3 minutes. The mic is disabled when your daily voice budget is exhausted; a warning appears in the composer when the remaining budget is low. While a recording is in progress, pressing Enter does not submit the draft — use the stop button (arrow-up) to finish recording first, or the ✕ button to cancel without transcribing.

**Context meter** (right side of the input)
Open it to inspect the token breakdown for the current conversation — input, output, system prompt, and MCP tools usage. Before you send the first message, the popover shows an **MCP Tools (estimated)** row with a pre-request estimate of how many tokens your connected servers' tool descriptions will consume. This estimate is UI-only and is not used in billing or actual session usage math. Once the model responds and the provider reports real input/output token counts, the estimate is replaced by the actual usage. If token counting is not supported for a particular server or model, the popover shows a **Token count unavailable** notice for that server instead.

## Skills

The Playground respects every skill you've configured in the [Skills](/inspector/skills) tab. A skill ends up in a conversation in one of two ways:

* **Inferred** — The model picks a skill based on your prompt, loads its instructions, and runs it against your MCP tools. Useful for observing skill behavior end-to-end.
* **Explicit** — Type `/` in the input and pick a skill from the **SKILLS** section of the picker. The skill content is front-loaded before your message.

## Prompts

Prompts are MCP prompt templates exposed by your servers via `prompts/list`. Type `/` in the input and pick one from the **PROMPTS** section. Selected prompts appear as expandable cards above the input — source server ID, description, arguments (required ones marked with `*`), and a preview of the first message — so you can review or remove them before sending.

## Workspace tools

When you are signed in as a project member, the Playground gives the model a set of built-in tools for interacting with the MCP servers saved in your project. These tools are not available to guests or in shared scenario sessions.

| Tool                         | What it does                                                                                                                                                                                                                                                                         |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `list_project_servers`       | List all servers saved in the project with their names, transport types, and enabled status.                                                                                                                                                                                         |
| `diagnose_server`            | Probe a server's URL, connect, initialize, and report its capabilities and any connection errors. Use this when a server won't connect or you want to check its health.                                                                                                              |
| `list_server_tools`          | List the tools a server exposes — names, descriptions, and input schemas. Paginated; pass `nextCursor` back as `cursor` for the next page.                                                                                                                                           |
| `call_server_tool`           | Execute a tool on a server and return its result. Runs with your own authorization and may have side effects. Get the tool name and parameter schema from `list_server_tools` first.                                                                                                 |
| `list_server_prompts`        | List the prompts a server exposes — names, descriptions, and arguments. Paginated.                                                                                                                                                                                                   |
| `get_server_prompt`          | Render a prompt from a server with the given arguments and return its messages.                                                                                                                                                                                                      |
| `list_server_resources`      | List the resources a server exposes — URIs, names, and MIME types. Paginated.                                                                                                                                                                                                        |
| `read_server_resource`       | Read one resource from a server by URI and return its contents.                                                                                                                                                                                                                      |
| `start_claude_readiness_run` | Grade a saved server against Anthropic's connector-directory rules. Returns a receipt with a run id — poll `get_readiness_run` for the verdict. Deterministic grading is free; `includeLlmObservations` adds an optional model pass that consumes MCPJam credits. Requires approval. |
| `start_openai_readiness_run` | Grade a saved server against OpenAI's app-directory rules. `submissionMode` is required and never inferred. Returns a receipt; poll `get_readiness_run` for the verdict. Requires approval.                                                                                          |
| `get_readiness_run`          | Read one readiness run: `status` (whether the run finished), `overallStatus` (the grade), and `llmObservations` (whether the optional model pass ran). A `completed` run can be `not-ready`.                                                                                         |
| `list_readiness_runs`        | List a project's readiness runs, newest first, optionally narrowed to one publisher or server.                                                                                                                                                                                       |
| `cancel_readiness_run`       | Stop a readiness run that is still going. The run's real terminal state arrives on a later `get_readiness_run`.                                                                                                                                                                      |
| `get_readiness_report`       | Read a finished run's findings, ordered most-consequential-first and capped. `truncated` and `totalFindings` indicate when you are seeing a subset.                                                                                                                                  |

Servers are identified by name or ID as saved in the project. Stdio servers are not supported for live operations — the model will receive a clear error rather than a connection failure.

When **Tool Approval** is on, the model pauses before `diagnose_server`, `call_server_tool`, `read_server_resource`, `start_claude_readiness_run`, and `start_openai_readiness_run` so you can approve or deny each one — these open a connection to one of your servers or spend credits. Read-only listing and polling tools never require approval, on any model. `cancel_readiness_run` is also direct — it stops traffic rather than starting it.

<Note>
  These tools are only available when you are signed in as a project member.
  They are not shown to guests or in scenario sessions.
</Note>

## Elicitation

When an MCP server needs additional input from you mid-tool-call, it can pause execution and ask a question. The Playground renders these requests as modal dialogs and resumes the tool call once you respond.

This applies to both model-initiated tool calls and tool calls triggered by an MCP App widget. When an App widget's tool call returns `input_required`, the same elicitation dialog appears overlaid above the widget — the App keeps rendering underneath — and only the final result is returned to the widget once all rounds are complete.

### Form elicitation

The server presents a form with fields defined by its schema. Fill in the fields and click **Accept** to submit your answer, or **Decline** to refuse. The tool call continues (or fails gracefully) based on your response.

### URL elicitation

The server asks you to visit a URL to complete an out-of-band step — for example, an OAuth authorization or a payment flow. The dialog shows:

* The **full destination URL** as plain text (never as a clickable link, so you must read it before opening it).
* The **host emphasized** so subdomain spoofing is visible at a glance.
* Warnings for unsafe conditions: unencrypted `http://` links, punycode hostnames that can mimic well-known domains, embedded credentials in the URL, and bare IP addresses.

Non-`http(s)` schemes (such as `javascript:` or `data:`) are blocked outright and cannot be opened.

Click **Open in new tab** to open the URL. If your browser blocks the pop-up, a notice appears with a **Copy link** button so you can open it manually. The dialog only records your consent once the tab actually opens — a blocked pop-up does not count as acceptance.

Click **Decline** to refuse, or close the dialog to cancel without an explicit choice.

### URL-required notices

If a tool call fails because the server requires you to complete an out-of-band interaction first (MCP error `-32042`), the Playground shows a URL-required notice. This is advisory — the tool call has already failed — so there is nothing to decline. Open the required URL(s), then retry the tool call or ask the model to try again.

When a server requires multiple interactions before a tool can run, the notice walks you through them one at a time and only dismisses once you have handled all of them.

## Temporary Browser override

When Browser is available in your deployment, a **Browser** selector appears above the Playground layout. It lets you enable or disable Browser for the current Playground session without changing the selected client's saved configuration.

The selector has three options:

* **Client default** — Uses whatever the selected client has configured. The label shows the current default in parentheses, for example `Client default (Off)`.
* **On for this Playground** — Enables Browser for this session, making the browser panel available even if the client has it off.
* **Off for this Playground** — Disables Browser for this session, hiding the browser panel even if the client has it on.

The override applies across chat, multi-host comparison columns, the workspace panel, and the fallback right rail. New chats started within the same Playground session keep the selection. The override resets when you change project, client, or environment mode, or when you leave and return to Playground.

Enabling Browser makes the browser panel available and lets you choose a location and grant permission inside that panel. Enabling the override does not grant browser permission on its own. Disabling the override does not revoke device permission or terminate a browser you are currently viewing.

When an environment is active, the selector is replaced by a notice that Browser is controlled by the environment's client and cannot be overridden.

A **Browser settings** link appears in the browser panel header when a client is selected. Clicking it opens the client's [Browser tab](/inspector/clients#browser-settings) where you can change the saved enablement, profile, and personal preferences. Changes made there do not affect the current Playground session's temporary override.

## Related Features

* [Clients](/inspector/clients) — Manage MCP server connections, transports, and OAuth.
* [Skills](/inspector/skills) — Configure skills that the Playground can use.
* [Projects](/inspector/projects) — Organize servers, sessions, and views into shared workspaces.

### Local Browser and WebMCP security

Local Browser and WebMCP inspection require **Allow Browser** on the current device. Shell permission is separate. Tool Approval keeps its existing chat setting: when off, the model can act without asking before each browser or page-tool call. Enabling page tools gives you and the chat shared control of the inspection page. Ordinary websites, local development servers and other local-network apps remain accessible; MCPJam’s own controller endpoints and local-file navigation are blocked.

Page text, screenshots and tool results used in chat may be sent to your selected model provider. Signed-in websites and local-network services can contain sensitive information. Exports may include page contents, tool inputs/results and screenshots; review them before sharing.

Electron inspection keeps website sign-ins in a separate profile for each account and project, including a separate standalone profile. The upgrade requires signing in to websites again: the old shared inspection profile is never imported. **Clear inspection site data** closes sessions for the selected profile and deletes its website storage/cache. **Delete old shared profile** removes the unused legacy profile. Neither action deletes previously saved transcripts, screenshots, exported files or cloud profile copies. Node inspection is temporary and does not retain sign-ins after closing.

Electron website downloads require your decision in a native save dialog. They are limited to 1 GiB and never opened automatically. Node website downloads are disabled. Camera, microphone, geolocation, device access and clipboard reads remain denied; explicit human paste and file selection remain available.

Local profiles are files on this machine protected by your OS account and filesystem permissions. Some browser secrets may use OS credential protection; this is not a promise that all profile data, screenshots or transcripts are encrypted. Uploading a Playground browser profile to a cloud computer is a separate action that copies its data outside this machine.


# Plugins
Source: https://docs.mcpjam.com/inspector/plugins

Import Agent Plugins 1.0 bundles as versioned, read-only projections of their skills and MCP servers

<Note>
  Plugins are available on accounts where the feature has been enabled. The **Add plugin** entry and the Plugins section only appear when the feature is active for your account. Importing a plugin requires **project admin** — it creates shared executable capabilities, so it carries the same bar as managing shared OAuth connections. Project members can view installed plugins; the feature flag only controls whether the surface appears, never who may import.
</Note>

A **plugin** is an [Agent Plugins 1.0](https://agent-plugins.org) bundle — a ZIP archive or folder — that MCPJam stores and projects as a versioned set of skills and MCP servers. Importing a plugin does not run any code from the bundle; MCPJam creates a read-only projection of the bundle's declared components and makes them available to your project.

## Bundle layout

Agent Plugins fixes where each component lives. MCPJam discovers components **only** from these locations:

```
my-plugin/
├── plugin.json          # required manifest, at the root
├── mcp.json             # optional MCP server configuration, at the root
├── skills/              # optional skills, one immediate child directory each
│   └── deploy-app/
│       ├── SKILL.md     # required per skill
│       └── scripts/     # supporting files, never executed at import
└── assets/              # optional icons, logos, screenshots
```

A `plugin.json` anywhere else in the bundle is treated as an ordinary file, an `mcp.json` outside the root is ignored with an `MCP_CONFIG_IGNORED` preview warning, and a `SKILL.md` deeper than one directory under `skills/` is not a skill. Directories the format does not define (`hooks/`, vendored folders, and so on) are preserved as plain files — they are not components and not errors.

### `plugin.json`

The manifest requires two fields:

* **`$schema`** — the canonical Agent Plugins schema identifier, for example `https://agent-plugins.org/schemas/1.0.0/plugin.schema.json`. MCPJam uses it to select its locally supported validation rules; the URL is **never fetched** during import. A missing or unsupported `$schema` rejects the bundle.
* **`name`** — the plugin's stable identity: 1–64 lowercase characters from `a-z`, `0-9`, `-`, and `.`. Dots are legal (`com.example.deploy` is a valid name), separators may not lead or trail, and the same separator may not repeat (`--` and `..` are invalid — mixed pairs like `foo.-bar` are allowed by the spec).

`version` is a free-form string — the format imposes no semver rule. The manifest is a **closed object**: unknown top-level fields are reported in the import preview and ignored — with one exception. A handful of *execution-ambiguous* names (`command`, `entrypoint`, `exec`, `install`, `install_script`, `postinstall`, `preinstall`, `run`, `script`, `scripts`) **reject the bundle outright** rather than being ignored, because silently dropping something that looks like an install or run hook is the wrong failure mode. Remove them from `plugin.json`; there is no preview to fix them from. The `extensions` field carries client-specific data under reverse-domain namespaces; MCPJam reads its own presentation metadata (display name, icon, logo) from the `com.mcpjam` namespace and preserves other namespaces without validating their *shape*. Two hygiene rules still apply to every namespace: secret-looking values are dropped rather than stored (reported as `MANIFEST_SECRET_FIELD_OMITTED`), and a namespace nested beyond the depth cap rejects the bundle with `VALUE_TOO_DEEP`. Round-tripping client-specific metadata is safe as long as it carries no credentials and stays reasonably flat.

```json theme={"theme":"css-variables"}
{
  "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
  "name": "com.example.deploy",
  "version": "2.1.0",
  "description": "Deploy tools and the workflows that drive them.",
  "extensions": {
    "com.mcpjam": { "display_name": "Deploy Tools", "icon": "assets/icon.png" }
  }
}
```

### `mcp.json`

MCP servers are declared in one root document requiring its own `$schema` and an `mcpServers` object. The Agent Plugins version in `mcp.json` must match the one in `plugin.json`.

Every server entry declares an explicit **`type`**, which is authoritative — MCPJam never infers the transport from the presence of `command` or `url`:

| `type`            | Transport           | Requires                                       |
| ----------------- | ------------------- | ---------------------------------------------- |
| `stdio`           | Local child process | `command` (plus optional `args`, `env`, `cwd`) |
| `streamable-http` | Streamable HTTP     | `url` (plus optional `headers`)                |
| `sse`             | Legacy HTTP+SSE     | `url` (plus optional `headers`)                |

```json theme={"theme":"css-variables"}
{
  "$schema": "https://agent-plugins.org/schemas/1.0.0/mcp.schema.json",
  "mcpServers": {
    "deploy-local": {
      "type": "stdio",
      "command": "./bin/server",
      "args": ["--data", "${PLUGIN_DATA}/state"],
      "env": { "MODE": "production", "API_KEY": "${API_KEY}" }
    },
    "deploy-remote": {
      "type": "streamable-http",
      "url": "https://mcp.example.com/mcp"
    }
  }
}
```

Remote URLs must use HTTPS; plain-HTTP loopback URLs (`http://localhost:…`) are accepted with a warning for local development.

### Plugin variables

Two placeholders are defined by the format, both resolved by MCPJam at launch time — never at import time:

| Placeholder      | Resolves to                                                           |
| ---------------- | --------------------------------------------------------------------- |
| `${PLUGIN_ROOT}` | The verified, content-addressed root of the materialized bundle       |
| `${PLUGIN_DATA}` | A writable per-plugin data directory, preserved across plugin updates |

Placeholders expand **only in `args`, `env` values, and `cwd`** — a placeholder in `command` invalidates that server entry. Commands are single executable tokens: a `./`-prefixed command resolves against the plugin root with containment enforced, so `./bin/server` points at the file shipped in the bundle and `./bin/../../outside` invalidates the entry. Materialization does not carry archive permission bits, so on macOS and Linux a shipped binary is written without its executable bit and will not spawn — prefer an interpreter command (`node`, `python`) with the script passed in `args`. `cwd` must start with `./`, `${PLUGIN_ROOT}`, or `${PLUGIN_DATA}`. A bundle's `env` may not define `PLUGIN_ROOT` or `PLUGIN_DATA` — those variables are client-controlled.

### Failure isolation

Problems are contained at the narrowest boundary the format defines:

| Problem                                                                                            | Effect                                                       |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| Archive violation (path traversal, symlink entry, size limits) or fatal manifest violation         | The whole import is rejected                                 |
| Invalid `mcp.json` document (bad JSON, wrong `$schema`, unknown top-level field, version mismatch) | MCP servers are disabled for the bundle; skills still import |
| One invalid server entry (unknown `type`, unknown field, placeholder in `command`, escaping path)  | Only that entry is **skipped**; sibling servers import       |
| One non-conforming skill                                                                           | Only that skill is skipped; sibling skills import            |

A **skip** in the import preview means exactly this: the component was declared but did not conform, so it is absent from the imported revision. Each skip appears in the preview's warnings as a `COMPONENT_SKIPPED` entry naming the component and the reason. A skipped server is a bundle problem to fix and re-import — not a runtime failure.

### Credential values are never imported

MCPJam screens every literal `env` and `headers` value in the bundle:

* **Non-secret literals are stored** — `{"MODE": "production"}` or an `X-Api-Version` header imports as-is, so portable plugins run without a setup step.
* **Secret-looking literals are never stored** — anything resembling a token, key, PEM block, or URL-embedded password is dropped with a warning and becomes a **required setup item**: you supply the value per server after installing.
* **`${VAR}` references** always become setup items — the bundle names the variable, you provide the value.

The setup section of the install summary lists these names. Values are always provided by you, per server; MCPJam never reads or stores credential values from a bundle.

## Where plugin servers run

* **`stdio` components run on the local MCPJam runtime today.** At connect time the bundle is materialized into a local content-addressed cache, `${PLUGIN_ROOT}`/`${PLUGIN_DATA}` are substituted, and the process spawns on your machine. Hosted VM execution for plugin stdio components is coming; the plugin card shows a **Local runtime** badge in the meantime.
* **`streamable-http` and `sse` components** are remote connections and work from any client.

## Importing a plugin

Plugin import needs a cloud project — signed-in or guest. Fully local, unsynced mode shows an explanation instead of the import picker.

1. Open the **Connect** tab (Servers).
2. Click the **⋯** actions menu in the top-right of the server list.
3. Click **Add plugin**.
4. In the dialog, choose your source:
   * **Choose a ZIP** — select a `.zip` plugin bundle up to 25 MB compressed.
   * **Choose a folder** — select the plugin's root folder; MCPJam zips it locally before uploading.
5. Review any preflight warnings shown before the upload starts.
6. Click **Continue** to upload and inspect the bundle.
7. Review the import preview (see [Preview](#preview) below).
8. Choose how to install:
   * **Install only** — stores the revision without making it the active one.
   * **Install and connect** — stores the revision and makes it the active one, so new attachments offer it.

### Preview

Before you confirm the install, MCPJam shows a preview of what the bundle declares:

| Section                | What it shows                                                                                                                                                                                          |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Identity               | Plugin name, version, and a short content-addressed revision ID                                                                                                                                        |
| Census                 | Counts of skills, MCP servers, app mappings, and assets                                                                                                                                                |
| Warnings               | Non-fatal issues, including every skipped component with its reason                                                                                                                                    |
| Unsupported components | One line per component MCPJam stores but does not run — lifecycle hooks, browser extensions, scheduled-task templates, and unrecognized manifest fields — each with its key and the reason it is inert |
| Setup required         | Environment variable and header names the bundle declares — you supply values per server after installing, not here                                                                                    |
| Skills                 | Declared skill model references and descriptions                                                                                                                                                       |
| MCP servers            | Server keys, transports, and any OAuth or credential requirements                                                                                                                                      |
| App mappings           | Metadata entries preserved from `.app.json` files                                                                                                                                                      |
| Assets                 | Icons and other static files included in the bundle                                                                                                                                                    |

### Content-addressed re-import

If you import a bundle whose bytes are identical to one already stored, MCPJam reuses the existing revision instead of creating a new one. The success screen says **Already imported** rather than **Plugin installed** to make this clear.

### Retrying a failed import

The dialog stays open across close and reopen so an in-progress import is not lost. If an import fails:

* **Commit failed** (the bundle was already inspected and previewed) — click **Retry install** to recommit the same import without re-uploading.
* **Inspection failed** (the bundle was uploaded but not yet inspected) — click **Resume inspection** to re-drive the inspect step.
* **Terminal failure** — click **Start a new import** to upload the bundle again.

## Plugins section

Once at least one plugin is installed, a **Plugins** section appears above the server grid on the Connect tab. Each plugin is shown as a group card.

### Plugin group card

The card shows the plugin's name and a rolled-up health badge:

| Badge              | Meaning                                                                  |
| ------------------ | ------------------------------------------------------------------------ |
| Ready              | All components of the active revision are ready to use                   |
| Needs auth         | At least one component requires authorization before its first tool call |
| Local runtime      | At least one component runs only from a local MCPJam process             |
| Computer required  | At least one component runs only inside a Project Computer               |
| No active revision | The plugin is installed but no revision has been made active             |
| Disabled           | The plugin has been disabled                                             |
| Checking…          | Setup status is still loading                                            |

Click the card header to expand it and see the active revision's components:

* **Revision ID** — the short content-addressed identity of the active bundle.
* **MCP servers** — each server's key, transport, placement, and readiness.
* **Skills** — model references for each declared skill, namespaced as `<plugin-name>/<skill-name>`.
* **Unsupported components** — a count of the same inert components the import preview itemizes. Only the count appears here; the per-component keys and reasons are shown at import time.
* **Other revisions** — ready revisions of the same plugin that are not currently active. Click **Activate** next to a revision to make it the active one.

### Plugin actions

Click the **⋮** menu on a plugin card to:

* **Disable / Enable** — temporarily disable a plugin without uninstalling it.
* **Uninstall** — soft-uninstall the plugin. Revisions already used by runs and sessions are preserved so existing history stays reproducible. Uninstall is blocked while a live environment still pins one of the plugin's versions.

## Credential setup

After installing a plugin, each MCP server component that requires credentials shows its readiness in the plugin card. The setup section of the install summary lists the environment variable and header names the bundle declared. You configure the actual values per server — MCPJam never reads or stores credential values from the bundle itself.

## Plugin provenance in runs and sessions

When an eval run or swarm simulation executes against an environment that pins a plugin, MCPJam records the exact bundle hash alongside the run. This record is immutable: re-importing the plugin after a run completes does not change what that run reports.

* **Eval run detail** — a **Plugins** row below the environment label shows each pinned plugin's name and short bundle hash. See [Plugin provenance on runs](/inspector/evals#plugin-provenance-on-runs).
* **Swarm session pane** — the transcript pane for a completed simulation shows the same plugin chips so you can tell which bundle produced that session's transcript.


# Projects
Source: https://docs.mcpjam.com/inspector/projects

Group your MCP servers into projects, with the ability to share them with your team

Each project saves its own set of MCP servers. Switch between projects with one click, connect multiple servers in each, and share any project with teammates so everyone works with the same configuration.

## Key Features

* **Multiple Servers** - Connect to several MCP servers within a single project
* **Save Your Setup** - Store all your server configurations in one place
* **Share Instantly** - Invite team members via email to access the same project
* **Real-Time Sync** - Changes update automatically for all project members
* **Switch Contexts** - Jump between different project setups with one click

## Getting Started

The combined org and project switcher sits at the top of the sidebar. The trigger shows your active **organization** in bold on the first line and the active **project** beneath it.

1. **Click the switcher** at the top of the sidebar to open the context menu
2. **Create a project** - Click **Create project** at the bottom of the list, give it a name, and choose which organization it belongs to
3. **Add your servers** - Configure your MCP servers in the Connect tab
4. **Enable auto-connect** - Use the **Auto-connect** toggle in the Connect tab header to automatically connect all project servers whenever a host opens
5. **Share with your team** - Click the share button and invite collaborators

<Info>
  When you switch projects, all servers disconnect automatically to give you a clean slate.
</Info>

## Switching Organizations

Opening the switcher lands on the **projects view** for your active organization. The organization name appears as a header row at the top of the list.

* **Drill into the org list** — click the organization header row to replace the projects view with a list of all your organizations. Click any row to switch to that organization; the URL updates to a project in the new org and the sidebar reloads with its projects.
* **Back** — click the **Organizations** back header to return to the projects view without switching.
* **New organization** — appears at the bottom of the org list when you can create another.
* Organization settings are accessed through **Settings › Organization**, not from the switcher.

If you were invited to a paid-seat organization but the seat hasn't been paid for yet, that organization appears greyed out in the list with a **Seat not paid yet** tooltip. You cannot open it until the org owner completes payment for your seat.

<Info>
  After completing an OAuth flow, you are returned to the organization you were in when the flow started, so you never land on the wrong organization after authenticating.
</Info>

## Creating and deleting projects

### Creating a project

Click **Create project** at the bottom of the switcher's projects view. A dialog opens with:

* **Name** — prefilled with a free name (e.g. "Project 3"). Edit it before confirming.
* **Organization** — a dropdown to choose which organization the project belongs to. Organizations where you don't have permission to create projects, or where your seat is still pending, are not shown.

Click **Create** (or press Enter) to create the project and switch to it automatically.

### Deleting a project

Hover over any project row in the switcher to reveal the **trash icon**. Clicking it opens a confirmation dialog showing the project name. Click **Delete** to confirm — this removes the project and all its servers permanently.

<Warning>
  Deleting a project is irreversible. All servers, sessions, and eval suites in the project are removed.
</Warning>

## Credits

A **See credits** row in the sidebar footer shows your organization's credit usage. Clicking it opens the organization billing page where you can see the full breakdown, your plan, and reset timing. The row is only visible when you are signed in and belong to an organization.

<iframe title="Projects demo" />

## Leaving an organization

If you are a member (not an owner or admin) of an organization, you will see an **Access restricted** screen when you open the organization settings. From that screen you can leave the organization:

1. Click **Leave organization** below the "Go to Servers" button.
2. Confirm in the dialog that appears.

You will lose access to the organization immediately and be redirected to the Servers page. To rejoin, you need to be re-invited by an admin or owner.

<Note>
  Owners cannot leave an organization — transfer ownership or delete the organization instead.
</Note>

## Sharing Projects

Project sharing lets your entire team work with the same servers. Set up your project once, invite your team by email, and everyone gets access to the same servers with the same configurations instantly.

### What Gets Shared

Members get access to every server in the project with the same configuration. Project admins can also configure which servers auto-connect and set per-server overrides (headers, timeout) from the Servers tab — these settings apply across every host in the project.

<Note>
  For security, authorization headers and OAuth tokens are never shared. Each team member authenticates with their own credentials when connecting to servers that require authentication.
</Note>

## Secret storage for env vars and headers

When you save a server that has **environment variables** (stdio) or **custom HTTP headers**, those values are stored as encrypted secrets. The next time you open the server's edit form, each row shows its name with the value masked behind a fixed placeholder.

### Viewing a saved value

Click the **eye icon** on a row to reveal that row's value. The eye toggles the value back to masked when you click it again. Only the row you click is uncovered — other rows stay masked.

When you first expand the Environment Variables or Connection overrides section on a saved server, the inspector fetches the variable and header names so each row can show which keys are configured. No values are fetched at that point; a value only travels to the browser when you click its eye.

<Warning>
  If you want to change the **authentication type** (for example, switching from Bearer Token to a different method) on an HTTP server that has hidden headers, you must reveal the relevant header rows first. Saving an auth change without revealing will show a validation error to prevent accidentally overwriting hidden headers.
</Warning>

### How updates work

* **Env vars**: Any edit to the env vars section (add, remove, or change a value) replaces the stored secret with the new set of values on save.
* **Custom headers**: Any edit to headers or the authentication type replaces the stored headers secret on save.
* If you save without revealing or editing a secret section, the existing stored secret is left unchanged.

### Changing a server's URL

Editing the path portion of a server's URL (for example, `/mcp` to `/mcp/v2`) keeps all saved credentials intact.

Changing the **origin** — the scheme, host, or port — clears every credential saved against that server: request headers, environment variables, the bearer token, and any OAuth access token, refresh token, and client secret. The edit form warns you before this happens and lists exactly what will be removed. You must check the acknowledgement box before Save becomes available.

This affects credentials that other project members may have added and that you may not be able to see.

After a cross-origin URL change, each member who had credentials saved for that server needs to re-enter them.

### URL history

The server detail panel includes a **URL history** section that shows every time the server's URL was changed, who made the change, and whether saved credentials were cleared as a result. This is visible to all project members regardless of plan, so anyone whose credentials were removed can see what happened and who made the edit.

### Real-Time Sync

Changes sync instantly between all project members. New servers and configuration changes update automatically for everyone.

### How to Share a Project

1. **Sign in** - Sharing requires authentication
2. **Click the member avatars** in the header (or the share button)
3. **Enter an email address** of the person you want to invite
4. **Click "Invite"** to send the invitation

Once they sign in with that email address, the shared project will appear in their project list.

### Managing Members

If you created the project, you can remove any member by clicking the X next to their name.

If you're not the creator, you can leave the project at any time. The creator can also remove you from the project.

## Project Settings

Open **Project Settings** (gear icon in the header) to manage project-level configuration.

### Default host config

The **Default Host Config** section sets the seed configuration applied to new scenarios, eval suites, and direct chat tabs when they are created. It controls the default model, system prompt, temperature, tool approval, connection settings, capabilities, and host context for anything new in the project.

Editing the default host config does **not** change existing scenarios or eval suites — each owns its own config once created.

The editor is only visible when you are signed in and have a saved project. Members without manage permissions see the editor in read-only mode.

#### Personal computer

When your deployment exposes computer-backed tools (such as **Bash**), a **Personal computer** toggle appears in the built-in tools section of the host config editor. Turning it on attaches a per-member cloud workstation — a persistent Linux sandbox — to the host.

* **Computer-backed tools are disabled until a computer is attached.** The checkbox for tools like Bash is greyed out with a hint ("Requires a personal computer") until you turn the toggle on.
* **Detaching the computer also removes computer-backed tool ids** from the config, so the saved config is always consistent.
* The toggle is hidden until the deployment enables at least one computer-backed tool, so it won't appear as a dead control before launch.

## What's Next

* **[Servers](/inspector/connecting-servers)** - Configure MCP servers to add to your projects
* **[Playground](/inspector/playground)** - Chat with the servers in your project across frontier models


# MCP protocol versions
Source: https://docs.mcpjam.com/inspector/protocol-versions

Pin the inspector to a specific MCP protocol version — including the 2026-07-28 stateless RC — to test how your server behaves across versions.

The inspector can speak more than one version of the MCP protocol. By default (**Automatic**), unpinned connections probe `server/discover` for 2026 support and fall back to the legacy `initialize` negotiation — on both HTTP and stdio. You can also pin a specific version when you want to test an exact revision — including the **2026-07-28 stateless RC** or any earlier stable release.

<Warning>
  **Migration required if you used the old Draft version.** The previous `DRAFT-2026-v1` placeholder has been retired and replaced with the upstream RC literal `2026-07-28`. If you had any servers pinned to `DRAFT-2026-v1`, you must re-select the protocol version in the inspector UI. Stored `DRAFT-2026-v1` pins are rejected by spec-conforming servers with a `-32004 UnsupportedProtocolVersionError`.
</Warning>

## Where the setting lives

There are three places to set a version, and they layer:

1. **Client → MCP Protocol** — the *host default* applied to every server attached to that Client.
2. **Add Server modal → Connection overrides → Protocol version** — a *per-server override* set at add time (available when adding a server to a shared project).
3. **Server card → Edit → Advanced settings → Protocol version** — a *per-server override* that wins over the host default.

Use the host default when you want every server in a Client to speak the same version. Use per-server overrides when you're testing a mix — for example, a server pinned to 2025-11-25 alongside one you're upgrading to Latest.

## Available versions

### Host default (client-level) dropdown

| Option                               | Stored value | Transport                                             | Use it when                                                                                                                                              |
| ------------------------------------ | ------------ | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Automatic**                        | `"auto"`     | HTTP + stdio: `server/discover`, then legacy fallback | Detect 2026 support from the server; otherwise use the version negotiated by `initialize`. This is the default.                                          |
| **Latest (2026-07-28)**              | `2026-07-28` | HTTP in MCPJam's preview (Streamable HTTP POST)       | You want to test your server against the newest stateless revision — no `initialize` handshake, `server/discover` for capabilities, per-request `_meta`. |
| **2025-11-25**                       | `2025-11-25` | HTTP, STDIO, SSE                                      | Pin to the 2025-11-25 stateful release. The initialize handshake offers only this version.                                                               |
| **2025-06-18**                       | `2025-06-18` | HTTP, STDIO, SSE                                      | Test against the June 2025 stable revision.                                                                                                              |
| **2025-03-26**                       | `2025-03-26` | HTTP, STDIO, SSE                                      | Test against the March 2025 stable revision.                                                                                                             |
| **Host default** *(per-server only)* | —            | —                                                     | Inherit whatever the Client's MCP Protocol tab is set to.                                                                                                |

**Automatic** is stored as the selection policy `"auto"` — it is not a wire literal and is never sent to the server. At connect time the SDK negotiates the era with the server rather than announcing a specific version up front. Picking any other option stores that exact version literal and pins the connection to that single revision.

When Automatic falls back to the legacy `initialize` handshake, it respects the client's **Supported protocol versions** list (if set). The SDK negotiates only within that list and rejects a server that counter-offers a version outside it. If no list is set, the SDK's built-in defaults apply. Choose an explicit version pin when you want strict, list-independent control.

**Latest** is derived, not hardcoded: it always labels the newest version in the SDK's known-version list. Every other known revision is shown as a bare date.

The 2026 RC applies the stateless model across transports. MCPJam's initial preview client is narrower: it currently supports the RC over Streamable HTTP POST. The per-server dropdown hides the RC option for STDIO and legacy SSE servers; if a host-level RC default reaches a non-HTTP server, the inspector fails the connection with a clear transport error instead of silently attempting the wrong protocol.

## Setting the host default

<Frame>
  <img alt="Client editor with the MCP Protocol tab open and the Protocol version dropdown showing Automatic, Latest (2026-07-28), 2025-11-25, 2025-06-18, and 2025-03-26 options" />
</Frame>

1. Open the **Clients** tab.
2. Pick the Client you want to edit (or create a new one).
3. Open the **MCP Protocol** tab.
4. In the **Protocol version** dropdown, pick a version:
   * **Automatic** — stored as `"auto"`; the SDK negotiates the era at connect time (default).
   * **Latest (2026-07-28)** — pin to the newest stateless revision.
   * **2025-11-25** — pin to the 2025-11-25 stateful release.
   * **2025-06-18** or **2025-03-26** — pin to an earlier stable revision.
5. Save.

Every server attached to this Client now connects with that version unless it has its own per-server override. A helper note appears below the dropdown whenever a version is pinned, describing what the pin stores.

## Overriding a single server

You can set the per-server protocol version either when adding a server or when editing an existing one.

### At add time (shared projects)

When adding a server to a shared project, the **Connection overrides** section of the Add Server modal includes a **Protocol version** picker. The 2026 RC option is only shown for HTTP servers.

1. Click **Add server** in the **Servers** tab.
2. Fill in the server details.
3. Expand **Connection overrides**.
4. Pick **Client default**, **Latest (2026-07-28)**, **2025-11-25**, or an earlier stable revision from the **Protocol version** dropdown.
5. Submit the form.

The pin is applied once the server is created and the connection is established.

### After adding (edit flow)

1. Go to the **Servers** tab.
2. Click the three dots on the server card → **Edit** (or open **View server info** → **Edit**).
3. Expand **Advanced settings**.
4. Find **Protocol version** and pick **Host default**, **Latest (2026-07-28)**, **2025-11-25**, or an earlier stable revision.
5. Save and reconnect the server.

The per-server override is the more specific signal — it wins over whatever the Client's MCP Protocol tab says.

## OAuth protocol alignment with MCP negotiation

When you add or edit an HTTP server with **OAuth 2.0** authentication, the **Authentication → Advanced → Protocol** dropdown lets you choose which OAuth spec era the connect flow uses. The default is **Auto**.

**Auto** is stored as intent; it is not converted into a permanent concrete pin when you save the server. Each OAuth flow resolves and records one concrete version in this order:

1. An explicit OAuth protocol selection.
2. An explicit per-server or host MCP wire pin.
3. The version freshly detected or negotiated with the MCP server.
4. `2025-11-25` when a `401` requires OAuth before the server can provide protocol evidence.

The last case is an authentication-gated compatibility fallback, not version detection. If you know an auth-gated server requires the 2026 draft behavior, select `2026-07-28` explicitly.

OAuth callback security follows the concrete version recorded for that flow. Every version validates `state`. The 2026-07-28 flow also validates a returned RFC 9207 `iss` against the discovered authorization-server issuer; 2025 flows retain compatibility and ignore callback `iss`.

## What changes when you pick Latest (2026-07-28)

If your server already speaks `2026-07-28`, you mostly won't notice. A few things to know when you're testing:

* **No `initialize` handshake.** The inspector connects, immediately fires `server/discover`, and uses the result to populate the server card's name, version, capabilities, and instructions.
* **Per-request metadata.** Every request the inspector sends carries `MCP-Protocol-Version: 2026-07-28` as an HTTP header and the same value inside `params._meta["io.modelcontextprotocol/protocolVersion"]`. `clientInfo` and `clientCapabilities` ride along on every request too.
* **No session IDs.** The inspector never sends an `mcp-session-id`. If your server returns one, the inspector discards it and surfaces a warning — your server isn't conforming to the stateless RC.
* **Cancellation is closing the stream.** For SSE responses, the inspector closes the response stream to cancel; your server should treat that as a cancel signal.

### What the inspector tells you

* If your server doesn't speak `2026-07-28`, the connection fails with a message naming the server and the pinned version — for example, `MCP server "my-server" doesn't support MCP protocol version 2026-07-28, which this client is pinned to.` The error banner and any connect/reconnect toast include a **Change protocol version** button that opens the client's **MCP Protocol** tab directly. Set the version to **Automatic** there to let the inspector negotiate whatever the server supports, or pick a version the server advertises.
* If your server is missing a capability the inspector needs for a request, you'll get `-32003 MissingRequiredClientCapability` back from your server — surface those in the Activity log to confirm they reach you.
* The **Activity** log (server card → **Activity**) is the source of truth — every request and response, success or error, lands there so you can verify headers and `_meta` content.

### Not yet supported in the RC client

The 2026 RC client is a **preview**. A handful of pieces from the SEP family aren't wired up yet — if you try them, the inspector throws a labeled error instead of silently no-op'ing:

* `subscriptions/listen` (long-lived notification stream)
* Server-initiated requests via MRTR / `InputRequiredResult` for sampling and listRoots embedded in responses (elicitation via `input_required` is supported — see [Multi-round input](#multi-round-input-mcp-2026-07-28-input_required))
* Resumption tokens
* Automatic detection (HTTP and stdio) falls back to the legacy `initialize` handshake when `server/discover` is not available. An explicit 2026 pin remains exact and does not fall back.

For everything else — `tools/list`, `tools/call`, `resources/*`, `prompts/*`, OAuth refresh, custom headers, progress notifications — the RC client behaves like the legacy one.

## Recommended testing flow

1. Build a Client called something like `Latest protocol sandbox` with **MCP Protocol → Latest** as the host default.
2. Attach the HTTP server you're upgrading.
3. Connect — confirm the server card shows the server info populated from `server/discover` (name, version, capabilities, instructions).
4. Run a `tools/list` and a `tools/call` from the **Tools** tab.
5. Watch the **Activity** log for the request `_meta` and the `MCP-Protocol-Version` header.
6. Try an unsupported version on a second test server to confirm your `-32004` error envelope looks right.
7. Once your server is happy on Latest, keep one Client on **Automatic** (or pinned to **2025-11-25**) so you can flip between versions without rewriting settings.

If you need to talk to a mix of servers on different versions in the same Client, use per-server overrides — the RC server stays pinned to `2026-07-28`, the rest fall back to the host default.

## Recommended reading

Background on the 2026-07-28 RC and the stateless direction the protocol is moving in:

* [MCP Is Growing Up](https://aaif.io/blog/mcp-is-growing-up/) — AAIF's overview of what the 2026-07-28 RC means for teams building agentic systems.
* [2026-07-28 release candidate announcement](https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/) — the official blog post walking through what the RC changes and why.
* [MCP specification (draft)](https://modelcontextprotocol.io/specification/draft) — the in-progress spec text the inspector's RC client targets.


# Skills
Source: https://docs.mcpjam.com/inspector/skills

Load and use skills to give your agents the context they need to use MCP tools effectively

<Note>
  The Skills tab is always accessible. **Cloud (hosted) skills** — the project store, upload button, and skill tree — are available on accounts where the feature has been enabled. **Skills over MCP** (server-provided skills from connected servers) require no account feature and are always shown. Local skills, read from your filesystem, also need no account feature.
</Note>

[Skills](https://agentskills.io/what-are-skills) are an open format that provide instructions on how to use MCP tools and complete workflows. MCPJam lets you discover skills from your filesystem (local) or your project (hosted), upload new ones, and use them in the Playground — either automatically based on the user's prompt or deterministically with the `/` command.

## Getting Started

1. **Open the Connect view** in MCPJam Inspector and select the **Skills** tab (Servers | Client | Computer | **Skills**)
2. **Load skills** — MCPJam automatically discovers skills from [supported directories](#skill-directories). You can also upload skills directly through the Skills tab, or run `npx skills` to install them
3. **Use skills in the Playground** — the LLM discovers them automatically, or you can inject them with the `/` command

<iframe title="Skills demo" />

## Skill Directories

MCPJam scans the following directories for skills. Each subdirectory is expected to contain a `SKILL.md` file. When duplicate skill names are found, the first match in search order wins.

**Global:**

```
~/.claude/skills/
~/.mcpjam/skills/
~/.agents/skills/
```

**Project-local:**

```
./.claude/skills/
./.mcpjam/skills/
./.agents/skills/
```

<Tip>
  If you have no skills installed, run `npx skills` to get started quickly.
</Tip>

## Finding skill IDs

Several CLI flags take a skill ID: `eval run --compose-skill`, `eval cases run --compose-skill`, and `environments ensure-adhoc --skill`. Use `mcpjam cloud skills list` to discover the IDs of the project-shared skills in your project:

```bash theme={"theme":"css-variables"}
mcpjam cloud skills list --project my-project
```

Each row includes a `pinnability` field that says whether the skill can be pinned into an environment or eval run, and why not if it can't. To see a skill's full SKILL.md body:

```bash theme={"theme":"css-variables"}
mcpjam cloud skills get --skill <id> --project my-project
```

Only `sharing: "project"` skills can be pinned. Personal drafts (`sharing: "user"`) are visible in the list but are not accepted by `--compose-skill` or `skillSelection`.

## Uploading Skills

You can upload skill folders directly from the Skills tab in the Connect view. The upload dialog supports drag-and-drop or file browsing and validates the `SKILL.md` frontmatter in real time.

* **Local:** Uploaded skills are stored in `~/.mcpjam/skills/`.
* **Hosted (cloud):** Cloud skills carry supporting files. A folder may contain `SKILL.md` plus scripts, references, and assets; the model reaches them with `listSkillFiles` and `readSkillFile` after loading the skill.

Hosted skill folders are bounded by three limits, enforced when the files are attached:

| Limit           | Value |
| --------------- | ----- |
| Per file        | 2 MB  |
| Files per skill | 50    |
| Total per skill | 20 MB |

<Note>
  Uploading a skill with a name that already exists will be rejected. Delete the existing skill first if you need to replace it.
</Note>

## Updating Skills

To pull the latest version of the `mcp-inspector` skill — which contains MCPJam's guidance for interpreting probe, doctor, OAuth, and conformance output — run:

```bash theme={"theme":"css-variables"}
npx skills update mcp-inspector
```

To update every installed skill at once:

```bash theme={"theme":"css-variables"}
npx skills update
```

## Where a turn's skills come from

A chat turn builds **one catalog** from every origin available to it, rather
than picking a single source:

| Origin                      | Reference              | Available on                             |
| --------------------------- | ---------------------- | ---------------------------------------- |
| Your project (Cloud Skills) | `code-review`          | anywhere you're signed in with a project |
| A local file                | `local/code-review`    | the desktop and npm app                  |
| A connected MCP server      | `<server>/code-review` | anywhere that server is connected        |
| A plugin                    | `<plugin>/code-review` | wherever the plugin is installed         |

This is why the Skills tab's Local/Cloud switch only changes what you are
**browsing**. It no longer decides what the model can use: a desktop turn signed
into a project offers your local files *and* your project skills, so a skill you
author in the tab is usable in the very next message.

Signed out, or with no project selected, a desktop turn simply has no project
family — it offers local files, exactly as before.

### Two skills with the same name

Local and project skills are namespaced apart on purpose, so nothing is hidden
and nothing is silently preferred. If you have a local `code-review` and a
project `code-review`, both appear and both load:

```
loadSkill("local/code-review")  → your file
loadSkill("code-review")        → the project's
```

**An exact reference always wins.** `code-review` *is* the project skill's
reference, so asking for it is not ambiguous — it is a direct hit, and the local
skill stays reachable under its own name. That is what keeps a project skill
addressable no matter what you happen to have on disk.

Ambiguity is the case where nothing is an exact hit: two *namespaced* skills
share a name and no skill answers to the bare name itself — a plugin
`docs-tools/code-review` alongside a local `local/code-review`, with no project
`code-review`. There the tool refuses and names the alternatives rather than
guessing:

```
"code-review" is ambiguous — it matches "docs-tools/code-review" and
"local/code-review". Use the full reference.
```

## Skills an agent writes for itself

Harness runs (Claude Code, Codex) execute in a sandbox with a real filesystem, so
the agent can write a skill mid-run the same way it would on your machine. After
a clean turn, MCPJam **adopts** any skill it finds in the sandbox's skills
directory into your project — otherwise the work would vanish with the sandbox.

Adopted skills are **personal**: visible to you, not to the rest of the project,
until you promote one. They are marked with a `computer-adopted` provenance, so
you can tell them from skills you authored.

Adoption is deliberately narrow. It does not run when:

* **the run used pinned skills** — a pinned set is a frozen snapshot, and writing
  back into the pool would change what the *next* eval arm resolves against,
  silently altering the thing under comparison;
* the turn failed, was aborted, or is paused waiting for approval;
* the run carries an execution scope (guest and swarm runs never adopt);
* the live skills fetch failed, so MCPJam cannot tell an agent-authored skill
  from one it simply failed to see as already managed.

<Note>
  Set `HARNESS_SKILL_ADOPTION_DISABLED=1` to turn adoption off entirely for a
  deployment.
</Note>

## Using Skills

### Progressive Disclosure

When you have skills available, MCPJam inlines the skill catalog — each skill's name and a one-line description — directly into the LLM's system prompt. The LLM decides which skills are relevant to your prompt and calls `loadSkill` to load only the full instructions it needs. Once a skill is loaded, the LLM can also browse and read its supporting files using `listSkillFiles` and `readSkillFile`.

The catalog is not a tool call. There is no general `listSkills` tool — a turn with no skills gets no skill tools and no catalog section at all, so an empty project never spends a discovery turn on tools that would return nothing.

#### Catalog Size Limit

The catalog is capped at **2% of the model's context window** (MCPJam follows OpenAI's plugin guidance here, measured in characters at roughly 4 characters per token). When a model definition carries no context length, the cap falls back to **8,000 characters**.

If your skills do not fit, MCPJam shortens descriptions first, and only omits whole skills as a last resort. An omission is always reported in the catalog itself:

```
(3 more skills could not be listed within this model's skill-metadata budget.)
```

An omitted skill is invisible to the model and cannot be loaded, so keep skill descriptions to a single line if you have many skills installed.

#### Catalog Fetch Failures (Hosted)

In the hosted app, the catalog is fetched from your project when the prompt is built, with a **3 second timeout**. If that fetch fails or times out, the turn proceeds with no skills rather than failing — the LLM sees no catalog and no skill tools. MCPJam records the failure separately from a genuinely empty project, so a run that silently lost its skills is distinguishable after the fact.

### Deterministic Injection

Type `/` in the input to open the skills popover. Select a skill to inject it directly into the conversation — this pre-loads the `SKILL.md` content before the LLM processes your message, so the skill is guaranteed to be used.

When you select a skill via `/`, it appears as a card above the input. You can expand the card to browse the skill's file tree and optionally select additional files to include. Only files you explicitly select are pre-loaded — by default, only the `SKILL.md` is injected.

The `/` picker works in both local and hosted modes. In the hosted app, it lists your project's cloud skills.

<iframe title="Deterministic skill injection demo" />

## LLM Tool Integration

When at least one skill is available, the LLM is given `loadSkill`, and — on surfaces that can serve supporting files — the two file tools as well:

| Tool                        | Description                                    | When advertised                                        |
| --------------------------- | ---------------------------------------------- | ------------------------------------------------------ |
| `loadSkill(name)`           | Load the full skill instructions by name       | Whenever the turn has at least one skill               |
| `listSkillFiles(name)`      | List all files in the skill directory          | Only when the turn's skills can carry supporting files |
| `readSkillFile(name, path)` | Read the content of a specific supporting file | Only when the turn's skills can carry supporting files |

The file tools are not advertised everywhere, because promising them for skills that have no files invites the model to go looking:

* **Local and hosted Playground chat** — all three tools.
* **Eval runs with pinned skills** — `loadSkill` only. Pinned eval skills are `SKILL.md`-only, so there are no files to browse.
* **Environment-resolved turns** — all three, but only when the resolved set actually contains a file-bearing skill.

Together with the inlined catalog, these tools let the LLM progressively load skill content as needed during a conversation, rather than injecting every skill's content upfront.

<Note>
  `listSkills` is not a general discovery tool. It appears only when a connected MCP server provides skills through the [Skills over MCP](https://aaif.io/blog/skills-over-mcp) extension (SEP-2640), and it lists those server-provided skills only. Use it to re-read the catalog mid-conversation; the catalog is already inlined in the system prompt at the start of each turn.
</Note>

## Skills over MCP (Server-Provided Skills)

When a connected MCP server advertises the SEP-2640 skills extension, MCPJam discovers its skills automatically and inlines them in the system prompt catalog alongside local and hosted skills — each entry attributed to its originating server (`MCP server "…"`). The model can choose a relevant skill from the catalog and load it with `loadSkill`, exactly as it would for any other skill origin.

Every server skill is verified before it is loaded: MCPJam fetches each file listed in the skill's manifest and checks its digest (and byte length, when the server provides one) before any content reaches the model.

Each server skill row shows the origin server's label as a badge directly on the row, so provenance is visible wherever the skill appears in the list. A server that has not mutually declared the extension contributes no rows — "this host and server never had a skills conversation" is a different fact from "this server has no skills".

The tab's single refresh button reloads both the project skill store and all server skill catalogs at once. It remains visible even when the project store is hidden, since server skills are the only list in that case.

### Inspecting a server's skills from the CLI

`mcpjam skills` asks a server what it serves and verifies the answer. It
connects directly, so it needs no MCPJam project and no API key:

```bash theme={"theme":"css-variables"}
mcpjam skills list --url https://example.com/mcp
mcpjam skills get --url https://example.com/mcp --uri "skill://acme/refunds/SKILL.md"
mcpjam skills read --url https://example.com/mcp \
  --skill-uri "skill://acme/refunds/SKILL.md" \
  --resource-uri "skill://acme/refunds/scripts/run.py"
```

`mcpjam resources list` will also show `skill://` URIs, because they are
resources — but it shows them as bytes with no manifest, no digest check, and
no reason when something is wrong. These commands run the same verified read
path the Playground uses, so what you get back is either verified content or a
**refusal** naming the check that failed (see the table below). A refusal is a
result, not an error: it prints to stdout and exits 0, because "this skill
cannot be verified, and here is why" is the answer you asked for.

<Note>
  `mcpjam skills` reads a **connected server's** skills. `mcpjam cloud skills`
  lists the Cloud Skills in your own MCPJam project. Same word, opposite
  direction.
</Note>

The skills extension is mutually declared, so these commands advertise it on
connect. If you pin a host persona that does not — `--host cursor`, say — the
command refuses rather than quietly advertising anyway, since the answer to
"what would Cursor see here?" is *nothing*:

```
$ mcpjam skills list --url https://example.com/mcp --host cursor
`skills` requires the advertised client capabilities to include the
`io.modelcontextprotocol/skills` extension ... That host would not see this
server's skills at all.
```

Agents driving MCPJam reach the same three operations as `list_server_skills`,
`get_server_skill` and `read_server_skill_file` — over `/api/v1`, the hosted
MCP server, and the in-app assistant.

### Checking that your own server serves skills correctly

`mcpjam server doctor` includes a **skills** check. Unlike the tool, resource
and prompt checks it verifies rather than counts, fetching a sample of skills
and checking each against its manifest:

```
- skills: ok (4 skills discovered. 4 verified against their manifests.)
- skills: error (1 of 2 sampled skills failed verification:
  skill://demo/tampered/SKILL.md (digest_mismatch).)
```

It reports `skipped` when the extension is not active on the connection, which
is the normal state for a server that serves no skills.

### Unloadable skills

A server skill can appear in the catalog but be refused for loading. MCPJam keeps the skill visible in the catalog — marked `unverifiable` — so the model stops retrying it and you can see what the server advertised. MCPJam declines to inject unverifiable content into a model turn. The reason is always shown:

| Reason               | What it means                                                                                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `no_resources`       | The server omitted the `resources` manifest the spec requires. MCPJam cannot verify the skill's contents without one.                                                           |
| `dynamic_resources`  | The server declared `"resources": "dynamic"` — a form the spec defines for skills generated per request. There is no manifest to verify against, so MCPJam declines to load it. |
| `too_many_resources` | The skill's manifest lists more than 512 files, which is the per-skill limit the spec requires hosts to support.                                                                |
| `too_large`          | The skill's manifest declares a total size over 16 MiB, which is the per-skill byte budget the spec requires hosts to support.                                                  |

A skill that is unloadable for a limit reason (`too_many_resources` or `too_large`) is a real skill whose manifest exceeds what any conforming host is required to handle — it is not a server bug. The skill remains visible in the catalog with its advertised manifest so you can inspect what was declared.

### Prompt injection cap

Verification uses the spec's per-skill budget (16 MiB). A separate, lower cap of **128 KiB** applies to how much skill content MCPJam will place in a single model turn. A skill that verifies correctly but exceeds this cap is refused rather than truncated — a clipped skill ends mid-sentence, and the model cannot distinguish that from a skill that simply said less.


# MCP Inspector
Source: https://docs.mcpjam.com/inspector/tools-prompts-resources

Manually test your MCP server's tools, resources, and prompts

The MCP Inspector provides a complete testing environment for your MCP server. Manually invoke tools, read resources, fetch prompts, and view all JSON-RPC messages between MCPJam and your server.

<Note>
  The Tools, Resources, and Prompts tabs only load data when the selected server
  is connected. If the server is disconnected, the tabs clear any previously
  loaded primitives and display a prompt to reconnect. Refresh, run, and read
  actions are disabled until the server is connected again.
</Note>

<Frame>
  <img alt="MCPJam MCP Inspector" />
</Frame>

## Tools

Test your MCP tools by manually invoking them with custom parameters:

* **View all tools** - Browse tool names, descriptions, and parameters
* **Search tools** - Quickly find specific tools by name
* **Invoke tools** - Trigger tools with custom input parameters
* **View results** - See tool outputs, including widget responses for ChatGPT apps and MCP apps
* **Elicitation support** - Handle elicitation requests from tools with a spec-complete form dialog
* **Multi-round input** - Tools that return `input_required` (MCP 2026-07-28) collect the needed input through the same dialog and retry automatically
* **Save requests** - Save frequently used tool invocations for quick testing
* **Quality badges** - See per-tool quality indicators in the sidebar for tools with definition issues

### Elicitation requests

When a tool triggers an MCP elicitation request, a dialog appears asking you to provide additional input. The dialog renders a form built from the server's schema and supports the full MCP elicitation subset:

* **Field labels and defaults** — Fields show the schema `title` when provided (the raw property key stays visible alongside it). Schema `default` values are pre-filled automatically.
* **Field types** — Strings, numbers, integers, booleans (checkbox), single-select dropdowns, and multi-select checkbox lists are all rendered natively. Fields with `format: email`, `uri`, `date`, or `date-time` use the matching browser input type.
* **Inline validation** — Clicking **Accept** validates all fields before submitting. Required fields, numeric `minimum`/`maximum`, string `minLength`/`maxLength`, `pattern`, and format constraints are all checked. Errors appear under the relevant field and block submission until resolved.
* **Server identity** — The dialog header identifies which server is requesting information. When the server supplies both a name and an ID, both are shown — the ID is the trusted anchor and is always visible.
* **Unsupported shapes** — Nested objects and other schema shapes outside the MCP elicitation spec fall back to a plain JSON textarea so you can still answer by hand.

**Decline** and **Cancel** are always available and bypass validation — they send the corresponding action to the server without any parameters.

### Multi-round input (MCP 2026-07-28 `input_required`)

When a tool call returns an `input_required` result (MCP 2026-07-28 §12), the inspector collects the embedded elicitation input through the same dialog and automatically retries the original call with your answers. This is distinct from the legacy `elicitation/create` flow:

* **Dialog title** — shows "Operation Needs Input" (vs "Elicitation Request" for legacy requests).
* **Dialog header** — reads "\[server] needs input to continue" (vs "\[server] is requesting information").
* **Multiple keys per round** — if the server embeds more than one input request in a single round, the dialog walks through them one at a time and submits all answers together.
* **Automatic retry** — after you respond, the inspector retries the original `tools/call` with your answers attached. You do not need to re-invoke the tool manually.
* **Decline and Cancel** — send the corresponding action back to the server; the server decides whether to abort or continue.

This behavior applies to local connections. The inspector must be connected to the server before the tool is invoked; the `elicitation` capability is advertised at connect time.

### OAuth scope step-up on tool calls

When a tool call returns a `403 insufficient_scope` error, the Tools tab automatically triggers a re-authorization flow. Inspector unions the scopes from your original OAuth grant with the scopes the server challenged, then redirects you to the authorization server to approve the expanded set. After you return from the consent screen, re-run the tool.

The step-up is bounded per browser session: if re-authorization does not resolve the scope error after the allowed number of attempts, Inspector surfaces the error instead of redirecting again. The counter resets when a tool call succeeds, so a future legitimate scope challenge starts fresh. Opening a new browser session also resets the counter.

<Note>
  Runtime scope step-up is only available on the local (non-hosted) tool-call surface. The hosted path does not yet forward the scope challenge.
</Note>

### Tool quality badges

When enabled, each tool in the sidebar may display a small badge showing the number of quality findings for that tool:

| Badge color | Meaning                                                                                                                                                           |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Red         | A spec-level error (for example, `inputSchema` missing or not an object)                                                                                          |
| Amber       | A quality warning (for example, undocumented parameters, a description referencing a parameter not in the schema, or a `required` name missing from `properties`) |

Hover over a badge to see the full list of findings. Tools with no issues show no badge.

Saved requests are stored in your browser's local storage and accessible from the Saved Requests tab.

## Resources

Inspect your MCP server's resources:

* **Browse resources** - View all available resources with names, URIs, and MIME types
* **Read resources** - Fetch and view resource content
* **Resource templates** - Test templated resources with dynamic parameters
* **Multi-round input** - Resources that return `input_required` (MCP 2026-07-28) collect the needed input through the elicitation dialog and retry the read automatically

### Subscription stream panel

When connected to a local server, the Resources tab shows a **Subscriptions** panel at the bottom. The panel lets you observe and control the subscription lifecycle the local manager owns.

<Note>
  The subscription panel is only available in local (non-hosted) mode and only when a server is connected.
</Note>

The panel branches on the negotiated protocol era:

**Modern era (`subscriptions/listen`, MCP 2026-07-28)**

The server supports a single `subscriptions/listen` stream. Because a listen filter is immutable for the life of a subscription, changing the desired filter closes the current stream and opens a new one automatically. The panel shows:

* **Desired filter** — checkboxes for `tools/list_changed`, `prompts/list_changed`, and `resources/list_changed`, plus per-URI watch toggles for each resource. Toggling any checkbox posts the new desired interests; the coordinator performs the close and reopen.
* **Stream details** — status (`Opening`, `Active`, `Closed by server (graceful)`, `Connection lost (remote close)`, `Cancelled locally`, `Error`), the MCP subscription id and whether it was `reported` or `observed`, the requested filter next to the acknowledged filter, any interests the server did not honor, the close reason, and the re-listen attempt count.
* **Cancel stream** — ends the current stream as a local abort.

**Legacy era (`resources/subscribe`)**

The server uses per-URI subscribe RPCs. The panel shows the same stream details but replaces the desired-filter checkboxes with per-URI **Subscribe** / **Unsubscribe** buttons backed by `resources/subscribe` and `resources/unsubscribe`.

**History**

Both eras append a History log below the stream list. Each entry records the kind of transition:

| Entry kind        | Meaning                                                                 |
| ----------------- | ----------------------------------------------------------------------- |
| `opened`          | A new stream was requested                                              |
| `acknowledged`    | The server confirmed the filter                                         |
| `reconnected`     | A re-listen attempt was made                                            |
| `closed-graceful` | The server closed the stream gracefully                                 |
| `closed-remote`   | The connection was lost (remote close)                                  |
| `cancelled`       | The stream was cancelled locally                                        |
| `error`           | The stream ended with an error                                          |
| `notification`    | A subscription notification arrived                                     |
| `rejected`        | A notification arrived for an interest the server did not agree to send |

Graceful and remote closes are always distinct entries — they are never flattened into a single "disconnected" event. Rejected notifications are shown rather than silently dropped, so you can see the absence the server chose.

## Prompts

Test your MCP prompts:

* **View prompts** - Browse prompt names, descriptions, and parameters
* **Fetch prompts** - Retrieve prompt content with custom arguments
* **Use in Playground** - Type `/` in the Playground to insert prompts directly into the conversation
* **Multi-round input** - Prompts that return `input_required` (MCP 2026-07-28) collect the needed input through the elicitation dialog and retry the fetch automatically

## JSON-RPC Logging

All communication between MCPJam and your MCP server is logged in real-time:

* **Request/response pairs** - View complete JSON-RPC messages
* **Message types** - See tool calls, resource reads, prompt fetches, and more
* **Error tracking** - Inspect errors and validation issues
* **Timestamps** - Track message timing and performance

The JSON-RPC logger helps you debug your MCP server implementation and understand the complete protocol flow.

### Inline HTTP headers

When you expand an outgoing JSON-RPC frame (a request sent to the server), a collapsed **HTTP headers** section appears below the JSON body. Expanding it shows the full HTTP exchange — request headers, response status, and any `Mcp-*` headers the server mirrored back.

The collapsed row names the first disagreement without requiring you to open anything: for example, `mcp-name disagrees with the body` tells you which header caused a routing failure before you inspect the exchange in detail. The row turns red when the exchange returned a 4xx/5xx status or the connection failed at the transport level.

`Mcp-Param-*` rows are judged too, whenever the log holds a `tools/list` for the tool: each mirrored argument shows ✓ when the header matches the value in the request body, and a failure verdict when it disagrees, was never sent, or names a parameter the tool never declared. When the tool's schema can't be recovered from the log the row says `unchecked — tool schema unavailable`, so an unverifiable header never reads as a passing one.

The section is absent when no exchange could be confidently matched to the frame, and hidden entirely when the exchange carries no cross-checked `Mcp-*` headers (for example, connections to servers running a protocol era before 2026-07-28).

The dedicated **HTTP** source filter in the traffic log still shows every exchange on its own terms, including ones that carry no single JSON-RPC frame.

### OAuth log entries

When a server uses OAuth, each step of the OAuth flow appears as a log entry in the traffic log alongside regular JSON-RPC messages. Use the **Source** filter dropdown and select **OAuth** to show only OAuth entries.

Each OAuth log entry displays a status indicator:

| Indicator | Meaning                                                                      |
| --------- | ---------------------------------------------------------------------------- |
| `oauth ✓` | Step completed successfully                                                  |
| `oauth ✗` | Step failed                                                                  |
| `oauth …` | Step is in progress                                                          |
| `oauth ↺` | Step failed but was recovered (for example, after re-registering the client) |

Click any entry to expand it and inspect the full step payload, including HTTP request and response details.


# View origins
Source: https://docs.mcpjam.com/inspector/view-origins

Which origin your MCP App's view runs at, and how to allowlist it with a third-party API

Your app's view runs in a sandboxed iframe at a **real URL**, on an origin separate from the Inspector itself. That matters whenever something outside MCPJam checks where a request came from — a referrer-restricted Google Maps key, an OAuth redirect URI, a CORS allowlist.

## Which origin to allowlist

<Tabs>
  <Tab title="Local">
    The view runs on the Inspector's scheme and port with the hostname swapped between `localhost` and `127.0.0.1`. The swap is what keeps the sandbox on a different origin from the app.

    Because the swap depends on which one you open the Inspector with, **allowlist both**:

    ```
    http://localhost:*
    http://127.0.0.1:*
    ```
  </Tab>

  <Tab title="Hosted">
    ```
    https://sandbox.mcpjam.com
    ```
  </Tab>
</Tabs>

To read the exact value rather than infer it, open a tool result, expand the CSP Workbench, and look at **Sandbox Stack → View origin**. It has a copy button.

<Note>
  Cross-origin requests carry the **origin**, not the full path — MCPJam sends
  `Referrer-Policy: strict-origin-when-cross-origin`. So a third party sees
  `http://127.0.0.1:6274/`, and an allowlist pattern keyed on the origin
  matches. A pattern keyed on a path will not.
</Note>

## CSP is a separate gate

Allowlisting your origin tells the third party to *accept* your requests. It does not let your widget *make* them — MCPJam's default policy denies everything not declared. Both have to pass.

Declare what the API needs in the UI resource's `_meta.ui.csp`:

```json theme={"theme":"css-variables"}
{
  "_meta": {
    "ui": {
      "csp": {
        "resourceDomains": [
          "https://maps.googleapis.com",
          "https://*.gstatic.com"
        ],
        "connectDomains": ["https://maps.googleapis.com"]
      }
    }
  }
}
```

`resourceDomains` covers scripts, images, styles and fonts; `connectDomains` covers `fetch`/XHR/WebSocket. A JavaScript SDK usually needs both. `frameDomains` is only for APIs you embed as a nested iframe — an embed widget, not a JS SDK.

Anything still blocked shows up in the Workbench's **Findings** tab with the directive that refused it.

## `_meta.ui.domain`

SEP-1865 lets a server request a dedicated origin with `_meta.ui.domain`. The format is host-specific and each host derives its own — Claude uses `sha256(<connector URL>)[:32] + ".claudemcpcontent.com"`, ChatGPT a per-plugin label — so one declared string cannot match every host.

MCPJam **derives** the origin it serves your view from rather than routing on what you declare; a server-chosen string would otherwise let one server claim another's origin, and with it that origin's cookies and storage. If you declare a domain, the Workbench's Findings tab reports whether it matches what MCPJam serves. A mismatch is informational — it is the normal state for a server already targeting Claude or ChatGPT — and simply means an allowlist keyed on that value will not match requests coming from MCPJam.

## Rendering mode

MCPJam mounts a view by writing its HTML into a blank document, the same way Claude does. A document written that way **without a doctype is parsed in quirks mode**, where the box model and percentage heights differ from what you probably tested against.

Start your resource HTML with `<!DOCTYPE html>`. The readiness report flags this as `claude.apps.design.doctype`.


# XAA Debugger
Source: https://docs.mcpjam.com/inspector/xaa-debugger

Configure and debug Cross-App Access flows with MCPJam as the test identity provider and client/agent

<iframe title="XAA Debugger walkthrough" />

The XAA (Cross-App Access) Debugger tests an MCP server and its authorization setup.

MCPJam acts as both:

* The **identity provider**, which signs the ID token and ID-JAG.
* The **client/agent**, which presents the ID-JAG and uses the access token.

## Before You Run the Test

### Trust MCPJam's Identity Provider

The XAA Flow header shows three URLs:

<Frame>
  <img alt="XAA Flow header showing the Issuer URL, OpenID Config, and JWKS URL" />
</Frame>

| URL               | Purpose                                              |
| ----------------- | ---------------------------------------------------- |
| **Issuer URL**    | The issuer MCPJam puts in its signed tokens.         |
| **OpenID Config** | Metadata describing the issuer and its signing keys. |
| **JWKS URL**      | MCPJam's public signing keys.                        |

Configure your authorization server with the values it requests. The **Issuer URL** identifies MCPJam's test identity provider, the **OpenID Config** URL provides discovery metadata, and the **JWKS URL** provides the public signing keys.

### Local and hosted issuers

If MCPJam is running locally and your authorization server is hosted in the
cloud, turn on **Use hosted issuer** in the XAA Flow header. A cloud server
cannot reach a local Issuer URL.

<Frame>
  <img alt="Use hosted issuer toggle in the XAA Flow header" />
</Frame>

### Other header controls

The header also has an **Identity assertion** (OIDC / SAML) toggle that sets the format MCPJam mints, covered under [Identity assertion format](#identity-assertion-format). It also has a **[Run as](#simulated-identity)** control for choosing the simulated identity.

## Add Your MCP Server

Select **Configure Server to Test** (or **Add Server**), enter the MCP server name and URL, then choose how MCPJam identifies itself to the authorization server.

### Server Configuration

<Frame>
  <img alt="Server configuration fields in the XAA Debugger" />
</Frame>

#### Registration Method

* **Pre-registered client:** Register MCPJam in your authorization server first,
  then enter the client ID and optional client secret it gives you.
* **Client metadata URL (CIMD):** MCPJam uses a client metadata URL as the
  client ID. Your authorization server must advertise CIMD support. The
  **Client authentication** control chooses whether MCPJam proves it owns that
  identity:
  * **Public (no client auth):** Your authorization server takes the metadata
    URL at face value. Nothing proves MCPJam owns that identity. This is the
    default.
  * **Confidential (private\_key\_jwt):** MCPJam holds a private key and signs
    each token request with it, proving it owns the identity. Choose this when
    your authorization server requires a confidential client. In the hosted
    inspector, this option requires you to be signed in with an active
    organization; MCPJam probes for the capability automatically and blocks
    Start/Run with a clear message if the requirement is not met.
* **Open dynamic registration (DCR):** MCPJam registers a client during the
  test. Your authorization server must provide an open registration endpoint.

#### Identity assertion format

Choose the protocol MCPJam's identity provider uses to authenticate users.

* **OIDC ID token:** MCPJam mints an OIDC ID token as the identity assertion.
  This is the default.
* **SAML assertion:** MCPJam mints a signed SAML 2.0 assertion, and the ID-JAG
  carries a `saml-nameid` subject identifier so a SAML-federated server can
  resolve the user.

#### Scopes

Enter optional permissions separated by spaces. MCPJam includes them when
requesting the ID-JAG and access token.

### Advanced Settings

<Frame>
  <img alt="Advanced settings in the XAA Debugger" />
</Frame>

#### Authorization Server Issuer

Leave it blank to use the authorization server MCPJam discovers from the MCP
server. Enter a URL to use a different authorization server.

#### Path-scoped authorization server

Keep this off for normal RFC 8414 behavior. Turn it on only when the discovered
URL and metadata issuer differ on the same origin.

#### Simulated identity

Choose the user MCPJam puts in the ID token. Leave the fields blank to use a
built-in demo user (`user-12345` / `demo.user@example.com`).

## Inspect the ID-JAG

<Frame>
  <img alt="ID-JAG Inspector showing the assertion claims, header, payload, and signature" />
</Frame>

Before MCPJam sends the ID-JAG to the authorization server, the right panel
opens the **ID-JAG Inspector**. Use it to inspect exactly what MCPJam created
and identify problems before the token exchange. MCPJam checks each piece of
information inside the ID-JAG and shows whether it looks correct or broken. It
also shows the relevant RFC or ID-JAG requirement for each check.

### Header

The JWT header shows:

* `alg`: the signing algorithm.
* `typ`: the token type.
* `kid`: the signing key identifier.

### Payload

The payload shows the values MCPJam puts into the ID-JAG:

| Claim                   | What it identifies                                      |
| ----------------------- | ------------------------------------------------------- |
| `iss`                   | MCPJam's test identity provider                         |
| `sub` and `email`       | The test user's identity                                |
| `aud`                   | The authorization server that should receive the ID-JAG |
| `resource`              | The MCP server the access is for                        |
| `client_id`             | The client identity used in the test                    |
| `jti`, `iat`, and `exp` | Replay protection and token lifetime                    |
| `scope`                 | The permissions requested for the MCP server            |

### Signature

The inspector shows the signed JWT and provides **Copy JWT** for further
inspection. The authorization server verifies the signature using MCPJam's
public signing keys.

## Negative Tests

<Frame>
  <img alt="Negative-test scorecard showing invalid ID-JAGs rejected" />
</Frame>

Checks that your authorization server rejects broken ID-JAGs. Run a successful
flow first, then click **Run negative tests**. A strict check passes when the
authorization server rejects the broken assertion. The MCP server is not called
during these checks.

The scorecard runs these strict rejection checks:

* Bad Signature
* Wrong Audience
* Expired
* Missing Claims
* Invalid `typ` Header
* Wrong Issuer
* Resource Mismatch
* Client ID Mismatch
* Unknown `kid`

It also reports two policy probes:

* Unknown Subject
* Scope Denial

No pass or fail here. Your server can accept or reject an unknown user, and can
trim the scopes it grants. Just confirm the result is what you expect.

These are targeted security and policy checks, not a full conformance test.

## The XAA Flow

The debugger shows the flow in the sequence diagram and logs each request and response:

1. **Discover the MCP server and authorization server.** MCPJam asks the MCP server which authorization server protects it, then reads the authorization server metadata.
2. **Resolve the client identity.** MCPJam uses the pre-registered client, performs DCR, or uses its CIMD URL.
3. **Simulate sign-in at MCPJam's identity provider.** MCPJam issues a test ID token.
4. **Request an ID-JAG.** MCPJam's identity provider exchanges the ID token for an ID-JAG.
5. **Request an access token using the ID-JAG.** MCPJam sends the ID-JAG to the authorization server's token endpoint using the JWT-bearer grant.
6. **Call the MCP server.** MCPJam sends the resulting access token to the MCP server.

## Debugging Failures

The right panel shows each flow step, its request and response details, and any
available token information. If a step fails, it also provides guidance for
diagnosing the problem. This may include:

* The authorization server does not advertise or accept the JWT-bearer grant.
* MCPJam's issuer or signing keys are not trusted.
* The client identity is missing, unregistered, or not allowed to use the grant.
* The aud, resource, or client\_id claim does not match the authorization server's configuration.
* The authorization server issues a token, but the MCP server rejects it.

Use the compatibility results after discovery and expand each flow step to inspect the exact request, response, and token claims.

## Related Standards

* [MCP Authorization](https://modelcontextprotocol.io/specification/draft/basic/authorization)
* [OAuth 2.0 Authorization Server Metadata (RFC 8414)](https://www.rfc-editor.org/rfc/rfc8414)
* [OAuth 2.0 JWT Bearer Token Grant (RFC 7523)](https://www.rfc-editor.org/rfc/rfc7523)
* [OAuth 2.0 Dynamic Client Registration (RFC 7591)](https://www.rfc-editor.org/rfc/rfc7591)


# Installation
Source: https://docs.mcpjam.com/installation

Here's a list of all ways to start MCPJam

MCPJam Inspector runs three ways: a hosted web app, a desktop app for Mac and Windows, or via your terminal. The web app is HTTPS-only and has no install. Terminal and Desktop support HTTP/S and local STDIO servers.

<CardGroup>
  <Card title="Web App" icon="globe" href="https://app.mcpjam.com">
    Open [app.mcpjam.com](https://app.mcpjam.com). HTTPS only. No install. Share servers with your team.
  </Card>

  <Card title="Desktop App" icon="monitor">
    Download for [Mac](https://github.com/MCPJam/inspector/releases/latest/download/MCPJam.Inspector.dmg) or [Windows](https://github.com/MCPJam/inspector/releases/latest/download/MCPJam-Inspector-Setup.exe). HTTP/S and local STDIO.
  </Card>

  <Card title="Terminal" icon="terminal">
    `npx @mcpjam/inspector@latest`. HTTP/S and local STDIO.
  </Card>
</CardGroup>

### Which one do I need?

| Capability               | Web App |        Desktop        |   Terminal (`npx`)  |        Docker       |
| ------------------------ | :-----: | :-------------------: | :-----------------: | :-----------------: |
| Install required         |   None  |  Mac/Windows download |       Node.js       |     Docker + Git    |
| HTTPS MCP servers        |    ✓    |           ✓           |          ✓          |          ✓          |
| HTTP MCP servers         |    —    |           ✓           |          ✓          |          ✓          |
| Local STDIO servers      |    —    |           ✓           |          ✓          |  ✓ (with `--` args) |
| Skills                   |    —    |           ✓           |          ✓          |          ✓          |
| Shareable server URLs    |    ✓    |           —           |          —          |          —          |
| Always on latest version |    ✓    | Re-download to update | Per-run (`@latest`) | Rebuild from source |

If your server is local (running on `localhost`), the web app cannot reach it — pick **Desktop** or **Terminal**. If you want teammates to one-click into the same server, pick **Web App**.

## Web app

Go to [app.mcpjam.com](https://app.mcpjam.com) in your browser. No install required. The web app accepts HTTPS MCP server URLs only — for HTTP or local STDIO servers, use the desktop or terminal options below. See [Hosted App](/hosted/overview) for more.

## Desktop app

Download the installer for your OS and run it:

* [Mac](https://github.com/MCPJam/inspector/releases/latest/download/MCPJam.Inspector.dmg)
* [Windows](https://github.com/MCPJam/inspector/releases/latest/download/MCPJam-Inspector-Setup.exe)

The desktop app supports HTTP/S and local STDIO servers, and does not require Node.js.

## Terminal

Run the command in your terminal:

<CodeGroup>
  ```bash npm theme={"theme":"css-variables"}
  npx @mcpjam/inspector@latest
  ```

  ```bash pnpm theme={"theme":"css-variables"}
  pnpm dlx @mcpjam/inspector@latest
  ```

  ```bash yarn theme={"theme":"css-variables"}
  yarn dlx @mcpjam/inspector@latest
  ```

  ```bash bun theme={"theme":"css-variables"}
  bunx @mcpjam/inspector@latest
  ```
</CodeGroup>

After installing the package, you should see a link to `localhost`. Open that link up to see the inspector.

<Info>
  **Widget rendering (Chromium):** On first startup, Inspector automatically downloads and installs the Playwright Chromium browser needed to render widgets. This happens in the background — you'll see download progress in the terminal. No manual `playwright install` step is required. Docker images already include Chromium and are unaffected.
</Info>

### From your editor (VS Code, Cursor, Windsurf)

There is no dedicated editor extension — and you don't need one. Open your editor's integrated terminal, run the same `npx @mcpjam/inspector@latest` command, and click the printed `localhost` link. The inspector runs alongside your dev server in any editor that has a terminal pane.

If you want the inspector to auto-launch from your project's `dev` script (so it boots whenever you start your server), see [Launch from Code](/inspector/launch-from-code).

## Docker

There is no published image — build one from source, then run it bound to localhost for security:

```bash theme={"theme":"css-variables"}
git clone https://github.com/MCPJam/inspector.git
cd inspector
docker build -t mcpjam/mcp-inspector:local -f mcpjam-inspector/Dockerfile .
docker run -p 127.0.0.1:6274:6274 mcpjam/mcp-inspector:local
```

The app is available at `http://127.0.0.1:6274`. Always use `-p 127.0.0.1:6274:6274` (not `-p 6274:6274`) to keep the inspector local-only. On macOS/Windows, connect to host MCP servers via `http://host.docker.internal:PORT` instead of `127.0.0.1`.

### Accessing over the network

Network access requires **Docker**. A native (`npx`) install binds to `127.0.0.1` (localhost only) and cannot be reached from another machine. The Docker image binds `0.0.0.0`, so it can.

For security, the inspector only issues its session token to `localhost`. When you open a Docker install from another machine (e.g. `http://192.168.1.50:6274`), you'll land on an authentication error. To allow a specific host, set `MCPJAM_ALLOWED_HOSTS` to that hostname and publish the port on your network interface (drop the `127.0.0.1:` prefix from `-p`):

```bash theme={"theme":"css-variables"}
docker run -p 6274:6274 -e MCPJAM_ALLOWED_HOSTS=192.168.1.50 mcpjam/mcp-inspector:local
```

For an IPv6 host, bracket the entry: `MCPJAM_ALLOWED_HOSTS=[fd00::50]`. Comma-separate multiple hosts. Wildcards like `*.example.com` are supported — if you use a wildcard entry, also set `MCPJAM_ALLOW_WILDCARD_ORIGINS=true` so API calls are accepted (without it, the session token is issued but requests still 403).

<Warning>
  Only add hosts you trust. Allowlisting a host also accepts it as a request origin, which exposes the local shell and agent-browser tools that are on by default in self-hosted mode. Set `MCPJAM_LOCAL_COMPUTER_ENABLED=false` and `MCPJAM_LOCAL_BROWSER_ENABLED=false` if you don't want that. Tunnel and relay domains are never allowed, even if listed.
</Warning>

### Docker with Arguments

You can pass command-line arguments to the Docker container:

```bash theme={"theme":"css-variables"}
# Launch with STDIO server
docker run -p 127.0.0.1:6274:6274 mcpjam/mcp-inspector:local -- \
  npx -y @modelcontextprotocol/server-everything

# Launch with HTTP server
docker run --rm -p 127.0.0.1:6274:6274 mcpjam/mcp-inspector:local -- \
  --url http://host.docker.internal:8080/mcp \
  --name "HTTP Server" \
  --tab tools
```

For more information check out our docs on [Launch from Code](https://docs.mcpjam.com/inspector/launch-from-code)

<Warning>
  **Important for macOS/Windows users:**

  * Access the app via `http://127.0.0.1:6274` (not `localhost`)
  * When connecting to MCP servers on your host machine, use `http://host.docker.internal:PORT` instead of `http://127.0.0.1:PORT`

  Example:

  ```bash theme={"theme":"css-variables"}
  # Your MCP server runs on host at: http://127.0.0.1:8080/mcp
  # In Docker, configure it as: http://host.docker.internal:8080/mcp
  ```
</Warning>

## Start with STDIO server

This will open the MCPJam inspector and connect to a STDIO using the commands you provided.

<Tabs>
  <Tab title="Python Server">
    ```bash theme={"theme":"css-variables"}
    # Starts MCPJam inspector with a FastMCP server
    npx @mcpjam/inspector@latest uv run fastmcp run /path/to/your/server.py
    ```

    This command assumes [`uv`](https://docs.astral.sh/uv/) is installed and on your PATH. If you see `Unknown command: uv` (fish), `command not found: uv` (bash/zsh), or `'uv' is not recognized` (Windows), install it first:

    ```bash theme={"theme":"css-variables"}
    curl -LsSf https://astral.sh/uv/install.sh | sh
    ```
  </Tab>

  <Tab title="Node.js Server">
    ```bash theme={"theme":"css-variables"}
    # NPX package
    npx @mcpjam/inspector@latest npx -y your-mcp-package

    # Direct Node.js execution
    npx @mcpjam/inspector@latest node /path/to/your/server.js
    ```
  </Tab>
</Tabs>

<Info>
  **Use Absolute Paths**: Always use absolute file paths (e.g.,
  `/Users/yourname/project/server.py`) to avoid path resolution issues.
</Info>

## Start with custom port

This will start the MCPJam inspector with a custom port number. The default port is `6274`.

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest --port 4000
```

The port must be a valid integer between 1 and 65535. Passing an invalid value (e.g. a string or out-of-range number) exits immediately with an error.

## Start with Ollama

This will open the MCPJam inspector and start an Ollama model with the `ollama serve <model>` command. Make sure you have Ollama installed.

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest --ollama llama3.2
```

## Start with Configuration File

You can use a configuration file to connect to multiple MCP servers at once:

```bash theme={"theme":"css-variables"}
# Using npx directly
npx @mcpjam/inspector@latest --config path/to/config.json --server myserver

# If running from the repository
npm run build
npm start -- --config path/to/config.json --server myserver
```

<Info>
  **Note**: When using `npm start`, you need the `--` separator to pass
  arguments to the underlying script.
</Info>

All servers in the config get added to your project. By default they all
connect on launch; pass `--server <name>` to only connect that one. If
you're not signed in, we'll send you to sign in and bring you right back.

Example configuration file structure:

```json config.json theme={"theme":"css-variables"}
{
  "mcpServers": {
    "everything": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-everything"],
      "env": {
        "HELLO": "Hello MCP!"
      }
    },
    "weather-server": {
      "command": "python",
      "args": ["weather_server.py"],
      "env": {
        "WEATHER_API_KEY": "your-api-key"
      }
    }
  }
}
```

The config parser accepts three equivalent wrapper shapes, so you can paste a config from any of these sources without reformatting:

| Shape                                                | Example                                       |
| ---------------------------------------------------- | --------------------------------------------- |
| `mcpServers` wrapper (MCPJam / Claude Desktop style) | `{ "mcpServers": { "my-server": { ... } } }`  |
| `mcp_servers` wrapper (OpenAI plugin style)          | `{ "mcp_servers": { "my-server": { ... } } }` |
| Direct server map (OpenAI `.mcp.json` style)         | `{ "my-server": { ... } }`                    |

Declaring both `mcp_servers` and `mcpServers` in the same file is an error. A bare single-server object (top-level `command` or `url` string) is not accepted — wrap it in one of the shapes above.

### Mixing STDIO and HTTP/SHTTP servers

You can mix STDIO and HTTP (Streamable HTTP / SHTTP) server entries in the same config. Each entry's transport is inferred from its fields: `command` + `args` means STDIO, `url` means HTTP/SHTTP.

```json config.json theme={"theme":"css-variables"}
{
  "mcpServers": {
    "local-everything": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-everything"],
      "env": {
        "HELLO": "Hello MCP!"
      }
    },
    "hosted-weather": {
      "url": "https://weather.example.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN_HERE"
      }
    }
  }
}
```

The terminal and desktop installs accept both shapes in the same file. The hosted web app accepts HTTP/SHTTP entries only — STDIO entries are ignored because the hosted runtime cannot spawn local processes.

For per-server `Authorization` headers, see [API keys → Playground BYOK](/reference/api-keys#3-playground-byok-bring-your-own-key).

## Start with Verbose Logging

Enable verbose HTTP request logging for debugging purposes. This is useful when troubleshooting connection issues with MCP servers.

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest --verbose
```

Or use the short form:

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector@latest -v
```

<Info>
  Verbose mode logs all HTTP requests made by the inspector, which can help
  diagnose connectivity problems or inspect server communication.
</Info>


# API keys
Source: https://docs.mcpjam.com/reference/api-keys

Which 'API key' is which in MCPJam — and the one you need to call the MCPJam API.

"API key" gets used for a few different things in MCPJam. The short version:

* To call the **[MCPJam API](/reference/public-api)**, you want the **MCPJam API key** (`sk_…`) — the first one below.
* Two others here are **third-party secrets you supply**, not keys MCPJam issues — they're listed only because the name collides.
* One older key (`mcpjam_…`) is **retired** and no longer works.

## MCPJam API key (`sk_…`)

The key for the **[MCPJam API](/reference/public-api)** — programmatic access to live MCP server diagnostics at `https://app.mcpjam.com/api/v1/...`. This is the one to reach for whenever you mean "the MCPJam API key."

**Where it lives:** [**Settings → API Keys**](https://app.mcpjam.com/settings/api-keys) in the hosted app — open Settings from the account menu in the sidebar, then click the **API Keys** tab. That link also takes you straight there (sign in if prompted; you'll land back on the page). Shown **exactly once** at creation — store it in your own secret manager or an environment variable.

**When it's used:** Every `Authorization: Bearer sk_…` request to `/api/v1/*` — validating servers, running the doctor, listing tools/prompts/resources, reading resources, exporting snapshots, and **saving `@mcpjam/sdk` eval results** (set `MCPJAM_API_KEY` to this key; see [Saving Results](/sdk/concepts/saving-results)).

**Errors you might see**

* `401 UNAUTHORIZED` → invalid or revoked key.
* `401` with `details.reason: "ORPHANED_KEY"` → the key lost its organization binding; create a new one from Settings.
* `403 FORBIDDEN` ("API keys cannot manage other API keys") → you pointed an `sk_…` key at the key-management endpoints; key management is UI-only.
* `429 RATE_LIMITED` → per-key rate limit (60/min, burst 10); honor `Retry-After`.

Full surface in the [MCPJam API reference](/reference/public-api).

## LLM provider key — not an MCPJam key

A secret from an LLM provider (OpenAI, Anthropic, Google, xAI, Groq, Mistral, ...). MCPJam never issues this — it just uses the key you paste in to call that provider on your behalf.

**Where it lives:** **Settings → LLM Providers**.

**When it's used:** Every Playground chat against a provider model that isn't MCPJam's free hosted model.

**Errors:** "Invalid API key for OpenAI / Anthropic / ..." → wrong, expired, or for a different project. ("MCPJam model limit reached" is a [different problem](/troubleshooting/common-errors#mcpjam-model-limit-reached).)

You don't need one to run inspector probes (Tools, Resources, Prompts, OAuth Debugger, conformance) — those only talk to MCP servers.

## Playground BYOK — not an MCPJam key

A per-server bearer token for an HTTP MCP server that requires `Authorization: Bearer ...`. MCPJam forwards it to that one server only; it never authenticates you to MCPJam.

**Where it lives:** **Servers card → Headers field**, or `config.json` under `mcpServers.<name>.headers.Authorization` (e.g. `"Authorization": "Bearer ${MY_SERVER_TOKEN}"`).

**When it's used:** Every request the inspector makes to that MCP server.

**Errors:** [`auth/http_401`](/troubleshooting/error-codes#unauthorized-401), [`auth/http_403`](/troubleshooting/error-codes#forbidden-403) — the target server rejected the token.

## Project API Key (`mcpjam_…`) — retired

<Warning>
  **Retired.** Project API keys no longer work: existing keys no longer authenticate, new keys can't be generated, and the `/sdk/v1/evals/*` ingestion endpoints they protected return `410 Gone` for every request. Eval reporting now runs on the **MCPJam API key (`sk_…`)** through the [MCPJam API](/reference/public-api) — see [Saving Results](/sdk/concepts/saving-results).
</Warning>

This was a project-scoped token for `@mcpjam/sdk` / CLI traffic to the hosted backend, read from the `MCPJAM_API_KEY` environment variable. If your CI logs show `410` reporting errors, this retirement is why. **To fix CI:** upgrade `@mcpjam/sdk`, replace the `MCPJAM_API_KEY` value with an `sk_…` key from [Settings → API keys](https://app.mcpjam.com/settings/api-keys), and (optionally) set `MCPJAM_PROJECT_ID` to keep filing results under the same project — otherwise they land in your org's Default project. Old SDK versions' default auto-save path warns and continues, so eval runs themselves keep passing either way.

## Quick reference

| Key                | Format            | Issued by                                                       | Use it for                                            |
| ------------------ | ----------------- | --------------------------------------------------------------- | ----------------------------------------------------- |
| **MCPJam API key** | `sk_…`            | [Settings → API Keys](https://app.mcpjam.com/settings/api-keys) | The [MCPJam API](/reference/public-api) (`/api/v1/*`) |
| LLM provider key   | provider-specific | the LLM provider                                                | Playground chat — *third-party*                       |
| Playground BYOK    | server-specific   | your MCP server                                                 | Authenticating to one MCP server — *third-party*      |
| Project API Key    | `mcpjam_…`        | — (no longer issued)                                            | **Retired** — no longer works                         |


# MCPJam API
Source: https://docs.mcpjam.com/reference/public-api

Programmatic access to the MCP servers in your MCPJam projects — diagnostics, tool calls, prompt rendering, and asynchronous eval runs.

<Warning>
  **The MCPJam API is in preview.** It works today and we use it ourselves, but
  it has not been generally announced and the surface may change — including in
  breaking ways — while we finish the design. Pin your integration to the
  behaviors documented on this page, build a [tolerant
  reader](#versioning--stability), and expect to revisit it. Feedback is very
  welcome on [Discord](https://discord.gg/JEnDtz8X6z) or
  [GitHub](https://github.com/MCPJam/inspector/issues).
</Warning>

<Card title="Create an API key" icon="key" href="https://app.mcpjam.com/settings/api-keys">
  Go straight to key management in the hosted app. If you're signed out,
  you'll be asked to sign in and then land right back on the API keys page.
</Card>

The MCPJam API lets you operate the MCP servers saved in your
[hosted](/hosted/overview) projects — from CI, scripts, or your own agents —
without opening the UI.

What you can do with it today:

* **Discover your resources** — list your projects, their servers, eval
  suites, and chat sessions, so every ID the other routes need is
  self-serve
* **Validate a server** — connect, initialize, and capture a capability snapshot
* **Run the doctor** — the probe → connect → initialize → capabilities workflow
* **Check OAuth requirements** — does this server need an OAuth grant?
* **List tools, prompts, and resources** — the server's MCP primitives
* **Call a tool / render a prompt** — execute primitives and get the MCP result back verbatim
* **Read a resource** by URI
* **Export a full snapshot** — tools, resources, and prompts as one JSON document
* **Manage a project's clients** — list, read, **create** (from a built-in
  template like `claude`/`chatgpt`/`cursor`, or from a full config), edit,
  duplicate, and delete the named, reusable configurations that define how
  MCPJam connects to and talks to your MCP servers. Edits are compare-and-set:
  read the client, then send its `configId` back as `expectedConfigId` for a
  config change and its `name` back as `expectedName` for a rename — a rename
  does not rotate the config, so the config token cannot detect a concurrent
  one — and a concurrent edit is rejected rather than silently reverted
* **Manage a project's environments** — list, read, create, edit, archive, and
  restore the named execution bundles (host + optional server group + optional
  pinned skills and plugin versions) that eval suites and journeys run against,
  and **preview** what one resolves to before launching it
* **List a project's scenarios** — name, access mode, attached servers, and
  share link — and **read one scenario's settings**: model, system prompt,
  tool-approval policy, and resolved servers
* **Author eval suites** — create a runnable suite (name, default model, servers, test cases) synchronously and get a `suiteId` back; no run is started and no credits are spent
* **Run eval suites asynchronously** — create a run from a saved suite, get a `202` + `runId`
  immediately, then poll status, per-iteration results (tool calls, token
  usage, latency), and full traces. Runs appear live in the hosted UI,
  tagged `source: "api"`
* **Import OAuth tokens** — complete OAuth yourself (e.g. the SDK's
  `runOAuthLogin`) and push the tokens; subsequent calls inject and refresh
  them server-side
* **Run a headless agent turn** — send a message history, get back the
  assistant reply, the platform operations it invoked, references to any
  resources it created (eval suites), and token usage; the model is pinned
  server-side and billed to the project

All endpoints operate on servers you have already added to a project in the
hosted inspector. Managing API keys remains UI-only — see
[Not in the API yet](#not-in-the-api-yet).

## Base URL

```text theme={"theme":"css-variables"}
https://app.mcpjam.com/api/v1
```

The API is path-versioned. All routes on this page are relative to the base
URL above.

## Authentication

Every request needs an MCPJam API key in the `Authorization` header:

```bash theme={"theme":"css-variables"}
curl -X POST \
  "https://app.mcpjam.com/api/v1/projects/$PROJECT_ID/servers/$SERVER_ID/tools" \
  -H "Authorization: Bearer $MCPJAM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### Creating a key

1. Open [**Settings → API keys**](https://app.mcpjam.com/settings/api-keys)
   in the hosted app (the card at the top of this page takes you there
   directly).
2. Click **Create API key**, name it, and pick the **organization** it will act in.
3. Copy the key (`sk_…`) immediately — **it is shown exactly once** and never
   stored by MCPJam in retrievable form.

Keys can be revoked from the same page at any time. Revocation takes effect
immediately.

### Scope

A key is bound to one MCPJam organization at creation and acts **as you,
inside that organization**. Per-request authorization still applies: a call
only succeeds if the key's owner can access that project and server. A key
can never reach projects outside its organization.

<Note>
  Two deliberate restrictions while in preview:

  * **API keys cannot manage API keys.** Requests to the key-management surface
    with an `sk_…` bearer fail with `403 FORBIDDEN`. Create and revoke keys in
    the UI.
  * **Guest sessions cannot use the API.** Sign in to create keys.
</Note>

### Keep keys secret

Treat an API key like a password:

* Load it from an environment variable or secret manager — never commit it to
  source control or ship it in client-side code.
* Scope keys narrowly: one key per integration, named so you can tell them apart.
* Rotate by creating a replacement key, switching traffic, then revoking the old one.
* If a key leaks, revoke it immediately in
  [**Settings → API keys**](https://app.mcpjam.com/settings/api-keys).

If you see `401 UNAUTHORIZED` with `details.reason: "ORPHANED_KEY"`, the key
is no longer bound to an organization and cannot be used — create a new key
from [Settings](https://app.mcpjam.com/settings/api-keys).

## Conventions

**Requests.** Operations on servers are `POST` with a JSON body (most accept
an empty `{}`). **Reads** — the catalog listings and eval-run polling — are
`GET` and take their options (`cursor`, `limit`, filters) as query
parameters.

**Responses.** Three envelope shapes, used consistently:

| Kind            | Shape                                                                               |
| --------------- | ----------------------------------------------------------------------------------- |
| Single resource | The resource object directly                                                        |
| Collection      | `{ "items": [...], "nextCursor": "..." }` — `nextCursor` omitted on the last page   |
| Error           | `{ "code": "...", "message": "...", "details": {...} }` with a matching HTTP status |

**Pagination.** Collections are cursor-based. Pass the previous response's
`nextCursor` as `cursor` in the next request (body field on `POST` lists,
query parameter on `GET` lists). Cursors are opaque — don't parse them.

**Authoring vs running suites.** `POST /eval-suites` creates a runnable suite
(the suite plus its test cases) **synchronously** and responds `201` with the
`suiteId`, WITHOUT running anything — author once, run later. `POST /eval-runs`
is the async counterpart: it validates and creates a run synchronously, then
detaches execution and responds `202` with a `runId`. Poll
`GET /eval-runs/{runId}` until `status` is `completed`, `failed`, or
`cancelled`.

**IDs.** Every identifier the API takes is discoverable through the API
itself: `GET /projects` lists your projects, `GET /projects/{projectId}/servers`
lists each project's servers, and `GET /projects/{projectId}/eval-suites`
lists its eval suites. Start from `GET /me` to confirm which account a key
acts as.

## Errors

Errors always use the canonical body `{ code, message, details? }`. The `code`
is stable and machine-readable; `message` is human-readable and may change;
`details` is an optional, unstructured bag.

| Code                    | HTTP | Meaning                                                                                                                                                                                                     |
| ----------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UNAUTHORIZED`          | 401  | Missing, invalid, revoked, or orphaned key                                                                                                                                                                  |
| `OAUTH_REQUIRED`        | 401  | The **target MCP server** needs an OAuth grant — distinct from a bad key                                                                                                                                    |
| `FORBIDDEN`             | 403  | Key is valid but not allowed to do this — or the **target MCP server** rejected the credentials MCPJam presented (retrying with a different MCPJam key will not help; reconnect or re-authorize the server) |
| `VALIDATION_ERROR`      | 400  | Malformed body or parameters                                                                                                                                                                                |
| `NOT_FOUND`             | 404  | Unknown project, server, or resource                                                                                                                                                                        |
| `CONFLICT`              | 409  | The resource isn't in a state that accepts this write — a stale `expectedRevision`, a duplicate name, or an environment that can't currently be launched. Re-read and retry                                 |
| `FEATURE_NOT_SUPPORTED` | 422  | The target server doesn't support this MCP capability                                                                                                                                                       |
| `RATE_LIMITED`          | 429  | Too many requests — see [Rate limits](#rate-limits)                                                                                                                                                         |
| `SERVER_UNREACHABLE`    | 502  | Could not connect to the target MCP server                                                                                                                                                                  |
| `TIMEOUT`               | 504  | The target MCP server connected but didn't respond in time                                                                                                                                                  |
| `INTERNAL_ERROR`        | 500  | Something failed on our side                                                                                                                                                                                |

New error codes may be **added** over time; treat unknown codes as
non-retryable failures unless the HTTP status says otherwise.

## Rate limits

Each key gets **60 requests per minute** sustained, with bursts up to **10**.
Exceeding it returns `429 RATE_LIMITED` with a `Retry-After` header (in
seconds):

```http theme={"theme":"css-variables"}
HTTP/1.1 429 Too Many Requests
Retry-After: 7

{ "code": "RATE_LIMITED", "message": "API key rate limit exceeded. Slow down and retry." }
```

Honor `Retry-After`, add jittered exponential backoff, and expect these limits
to be tuned during the preview.

Other `429`s — the per-minute brakes and daily budgets described with the
endpoints below — carry `Retry-After` whenever the refusal knows when it
lifts, which is the ordinary case. Treat it as a hint that is usually there
rather than a guarantee: a refusal that reaches you without one still needs
your backoff, so **never** block waiting for a header that may not come.

## Endpoints

Each endpoint is fully documented — request and response schemas, examples,
and an interactive playground — under **Endpoints** in the sidebar.

**Catalog** — discover the IDs everything else takes:

| Endpoint                                                        | What it does                                                                                                                                |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /me`                                                       | The account behind the key                                                                                                                  |
| `GET /organizations`                                            | The organizations the caller belongs to — where `organizationId` comes from. A key is bound to one organization and only ever sees that one |
| `GET /projects`                                                 | Projects the key can access (optionally `?organizationId=`)                                                                                 |
| `GET /projects/{projectId}/servers`                             | The project's saved MCP servers                                                                                                             |
| `GET /projects/{projectId}/eval-suites`                         | The project's eval suites, with latest-run summaries                                                                                        |
| `GET`, `POST /projects/{projectId}/clients`                     | List clients, or create one from a template or full config                                                                                  |
| `GET`, `PATCH`, `DELETE /projects/{projectId}/clients/{client}` | Read, edit, or delete a client. The path segment takes a client **name or ID**                                                              |
| `POST /projects/{projectId}/clients/{client}/servers`           | Replace the client's server attachments (a replacement — omitted servers are detached)                                                      |
| `POST /projects/{projectId}/clients/{client}/duplicate`         | Copy a client's current config into a new client                                                                                            |
| `GET /chat-sessions`                                            | Chat sessions (`?projectId=&status=&limit=&before=`)                                                                                        |

**Project environments** — the named execution bundles (one host, optionally a
standalone server group, optionally pinned skills and plugin versions) that eval
suites and journeys run against. Reads need project membership; **every write
needs project admin**. Writes are revisioned: pass the `revision` you last read
back as `expectedRevision`, and a stale value returns `CONFLICT` (409) instead
of overwriting a concurrent edit.

| Endpoint                                                          | What it does                                                                                                          |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `GET`, `POST /projects/{projectId}/environments`                  | List environments (`?includeArchived=true` to include archived ones), or create one                                   |
| `GET`, `PATCH /projects/{projectId}/environments/{environmentId}` | Read one environment, or edit it (send `null` to clear `serverAttachmentId`, `skillSelection`, or `pluginVersionIds`) |
| `GET /projects/{projectId}/environments/{environmentId}/resolve`  | Preview the host config, closed server set, and pinned plugin versions this environment resolves to right now         |
| `POST /projects/{projectId}/environments/{environmentId}/archive` | Archive it — reversible, and frees the name for a new one                                                             |
| `POST /projects/{projectId}/environments/{environmentId}/restore` | Restore an archived environment                                                                                       |

Archive and restore are explicit sub-actions rather than a `DELETE` because the
row is kept and restore has real semantics: it re-checks the name and drops
plugin pins whose version no longer exists.

**Server diagnostics & primitives** (`/projects/{projectId}/servers/{serverId}/...`):

| Endpoint                       | What it does                                                                |
| ------------------------------ | --------------------------------------------------------------------------- |
| `POST .../validate`            | Connect, initialize, and capture a capability snapshot                      |
| `POST .../doctor`              | Full health workflow: probe → connect → initialize → capabilities           |
| `POST .../check-oauth`         | Does this server require an OAuth grant?                                    |
| `POST .../tools`               | List the server's tools                                                     |
| `POST .../tools/call`          | Execute a tool; returns the MCP `CallToolResult` plus additive `durationMs` |
| `POST .../prompts`             | List the server's prompts                                                   |
| `POST .../prompts/get`         | Render a prompt; returns the MCP `GetPromptResult` verbatim                 |
| `POST .../resources`           | List the server's resources                                                 |
| `POST .../resources/read`      | Read one resource by URI                                                    |
| `POST .../export`              | One JSON snapshot of tools, resources, and prompts                          |
| `POST .../oauth/import-tokens` | Store OAuth tokens you obtained yourself; later calls inject + refresh them |

**Eval runs** (`/projects/{projectId}/...`):

| Endpoint                                                   | What it does                                                                                                                                                                                                                                                                                                                                                       |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `POST .../eval-suites`                                     | Author-only: create a suite + its test cases from `name` + `serverIds` + a default `model` + `tests`, WITHOUT running it; responds `201` + `suiteId`. Each test's case body is an ordered `steps` array (prompt / toolCall / interact / assert); per-test `model`/`provider`/`runs` are optional and fall back to suite defaults                                   |
| `POST .../eval-runs`                                       | Create a run from a `suiteId` (rerun; `serverIds` optional — defaults to the suite's saved servers) or `suiteName` + inline `tests` + `serverIds` (new suite). `environmentId` (requires `suiteId`) runs against one of the suite's attached environments instead. Responds `202` + `runId`                                                                        |
| `PATCH .../eval-suites/{suiteId}`                          | Edit suite settings, including `environmentIds` — the project environments the suite runs against (non-empty array sets/replaces, `null` clears)                                                                                                                                                                                                                   |
| `PATCH .../eval-suites/{suiteId}/schedule`                 | Enable/disable scheduled runs; `environmentId` pins which environment they launch                                                                                                                                                                                                                                                                                  |
| `GET .../eval-runs/{runId}`                                | Run status, result, and summary — poll until terminal. Carries `gateWaiver`: the audited override in force over this run's gate, or `null`                                                                                                                                                                                                                         |
| `GET .../eval-runs/{runId}/gate`                           | The stored suite quality-gate report for this run (`passed` / `failed` / `non_gateable` / `not_configured`). `not_configured` is a 200 report, never a missing route. A deployment that predates the evaluator answers `422 FEATURE_NOT_SUPPORTED`                                                                                                                 |
| `GET .../eval-runs/{runId}/decision-summary`               | **Start here when a run did not pass.** The canonical run decision contract: the verdict and where it came from, counts labelled with the population they count, the run's own verdict decision, and per-trial diagnostics with the user-value chain, the first failed stage, the failure category, evidence scoped to that stage, and one next action (paginated) |
| `POST .../eval-runs/{runId}/gate-waivers`                  | Override a **failing** run's release gate until `expiresAt` (required, future, max 30 days out), with a `reason` (required, max 500 chars). Manage tier. Does NOT change the run's `result` — the run keeps its verdict and the waiver is reported alongside it. `409` when one is already in force, returning the existing waiver                                 |
| `GET .../eval-runs/{runId}/gate-waivers`                   | The waiver in force over a run, or `null`. Readable by anyone who can view the run                                                                                                                                                                                                                                                                                 |
| `DELETE .../eval-runs/{runId}/gate-waivers/{waiverId}`     | Revoke a waiver, putting the gate back. Idempotent — a second call reports the original revocation rather than restamping it                                                                                                                                                                                                                                       |
| `GET .../eval-runs/{runId}/iterations`                     | Per-iteration results: tool calls, token usage, latency (paginated)                                                                                                                                                                                                                                                                                                |
| `GET .../eval-runs/{runId}/iterations/{iterationId}/trace` | Full trace: messages + expected-vs-actual analysis                                                                                                                                                                                                                                                                                                                 |
| `GET .../eval-suites/{suiteId}/runs`                       | Recent runs for a suite, newest first                                                                                                                                                                                                                                                                                                                              |

**Eval result ingestion** (`/projects/{projectId}/eval-ingest/...`) — how `@mcpjam/sdk` saves results from eval runs executed outside the platform (local dev, CI). The `{projectId}` segment accepts the literal `default` for the key org's Default project. Most users never call these directly — set `MCPJAM_API_KEY` and the [SDK reporter](/sdk/concepts/saving-results) does:

| Endpoint                                    | What it does                                                          |
| ------------------------------------------- | --------------------------------------------------------------------- |
| `POST .../eval-ingest/report`               | One-shot: create a run from finished results and finalize it          |
| `POST .../eval-ingest/runs/start`           | Chunked flow: create (or idempotently reuse) a run by `externalRunId` |
| `POST .../eval-ingest/runs/iterations`      | Append an iteration batch to a started run                            |
| `POST .../eval-ingest/runs/finalize`        | Finalize a chunked run (idempotent)                                   |
| `POST .../eval-ingest/artifacts/upload-url` | Mint an upload URL for eval artifacts (JUnit XML, Jest/Vitest JSON)   |

**Agent** (`/projects/{projectId}/agent`) — a headless assistant turn for
building conversational surfaces (the MCPJam Slack app is the first consumer):

| Endpoint         | What it does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `POST .../agent` | Run ONE assistant turn over a `messages` history (`{role: "user"\|"assistant", content}` items, max 50 messages × 8 KB each, 96 KB total). The model is pinned server-side and billed to the project. The turn can read project servers/suites/runs and **create eval suites**; it can NOT start runs — kick those off yourself with `POST .../eval-runs`. Responds synchronously with `reply`, the `toolCalls` it made, `createdResources` (ids + app links), and token `usage`. Capped at 4 concurrent turns per organization and \~90 s wall clock. **Not idempotent — never blind-retry**: dedupe on your own trigger identity, and on an error response check `details.createdResources` (and, after a lost response, list suites) before retrying, or you'll create duplicates |

**Scenarios** (`/projects/{projectId}/scenarios...`) — read-only:

| Endpoint                         | What it does                                                                            |
| -------------------------------- | --------------------------------------------------------------------------------------- |
| `GET .../scenarios`              | List the project's published scenarios: name, access mode, attached servers, share link |
| `GET .../scenarios/{scenarioId}` | One scenario's settings: model, system prompt, tool-approval policy, resolved servers   |

<Note>
  **Swarms and User Testing are enabled per organization.** Everything in the
  four sections below is behind that gate, enforced server-side. If it is not
  on for your organization, the reads generally return empty and every **write**
  — creating a persona or journey, launching a run, publishing a scenario,
  requesting insights — answers `403` with a message naming the feature, not a
  `404` and not a validation error.

  Five operations are deliberately outside the gate, because every one of them
  *reduces* exposure and spend: cancelling a journey run, unpublishing a
  scenario, and archiving a persona, journey or swarm (each one's `DELETE`). An
  organization that loses access can still stop what is running and clean up
  what it authored.

  Ask us on [Discord](https://discord.gg/JEnDtz8X6z) if you want it turned on.
</Note>

**Swarms — authoring** (`/projects/{projectId}/personas...`, `.../journeys...`,
`.../swarms...`) — the definitions a swarm run executes. A **persona** is a
reusable synthetic character; a **journey** is the task you point one at; a
**swarm** is an authoring container holding defaults for the journeys made
under it. Nothing here starts anything — launching is
`POST .../journeys/{journeyId}/runs`.

| Endpoint                          | What it does                                                                                               |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `GET .../personas`                | List the project's personas                                                                                |
| `POST .../personas`               | Create a persona (accepts `Idempotency-Key`)                                                               |
| `GET .../personas/{personaId}`    | One persona                                                                                                |
| `PATCH .../personas/{personaId}`  | Update a persona. Journeys reference it by id, so renaming is safe                                         |
| `DELETE .../personas/{personaId}` | Delete a persona. **409** while journeys still reference it                                                |
| `GET .../journeys`                | List the project's journeys                                                                                |
| `POST .../journeys`               | Create a journey (accepts `Idempotency-Key`). `personaId` and `swarmId` must be in this project            |
| `GET .../journeys/{journeyId}`    | One journey                                                                                                |
| `PATCH .../journeys/{journeyId}`  | Update a journey. `null` CLEARS `environmentIds`; `sessionsPerTarget` and `maxTurns` must be sent together |
| `DELETE .../journeys/{journeyId}` | **Archive** a journey — existing runs keep pointing at the definition they executed                        |
| `GET .../swarms`                  | List swarm containers                                                                                      |
| `POST .../swarms`                 | Create a swarm container (accepts `Idempotency-Key`)                                                       |
| `GET .../swarms/{swarmId}`        | One swarm                                                                                                  |
| `PATCH .../swarms/{swarmId}`      | Update the container's defaults. Not a cascade — existing journeys keep their own values                   |
| `DELETE .../swarms/{swarmId}`     | **Archive** a swarm. The journeys under it stay launchable                                                 |

**Swarms — generation** — drafts a model writes for you. Both routes return
candidates and **persist nothing**: feed what you want to keep to the create
routes above. That is also why neither accepts an `Idempotency-Key` — a call
with no effect has nothing to de-duplicate, and offering one would imply the
drafts are stable across retries when they are not.

| Endpoint                     | What it does                                                                                        |
| ---------------------------- | --------------------------------------------------------------------------------------------------- |
| `POST .../personas/generate` | Draft a slate of personas (and journeys for each) grounded in a server attachment or an environment |
| `POST .../journeys/generate` | Draft journeys for a persona passed **by value**, so you can draft before you save                  |

<Warning>
  The generation routes **spend** — they run models on your organization's
  account. They are metered two ways, and the two mean different waits: a
  per-minute burst brake (`429`, retry in seconds) and your plan's daily
  budget (`429`, resets at UTC midnight). Both normally carry `Retry-After` —
  honor it rather than guessing, and fall back to your own backoff on the
  refusal that arrives without one.
</Warning>

**Swarms — runs** (`/projects/{projectId}/journeys/{journeyId}/runs`,
`.../journey-runs/...`) — launching a journey and reading what it produced:

| Endpoint                                | What it does                                                                                |
| --------------------------------------- | ------------------------------------------------------------------------------------------- |
| `GET .../journeys/{journeyId}/runs`     | That journey's runs, newest first                                                           |
| `POST .../journeys/{journeyId}/runs`    | **Launch.** Answers `202` — the fan-out has started, not finished. Send `Idempotency-Key`   |
| `GET .../journey-runs/{runId}`          | The full run: counts, per-target breakdown, per-session `attempts`                          |
| `GET .../journey-runs/{runId}/sessions` | The chat sessions the run produced (summaries, not transcripts)                             |
| `POST .../journey-runs/{runId}/cancel`  | Stop a run. Idempotent — a second cancel is `200` with `alreadyCanceled: true`, not a `409` |

Three things worth knowing before you write the client:

* **Always send `Idempotency-Key` on a launch.** It spends model credits, so a
  retry of a dropped response must not run the journey twice. Replaying a key
  returns the original run with `deduped: true` and starts no second runner —
  and consumes no quota. Omitting the header is read as a request that declined
  to identify itself, and gets a fresh run every time.
* **Check `canceled` before reporting a failure.** A stopped run carries
  `status: "failed"`, because cancellation is recorded as a marker rather than
  a status of its own. `stale: true` is the third case: the runner went silent
  and the watchdog settled the run.
* **`targetId`, not `hostId`, identifies a target.** Two environments can
  resolve to the same host with different servers, so a run can hold two
  targets that share a `hostId`.

<Warning>
  Launching **spends**: a run fans out into `targets × sessionsPerTarget`
  sessions. Two different `429`s can come back — the per-minute burst brake
  (retry in seconds) and your plan's daily launch cap (resets at UTC midnight).
  Both normally carry `Retry-After`; back off on your own when one does not.
</Warning>

**Swarms — insights** — what a run revealed. Read these in order of cost: the
roll-up and the scorecard are deterministic and free, the findings registry
aggregates them, and wave insights run a model.

| Endpoint                                          | What it does                                                                |
| ------------------------------------------------- | --------------------------------------------------------------------------- |
| `GET .../journeys-overview`                       | Project roll-up: recent runs, goal-completion rates, repeat failures, trend |
| `GET .../journey-runs/{runId}/scorecard`          | One run's deterministic rubric result. No model, no spend                   |
| `GET .../journey-findings`                        | Criteria that keep failing, tracked across waves                            |
| `POST .../journey-findings/{findingId}/dismiss`   | Hide a finding                                                              |
| `POST .../journey-findings/{findingId}/undismiss` | Un-hide it                                                                  |
| `GET .../waves/{waveId}/insights`                 | Poll a wave's generated insights. **404 = nobody asked**                    |
| `POST .../waves/{waveId}/insights`                | Request them. `202` — scheduled, not done                                   |
| `DELETE .../waves/{waveId}/insights`              | Cancel a generation still `pending`                                         |

Two arithmetic traps worth naming, because both produce numbers that look
plausible and are wrong:

* **Divide by `sessionsGraded`, never by the session total.** Three failures
  out of four graded sessions, in a run that attempted forty, is 75% — not
  7.5%.
* **`failedGradingCount` is not `failCount`.** A crashed judge is not a
  regression, and folding the two together makes one look like the other.

And one modelling note: a finding's `status` is a lifecycle —
`new | recurring | regressed | resolved`. `dismissed` is *not* one of them.
Dismissal is the orthogonal `dismissedAt`, so a finding can be both recurring
and dismissed, and hiding one never claims it stopped happening.

<Warning>
  Requesting wave insights **spends**, and draws on the `insightsPerDay`
  ledger that is **shared** with eval-run and user-testing insights — burning
  it here takes it from there too. Poll the `GET` rather than re-requesting;
  `force: true` deliberately spends a second time.

  Three refusals, three different meanings: a short `429` is the per-minute
  brake (wait seconds), a `429` naming `insightsPerDay` is the daily ledger
  (its `Retry-After`, when present, counts to UTC midnight), and a `403` means
  the feature is not available to your organization — waiting will never help.
</Warning>

**User testing** (`/projects/{projectId}/environments/{environmentId}/scenario`,
`.../user-testing/scenarios/...`) — publishing an environment for real visitors,
reading what they did, and controlling who can reach it. A **scenario** is a
published environment: one per environment, addressed by its own id once it
exists.

| Endpoint                                                                   | What it does                                                                                                          |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `PUT .../environments/{environmentId}/scenario`                            | **Publish.** Idempotent — republishing returns the existing scenario, and `created: false` says so. Project **admin** |
| `DELETE .../environments/{environmentId}/scenario`                         | **Unpublish.** The link dies and live sessions end. `deleted: false` when there was nothing published                 |
| `PATCH .../user-testing/scenarios/{scenarioId}`                            | Rename, re-describe, or change `mode`                                                                                 |
| `GET .../user-testing/scenarios/{scenarioId}/sessions`                     | What visitors did: one row per session, with feedback and a first-message preview                                     |
| `GET .../user-testing/scenarios/{scenarioId}/sessions/{sessionId}`         | One session's transcript, paged. Tool payloads and blobs are dropped                                                  |
| `PUT .../user-testing/scenarios/{scenarioId}/guest-execution`              | The spend dial for anonymous visitors: daily credits, computer starts, concurrency                                    |
| `POST .../user-testing/scenarios/{scenarioId}/rotate-link`                 | **Mint a new share link.** No body — the new secret is minted server-side                                             |
| `PUT .../user-testing/scenarios/{scenarioId}/members`                      | Invite someone (or change their role) on an `invited_only` scenario                                                   |
| `DELETE .../user-testing/scenarios/{scenarioId}/members/{memberIdOrEmail}` | Remove someone. Their grant is revoked, ending their live sessions                                                    |
| `POST .../user-testing/scenarios/{scenarioId}/rebind`                      | Point the scenario at a different environment in the same project                                                     |

<Warning>
  **Rotating the link does not evict anyone who already used it.** It stops the
  old URL from granting NEW access; the grants already redeemed under it stay
  valid. That is the single most important thing to know here, because rotation
  is what you reach for when a link leaks — and on its own it does not undo the
  leak. Rotate *and* remove the members you did not intend to have.

  What DOES take effect at once, ending live sessions rather than waiting for
  expiry: unpublishing, tightening `mode`, and removing a member. Those bump
  the scenario's `accessVersion`, and a session minted under an older version
  stops working. None of them reverse by calling the opposite — a re-invited
  member gets a new session, not their old one.

  Read `accessVersion` off the publish and rebind responses. The update and
  rotate responses do not carry it.
</Warning>

**User testing — what visitors did** — the analytics reads, and the model pass
over them. Read these in cost order: metrics, usage and signals are
deterministic and free; the insights request runs a model.

| Endpoint                                                                      | What it does                                                                                               |
| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `GET .../user-testing/scenarios/{scenarioId}`                                 | The scenario, plus its insights envelope when you may have it                                              |
| `GET .../user-testing/scenarios/{scenarioId}/metrics`                         | Volumes, completion and feedback rates. `?population=real\|synthetic` splits real visitors from swarm runs |
| `GET .../user-testing/scenarios/{scenarioId}/usage`                           | Which models, tools and servers the sessions used. **Check `scan.truncated`** before quoting a rate        |
| `GET .../user-testing/scenarios/{scenarioId}/signals`                         | What the miner found in the current window — and where you get the `windowId`                              |
| `GET .../user-testing/scenarios/{scenarioId}/findings`                        | Every finding raised over this scenario, dismissed ones included                                           |
| `GET .../user-testing/scenarios/{scenarioId}/windows/{windowId}/insights`     | One window's insights. **`404` means nobody asked**, which is not the same as `pending`                    |
| `POST .../user-testing/scenarios/{scenarioId}/insights`                       | **Spends.** `202` with the `windowId` to poll. `409` if the window has not been mined yet                  |
| `DELETE .../user-testing/scenarios/{scenarioId}/insights`                     | Release a generation stuck `pending`. Takes `windowId` in the body                                         |
| `POST .../user-testing/scenarios/{scenarioId}/findings/{findingId}/dismiss`   | Dismiss a finding. Keyed on its stable id, so it stays dismissed when insights regenerate                  |
| `POST .../user-testing/scenarios/{scenarioId}/findings/{findingId}/undismiss` | Undo that                                                                                                  |

The five analytics shapes above are documented as **open objects** on purpose.
Their upstream projections grow with the product and the SDK types them the
same way, so pinning a field list would turn every new metric into a spec
violation. Read what you recognize; ignore the rest.

<Warning>
  Requesting insights **spends**, and draws on the same `insightsPerDay` ledger
  as eval-run and swarm wave insights — one budget, three producers. Poll the
  window read rather than re-requesting; `force: true` deliberately spends a
  second time, which is why it is never inferred from anything else.
</Warning>

**Planning** — one read that answers "may I?" before you try:

| Endpoint                                    | What it does                                                                         |
| ------------------------------------------- | ------------------------------------------------------------------------------------ |
| `GET .../projects/{projectId}/capabilities` | The caller's role, the beta gate's state, plan limits, and the booleans to branch on |

<Note>
  `capabilities` is a **planning aid, not a gate**. Every write still enforces
  independently, so a `true` here that races a flag flip costs you a clean
  `403` rather than an incorrect success — and nothing should consult it
  *instead of* trying. It exists because agent surfaces are static: an MCP tool
  catalog is built with no organization in hand, so the alternative is
  attempting the write and reading the failure, by which point the agent has
  usually already told someone what it was about to do.
</Note>

The same surface is available as an [OpenAPI
specification](https://raw.githubusercontent.com/MCPJam/inspector/main/docs/reference/openapi.json)
for Postman, Swagger UI, or client generation.

## Run evals from the API

The shortest useful loop — create a run with one inline test, then poll:

```bash theme={"theme":"css-variables"}
# 1. Create (responds 202 immediately; -f + jq -e fail fast on errors)
RUN_ID=$(curl -fsS -X POST \
  "https://app.mcpjam.com/api/v1/projects/$PROJECT_ID/eval-runs" \
  -H "Authorization: Bearer $MCPJAM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "suiteName": "smoke",
    "serverIds": ["'$SERVER_ID'"],
    "tests": [{
      "title": "echo works",
      "runs": 1,
      "model": "anthropic/claude-haiku-4.5",
      "provider": "anthropic",
      "steps": [
        { "id": "s1", "kind": "prompt", "prompt": "Use the echo tool to say hi" },
        {
          "id": "s2",
          "kind": "assert",
          "assertion": {
            "type": "toolCalledWith",
            "toolName": "echo",
            "args": { "args": { "text": "hi" } }
          }
        }
      ]
    }]
  }' | jq -er .runId)

# 2. Poll until status is completed | failed | cancelled
curl -s "https://app.mcpjam.com/api/v1/projects/$PROJECT_ID/eval-runs/$RUN_ID" \
  -H "Authorization: Bearer $MCPJAM_API_KEY"

# 3. If it did not pass, read the decision summary FIRST
curl -s "https://app.mcpjam.com/api/v1/projects/$PROJECT_ID/eval-runs/$RUN_ID/decision-summary" \
  -H "Authorization: Bearer $MCPJAM_API_KEY"

# 4. Or go straight to per-iteration tool calls, token usage, latency
curl -s "https://app.mcpjam.com/api/v1/projects/$PROJECT_ID/eval-runs/$RUN_ID/iterations" \
  -H "Authorization: Bearer $MCPJAM_API_KEY"
```

The decision summary composes steps 2 and 4 into one versioned object so every
client reads a run the same way. Four things to know before gating on it:

* **`measurementUnit` ships with every count.** Under verdict policy v2 the
  counts are case-execution *variants*, with repetitions as trials inside them;
  on a legacy run they are trials. The same suite reports a different total
  under each, so a count without its unit is not a fact.
* **It explains the verdict; it never re-decides it.** The run's own decision is
  the authority and is carried through on `decision`. The per-trial
  `diagnostics` sit underneath it — a case can pass with a failing trial in it,
  so tallying diagnostics gives a different answer than the platform reached.
* **`verdict: "notEstablished"` is not a failure and not `inconclusive`.** It
  means no verdict exists at all (unfinished, stopped early, or undecidable);
  `undecided.reason` says which. `inconclusive` *is* a decision — the validity
  phase ran and withheld a verdict.
* **`diagnostics.complete` says whether you have the whole list.** A page
  fetched with a `cursor` is never complete, and `scannedIterations` separates
  "nothing failed" from "we did not look".

Evidence is scoped to the claim it supports: for a measured failure the span
ids, prompt indexes and reasons come from the first failed stage's row alone,
and `evidence.tracePath` is an API-relative path you can append to your base URL
to read the full trace.

`model` takes an id from the hosted catalog — `provider/name` form, e.g.
`anthropic/claude-haiku-4.5` — and runs on your organization's credits.
Provider-native ids (e.g. `claude-sonnet-4-5`) are bring-your-own-key: pass
the key in `modelApiKeys`. A model the API can't execute is rejected at
create time with `VALIDATION_ERROR`; `details.hostedModels` lists the valid
hosted ids for that provider.

Reruns are even shorter: `{ "suiteId": "..." }` reruns the suite exactly as
configured, connecting the suite's saved server selection (the `202` response
lists the resolved `servers`). Pass `serverIds` to override the selection; a
suite with no saved selection requires it (`VALIDATION_ERROR` with
`details.reason: "NO_SAVED_SERVER_SELECTION"` otherwise).

### Running against a project environment

A [project environment](#endpoints) is launchable only through a suite that has
it **attached**. Attach them first with
`PATCH /projects/{projectId}/eval-suites/{suiteId}` and an `environmentIds`
array (send `null` to detach them all; `[]` is rejected — use `null`), then
pass `environmentId` on the run. It must be one of that suite's attached
environments; anything else is a `400` with
`details.reason: "ENVIRONMENT_NOT_ATTACHED"`, raised before any case is
authored or any server connected.

Omitting `environmentId` is meaningful, and depends on the suite:

* **no** attached environments → the run uses the saved server selection, as before;
* **exactly one** attached → that environment is used automatically. The `202`
  response's `environment` field says so;
* **several** attached → `400` with `details.reason: "ENVIRONMENT_REQUIRED"`,
  naming the candidates.

The environment supplies the closed server set — including the servers its
pinned plugin versions contribute — so `serverIds` is not required and is
**rejected** alongside it (`400`), rather than accepted and ignored: honoring
both would connect a different set than the run is stamped with. The same goes
for a `serverIds` override on an environment-based suite
(`details.reason: "ENVIRONMENT_SERVERS_NOT_OVERRIDABLE"`).

The run is pinned to the environment revision resolved at launch: if the
environment changes in between, the run is rejected with `CONFLICT` (409)
rather than executing against a different configuration than the one whose
tools were captured. Use
`GET /projects/{projectId}/environments/{environmentId}/resolve` to see what it
will connect before launching.

Every run records which environment it used. `GET .../eval-runs/{runId}` (and
the run listings) return `environment: { id, name, revision }` — read from the
run's immutable snapshot, not the suite's current attachments — or `null` for a
run that used a saved server selection.

Per-organization
concurrency is capped (default **2** concurrent runs); exceeding it returns
`429` with `details.reason: "CONCURRENT_RUN_LIMIT"` — wait for an active run
to finish.

If a server answers `401 OAUTH_REQUIRED`, complete the OAuth flow yourself
(the SDK's `runOAuthLogin` handles interactive, headless, and
client-credentials flows) and push the result to
`POST .../oauth/import-tokens` once. Subsequent calls inject the stored token
and refresh it server-side.

## Not in the API yet

To set expectations while in preview, these are **not** available over the API
today (most exist in the hosted inspector UI):

* Creating or revoking API keys (UI-only by design — see [Authentication](#authentication))
* Chat and conformance suites
* Browser-based OAuth flows initiated by the API (use
  `oauth/import-tokens` after completing OAuth yourself)

If one of these blocks you, tell us on
[Discord](https://discord.gg/JEnDtz8X6z) — it directly shapes what we
stabilize first.

## Versioning & stability

* The API is **path-versioned** (`/api/v1`). When it reaches general
  availability, breaking changes will require a new version path.
* **During the preview**, breaking changes to v1 may still happen; we'll note
  them in the [changelog](/changelog/overview).
* **Additive changes** — new endpoints, new optional request fields, new
  response fields, new error codes — are considered non-breaking and can ship
  at any time. Write clients that ignore unknown fields.
* Error **codes** are stable identifiers; error **messages** are not.

### Browser tools on chat sessions

<Note>
  Hosted browser availability depends on the account and deployment. The browser workspace must be enabled to open the live pane in the Playground.
</Note>

`POST /chat-sessions/messages` accepts an optional `browser: { policy?, profileId? }`. Attach it on every turn that should browse. The initial grant and profile are fixed; the current host policy and pinned read-only mode can narrow the effective permissions. Responses and traces include the read-only `chatSessionId`, browser metadata, and screenshot URLs without inline image data.

| Method | Path                                          | Purpose                                                                                                                      |
| ------ | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| POST   | `/chat-sessions/browser`                      | Create an empty API session and its browser with `projectId`, `policy`, optional `profileId`, and a stable `idempotencyKey`. |
| POST   | `/chat-sessions/{sessionId}/browser/open`     | Open or wake the session browser.                                                                                            |
| POST   | `/chat-sessions/{sessionId}/browser/command`  | Execute a browser command under the session policy with a stable `commandId`.                                                |
| POST   | `/chat-sessions/{sessionId}/browser/note`     | Append a command-ledger note.                                                                                                |
| POST   | `/chat-sessions/{sessionId}/browser/trace`    | Read command receipts and screenshot evidence.                                                                               |
| POST   | `/chat-sessions/{sessionId}/browser/artifact` | Resolve a command screenshot URL.                                                                                            |
| POST   | `/chat-sessions/{sessionId}/browser/close`    | Release the session desktop.                                                                                                 |

Only the owning user can drive an API session's browser. All inputs use the public `sessionId`; conversation and browser identifiers in responses are informational. Capacity refusals before execution release the turn lease so the same idempotency key can be retried. An execution with an unknown outcome must be inspected before attempting a new action.


# Connecting to MCP Servers
Source: https://docs.mcpjam.com/sdk/concepts/connecting-servers

Learn how to connect to local and remote MCP servers

The `MCPClientManager` is your gateway to MCP servers. It handles connections, manages multiple servers, and provides a unified interface for calling tools, reading resources, and accessing prompts.

## When to Use MCPClientManager

Use `MCPClientManager` when you need to:

* Connect to one or more MCP servers
* Execute tools programmatically (without an LLM)
* Build applications that aggregate tools from multiple servers
* Set up test environments for your MCP server

## Two Types of Servers

MCP servers come in two transport types:

### STDIO Servers (Local)

STDIO servers run as subprocesses on your machine. You provide a command and arguments, and the SDK spawns the process.

```typescript theme={"theme":"css-variables"}
import { MCPClientManager } from "@mcpjam/sdk";

const manager = new MCPClientManager({
  myServer: {
    command: "node",
    args: ["./my-server.js"],
    env: {
      API_KEY: process.env.MY_API_KEY,
    },
  },
});
```

**Best for:**

* Local development
* Servers you're building
* CLI-based MCP servers
* Servers requiring local resources

<Note>
  Unconfigured stdio connections auto-negotiate the MCP era: the SDK probes `server/discover` on a sibling process to detect 2026 support, then falls back to the legacy `initialize` handshake if the server doesn't respond. This means your server binary may be spawned a second time during the probe. The probe is bounded (at most one extra spawn) and only occurs when no explicit protocol version pin is set.
</Note>

### HTTP Servers (Remote)

HTTP servers connect via SSE (Server-Sent Events) or Streamable HTTP over the network.

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  asana: {
    url: "https://mcp.asana.com/sse",
    requestInit: {
      headers: {
        Authorization: `Bearer ${process.env.ASANA_TOKEN}`,
      },
    },
  },
});
```

**Best for:**

* Production MCP servers
* Third-party integrations (Asana, Slack, etc.)
* Shared team servers
* Serverless deployments

### HTTP Authentication Options

For HTTP servers, `MCPClientManager` supports four auth patterns:

* `accessToken` for a static bearer token
* `refreshToken` + `clientId` for non-interactive OAuth token exchange with automatic refresh
* `authProvider` when you need to provide a custom MCP SDK OAuth provider
* `onUnauthorized` for custom 401 recovery when you manage tokens externally

If you already have a refresh token, you can let the SDK handle access token exchange and refresh for you:

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  asana: {
    url: "https://mcp.asana.com/mcp",
    refreshToken: process.env.ASANA_REFRESH_TOKEN!,
    clientId: process.env.ASANA_CLIENT_ID!,
    clientSecret: process.env.ASANA_CLIENT_SECRET,
  },
});
```

With this configuration, the SDK:

* exchanges the refresh token for an access token during connection
* automatically retries with a fresh access token when the server returns an auth challenge
* stores rotated refresh tokens returned by the authorization server

<Note>
  `refreshToken` is only supported for HTTP servers. When you use it, do not also set `accessToken`, `authProvider`, or an `Authorization` header in `requestInit.headers`.
</Note>

#### Custom 401 recovery with onUnauthorized

Use `onUnauthorized` when you supply an `accessToken` but manage token rotation yourself — for example, when tokens come from a secrets vault or a backend service rather than a standard OAuth flow:

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  myServer: {
    url: "https://mcp.example.com/mcp",
    accessToken: currentToken,
    onUnauthorized: async ({ serverId }) => {
      // Call your own token endpoint to get a fresh token
      const { accessToken } = await myTokenService.refresh(serverId);
      return { accessToken };
    },
  },
});
```

When the server returns HTTP 401, the SDK:

1. Calls `onUnauthorized` once to obtain a fresh access token.
2. Strips any stale `Authorization` header from `requestInit`.
3. Rebuilds the transport with the new token and retries the operation.
4. If the retry also returns 401, the error is surfaced to the caller.

<Note>
  `onUnauthorized` only fires on strict HTTP 401 responses — not 403. It is also skipped when `refreshToken` or `authProvider` is configured, since those patterns handle token refresh internally. Parallel 401s for the same server share a single refresh call.
</Note>

## Connecting and Disconnecting

After configuring servers, you must explicitly connect:

```typescript theme={"theme":"css-variables"}
// Connect to a specific server
await manager.connectToServer("myServer");

// Now you can use it
const tools = await manager.listTools("myServer");
console.log("Available tools:", tools.tools.map((tool) => tool.name));

// Always disconnect when done
await manager.disconnectServer("myServer");
```

<Note>
  Always disconnect servers when you're done, especially in tests. This cleans up subprocess handles and network connections.
</Note>

## Working with Multiple Servers

`MCPClientManager` can manage multiple servers at once:

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  math: {
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-everything"],
  },
  asana: {
    url: "https://mcp.asana.com/sse",
    requestInit: {
      headers: { Authorization: `Bearer ${token}` },
    },
  },
  github: {
    url: "https://mcp.github.com/sse",
    requestInit: {
      headers: { Authorization: `Bearer ${ghToken}` },
    },
  },
});

// Connect to all
await Promise.all([
  manager.connectToServer("math"),
  manager.connectToServer("asana"),
  manager.connectToServer("github"),
]);

// Aggregate tools from all servers
const allTools = await manager.getTools();
// Now allTools contains tools from math, asana, and github
```

## Executing Tools Directly

You can call tools without involving an LLM, useful for unit tests:

```typescript theme={"theme":"css-variables"}
// Direct tool execution
const result = await manager.executeTool("math", "add", { a: 5, b: 3 });
console.log(result); // 8

// Test your server's tools deterministically
expect(result).toBe(8);
```

## Integration with HostRunner

The most common pattern is connecting servers, then creating a `HostRunner` with their tools:

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  myServer: { command: "node", args: ["./server.js"] },
});

await manager.connectToServer("myServer");

// Get tools for HostRunner
const tools = await manager.getTools();

// Create agent with those tools
const agent = new HostRunner({
  tools,
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

// Now prompt the agent
const result = await agent.run("Add 2 and 3");
```

## Error Handling

Connection failures throw errors. Wrap in try/catch for production code:

```typescript theme={"theme":"css-variables"}
try {
  await manager.connectToServer("myServer");
} catch (error) {
  console.error("Failed to connect:", error.message);
  // Handle gracefully - maybe fall back to another server
}
```

## Health Checks

Verify a server is responsive:

```typescript theme={"theme":"css-variables"}
try {
  await manager.pingServer("myServer");
} catch {
  console.warn("Server not responding, reconnecting...");
  await manager.disconnectServer("myServer");
  await manager.connectToServer("myServer");
}
```

`pingServer` is era-aware: on a modern (2026-07-28) connection it probes with `server/discover`; on a legacy connection it uses `ping`. Callers don't need to handle the difference.

## Next Steps

<CardGroup>
  <Card title="Testing with LLMs" icon="bot" href="/sdk/concepts/testing-with-llms">
    Connect LLMs to your MCP tools
  </Card>

  <Card title="MCPClientManager Reference" icon="book" href="/sdk/reference/mcp-client-manager">
    Full API documentation
  </Card>
</CardGroup>


# Testing Across Providers
Source: https://docs.mcpjam.com/sdk/concepts/multi-provider

Compare your MCP server's performance across different LLMs

Different LLMs interpret tool descriptions differently. A tool that works perfectly with Claude might struggle with GPT-4, or vice versa. Testing across providers ensures your MCP server works reliably for all users.

## Why Test Multiple Providers?

Your users connect to MCP servers from various clients:

* Claude Desktop (Anthropic)
* ChatGPT plugins (OpenAI)
* Cursor (various models)
* Custom apps (any provider)

Each LLM has different:

* Tool calling capabilities
* Interpretation of descriptions
* Handling of complex arguments
* Response patterns

## Supported Providers

The SDK supports 10 providers out of the box:

| Provider       | Model Format           | Example                                                |
| -------------- | ---------------------- | ------------------------------------------------------ |
| Anthropic      | `anthropic/model`      | `anthropic/claude-sonnet-4-20250514`                   |
| OpenAI         | `openai/model`         | `openai/gpt-4o`                                        |
| Google         | `google/model`         | `google/gemini-1.5-pro`                                |
| Azure          | `azure/model`          | `azure/gpt-4o`                                         |
| Amazon Bedrock | `bedrock/model`        | `bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| Mistral        | `mistral/model`        | `mistral/mistral-large-latest`                         |
| DeepSeek       | `deepseek/model`       | `deepseek/deepseek-chat`                               |
| Ollama         | `ollama/model`         | `ollama/llama3`                                        |
| OpenRouter     | `openrouter/org/model` | `openrouter/anthropic/claude-3-opus`                   |
| xAI            | `xai/model`            | `xai/grok-beta`                                        |

## Comparing Providers

Create agents for each provider and run the same tests:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager({
  myServer: { command: "node", args: ["./server.js"] },
});
await manager.connectToServer("myServer");

const tools = await manager.getTools();

// Test definition (reused across providers)
const additionTest = new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 2 and 3");
    return r.hasToolCall("add");
  },
});

// Providers to test
const providers = [
  { model: "anthropic/claude-sonnet-4-20250514", key: "ANTHROPIC_API_KEY" },
  { model: "openai/gpt-4o", key: "OPENAI_API_KEY" },
  { model: "google/gemini-1.5-pro", key: "GOOGLE_GENERATIVE_AI_API_KEY" },
];

// Run across all providers
for (const { model, key } of providers) {
  const apiKey = process.env[key];
  if (!apiKey) continue;

  const agent = new HostRunner({ tools, model, apiKey });
  await additionTest.run(agent, { iterations: 20 });

  console.log(`${model}: ${(additionTest.accuracy() * 100).toFixed(1)}%`);
}
```

## Provider Comparison Script

A complete script for benchmarking:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalSuite, EvalTest } from "@mcpjam/sdk";

async function compareProviders() {
  // Setup
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");
  const tools = await manager.getTools();

  // Build test suite
  const suite = new EvalSuite({ name: "Tool Selection" });

  suite.add(new EvalTest({
    id: "c_add",
    name: "add",
    test: async (a) => (await a.run("Add 2+3")).hasToolCall("add"),
  }));

  suite.add(new EvalTest({
    id: "c_echo",
    name: "echo",
    test: async (a) => (await a.run("Echo 'hello'")).hasToolCall("echo"),
  }));

  // Providers
  const providers = [
    { name: "Claude", model: "anthropic/claude-sonnet-4-20250514", key: "ANTHROPIC_API_KEY" },
    { name: "GPT-4o", model: "openai/gpt-4o", key: "OPENAI_API_KEY" },
    { name: "Gemini", model: "google/gemini-1.5-pro", key: "GOOGLE_GENERATIVE_AI_API_KEY" },
  ];

  const results: Record<string, number> = {};

  for (const { name, model, key } of providers) {
    const apiKey = process.env[key];
    if (!apiKey) {
      console.log(`⏭️  ${name}: Skipped (no API key)`);
      continue;
    }

    const agent = new HostRunner({ tools, model, apiKey, temperature: 0.1 });

    console.log(`🧪 Testing ${name}...`);
    await suite.run(agent, { iterations: 20, concurrency: 3 });

    results[name] = suite.accuracy();
  }

  // Report
  console.log("\n📊 Results:");
  console.log("─".repeat(30));

  for (const [name, accuracy] of Object.entries(results)) {
    const bar = "█".repeat(Math.round(accuracy * 20));
    console.log(`${name.padEnd(10)} ${bar} ${(accuracy * 100).toFixed(1)}%`);
  }

  await manager.disconnectServer("everything");
}

compareProviders();
```

## Custom Providers

Add your own OpenAI or Anthropic-compatible endpoints:

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "my-provider/gpt-4",
  apiKey: process.env.MY_API_KEY,
  customProviders: {
    "my-provider": {
      name: "my-provider",
      protocol: "openai-compatible",
      baseUrl: "https://api.my-provider.com/v1",
      modelIds: ["gpt-4", "gpt-3.5-turbo"],
    },
  },
});
```

### LiteLLM Proxy

Test many models through a single proxy:

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "litellm/gpt-4",
  apiKey: process.env.LITELLM_API_KEY,
  customProviders: {
    litellm: {
      name: "litellm",
      protocol: "openai-compatible",
      baseUrl: "http://localhost:8000",
      modelIds: ["gpt-4", "claude-3-sonnet", "gemini-pro"],
      useChatCompletions: true,
    },
  },
});
```

## Interpreting Results

When comparing providers, look for:

### Consistent High Performance

All providers score >90%? Your tool descriptions are clear and well-documented.

### One Provider Struggling

If Claude works but GPT-4 doesn't, your descriptions might use Claude-specific patterns. Review and generalize.

### All Providers Struggling

Low accuracy across the board suggests ambiguous tool names or descriptions. Improve your MCP server's documentation.

### High Variance

If the same provider gives 70% one run and 95% the next, try:

* Lower temperature
* More iterations
* Clearer prompts in tests

## Next Steps

<CardGroup>
  <Card title="LLM Providers Reference" icon="book" href="/sdk/reference/llm-providers">
    All providers and configuration options
  </Card>

  <Card title="Running Evals" icon="chart-bar" href="/sdk/concepts/running-evals">
    Statistical evaluation basics
  </Card>
</CardGroup>


# Running Evals
Source: https://docs.mcpjam.com/sdk/concepts/running-evals

Measure tool accuracy with statistical evaluations

LLMs are probabilistic, so the same prompt might produce different results each time. Running a test once tells you it *can* work, not that it *reliably* works. Evals run the same test many times and measure statistical accuracy.

## Why Run Evals?

A single test pass can be misleading:

* The LLM might get lucky on one attempt
* Temperature introduces randomness
* Different phrasings might fail where others succeed

Running 30+ iterations gives you confidence in real-world performance:

* "This tool is called correctly 97% of the time"
* "Arguments are correct in 90% of cases"
* "Average latency is 1.2 seconds"

## EvalTest: Single Test Scenario

`EvalTest` runs one test function multiple times:

```typescript theme={"theme":"css-variables"}
import { EvalTest } from "@mcpjam/sdk";

const test = new EvalTest({
  id: "c_addition_accuracy",
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add"); // Return true/false
  },
});

const result = await test.run(agent, { iterations: 30 });

console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
console.log(`${result.successes}/${result.iterations} passed`);
```

### Writing Test Functions

Test functions receive an `HostExecutor` (implemented by `HostRunner` and mock agents) and return `boolean`:

```typescript theme={"theme":"css-variables"}
// Simple: did the right tool get called?
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
}

// Detailed: check arguments too
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
}

// Complex: multi-step workflow
test: async (agent) => {
  const r1 = await agent.run("Create a project called 'Test'");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
}
```

### Run Options

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,      // How many times to run
  concurrency: 5,      // Parallel runs (careful with rate limits)
  retries: 2,          // Retry failures
  timeoutMs: 30000,    // Per-test timeout
  mcpjam: {
    // Auto-save is enabled when MCPJAM_API_KEY is available
    suiteName: "SDK eval smoke",
    strict: false, // warn by default; true to fail CI on upload errors
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
});
```

### Metrics

After running, access various metrics:

```typescript theme={"theme":"css-variables"}
test.accuracy();           // Success rate (0.0 - 1.0)
test.precision();          // Precision metric
test.recall();             // Recall metric
test.averageTokenUse();    // Avg tokens per iteration
```

## EvalSuite: Multiple Tests

Group related tests together:

```typescript theme={"theme":"css-variables"}
import { EvalSuite, EvalTest } from "@mcpjam/sdk";

const suite = new EvalSuite({ name: "Math Operations" });

suite.add(new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 5 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  id: "c_multiplication",
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 6");
    return r.hasToolCall("multiply");
  },
}));

suite.add(new EvalTest({
  id: "c_division",
  name: "division",
  test: async (agent) => {
    const r = await agent.run("Divide 20 by 4");
    return r.hasToolCall("divide");
  },
}));

const result = await suite.run(agent, { iterations: 30 });
console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);
```

## Save Results to MCPJam

Both `EvalTest` and `EvalSuite` can automatically save results to MCPJam when a run completes. Set `MCPJAM_API_KEY` to an **MCPJam API key (`sk_…`)** from Settings → API keys and results are saved automatically:

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});
```

Results are filed under the project in `MCPJAM_PROJECT_ID` (or the `project` option), defaulting to your organization's **Default** project. For manual save APIs, CI metadata, and artifact uploads, see the [Save Results to MCPJam](/sdk/concepts/saving-results) guide.

### Tool errors vs. `passed` on upload

By default, when results are **built from traces** (auto-save and reporter helpers), a failed tool execution (MCP `isError`, errored tool spans, or error tool-results in messages) sets `passed: false` for that iteration even if your test function returned `true`. That keeps CI pass rate aligned with real server behavior.

To disable that for a run (for example you only assert which tools were called, not that every call succeeded), set `failOnToolError: false` on the `mcpjam` block:

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
    failOnToolError: false,
  },
});
```

See [Eval reporting — Tool execution and `passed`](/sdk/reference/eval-reporting#tool-execution-and-passed) for full detail and [`createEvalRunReporter`](/sdk/reference/eval-reporting#createevalrunreporter) overrides.

### Predicate gate

Predicates express richer pass/fail criteria than tool-call matching — without an LLM judge. Each predicate is a pure function of the iteration transcript, so it produces the same verdict every time, which makes it safe to use as a CI release gate.

Predicates are authored in the corpus JSON (the Inspector's per-case **Checks** editor writes the same shape) and threaded through the runner automatically; there is no predicate field to set on individual uploaded results. A case passes iff every **gating** predicate passes — an advisory one is recorded and never fails the case, and neither does one that could not be scored. An absent or empty list means no predicate gate.

The built-in predicate types. **Measures** is the link of the [user-value chain](/sdk/concepts/user-value-chain) the check is filed at — nothing files at Connection or Discovery, which the runner decides for you:

| Type                            | Measures   | What it checks                                                                      |
| ------------------------------- | ---------- | ----------------------------------------------------------------------------------- |
| `toolCalledWith`                | Selection  | A specific tool was called with matching args (partial/exact/ignore mode)           |
| `toolCalledAtLeastOnce`         | Selection  | A tool was called at least once                                                     |
| `toolNeverCalled`               | Selection  | A forbidden tool was never called                                                   |
| `onlyToolsCalled`               | Selection  | Every tool called was in an allowed set; an empty set claims no tool was called     |
| `firstToolWas`                  | Selection  | The first tool call in the transcript was a specific tool                           |
| `responseContains`              | User value | The final assistant message contains a substring                                    |
| `responseMatches`               | User value | The final assistant message matches a regex                                         |
| `noToolErrors`                  | Response   | No tool produced an error (MCP `isError` or transport failure)                      |
| `finalAssistantMessageNonEmpty` | User value | The final assistant message is non-empty                                            |
| `tokenBudgetUnder`              | User value | Total token usage is strictly under a budget                                        |
| `widgetRendered`                | User value | At least one MCP App widget rendered successfully (optionally scoped to one tool)   |
| `widgetRenderLatencyUnder`      | User value | Every rendered widget mounted in strictly under a millisecond budget                |
| `widgetNoConsoleErrors`         | User value | No widget render captured console errors                                            |
| `turnCountUnder`                | User value | The iteration used strictly fewer than N user turns                                 |
| `noEndingQuestion`              | User value | **Observation.** The final message does not end with a question. Warn/Report only   |
| `toolLatencyUnder`              | Response   | Every observed call settled under a millisecond ceiling                             |
| `toolResultContains`            | Response   | A tool result contains a substring                                                  |
| `toolResultMatchesSchema`       | Response   | Every tool result matches an authored JSON Schema (any JSON root)                   |
| `toolResultSizeUnder`           | Response   | Every tool result is under a byte ceiling, measured before our storage cap          |
| `argumentsMatchToolSchema`      | Tool call  | Every call's arguments validate against the tool's declared inputSchema             |
| `noRepeatedIdenticalCall`       | Tool call  | **Observation.** No identical call repeated back-to-back. Warn/Report only          |
| `toolCallCountUnder`            | Selection  | Strictly fewer than N tool calls                                                    |
| `toolCalledBefore`              | Selection  | Every call to one tool was preceded by a call to another                            |
| `noDeprecatedToolCalled`        | Selection  | **Observation.** No called tool's description marks it deprecated. Warn/Report only |
| `noDestructiveToolCalled`       | Selection  | No called tool declares `destructiveHint`                                           |
| `toolErrorNamesInput`           | Response   | **Observation.** Tool errors name an input key or a sent value. Warn/Report only    |
| `fullPageHasContinuation`       | Response   | **Observation.** Full pages carry continuation metadata. Warn/Report only           |

When an iteration records no widget render observations, the three `widget*` predicates fail rather than pass — a missing signal counts as a failure. See [Predicate gate reference](/sdk/reference/eval-reporting#predicate-gate) for the full type definitions and a code example, and [The user-value chain](/sdk/concepts/user-value-chain) for what a set of checks leaves unmeasured.

### Suite Results

```typescript theme={"theme":"css-variables"}
// Overall accuracy
console.log(`Suite: ${(suite.accuracy() * 100).toFixed(1)}%`);

// Per-test breakdown
for (const test of suite.getAll()) {
  console.log(`  ${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}

// Get specific test
const addTest = suite.get("addition");
console.log(`Addition accuracy: ${addTest.accuracy()}`);
```

## Choosing Iteration Count

| Scenario         | Iterations | Why                              |
| ---------------- | ---------- | -------------------------------- |
| Quick smoke test | 10         | Fast feedback during development |
| Regular testing  | 30         | Good statistical significance    |
| Pre-release      | 50-100     | High confidence before shipping  |
| Benchmarking     | 100+       | Comparing models or changes      |

## Best Practices

### Use Low Temperature

More deterministic results for testing:

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  // ...
  temperature: 0.1,
});
```

### Handle Rate Limits

Reduce concurrency for rate-limited APIs:

```typescript theme={"theme":"css-variables"}
await suite.run(agent, {
  iterations: 30,
  concurrency: 2, // Avoid hitting rate limits
});
```

### Test Edge Cases

Don't just test the happy path:

```typescript theme={"theme":"css-variables"}
suite.add(new EvalTest({
  id: "c_handles_empty_input",
  name: "handles-empty-input",
  test: async (agent) => {
    const r = await agent.run("Add numbers"); // No numbers given
    return !r.hasError(); // Should handle gracefully
  },
}));

suite.add(new EvalTest({
  id: "c_handles_large_numbers",
  name: "handles-large-numbers",
  test: async (agent) => {
    const r = await agent.run("Add 999999999 and 1");
    return r.hasToolCall("add");
  },
}));
```

### Set Quality Thresholds

Fail CI if accuracy drops below a threshold:

```typescript theme={"theme":"css-variables"}
await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`Accuracy ${suite.accuracy()} below 90% threshold`);
  process.exit(1);
}
```

## Bring your own host

`HostRunner` is the right starting point when you just need an LLM + tools + a model string. For richer setups — pinning a host style/profile, applying sandbox/permission policy, or running the same configuration in CI that the MCPJam Inspector uses for its Playground — start from a `Host` spec instead:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, Host, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager();
await manager.connectToServer("everything", {
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-everything"],
});

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-5.6-sol",
  systemPrompt: "You are a helpful assistant.",
}).requireServer("everything");

// Bind the spec to the live manager. apiKey lives on the runtime, not per-call.
const runtime = host.withManager(manager, {
  apiKey: process.env.OPENAI_API_KEY!,
});

const test = new EvalTest({
  id: "c_add",
  name: "add",
  test: async (runner) => (await runner.run("Add 2+3")).hasToolCall("add"),
});
await test.run(runtime, { iterations: 10 });
```

`HostRuntime` (returned by `host.withManager(...)`) snapshots the live `Host` on every `.run()`, so mutating the bound host between iterations takes effect on the next iteration. `HostRunner` snapshots once at construction. Pick `HostRunner` for a static spec, `HostRuntime` when you want the spec editable across calls.

**Personal computer.** A `Host` can declare a personal cloud workstation via the `computer` property (type `HostComputer`, exported from `@mcpjam/sdk`). When set, the scenario surfaces a `bash` tool and a web terminal for signed-in members. Use `setComputer()` to attach or detach it fluently:

```typescript theme={"theme":"css-variables"}
import { Host, type HostComputer } from "@mcpjam/sdk";

// Attach with the default shape
const host = new Host({ style: "mcpjam", model: "openai/gpt-5.6-sol" })
  .setComputer();

// Attach with an initial working directory
host.setComputer({ kind: "personal", toolset: "bash", workdir: "/srv/app" });

// Detach
host.setComputer(null);
```

Or pass it directly in the constructor:

```typescript theme={"theme":"css-variables"}
const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-5.6-sol",
  computer: { kind: "personal", toolset: "bash" },
});
```

`null` and an absent `computer` field hash identically, so existing host configs are unaffected. The `computer` field is automatically stripped from eval wire configs — evals never target personal computers.

**Statelessness.** `HostRuntime.run()` does NOT auto-replay history across turns. Use `PromptOptions.context` to thread context explicitly:

```typescript theme={"theme":"css-variables"}
const r1 = await runtime.run("Who am I?");
const r2 = await runtime.run("List my projects", { context: [r1] });
```

**Eval reporting (Stage 5).** When you upload results to MCPJam and the reporter can resolve a host snapshot, it sends the canonical host config alongside the run (`{ hostConfig, hostConfigHash }`) so the persisted row in the Evals UI shows the exact host you ran with — the v1 ingest surface always accepts the pair. The snapshot comes from `executor.getHostSnapshot?.()`, falling back to `MCPJamReportingConfig.host`; result-only uploads with neither source omit the pair. Uploaded results don't yet carry a per-iteration `hostSnapshot` field, so the pair stays run-level for now.

## Generate evals from the Inspector

You can also generate eval code from the MCPJam Inspector. Click **⋮ → Copy markdown for evals** on any server card, then paste it into an LLM. See the [Quickstart](/sdk/quickstart#generate-evals-from-the-inspector) for details.

## Next Steps

<CardGroup>
  <Card title="Testing Across Providers" icon="cpu" href="/sdk/concepts/multi-provider">
    Compare performance across LLMs
  </Card>

  <Card title="EvalTest Reference" icon="book" href="/sdk/reference/eval-test">
    Full EvalTest API
  </Card>

  <Card title="EvalSuite Reference" icon="book" href="/sdk/reference/eval-suite">
    Full EvalSuite API
  </Card>

  <Card title="Save Results to MCPJam" icon="cloud-upload" href="/sdk/concepts/saving-results">
    Auto-save, manual APIs, CI metadata, and artifact upload
  </Card>
</CardGroup>


# Save Results to MCPJam
Source: https://docs.mcpjam.com/sdk/concepts/saving-results

Save eval results to the MCPJam dashboard with an MCPJam API key (sk_…) for tracking and CI visibility

<Warning>
  Reporting authenticates with **MCPJam API keys (`sk_…`)** from Settings → API keys. The legacy project API keys (`mcpjam_…`) are retired and no longer work anywhere — see [API keys](/reference/api-keys).
</Warning>

After running evals, you can save results to MCPJam to track accuracy over time, compare across branches, and get visibility in the CI Evals dashboard.

<Frame>
  <img alt="CI Runs overview" />
</Frame>

## Setup

Create an MCPJam API key (`sk_…`) under **Settings → API keys** and export it:

```bash theme={"theme":"css-variables"}
export MCPJAM_API_KEY=sk_...
# Optional: file results under a specific project (defaults to your org's Default project).
export MCPJAM_PROJECT_ID=<project id>
```

That's it. Both `EvalTest` and `EvalSuite` auto-save results when this key is available. Results land in the project named by `MCPJAM_PROJECT_ID` (or the `project` option in the `mcpjam:` config), falling back to your organization's **Default** project.

After each successful upload the SDK prints a link to the run it just created, so you can open it straight from the terminal:

```text theme={"theme":"css-variables"}
[mcpjam/sdk] View run: https://app.mcpjam.com/evals/suite/jd7.../runs/kn2...?project=jh4...
```

One line per run, on by default. See [the printed run URL](/sdk/reference/eval-reporting#the-printed-run-url) for when the `?project=` param is omitted and when nothing prints at all.

**Attach the connected `MCPClientManager` to [`HostRunner`](/sdk/reference/host-runner)** (or pass `agent` / `mcpClientManager` to manual reporting APIs) when you need either of the following:

1. **MCP App / widget replay in Evals traces** — After each MCP App tool call, the agent uses the manager’s [`readResource`](/sdk/reference/mcp-client-manager#readresource) to fetch HTML from the tool’s `ui.resourceUri` and fills [`widgetSnapshots`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) on [`PromptResult`](/sdk/reference/prompt-result). Without the manager, traces still upload (messages + spans) but **widgets will not replay** in the dashboard. The tool’s JSON result alone is not enough for offline iframe replay.
2. **Replay credentials (authenticated HTTP MCP)** — The SDK can persist server connection details for debugging and reruns. You do not need to build a second secret object yourself; replay config is derived automatically when the agent or manager is attached.

## Auto-Save from EvalTest

When `MCPJAM_API_KEY` is set, `EvalTest.run()` automatically saves results:

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});
```

**Tool execution:** Auto-saved payloads use the same **pass/fail rules** as the [eval reporting reference](/sdk/reference/eval-reporting#tool-execution-and-passed): failed tool calls default to `passed: false` unless you set `failOnToolError: false` on the `mcpjam` object.

For authenticated HTTP servers:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner } from "@mcpjam/sdk";

const manager = new MCPClientManager({
  asana: {
    url: process.env.MCP_SERVER_URL!,
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
    clientSecret: process.env.MCP_CLIENT_SECRET,
  },
});

await manager.connectToServer("asana");

const agent = new HostRunner({
  tools: await manager.getTools(),
  model: "openai/gpt-5.6-sol",
  apiKey: process.env.OPENAI_API_KEY!,
  mcpClientManager: manager,
});

await test.run(agent, {
  iterations: 30,
  mcpjam: { suiteName: "Asana Eval" },
});
```

To disable auto-save for a specific run:

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,
  mcpjam: { enabled: false },
});
```

## Auto-Save from EvalSuite

Suites can be configured at construction or run time:

```typescript theme={"theme":"css-variables"}
const suite = new EvalSuite({
  name: "Math Operations",
  mcpjam: {
    suiteName: "Math Eval",
    ci: {
      branch: process.env.GITHUB_REF_NAME,
      commitSha: process.env.GITHUB_SHA,
      runUrl: `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`,
    },
  },
});
```

When a suite runs, individual `EvalTest` auto-saves are suppressed to avoid duplicate uploads. The suite consolidates all test results into a single run.

## Manual Save APIs

For more control — custom test runners, CI post-steps, or framework-agnostic flows — the SDK provides dedicated APIs:

```typescript theme={"theme":"css-variables"}
import {
  reportEvalResults,
  reportEvalResultsSafely,
  createEvalRunReporter,
  uploadEvalArtifact,
} from "@mcpjam/sdk";

// 1) One-shot save (strict — throws on failure)
await reportEvalResults({
  suiteName: "Nightly",
  mcpClientManager: manager,
  results: [{ caseTitle: "healthcheck", passed: true }],
});

// 2) One-shot save (safe — returns null on failure)
const result = await reportEvalResultsSafely({
  suiteName: "Nightly",
  results: [{ caseTitle: "healthcheck", passed: true }],
});

// 3) Incremental reporter (long-running processes)
// Assumes `agent` was created with `mcpClientManager: manager`
const reporter = createEvalRunReporter({
  suiteName: "Incremental",
  agent,
});
await reporter.record({ caseTitle: "step-1", passed: true });
await reporter.record({ caseTitle: "step-2", passed: false, error: "timeout" });
const output = await reporter.finalize();

// 4) Artifact upload (JUnit XML, Jest JSON, Vitest JSON)
await uploadEvalArtifact({
  suiteName: "JUnit import",
  format: "junit-xml",
  artifact: junitXmlString,
});
```

`reportEvalResults()` and `createEvalRunReporter()` resolve replay credentials in this order:

1. `serverReplayConfigs` if you pass it explicitly
2. `agent.getServerReplayConfigs()`
3. `mcpClientManager.getServerReplayConfigs()`

Most users should pass `agent` or `mcpClientManager` and let the SDK derive replay credentials automatically. Use `serverReplayConfigs` only as an advanced override.

When replay configs are inferred from `agent` or `mcpClientManager`, the SDK limits them to the `serverNames` you attach to the run when `serverNames` is provided.

**Manual reporters (Vitest/Jest hooks):** Pass `agent` or `mcpClientManager` into [`createEvalRunReporter`](/sdk/reference/eval-reporting#createevalrunreporter) — not only on `HostRunner` — and call **`await reporter.finalize()` before `await manager.disconnectAllServers()`** so replay config is still available at upload time. See [Replay metadata for the MCPJam UI](/sdk/reference/eval-reporting#replay-metadata-for-the-mcpjam-ui).

## CI Metadata

Attach CI/CD context to your eval runs for traceability in the dashboard:

```typescript theme={"theme":"css-variables"}
await reportEvalResults({
  suiteName: "Nightly",
  results: [...],
  ci: {
    provider: "github",
    branch: process.env.GITHUB_REF_NAME,
    commitSha: process.env.GITHUB_SHA,
    runUrl: `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`,
    pipelineId: process.env.GITHUB_WORKFLOW,
    jobId: process.env.GITHUB_JOB,
  },
});
```

Each iteration records the expected and actual tool calls side by side, along with the model's reasoning trace, so you can pinpoint exactly why a test passed or failed:

<Frame>
  <img alt="Test case detail view" />
</Frame>

## Next Steps

<CardGroup>
  <Card title="Running Evals" icon="chart-bar" href="/sdk/concepts/running-evals">
    Learn about EvalTest, EvalSuite, and iteration strategies
  </Card>

  <Card title="Saving Results Reference" icon="book" href="/sdk/reference/eval-reporting">
    Full API reference for all saving and reporting methods
  </Card>
</CardGroup>


# Testing with LLMs
Source: https://docs.mcpjam.com/sdk/concepts/testing-with-llms

Use HostRunner to run prompts and inspect tool calls

The `HostRunner` connects an LLM to your MCP tools, letting you test how models interact with your server. It handles the agentic loop (prompt → tool call → result → response) and gives you rich inspection capabilities.

## Why Test with LLMs?

Unit tests verify your tools work correctly in isolation. But in production, an LLM decides *which* tool to call and *what arguments* to pass. Testing with real LLMs catches issues like:

* Ambiguous tool descriptions that confuse the model
* Missing or unclear parameter documentation
* Tools that the model consistently misuses
* Edge cases in natural language interpretation

## Creating a HostRunner

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner } from "@mcpjam/sdk";

// Connect to your MCP server
const manager = new MCPClientManager({
  myServer: {
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-everything"],
  },
});
await manager.connectToServer("myServer");

// Create an agent with your tools
const agent = new HostRunner({
  tools: await manager.getTools(),
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
  mcpClientManager: manager,
});
```

Pass the same **`mcpClientManager`** instance when you use **MCP App** tools and upload evals to MCPJam: that enables [`getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) (HTML from `readResource` on `ui.resourceUri`) so traces can replay widgets offline. See the [HostRunner reference](/sdk/reference/host-runner).

## Running Prompts

Send natural language prompts and inspect what happens:

```typescript theme={"theme":"css-variables"}
const result = await agent.run("What is 15 plus 27?");

// What did the model say?
console.log(result.text);
// "15 plus 27 equals 42."

// What tools did it call?
console.log(result.toolsCalled());
// ["add"]

// What arguments did it pass?
console.log(result.getToolArguments("add"));
// { a: 15, b: 27 }
```

## The PromptResult Object

Every prompt returns a `PromptResult` with rich inspection methods:

```typescript theme={"theme":"css-variables"}
const result = await agent.run("...");

// Tool inspection
result.toolsCalled();           // string[] - names of tools called
result.hasToolCall("add");      // boolean - was this tool called?
result.getToolCalls();          // detailed tool call info
result.getToolArguments("add"); // arguments for a specific tool

// Performance
result.e2eLatencyMs();          // total time
result.llmLatencyMs();          // time in LLM API
result.mcpLatencyMs();          // time executing tools
result.totalTokens();           // tokens used

// Error handling
result.hasError();              // did something go wrong?
result.getError();              // error message if so

// MCP Apps (when agent was created with mcpClientManager)
result.getWidgetSnapshots();    // HTML snapshots for MCPJam trace replay
```

<Info>
  `HostRunner` never throws exceptions. Errors are captured in the result, making it safe to run many tests without try/catch blocks.
</Info>

## Multi-Turn Conversations

Pass previous results as context to maintain conversation history:

```typescript theme={"theme":"css-variables"}
// First turn
const r1 = await agent.run("Create a task called 'Buy groceries'");

// Second turn (model sees the first exchange)
const r2 = await agent.run("Mark it as high priority", {
  context: r1,
});

// Third turn (model sees both previous exchanges)
const r3 = await agent.run("Now show me all my tasks", {
  context: [r1, r2],
});
```

This is essential for testing workflows that span multiple interactions.

## Tuning Agent Behavior

### System Prompt

Guide the model's behavior:

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
  systemPrompt: "You are a task management assistant. Always confirm before deleting tasks.",
});
```

### Temperature

Control randomness (lower = more deterministic):

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  // ...
  temperature: 0.1, // More consistent for testing
});
```

### Max Steps

Limit the agentic loop iterations:

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  // ...
  maxSteps: 5, // Stop after 5 tool calls
});
```

## Control Multi-Step Loops with stopWhen

Use `stopWhen` when you want to stop the multi-step loop after a particular step completes:

```typescript theme={"theme":"css-variables"}
import { hasToolCall } from "@mcpjam/sdk";

// Stop after the step where the tool is called
const result = await agent.run("Search for open tasks", {
  stopWhen: hasToolCall("search_tasks"),
});

expect(result.hasToolCall("search_tasks")).toBe(true);
```

<Tip>
  `stopWhen` does not skip tool execution. It controls whether the prompt loop continues after the current step completes, and `HostRunner` also applies `stepCountIs(maxSteps)` as a safety guard.
</Tip>

## Bound Prompt Runtime with timeout

Use `timeout` when you want to bound how long `HostRunner.run()` can run:

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Run a long workflow", {
  timeout: { totalMs: 10_000, stepMs: 2_500 },
});

if (result.hasError()) {
  console.error(result.getError());
}
```

<Tip>
  `timeout` accepts `number`, `totalMs`, `stepMs`, and `chunkMs`. In practice, `number` and `totalMs` cap the full prompt, `stepMs` caps each step, and `chunkMs` mainly matters in streaming flows. The runtime creates an internal abort signal, so tools can stop early if their implementation respects the provided `abortSignal`.
</Tip>

## Writing Assertions

Use validators to assert tool call behavior:

```typescript theme={"theme":"css-variables"}
import { matchToolCalls, matchToolCallWithArgs } from "@mcpjam/sdk";

const result = await agent.run("Add 10 and 5");

// Check the right tool was called
expect(matchToolCalls(["add"], result.toolsCalled())).toBe(true);

// Check the arguments were correct
expect(
  matchToolCallWithArgs("add", { a: 10, b: 5 }, result.getToolCalls())
).toBe(true);
```

## Performance Insights

Understand where time is spent:

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Run a complex operation");

const latency = result.getLatency();
console.log(`Total: ${latency.e2eMs}ms`);
console.log(`  LLM: ${latency.llmMs}ms (${(latency.llmMs/latency.e2eMs*100).toFixed(0)}%)`);
console.log(`  Tools: ${latency.mcpMs}ms (${(latency.mcpMs/latency.e2eMs*100).toFixed(0)}%)`);

// Token usage for cost estimation
console.log(`Tokens: ${result.totalTokens()}`);
```

## Next Steps

<CardGroup>
  <Card title="Running Evals" icon="chart-bar" href="/sdk/concepts/running-evals">
    Run statistical evaluations
  </Card>

  <Card title="HostRunner Reference" icon="book" href="/sdk/reference/host-runner">
    Full API documentation
  </Card>

  <Card title="PromptResult Reference" icon="book" href="/sdk/reference/prompt-result">
    All result methods
  </Card>

  <Card title="Validators Reference" icon="book" href="/sdk/reference/validators">
    Assertion functions
  </Card>
</CardGroup>


# The User-Value Chain
Source: https://docs.mcpjam.com/sdk/concepts/user-value-chain

Which link of the chain each check you write measures

A run of one eval case walks six links, in order. MCPJam reports where it stopped being good, rather than a single pass or fail, because the six answer different questions and have different owners.

| # | Link           | The question it answers                                     | What good looks like           |
| - | -------------- | ----------------------------------------------------------- | ------------------------------ |
| 1 | **Connection** | Could the client reach the server and initialize a session? | Session connected              |
| 2 | **Discovery**  | Did the client receive usable primitives and metadata?      | Tools and resources discovered |
| 3 | **Selection**  | Did the model choose the right tool for the request?        | Right tool selected            |
| 4 | **Tool call**  | Was the call made with usable arguments?                    | Valid call made                |
| 5 | **Response**   | Did the server return data the model could use?             | Usable response returned       |
| 6 | **User value** | Was the user's actual request satisfied?                    | Request satisfied              |

Most documentation describes the chain as something you read after a run. This page is the other direction: when you write a check, you are choosing a link.

## Which links you can write a check for

Four of the six. Every [predicate](/sdk/reference/eval-reporting#predicate-gate) you author is filed at one link, and the count per link is uneven:

| Link       | Checks you can author |
| ---------- | --------------------- |
| Connection | none                  |
| Discovery  | none                  |
| Selection  | 9                     |
| Tool call  | 2                     |
| Response   | 7                     |
| User value | 9                     |

**Connection and Discovery have no authorable checks, and this is deliberate.** The runner decides both before your case runs, from what it observed setting the run up: whether `initialize` completed, whether `tools/list` answered, and — when a connection failed — whether an egress canary proves the network path out of the runner works at all. That last one is why there is nothing to author. From inside the runner, "their server is down" and "our egress is broken" look identical, and a check you wrote could not tell them apart. Absent positive evidence that our own side worked, the honest answer is that the link was not measured, not that the server failed.

So a suite with checks on every kind still measures Connection and Discovery exactly as much as a suite with none.

## The stage is where the evidence is filed

A check's link says where its evidence is recorded. It does not assert that a failure originated there.

`noToolErrors` is the clearest case. It files at **Response**, because a tool error is the server's answer. Until analyzer version 11 it filed at User value, and the same defect was counted twice: the analyzer already failed Response on an observed tool error, while the predicate row failed User value. Which link a reader saw as the first break depended on which row they looked at first.

Two consequences worth keeping in mind:

* A failure at one link is frequently caused upstream of it. Selection failing because two tools have near-identical descriptions is a Discovery problem wearing a Selection label.
* Moving a check to a different link changes where historical failures are attributed, so it is a versioned analyzer change rather than an edit anyone can make locally. The three `widget*` kinds are current candidates to move to Response.

## Graders that are not predicates

Three graders file at a link without being checks you write in a list:

| Grader                | Link       | What it is                                                                                                                       |
| --------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Tool-call matcher     | Selection  | The `expectedToolCalls` comparison, configured by [match options](/sdk/reference/validators) rather than authored as a predicate |
| Goal completion judge | User value | The hosted LLM judge scoring whether the request was satisfied                                                                   |
| Groundedness judge    | User value | Advisory only, run on demand; it has no score definition and never decides a link                                                |

A **gating** `toolCalledWith` is promoted into the matcher's expectations and graded there. An **advisory** one stays a predicate row, because promoting it would create an expectation that can fail the trial, which an advisory check must never do.

## What a suite is not measuring

Coverage is the question the six links exist to answer, and it is easy to write a plausible-looking suite that leaves most of the chain untouched.

A case with `toolCalledWith`, `noToolErrors` and `responseContains` measures Selection, Response and User value. It says nothing about Tool call — whether the arguments the model sent were valid against the tool's own schema — and nothing about Connection or Discovery, which no check can reach.

To cover Tool call, add `argumentsMatchToolSchema`. To read Connection and Discovery, read the run's chain rather than authoring anything: the stage rows report what the runner observed.

Two habits keep coverage honest:

* Read the chain on a passing run, not only a failing one. Six links reading `notMeasured` is not the same as six links passing, and only one of those is worth shipping on.
* Treat a link with no check as unmeasured rather than fine. `notMeasured` is an absence, and absence is not a pass.

## Observations never decide a link

Five kinds are marked **Observation** in the reference tables: `noEndingQuestion`, `noRepeatedIdenticalCall`, `noDeprecatedToolCalled`, `toolErrorNamesInput` and `fullPageHasContinuation`.

Each is a heuristic that can be right about what it saw and wrong about what it means. A poll loop and a wasteful retry are the same shape; a full page is not proof that more results exist; "Rate limited. Retry in 30 seconds." names no input key and is a good error message. So they carry one rule everywhere: `role: "advisory"` is required, a gating one is refused when written, and they are recorded beside a verdict without changing it.

They still file at a link, which is what places them in the right group when you are reading what a suite measures. They do not make that link pass or fail.

## Where to go next

* [Predicate gate reference](/sdk/reference/eval-reporting#predicate-gate) — every kind, its fields, and the link it measures
* [Running evals](/sdk/concepts/running-evals) — authoring a suite and reading its results
* [Validators](/sdk/reference/validators) — the tool-call matcher that files at Selection


# @mcpjam/sdk
Source: https://docs.mcpjam.com/sdk/index

MCP server unit testing, end-to-end (e2e) testing, and server evals

The [**@mcpjam/sdk**](https://www.npmjs.com/package/@mcpjam/sdk) is a TypeScript SDK for testing, evaluating, and building applications with MCP (Model Context Protocol) servers. It gives you the tools to check your MCP server works reliably across different LLMs and environments.

## Who is this for?

* **MCP Server Developers** - Test your tools work correctly with real LLMs
* **MCP Client Developers** - Build robust multi-server applications
* **App Marketplace Maintainers** - Evaluate server quality and compatibility
* **SDK Developers** - Integrate MCP capabilities into your products

## What can you do with it?

<CardGroup>
  <Card title="Unit Testing" icon="flask-conical">
    Test MCP primitives (tools, resources, prompts) deterministically without LLM calls
  </Card>

  <Card title="End-to-End Testing" icon="refresh-cw">
    Simulate real user interactions by connecting LLMs to your MCP servers
  </Card>
</CardGroup>

CI-friendly conformance is built into the same package:

* [Protocol Conformance SDK](/sdk/reference/protocol-conformance)
* [OAuth Conformance SDK](/sdk/reference/oauth-conformance)
* [MCP Apps Conformance SDK](/sdk/reference/apps-conformance)

Use the suite classes inside Jest or Vitest, then convert results into shared JSON or JUnit XML reporters with `toConformanceReport(...)` and `renderConformanceReportJUnitXml(...)`.

## Installation

```bash theme={"theme":"css-variables"}
npm install @mcpjam/sdk
```

## Quick Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalTest } from "@mcpjam/sdk";

// Connect to your MCP server
const manager = new MCPClientManager({
  myServer: {
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-everything"],
  },
});

// Create a runner with LLM + MCP tools
const runner = new HostRunner({
  tools: await manager.getToolsForAiSdk(),
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

// Run a prompt and inspect results
const result = await runner.run("Add 2 and 3");
console.log(result.toolsCalled());     // ["add"]
console.log(result.e2eLatencyMs());    // 1234

// Run statistical evaluation
const test = new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (runner) => {
    const r = await runner.run("Add 2+3");
    return r.hasToolCall("add");
  },
});
await test.run(runner, { iterations: 30 });
console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
```

<Note>
  **Renames in 1.11.** `TestAgent` → `HostRunner`, `EvalAgent` → `HostExecutor`, `.prompt()` → `.run()`, `Host.addServer()` → `Host.requireServer()`. No deprecation aliases. The same code with the old names will not type-check against `@mcpjam/sdk@>=1.11`.
</Note>

## Spec-first: `Host` + `HostRuntime`

For richer setups — pinning a host style/profile, applying sandbox/permission policy, or running the same configuration in CI that the MCPJam Inspector uses — start from a `Host` spec instead of constructing a runner directly:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, Host, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager();
await manager.connectToServer("everything", {
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-everything"],
});

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-4o",
}).requireServer("everything");

// Bind the spec to the live manager. apiKey lives on the runtime, not per-call.
const runtime = host.withManager(manager, {
  apiKey: process.env.OPENAI_API_KEY!,
});

const test = new EvalTest({
  id: "c_add",
  name: "add",
  test: async (runner) => (await runner.run("Add 2+3")).hasToolCall("add"),
});
await test.run(runtime, { iterations: 10 });
```

See [Run evals from your own runtime](/sdk/concepts/running-evals#bring-your-own-host) for the full Host / HostRunner / HostRuntime story.


# Migrating to SDK 3.0
Source: https://docs.mcpjam.com/sdk/migrating-to-v3

What changed, why a passing test can start failing, and how to upgrade.

SDK 3.0 makes code-first evals **enforce what they declare**. Three assertions the SDK accepted but never actually evaluated locally now gate every iteration, and two metrics stop returning a number that was not what their name promised.

Most upgrades need no code changes. The exception is deliberate: a test that declared `expectedToolCalls` and never had them checked can start failing. That is the bug being fixed — the same run was already failing in the MCPJam dashboard, which recomputed the match server-side.

## Why this changed

Before 3.0, `expectedToolCalls` was upload metadata. The local verdict came from your `test` function alone; the platform then re-derived the tool-call match from the case snapshot. The two could disagree, and when they did the SDK was the one telling you what you wanted to hear:

```typescript theme={"theme":"css-variables"}
// SDK 2.x — both of these were true at once
test.accuracy(); // 1.0  — every iteration "passed" locally
// …while the run for those same iterations showed FAILED in MCPJam,
// because the expected tool call was never made.
```

A verdict you cannot trust is worse than no verdict. 3.0 evaluates the expectation where the iteration runs, so local and hosted agree.

## Breaking changes

### `expectedToolCalls` is enforced locally

If a test declares expected tool calls, they are matched during the run and a mismatch fails the iteration.

```typescript theme={"theme":"css-variables"}
const test = new EvalTest({
  name: "books the venue",
  expectedToolCalls: [{ toolName: "book_venue" }],
  test: async (executor) => {
    const r = await executor.run("Book Friday.");
    return r.hasToolCall("book_venue");
  },
});
```

**If this newly fails**, pick whichever is true:

<AccordionGroup>
  <Accordion title="The expectation was wrong or stale">
    Fix or delete it. The tool may have been renamed, or the agent legitimately solves the task another way now.
  </Accordion>

  <Accordion title="The expectation was over-specified">
    Relax the matcher rather than the assertion:

    ```typescript theme={"theme":"css-variables"}
    matchOptions: { argumentMatching: "ignore" }   // names must match, arguments need not
    matchOptions: { maxExtraToolCalls: null }      // extra calls are fine (the default)
    ```
  </Accordion>

  <Accordion title="It was only ever documentation">
    Remove `expectedToolCalls`. A test with no expectations behaves exactly as it did in 2.x — your `test` function is the whole verdict.
  </Accordion>
</AccordionGroup>

<Note>
  Tests that declare neither `expectedToolCalls` nor `predicates` are unaffected by this release.
</Note>

### `precision()` and `recall()` return real values

In 2.x, `precision()`, `recall()` and `truePositiveRate()` all did `return this.accuracy()` — three names for the pass rate. They now compute genuine micro-averaged values from the tool-call matches, and **throw** when no test in the run declared `expectedToolCalls`, because there is nothing to compute them from.

```typescript theme={"theme":"css-variables"}
// 2.x: precision() === recall() === accuracy(), always
// 3.0: real values, or a thrown error if there are no expectations
test.accuracy(); // unchanged — the pass rate
```

If you were reading `precision()` as a stand-in for accuracy, call [`accuracy()`](/sdk/reference/eval-test#accuracy) directly.

### `falsePositiveRate()` is deprecated

It returned `failures / iterations` — the failure rate, not a false-positive rate. Use [`unexpectedToolCallRate()`](/sdk/reference/eval-test#unexpectedtoolcallrate), the fraction of expectation-bearing iterations that made a call nobody asked for. `falsePositiveRate()` still returns the legacy value for runs without expectations, so existing dashboards do not move.

## New in 3.0

### Predicates work code-first

The deterministic check engine was hosted-only. It now runs in your test file, gating the iteration **and** reporting the same verdicts, so the dashboard's check chips and cross-run criterion trends light up for code-first runs.

```typescript theme={"theme":"css-variables"}
const test = new EvalTest({
  name: "answers cleanly",
  predicates: [
    { type: "noToolErrors" },
    { type: "responseContains", needle: "confirmed" },
    { type: "turnCountUnder", max: 4 },
  ],
  test: async (executor) => {
    const r = await executor.run("Book Friday and confirm.");
    return r.text.length > 0;
  },
});
```

An iteration passes only if every predicate passes — independently of `failOnToolError`. See the [predicate gate](/sdk/reference/eval-reporting#predicate-gate) for the full list of types.

<Warning>
  The three widget predicates (`widgetRendered`, `widgetRenderLatencyUnder`, `widgetNoConsoleErrors`) read render observations that only a hosted run captures, and they fail closed. `EvalTest` rejects them **at construction** rather than failing every iteration with a confusing reason — move those cases to a hosted suite.
</Warning>

### `matchOptions`

How `expectedToolCalls` is matched, layered suite → case and validated when the test is constructed:

```typescript theme={"theme":"css-variables"}
new EvalSuite({
  name: "booking",
  matchOptions: { toolCallOrder: "strict" }, // default for every test
});
```

Full option table in the [`EvalTest` reference](/sdk/reference/eval-test#matchoptions).

## Upgrade checklist

<Steps>
  <Step title="Bump the package">
    ```bash theme={"theme":"css-variables"}
    npm install @mcpjam/sdk@^3
    ```
  </Step>

  <Step title="Run your suite">
    Failures here are expectations that were never being checked. Work through them with the three options above.
  </Step>

  <Step title="Replace deprecated metric calls">
    `falsePositiveRate()` → `unexpectedToolCallRate()`. If you read `precision()` or `recall()` expecting the pass rate, switch to `accuracy()`.
  </Step>

  <Step title="Optional: adopt predicates">
    Anything your `test` function asserts about the transcript — a tool was called, the reply contains a phrase, no tool errored — is expressible as a predicate, which makes it visible in the dashboard instead of hidden inside a boolean.
  </Step>
</Steps>

## Unchanged

Connecting to servers, `HostRunner` / `HostRuntime`, prompt execution and multi-turn context threading, reporting configuration, and the run URL printed after an upload all behave as they did in 2.x.


# Quickstart
Source: https://docs.mcpjam.com/sdk/quickstart

Get started with @mcpjam/sdk in 5 minutes

This guide walks you through installing the SDK, connecting to an MCP server, and running your first evaluation.

## Prerequisites

* Node.js 18+
* An LLM API key (Anthropic, OpenAI, etc.)

## Installation

```bash theme={"theme":"css-variables"}
npm install @mcpjam/sdk
```

## Step 1: Connect to an MCP Server

```typescript theme={"theme":"css-variables"}
import { MCPClientManager } from "@mcpjam/sdk";

const manager = new MCPClientManager({
  everything: {
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-everything"],
  },
});

await manager.connectToServer("everything");

// List available tools
const tools = await manager.listTools("everything");
console.log("Tools:", tools.tools.map((t) => t.name));
```

<Note>
  `@modelcontextprotocol/server-everything` is a reference MCP server with sample tools like `add`, `echo`, and `longRunningOperation`.
</Note>

If you're connecting to an OAuth-protected HTTP server and already have a refresh token, pass `refreshToken` and `clientId` instead of a static access token:

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  remoteServer: {
    url: "https://mcp.example.com/mcp",
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
    clientSecret: process.env.MCP_CLIENT_SECRET,
  },
});

await manager.connectToServer("remoteServer");
```

The SDK will exchange the refresh token for an access token and automatically refresh it when needed.

## Step 2: Execute Tools Directly

Call tools without an LLM, useful for unit tests:

```typescript theme={"theme":"css-variables"}
const result = await manager.executeTool("everything", "add", {
  a: 5,
  b: 3,
});
console.log("5 + 3 =", result); // 8
```

## Step 3: Create a HostRunner

Connect an LLM to your MCP tools:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner } from "@mcpjam/sdk";

const agent = new HostRunner({
  tools: await manager.getTools(),
  model: "anthropic/claude-sonnet-5",
  apiKey: process.env.ANTHROPIC_API_KEY,
  mcpClientManager: manager,
});
```

## Step 4: Run Prompts

```typescript theme={"theme":"css-variables"}
const result = await agent.run("What is 15 plus 27?");

console.log("Response:", result.text);
console.log("Tools called:", result.toolsCalled());
console.log("Arguments:", result.getToolArguments("add"));
console.log("Latency:", result.e2eLatencyMs(), "ms");
```

## Step 5: Write a Test

```typescript theme={"theme":"css-variables"}
import { matchToolCalls } from "@mcpjam/sdk";
import { describe, it, expect } from "vitest";

describe("Math MCP Server", () => {
  it("should call add for addition", async () => {
    const result = await agent.run("Add 10 and 5");
    expect(matchToolCalls(["add"], result.toolsCalled())).toBe(true);
  });
});
```

## Step 6: Run Statistical Evaluations

```typescript theme={"theme":"css-variables"}
import { EvalTest } from "@mcpjam/sdk";

const test = new EvalTest({
  id: "c_addition_accuracy",
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add");
  },
});

const run = await test.run(agent, { iterations: 30 });

console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
console.log(`${run.successes}/${run.iterations} passed`);
// "29/30 passed"
```

## Complete Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalSuite, EvalTest } from "@mcpjam/sdk";

async function main() {
  // Setup
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-5",
    apiKey: process.env.ANTHROPIC_API_KEY,
    mcpClientManager: manager,
  });

  // Quick test
  const result = await agent.run("Echo 'Hello!'");
  console.log("Response:", result.text);
  console.log("Tools:", result.toolsCalled());

  // Run eval suite
  const suite = new EvalSuite({ name: "Basic Operations" });

  suite.add(new EvalTest({
    id: "c_echo",
    name: "echo",
    test: async (a) => (await a.run("Echo 'test'")).hasToolCall("echo"),
  }));

  suite.add(new EvalTest({
    id: "c_add",
    name: "add",
    test: async (a) => (await a.run("Add 1 and 2")).hasToolCall("add"),
  }));

  await suite.run(agent, { iterations: 10 });

  console.log(`\nSuite accuracy: ${(suite.accuracy() * 100).toFixed(1)}%`);

  // Cleanup
  await manager.disconnectServer("everything");
}

main();
```

## Generate evals from the Inspector

You can also generate eval code directly from the MCPJam Inspector:

1. Connect your MCP server in the Inspector
2. Click the **⋮** menu on the server card
3. Select **Copy markdown for evals**
4. Paste the copied markdown into an LLM (Claude, ChatGPT, etc.)
5. The LLM will generate eval test code using `@mcpjam/sdk`

<Frame>
  <img alt="Copy markdown for evals menu option" />
</Frame>

The copied markdown is a ready-to-use prompt that includes your server's capabilities and the full `@mcpjam/sdk` API reference, so the LLM has everything it needs to write your eval tests.

<Tip>
  To save results to the MCPJam Evals dashboard, set `MCPJAM_API_KEY` to an **MCPJam API key (`sk_…`)** from Settings → API keys and add a `mcpjam: { suiteName: "…" }` block to `run()`. The retired project API keys (`mcpjam_…`) no longer work — see [Saving Results](/sdk/concepts/saving-results).
</Tip>

## Next Steps

<CardGroup>
  <Card title="Connecting to Servers" icon="plug" href="/sdk/concepts/connecting-servers">
    STDIO vs HTTP servers
  </Card>

  <Card title="Testing with LLMs" icon="bot" href="/sdk/concepts/testing-with-llms">
    Prompting and result inspection
  </Card>

  <Card title="Running Evals" icon="chart-bar" href="/sdk/concepts/running-evals">
    Statistical evaluations
  </Card>

  <Card title="API Reference" icon="book" href="/sdk/reference/mcp-client-manager">
    Complete API docs
  </Card>
</CardGroup>


# MCP Apps Conformance SDK
Source: https://docs.mcpjam.com/sdk/reference/apps-conformance

Programmatic MCP Apps surface validation with MCPAppsConformanceTest

The MCP Apps Conformance SDK lets you validate the server-side MCP Apps surface your server exposes through tools and `ui://` resources.

Use it when you want the same checks as the CLI's [`apps conformance`](/cli/apps-conformance) command, but inside your own test runner or CI pipeline.

<Note>
  This currently validates the **server-side** MCP Apps surface only. It does
  not prove full host-side SEP-1865 behavior such as `ui/initialize`,
  sandbox-proxy forwarding, or host notification ordering.
</Note>

## Import

```typescript theme={"theme":"css-variables"}
import {
  MCPAppsConformanceSuite,
  MCPAppsConformanceTest,
  renderConformanceReportJUnitXml,
  renderConformanceReportJson,
  toConformanceReport,
} from "@mcpjam/sdk";
```

## Basic usage

```typescript theme={"theme":"css-variables"}
const test = new MCPAppsConformanceTest({
  url: "https://your-server.com/mcp",
  timeout: 30_000,
});

const result = await test.run();

console.log(result.passed);   // true
console.log(result.outcome);  // "passed" | "failed" | "incomplete"
console.log(result.summary);  // "7/7 checks passed, 0 failed, 0 could not run, 0 not applicable"
console.log(result.discovery.uiToolCount);
```

For a stdio server:

```typescript theme={"theme":"css-variables"}
const test = new MCPAppsConformanceTest({
  command: "node",
  args: ["server.js"],
  timeout: 30_000,
});
```

## Suite usage

```typescript theme={"theme":"css-variables"}
import { writeFileSync } from "node:fs";

const suite = new MCPAppsConformanceSuite({
  name: "Apps CI",
  target: {
    url: "https://your-server.com/mcp",
    timeout: 30_000,
  },
  defaults: {
    checkIds: [
      "ui-tools-present",
      "ui-tool-metadata-valid",
    ],
  },
  runs: [
    {},
    {
      label: "resources",
      checkIds: [
        "ui-resources-readable",
        "ui-resource-contents-valid",
        "ui-resource-meta-valid",
      ],
    },
  ],
});

const result = await suite.run();
const report = toConformanceReport(result);

writeFileSync(
  "apps-conformance.junit.xml",
  renderConformanceReportJUnitXml(report),
);
writeFileSync(
  "apps-conformance.report.json",
  JSON.stringify(renderConformanceReportJson(report), null, 2),
);
```

`MCPAppsConformanceSuiteConfig` is shaped for CI matrices:

| Property   | Type                                                             | Required | Description                                  |
| ---------- | ---------------------------------------------------------------- | -------- | -------------------------------------------- |
| `name`     | `string`                                                         | No       | Suite label shown in summaries and JUnit XML |
| `target`   | `MCPServerConfig`                                                | Yes      | Shared HTTP or stdio target                  |
| `defaults` | `Partial<Omit<MCPAppsConformanceConfig, keyof MCPServerConfig>>` | No       | Shared defaults applied to every run         |
| `runs`     | `Array<... & { label?: string }>`                                | Yes      | Individual check selections and labels       |

## MCPAppsConformanceConfig

`MCPAppsConformanceConfig` extends the standard `MCPServerConfig`, so it accepts the same HTTP and stdio connection settings as `MCPClientManager`.

Additional property:

| Property   | Type               | Required | Default    | Description                           |
| ---------- | ------------------ | -------- | ---------- | ------------------------------------- |
| `checkIds` | `MCPAppsCheckId[]` | No       | all checks | Run only the selected MCP Apps checks |

Example with custom headers and a focused check set:

```typescript theme={"theme":"css-variables"}
const test = new MCPAppsConformanceTest({
  url: "https://your-server.com/mcp",
  requestInit: {
    headers: {
      Authorization: `Bearer ${process.env.TOKEN}`,
    },
  },
  checkIds: [
    "ui-resources-readable",
    "ui-resource-contents-valid",
  ],
});
```

## Check ids

Available checks:

* `ui-tools-present`
* `ui-tool-metadata-valid`
* `ui-tool-input-schema-valid`
* `ui-listed-resources-valid`
* `ui-resources-readable`
* `ui-resource-contents-valid`
* `ui-resource-meta-valid`

## Result shape

`run()` returns an `MCPAppsConformanceResult`.

| Property           | Type                                   | Description                                                                                     |
| ------------------ | -------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `passed`           | `boolean`                              | True only when `outcome` is `"passed"`. A check that could not run keeps this `false`.          |
| `outcome`          | `"passed" \| "failed" \| "incomplete"` | `"failed"` on any violation; `"incomplete"` when nothing failed but a selected check never ran. |
| `incompleteReason` | `string \| undefined`                  | Present when `outcome` is `"incomplete"`: which checks did not run and what to change.          |
| `target`           | `string`                               | URL or stdio command under test                                                                 |
| `checks`           | `MCPAppsCheckResult[]`                 | Individual check results                                                                        |
| `summary`          | `string`                               | Human-readable summary (`N/M checks passed, X failed, Y could not run, Z not applicable`)       |
| `durationMs`       | `number`                               | Total duration                                                                                  |
| `categorySummary`  | `Record<"tools" \| "resources", ...>`  | Per-category `total`, `passed`, `failed`, `skipped`, and `couldNotRun` counts                   |
| `discovery`        | object                                 | Counts for tools and UI resources discovered during the run                                     |

Each `MCPAppsCheckResult` includes:

* `id`
* `category`
* `title`
* `description`
* `status` — `"passed"`, `"failed"`, or `"skipped"`
* `skipReason` — set whenever `status` is `"skipped"`: `"not-applicable"` (no MCP Apps tools or UI resources to check) or `"could-not-run"` (tools/list did not complete, connection failed)
* `durationMs`
* optional `details`
* optional `warnings`
* optional `error`

`MCPAppsConformanceSuite.run()` returns an `MCPAppsConformanceSuiteResult`:

| Property     | Type                                                  | Description                  |
| ------------ | ----------------------------------------------------- | ---------------------------- |
| `name`       | `string`                                              | Suite name                   |
| `target`     | `string`                                              | Shared target under test     |
| `passed`     | `boolean`                                             | `true` iff every run passed  |
| `results`    | `Array<MCPAppsConformanceResult & { label: string }>` | Per-run results              |
| `summary`    | `string`                                              | Human-readable suite summary |
| `durationMs` | `number`                                              | Total suite duration         |

## Outcomes: passed, failed, incomplete

A run reports one of three outcomes, and `passed` is `true` only for the first:

| `outcome`    | CLI exit code | Meaning                                                                                                      |
| ------------ | ------------- | ------------------------------------------------------------------------------------------------------------ |
| `passed`     | `0`           | Every selected check either ran and passed, or does not apply to this server.                                |
| `failed`     | `1`           | At least one check found a violation.                                                                        |
| `incomplete` | `3`           | Nothing failed, but at least one selected check could not be run, so the run does not establish conformance. |

An incomplete run carries a root `incompleteReason` naming the checks that did not run and what to change; non-quiet CLI runs also print it to stderr. For a suite, the exit code is the worst of all runs (failure outranks incomplete).

## CI reporting

All three conformance domains use the same shared reporting helpers:

* `toConformanceReport(result)` normalizes protocol, OAuth, and apps runs into a single report shape.
* `renderConformanceReportJUnitXml(report)` emits redacted JUnit XML for CI dashboards.
* `renderConformanceReportJson(report)` returns the redacted JSON-ready object for artifact uploads.

That makes the SDK and CLI JUnit output byte-identical for the same result.

## What the runner validates

The current runner checks:

1. At least one tool advertises MCP Apps UI metadata.
2. Tool metadata uses a valid `ui://` resource URI and valid `visibility` values.
3. Tool `inputSchema` is a non-null JSON Schema object.
4. Listed UI resources use `ui://` and `text/html;profile=mcp-app`.
5. Referenced UI resources are readable via `resources/read`.
6. Resource payloads provide exactly one HTML document through `text` or `blob`.
7. `_meta.ui.csp`, `permissions`, `domain`, and `prefersBorder` use valid shapes.

## Notes

* The runner always advertises the MCP Apps UI extension capability so servers do not hide their MCP Apps surface when custom `clientCapabilities` are supplied.
* Deprecated `_meta["ui/resourceUri"]` is accepted but surfaced as a warning.
* Tool name SHOULD validations (length, character set, uniqueness) surface as warnings, not failures.
* HTML validation is intentionally lightweight. It verifies the expected MIME type and document-style HTML payload, not the full browser lifecycle.
* For a runnable project that writes JUnit XML from Vitest, see [`examples/conformance/basic/`](https://github.com/MCPJam/inspector/tree/main/examples/conformance/basic).


# Saving Eval Results
Source: https://docs.mcpjam.com/sdk/reference/eval-reporting

API reference for saving eval results to MCPJam

<Warning>
  Reporting authenticates with **MCPJam API keys (`sk_…`)** from Settings → API keys and uploads through the [MCPJam public API](/reference/public-api) (`/api/v1/projects/:projectId/eval-ingest/*`). The legacy project API keys (`mcpjam_…`) and their `/sdk/v1/evals/*` endpoints are retired — old SDK versions pointing there receive `410 Gone`. Upgrade `@mcpjam/sdk` to keep uploading.
</Warning>

The SDK provides APIs to save eval results to MCPJam for visualization in the CI Evals dashboard. Results can be saved automatically via `EvalTest`/`EvalSuite`, or manually using the APIs below.

## Environment Variables

| Variable            | Required | Default                  | Description                                         |
| ------------------- | -------- | ------------------------ | --------------------------------------------------- |
| `MCPJAM_API_KEY`    | Yes      | -                        | An MCPJam API key (`sk_…`) from Settings → API keys |
| `MCPJAM_PROJECT_ID` | No       | org's Default project    | Project the results are filed under                 |
| `MCPJAM_BASE_URL`   | No       | `https://app.mcpjam.com` | MCPJam API base URL override                        |

Use `MCPJAM_BASE_URL` only when you need to override the default ingest host, such as internal development against a non-production backend.

`MCPJAM_API_KEY` controls whether results are uploaded. **Replay credential** capture only happens when you provide `serverReplayConfigs`, `agent`, or `mcpClientManager`. **MCP App widget snapshots** in each result’s trace (for iframe replay in MCPJam) come from [`PromptResult.getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots), which is only populated when [`HostRunner`](/sdk/reference/host-runner) was constructed with [`mcpClientManager`](/sdk/reference/host-runner#parameters).

***

## Tool execution and `passed`

Uploaded iterations have a single boolean `passed` that drives pass rate on the dashboard. The SDK distinguishes:

1. **Structural pass** — Your test returned `true`, or expected tool calls were satisfied (Inspector/UI flows use shared matching logic).
2. **Tool execution** — The trace shows a real tool failure: MCP results with `isError: true`, timeline spans where a tool step ended in `error`, tool-result parts the UI would treat as errors, or a runner-level `iterationError` after a thrown tool step.

**Default behavior:** When the SDK **derives** `passed` from a trace (for example `EvalTest` / `EvalSuite` auto-save, [`PromptResult.toEvalResult()`](/sdk/reference/prompt-result#toevalresult), `createEvalRunReporter` helpers, or Inspector suite runs), structural success is **not enough** if tool execution failed — the iteration is recorded as **failed** unless you opt out.

**Opt out:** set `failOnToolError: false` on [`MCPJamReportingConfig`](#mcpjamreportingconfig) (global for that reporter or auto-save), or pass it on specific helper options (`addFromPrompt` / `recordFromRun` / `RunToEvalResultsOptions`, etc.). Use this when you only care that the model *invoked* the right tools with the right arguments, not that every MCP call returned a success payload.

**Manual `reportEvalResults`:** If you build `results[]` yourself and set `passed` explicitly, MCPJam stores your values as-is. The execution gate applies to code paths that **compute** `passed` from prompts, traces, and iterations.

**Programmatic reuse:** `@mcpjam/sdk` also exports `finalizePassedForEval`, `traceIndicatesToolExecutionFailure`, `isCallToolResultError`, and `traceMessagePartIndicatesToolFailure` for custom ingestion pipelines. The `@mcpjam/sdk/predicates` subpath exports the full predicate evaluator (`evaluatePredicate`, `evaluatePredicates`, `allPredicatesPassed`, `buildIterationTranscript`), the tool-error extractor `extractToolErrors`, and all predicate types.

***

## reportEvalResults()

One-shot reporting. Sends all results in a single call. Throws on failure.

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, reportEvalResults } from "@mcpjam/sdk";
```

### Signature

```typescript theme={"theme":"css-variables"}
reportEvalResults(input: ReportEvalResultsInput): Promise<ReportEvalResultsOutput>
```

### Example

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  asana: {
    url: process.env.MCP_SERVER_URL!,
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
    clientSecret: process.env.MCP_CLIENT_SECRET,
  },
});

await manager.connectToServer("asana");

const output = await reportEvalResults({
  suiteName: "Nightly",
  mcpClientManager: manager,
  results: [
    { caseTitle: "healthcheck", passed: true },
    { caseTitle: "tool-selection", passed: true, durationMs: 1200 },
    { caseTitle: "edge-case", passed: false, error: "Wrong tool called" },
  ],
  passCriteria: { minimumPassRate: 90 },
  ci: {
    branch: "main",
    commitSha: "abc123",
  },
});

console.log(`Run ${output.runId}: ${output.result}`);
// "Run abc123: passed"
console.log(`${output.summary.passed}/${output.summary.total} passed`);
```

***

## reportEvalResultsSafely()

Same as `reportEvalResults()`, but returns `null` instead of throwing on failure. Warnings are logged to the console.

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, reportEvalResultsSafely } from "@mcpjam/sdk";
```

### Signature

```typescript theme={"theme":"css-variables"}
reportEvalResultsSafely(input: ReportEvalResultsInput): Promise<ReportEvalResultsOutput | null>
```

### Example

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  asana: {
    url: process.env.MCP_SERVER_URL!,
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
  },
});

await manager.connectToServer("asana");

const output = await reportEvalResultsSafely({
  suiteName: "Nightly",
  mcpClientManager: manager,
  results: [{ caseTitle: "healthcheck", passed: true }],
});

if (output) {
  console.log(`Reported: ${output.summary.passRate * 100}% pass rate`);
} else {
  console.log("Reporting failed (non-blocking)");
}
```

<Note>
  Use `reportEvalResultsSafely()` when you don't want eval reporting failures to break your CI pipeline. Use `reportEvalResults()` (strict) when reporting is critical.
</Note>

***

## createEvalRunReporter()

Creates an incremental reporter for long-running processes. Results are buffered and flushed in batches (up to 200 results or 1MB per batch).

```typescript theme={"theme":"css-variables"}
import { createEvalRunReporter } from "@mcpjam/sdk";
```

### Signature

```typescript theme={"theme":"css-variables"}
createEvalRunReporter(input: CreateEvalRunReporterInput): EvalRunReporter
```

`CreateEvalRunReporterInput` accepts the same replay source fields as `reportEvalResults()`: `serverReplayConfigs`, `agent`, and `mcpClientManager`.

### EvalRunReporter Methods

| Method                         | Description                                         |
| ------------------------------ | --------------------------------------------------- |
| `add(result)`                  | Buffer a result (no network call)                   |
| `record(result)`               | Buffer a result and auto-flush when buffer is large |
| `flush()`                      | Upload all buffered results                         |
| `finalize()`                   | Flush remaining results and finalize the run        |
| `getBufferedCount()`           | Number of results in the buffer                     |
| `getAddedCount()`              | Total results added (including flushed)             |
| `setExpectedIterations(count)` | Set expected iteration count for progress tracking  |

#### PromptResult Helpers

| Method                                       | Description                                         |
| -------------------------------------------- | --------------------------------------------------- |
| `addFromPrompt(promptResult, overrides?)`    | Convert a `PromptResult` and buffer it              |
| `recordFromPrompt(promptResult, overrides?)` | Convert a `PromptResult`, buffer it, and auto-flush |

#### EvalTest/EvalSuite Run Helpers

| Method                                  | Description                                    |
| --------------------------------------- | ---------------------------------------------- |
| `addFromRun(run, options)`              | Convert all iterations from an `EvalTest` run  |
| `recordFromRun(run, options)`           | Convert and auto-flush from an `EvalTest` run  |
| `addFromSuiteRun(suiteRun, options)`    | Convert all iterations from an `EvalSuite` run |
| `recordFromSuiteRun(suiteRun, options)` | Convert and auto-flush from an `EvalSuite` run |

### Example

```typescript theme={"theme":"css-variables"}
// Assumes `agent` was created with `mcpClientManager: manager`
const reporter = createEvalRunReporter({
  suiteName: "Integration Tests",
  passCriteria: { minimumPassRate: 85 },
  agent,
  ci: {
    branch: process.env.GITHUB_REF_NAME,
    commitSha: process.env.GITHUB_SHA,
  },
});
```

### Example with manual replay source resolution

```typescript theme={"theme":"css-variables"}
const reporter = createEvalRunReporter({
  suiteName: "Integration Tests",
  passCriteria: { minimumPassRate: 85 },
  mcpClientManager: manager,
});

// Add results as tests complete
await reporter.record({ caseTitle: "test-1", passed: true, durationMs: 500 });
await reporter.record({ caseTitle: "test-2", passed: false, error: "timeout" });
await reporter.record({ caseTitle: "test-3", passed: true });

// Finalize the run
const output = await reporter.finalize();
console.log(`${output.summary.passed}/${output.summary.total} passed`);
```

### Replay credential sources

Authenticated HTTP evals can securely persist replay credentials for reruns and debugging. Manual reporting APIs resolve replay configs in this order:

1. `serverReplayConfigs`
2. `agent.getServerReplayConfigs()`
3. `mcpClientManager.getServerReplayConfigs()`

Prefer passing `agent` or `mcpClientManager` directly. Use `serverReplayConfigs` only when you need a low-level override.

If you also provide `serverNames`, inferred replay configs are filtered to those server IDs before upload. Explicit `serverReplayConfigs` are left unchanged.

### Replay metadata for the MCPJam UI

Uploaded runs can show **Replay this run** / server-side MCP replay when the ingest payload includes derived **`serverReplayConfigs`** (stored as **`hasServerReplayConfig`** on the run). In practice:

* **HTTP MCP (`url`)** — Replay configs are built for typical streamable HTTP connections. **Stdio** transports **do not** produce entries from [`MCPClientManager.getServerReplayConfigs()`](/sdk/reference/mcp-client-manager); use HTTP when you need dashboard replay.
* **`HostRunner` vs reporter** — Putting `mcpClientManager` on [`HostRunner`](/sdk/reference/host-runner) fills **MCP App widget snapshots** on [`PromptResult`](/sdk/reference/prompt-result). The **reporter** (and one-shot `report*`) resolves **server replay** from its **own** `agent` / `mcpClientManager` fields. Pass `agent` or `mcpClientManager` into [`createEvalRunReporter`](#createevalrunreporter) as well; **agent-only** wiring can still upload traces and widgets but **omit** `hasServerReplayConfig`.
* **Teardown order** — `finalize()` / one-shot reporting calls `getServerReplayConfigs()` against **connected** registrations. In `afterAll`, run **`await reporter.finalize()`** (or `reportEvalResults`) **before** **`await manager.disconnectAllServers()`**. Disconnecting first clears manager state and uploads **without** replay metadata.

**LLM API keys** are **not** stored on the run. Replaying in the MCPJam UI still requires provider keys (e.g. OpenRouter) in **Settings** for your suite’s models.

### Using with PromptResult

```typescript theme={"theme":"css-variables"}
// Pass `agent` or `mcpClientManager` when you need server replay metadata in MCPJam
const reporter = createEvalRunReporter({ suiteName: "Prompt Tests", agent });

const result = await agent.run("Add 2 and 3");
reporter.addFromPrompt(result, {
  caseTitle: "addition",
  passed: result.hasToolCall("add"),
});

const output = await reporter.finalize();
```

### Using with EvalTest Runs

```typescript theme={"theme":"css-variables"}
const reporter = createEvalRunReporter({ suiteName: "Full Suite" });

const test = new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (agent) => (await agent.run("Add 2+3")).hasToolCall("add"),
});

const run = await test.run(agent, { iterations: 10 });
await reporter.recordFromRun(run, { casePrefix: "addition" });

const output = await reporter.finalize();
```

***

## uploadEvalArtifact()

Parses test artifacts (JUnit XML, Jest JSON, Vitest JSON) and reports the results to MCPJam.

```typescript theme={"theme":"css-variables"}
import { uploadEvalArtifact } from "@mcpjam/sdk";
```

### Signature

```typescript theme={"theme":"css-variables"}
uploadEvalArtifact(input: UploadEvalArtifactInput): Promise<ReportEvalResultsOutput>
```

### Supported Formats

| Format          | Description                             |
| --------------- | --------------------------------------- |
| `"junit-xml"`   | JUnit XML test reports                  |
| `"jest-json"`   | Jest JSON output (`--json` flag)        |
| `"vitest-json"` | Vitest JSON reporter output             |
| `"custom"`      | Custom parser via `customParser` option |

### Example

```typescript theme={"theme":"css-variables"}
import { readFileSync } from "fs";

// Upload JUnit XML
await uploadEvalArtifact({
  suiteName: "CI Results",
  format: "junit-xml",
  artifact: readFileSync("test-results.xml", "utf-8"),
});

// Upload Jest JSON
await uploadEvalArtifact({
  suiteName: "Jest Results",
  format: "jest-json",
  artifact: readFileSync("jest-results.json", "utf-8"),
});

// Custom parser
await uploadEvalArtifact({
  suiteName: "Custom",
  format: "custom",
  artifact: myData,
  customParser: (data) => [
    { caseTitle: "test-1", passed: true },
    { caseTitle: "test-2", passed: false, error: "failed" },
  ],
});
```

***

## Types

### ReportEvalResultsInput

```typescript theme={"theme":"css-variables"}
type ReportEvalResultsInput = MCPJamReportingConfig & {
  suiteName: string;
  results: EvalResultInput[];
  agent?: {
    getServerReplayConfigs?: () => MCPServerReplayConfig[] | undefined;
  };
  executor?: {
    getHostSnapshot?: () => HostJson | undefined;
  };
  mcpClientManager?: MCPClientManager;
};
```

Fields specific to `ReportEvalResultsInput` (everything else comes from [`MCPJamReportingConfig`](#mcpjamreportingconfig)):

| Property           | Type                                                                      | Required | Description                                                                                                                                                                                                                                                                                                                                                                |
| ------------------ | ------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `suiteName`        | `string`                                                                  | Yes      | Suite name for the run                                                                                                                                                                                                                                                                                                                                                     |
| `results`          | `EvalResultInput[]`                                                       | Yes      | The results to upload                                                                                                                                                                                                                                                                                                                                                      |
| `agent`            | `{ getServerReplayConfigs?: () => MCPServerReplayConfig[] \| undefined }` | No       | Preferred replay source for manual reporting; use an executor (`HostRunner` / `HostRuntime`) created with `mcpClientManager` so results can include [`widgetSnapshots`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) from [`PromptResult.toEvalResult()`](/sdk/reference/prompt-result). Field name predates the Stage 4 rename; any `HostExecutor` is accepted. |
| `executor`         | `{ getHostSnapshot?: () => HostJson \| undefined }`                       | No       | Host-snapshot fallback for the run-level host config. Consulted when no per-iteration `hostSnapshot` is available; any object exposing `getHostSnapshot()` (e.g. `HostRunner`, `HostRuntime`) qualifies. See [Run-level host snapshot](#run-level-host-snapshot)                                                                                                           |
| `mcpClientManager` | `MCPClientManager`                                                        | No       | Replay source when no `agent` is provided; does not populate `widgetSnapshots` unless your `results[].widgetSnapshots` or trace payloads already include them                                                                                                                                                                                                              |

### MCPJamReportingConfig

| Property              | Type                          | Required | Description                                                                                                                                                                                                                                                                                     |
| --------------------- | ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`             | `boolean`                     | No       | Enable/disable reporting (default: `true`)                                                                                                                                                                                                                                                      |
| `apiKey`              | `string`                      | No       | MCPJam API key (`sk_…`; falls back to `MCPJAM_API_KEY` env var)                                                                                                                                                                                                                                 |
| `baseUrl`             | `string`                      | No       | MCPJam API base URL override (useful for internal development or tests)                                                                                                                                                                                                                         |
| `project`             | `string`                      | No       | Project id results are filed under (falls back to `MCPJAM_PROJECT_ID`, then the org's Default project)                                                                                                                                                                                          |
| `suiteName`           | `string`                      | No       | Suite name for the run                                                                                                                                                                                                                                                                          |
| `suiteDescription`    | `string`                      | No       | Description of the suite                                                                                                                                                                                                                                                                        |
| `serverNames`         | `string[]`                    | No       | MCP server names being tested                                                                                                                                                                                                                                                                   |
| `serverReplayConfigs` | `MCPServerReplayConfig[]`     | No       | Advanced override for replay credential capture                                                                                                                                                                                                                                                 |
| `notes`               | `string`                      | No       | Free-form notes                                                                                                                                                                                                                                                                                 |
| `passCriteria`        | `{ minimumPassRate: number }` | No       | Pass threshold (0-100)                                                                                                                                                                                                                                                                          |
| `failOnToolError`     | `boolean`                     | No       | When not `false`, results derived from traces treat tool execution failures as failed iterations (default: strict). See [Tool execution and `passed`](#tool-execution-and-passed)                                                                                                               |
| `strict`              | `boolean`                     | No       | Throw on upload errors (`false` = warn only)                                                                                                                                                                                                                                                    |
| `externalRunId`       | `string`                      | No       | Custom run ID (auto-generated if omitted)                                                                                                                                                                                                                                                       |
| `framework`           | `string`                      | No       | Test framework name (e.g., `"jest"`, `"vitest"`)                                                                                                                                                                                                                                                |
| `ci`                  | `EvalCiMetadata`              | No       | CI/CD pipeline context                                                                                                                                                                                                                                                                          |
| `expectedIterations`  | `number`                      | No       | Expected total iterations for progress tracking                                                                                                                                                                                                                                                 |
| `tags`                | `string[]`                    | No       | Free-form tags attached to the run for filtering in the MCPJam dashboard                                                                                                                                                                                                                        |
| `host`                | `Host`                        | No       | Explicit host snapshot override. The reporter prefers `iteration.hostSnapshot` (per-iteration capture from `HostRuntime`) and falls back to `executor.getHostSnapshot?.()`; supply `host` only when neither source is available. See [Run-level host snapshot](#run-level-host-snapshot) below. |

The replay sources `agent` and `mcpClientManager` are fields of [`ReportEvalResultsInput`](#reportevalresultsinput), not of `MCPJamReportingConfig` — pass them on the reporting call itself.

### Run-level host snapshot

When a usable host snapshot is available, the reporter includes a normalized + content-hashed host config alongside the run body (the v1 ingest surface always accepts the pair — the old per-`baseUrl` capability probe is gone):

```jsonc theme={"theme":"css-variables"}
POST /api/v1/projects/{projectId}/eval-ingest/runs/start
{
  "suiteName": "...",
  "results": [...],
  "hostConfig":     {/* canonical HostConfigInputV2 with serverIds stripped */},
  "hostConfigHash": "sha256(canonical)"
}
```

**Source order (highest priority first):**

1. `iteration.hostSnapshot` — per-iteration capture from `HostRuntime` (Stage 4). Reflects the bound `Host` state at the iteration's end. Not yet carried by `EvalResultInput`, so today the reporter falls through to 2 and 3.
2. `executor.getHostSnapshot?.()` — fallback for executors that don't expose per-iteration snapshots.
3. `MCPJamReportingConfig.host` — explicit override, compatibility path.

**Pass-1 homogeneity gate.** The reporter only sends a run-level `{ hostConfig, hostConfigHash }` when all available iteration snapshots canonicalize to the same hash. Heterogeneous runs (e.g. mutating the bound `Host` between iterations) omit the run-level field — per-iteration wire support is a later stage.

**Fail-safe omission.** Any error while resolving or canonicalizing the snapshot — malformed `hostSnapshot`, an executor that throws, non-canonicalizable host JSON — logs a warning and omits the wire pair rather than failing the eval upload. The body shape stays the same without it.

**Server-id normalization.** Runtime-manager identifiers (`serverIds`, `optionalServerIds`, `serverConnectionOverrides`) are stripped by `normalizeSdkEvalHostConfigForWire` on both ends so SDK runtime ids like `"everything"` never reach the backend's `validateServerScope` as `Id<'servers'>`. The persisted `hostConfigsV2` row's hash will differ from the wire `hostConfigHash` because suite-resolved Convex server ids are layered on top at storage time — the wire hash is a transport-integrity check, not the storage id.

### MCPServerReplayConfig

Advanced replay override. Most users should not construct this manually.

| Property       | Type      | Required | Description                                       |
| -------------- | --------- | -------- | ------------------------------------------------- |
| `serverId`     | `string`  | Yes      | MCP server identifier                             |
| `url`          | `string`  | Yes      | MCP server URL                                    |
| `preferSSE`    | `boolean` | No       | Prefer SSE transport for replay                   |
| `accessToken`  | `string`  | No       | Static bearer token for replay                    |
| `refreshToken` | `string`  | No       | Refresh token for replay                          |
| `clientId`     | `string`  | No       | OAuth client ID, required with `refreshToken`     |
| `clientSecret` | `string`  | No       | OAuth client secret when needed for token refresh |

### EvalCiMetadata

| Property     | Type     | Description                                |
| ------------ | -------- | ------------------------------------------ |
| `provider`   | `string` | CI provider (e.g., `"github"`, `"gitlab"`) |
| `pipelineId` | `string` | Pipeline/workflow identifier               |
| `jobId`      | `string` | Job identifier                             |
| `runUrl`     | `string` | URL to the CI run                          |
| `branch`     | `string` | Git branch name                            |
| `commitSha`  | `string` | Git commit SHA                             |

### EvalResultInput

| Property              | Type                                          | Required | Description                                                                                                                                                                                                                                                                                                                              |
| --------------------- | --------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `caseTitle`           | `string`                                      | Yes      | Test case title                                                                                                                                                                                                                                                                                                                          |
| `passed`              | `boolean`                                     | Yes      | Whether the test passed                                                                                                                                                                                                                                                                                                                  |
| `query`               | `string`                                      | No       | The prompt/query sent                                                                                                                                                                                                                                                                                                                    |
| `durationMs`          | `number`                                      | No       | Test duration in ms                                                                                                                                                                                                                                                                                                                      |
| `provider`            | `string`                                      | No       | LLM provider name                                                                                                                                                                                                                                                                                                                        |
| `model`               | `string`                                      | No       | Model identifier                                                                                                                                                                                                                                                                                                                         |
| `expectedToolCalls`   | `EvalExpectedToolCall[]`                      | No       | Expected tool calls                                                                                                                                                                                                                                                                                                                      |
| `actualToolCalls`     | `EvalExpectedToolCall[]`                      | No       | Actual tool calls made                                                                                                                                                                                                                                                                                                                   |
| `tokens`              | `{ input?, output?, total? }`                 | No       | Token usage                                                                                                                                                                                                                                                                                                                              |
| `error`               | `string`                                      | No       | Error message                                                                                                                                                                                                                                                                                                                            |
| `errorDetails`        | `string`                                      | No       | Detailed error info                                                                                                                                                                                                                                                                                                                      |
| `trace`               | `EvalTraceInput`                              | No       | Conversation trace                                                                                                                                                                                                                                                                                                                       |
| `externalIterationId` | `string`                                      | No       | Custom iteration ID                                                                                                                                                                                                                                                                                                                      |
| `externalCaseId`      | `string`                                      | No       | Custom case ID                                                                                                                                                                                                                                                                                                                           |
| `caseId`              | `string`                                      | No       | The case's declared identity ([`EvalTestConfig.id`](/sdk/reference/eval-test)). An `EvalSuite` puts this on every result automatically. The backend resolves by it first and adopts it onto a case that resolved by content hash, which is what lets a renamed test keep its history. Must equal `externalCaseId` when both are present. |
| `intent`              | `string \| null`                              | No       | Analytics grouping label for this case. `null` explicitly records an unlabelled modern producer; omit the field to preserve legacy wire behavior. Never affects scoring or the verdict.                                                                                                                                                  |
| `metadata`            | `Record<string, string \| number \| boolean>` | No       | Custom metadata; values must be scalars. When the runner evaluates a case's predicates, it persists `{ predicate, passed, reason, scope? }[]` under `metadata.predicates` automatically.                                                                                                                                                 |
| `isNegativeTest`      | `boolean`                                     | No       | Whether this is a negative test                                                                                                                                                                                                                                                                                                          |
| `advancedConfig`      | `Record<string, unknown>`                     | No       | Advanced case configuration recorded with the iteration (e.g. `steps`, `system`, `temperature`)                                                                                                                                                                                                                                          |
| `matchOptions`        | `EvalMatchOptions`                            | No       | Per-result match options. When present, the inspector snapshots these onto the iteration so historical pass/fail computation honors them. See [`EvalMatchOptions`](/sdk/reference/validators#evalmatchoptions)                                                                                                                           |
| `widgetSnapshots`     | `EvalWidgetSnapshotInput[]`                   | No       | MCP App HTML replay payloads (typically from [`getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) on [`PromptResult`](/sdk/reference/prompt-result)). Omitted when not using MCP Apps or when `HostRunner` had no `mcpClientManager`                                                                          |

### Predicate gate

Predicates add a deterministic, state-based assertion layer on top of tool-call matching. Each predicate is a pure function of the iteration transcript — same transcript, same verdict — which makes it suitable as a CI release gate.

Predicates can be authored two ways: on a hosted suite or case (in the corpus JSON, or through the Inspector's per-case **Checks** editor), or **code-first** from SDK 3.0 onward by passing `predicates` to [`EvalTest`](/sdk/reference/eval-test) or [`EvalSuite`](/sdk/reference/eval-suite). Either way the runner evaluates them and persists per-predicate verdicts under [`metadata.predicates`](#predicate-results-in-metadata). A case passes the predicate gate iff every **gating** predicate passes. A predicate authored `role: "advisory"` — which every observation kind must be — is recorded and surfaced, and never fails the case; so is one whose `status` is `"error"`, a check that could not be scored at all. An absent or empty list means no predicate gate (the case is judged solely by tool-call matching and `failOnToolError`).

<Note>
  Before SDK 3.0 the predicate engine was hosted-only: `EvalResultInput` had no predicate field, and a code-first run could not evaluate them. Code-first predicates now gate the iteration locally **and** report the same verdicts, so a suite reads identically whether it ran from your test file or from the platform.
</Note>

The three widget predicates — `widgetRendered`, `widgetRenderLatencyUnder`, `widgetNoConsoleErrors` — read render observations that only a hosted run captures, and they fail closed. `EvalTest` therefore **rejects them at construction** rather than failing every iteration with a confusing reason; move those cases to a hosted suite.

#### Predicate types

The **Measures** column is the link of the [user-value chain](/sdk/concepts/user-value-chain) each kind's evidence is filed at. It is where the evidence is recorded, not a claim about where a failure originated. No predicate files at Connection or Discovery: the runner decides those two from what it observed setting the run up, before your case runs.

| Type                            | Fields                                                        | Measures   | Description                                                                                                                                                                                                                                                                                                                 |
| ------------------------------- | ------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `toolCalledWith`                | `toolName`, `args` ([`ArgMatcher`](#argmatcher)), `minCount?` | Selection  | A call to `toolName` whose args satisfy `args` occurred at least `minCount` (default 1) times.                                                                                                                                                                                                                              |
| `toolCalledAtLeastOnce`         | `toolName`                                                    | Selection  | `toolName` was called at least once (args irrelevant).                                                                                                                                                                                                                                                                      |
| `toolNeverCalled`               | `toolName`                                                    | Selection  | `toolName` was never called (forbidden-tool check).                                                                                                                                                                                                                                                                         |
| `onlyToolsCalled`               | `toolNames`                                                   | Selection  | Every tool called appears in `toolNames`. An empty array is the "no tool was called" claim. Generalizes `toolNeverCalled` from one forbidden tool to an allowed set.                                                                                                                                                        |
| `firstToolWas`                  | `toolName`                                                    | Selection  | The first tool call observed in the transcript was `toolName`.                                                                                                                                                                                                                                                              |
| `responseContains`              | `needle`, `caseSensitive?`                                    | User value | The final assistant message contains `needle`. Case-insensitive by default.                                                                                                                                                                                                                                                 |
| `responseMatches`               | `pattern`                                                     | User value | The final assistant message matches the regular expression `pattern` (regex source string, no flags).                                                                                                                                                                                                                       |
| `noToolErrors`                  | —                                                             | Response   | No tool produced an error (neither MCP `isError: true` nor a JSON-RPC/transport failure).                                                                                                                                                                                                                                   |
| `finalAssistantMessageNonEmpty` | —                                                             | User value | The final assistant message is a non-empty, non-whitespace string.                                                                                                                                                                                                                                                          |
| `tokenBudgetUnder`              | `tokens`                                                      | User value | Total token usage for the iteration is strictly under `tokens`. Fails when usage was not measured.                                                                                                                                                                                                                          |
| `widgetRendered`                | `toolName?`                                                   | User value | At least one MCP App widget render observation (narrowed to `toolName` when set) has `status === "rendered"`. Fails when the iteration recorded no render observations in scope.                                                                                                                                            |
| `widgetRenderLatencyUnder`      | `ms`, `toolName?`                                             | User value | Every rendered widget observation (narrowed to `toolName` when set) mounted in strictly under `ms` milliseconds. Fails when nothing in scope rendered — an unrendered widget has no latency to check.                                                                                                                       |
| `widgetNoConsoleErrors`         | `toolName?`                                                   | User value | No widget render observation (narrowed to `toolName` when set) captured console errors. Fails when the iteration recorded no render observations in scope.                                                                                                                                                                  |
| `turnCountUnder`                | `turns`                                                       | User value | The iteration used strictly fewer than `turns` user turns. Fails when the transcript carries no turn count.                                                                                                                                                                                                                 |
| `noEndingQuestion`              | —                                                             | User value | **Observation.** The final assistant message's last non-empty line does not end with `?`. Cannot tell an offer ("Would you like a breakdown?") from a request for missing input, so it is Warn/Report only — `role: "advisory"` is required and a gating one is refused.                                                    |
| `toolLatencyUnder`              | `ms`, `toolName?`                                             | Response   | Every observed call in scope settled in strictly under `ms`. Reports `status: "error"` when the run captured no per-call timing.                                                                                                                                                                                            |
| `toolResultContains`            | `needle`, `caseSensitive?`, `toolName?`                       | Response   | A tool result in scope contains `needle`, searched over the model-visible text and the structured payload.                                                                                                                                                                                                                  |
| `toolResultMatchesSchema`       | `schema`, `toolName?`                                         | Response   | Every tool result in scope validates against the authored JSON Schema. Any JSON root — 2026-07-28 dropped the object-only restriction on `structuredContent`.                                                                                                                                                               |
| `toolResultSizeUnder`           | `maxBytes`, `toolName?`                                       | Response   | Every tool result in scope is strictly under `maxBytes`, measured on what the server returned and **before** the transcript's own 64 000-character text cap. Bytes, not tokens.                                                                                                                                             |
| `argumentsMatchToolSchema`      | `toolName?`                                                   | Tool call  | Every call's arguments validate against the tool's declared `inputSchema`, classified as `missing-required`, `wrong-type`, `bad-enum` or `hallucinated-param`. A key merely absent from `properties` is legal and is reported, never failed. Schema validity does not establish that the arguments match the user's intent. |
| `noRepeatedIdenticalCall`       | `toolName?`                                                   | Tool call  | **Observation.** No call repeated the one immediately before it with equal arguments. A poll loop and a transient retry are the same shape and both correct, so it is Warn/Report only.                                                                                                                                     |
| `toolCallCountUnder`            | `count`, `toolName?`                                          | Selection  | Strictly fewer than `count` calls in scope. Total calls, which is not the same as calls before the right tool.                                                                                                                                                                                                              |
| `toolCalledBefore`              | `toolName`, `beforeToolName`                                  | Selection  | Every call to `beforeToolName` was preceded by a call to `toolName`. Vacuously true when `beforeToolName` never ran.                                                                                                                                                                                                        |
| `noDeprecatedToolCalled`        | —                                                             | Selection  | **Observation.** No called tool's own description marks it deprecated. Anchored to self-deprecation, so "Replaces the deprecated `old_search` tool" does not fire. Warn/Report only.                                                                                                                                        |
| `noDestructiveToolCalled`       | —                                                             | Selection  | No called tool declares `annotations.destructiveHint`. Reports `status: "error"` when no tool in the inventory declares annotations at all.                                                                                                                                                                                 |
| `toolErrorNamesInput`           | `toolName?`                                                   | Response   | **Observation.** Every tool error message names one of that tool's input keys or a value the call sent. It does not measure recovery quality — "Rate limited. Retry in 30 seconds." names nothing and is a good message. Warn/Report only.                                                                                  |
| `fullPageHasContinuation`       | `toolName?`                                                   | Response   | **Observation.** A page whose length equals its requested limit carries recognized continuation metadata. A full page is not proof that more results exist, so this reports rather than fails. Warn/Report only.                                                                                                            |

#### ArgMatcher

Used by `toolCalledWith` to specify expected arguments and matching mode.

| Property           | Type                               | Description                                                                                                                                          |
| ------------------ | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `args`             | `Record<string, unknown>`          | Expected argument shape.                                                                                                                             |
| `argumentMatching` | `"partial" \| "exact" \| "ignore"` | Matching mode. `"partial"` (default) checks only the keys present in `args`; `"exact"` requires deep equality; `"ignore"` skips argument comparison. |

#### A predicate never passes by default

A predicate that cannot be evaluated fails the case instead of silently passing. That covers malformed predicates (unknown type, missing required fields, invalid `minCount`, empty `needle`/`pattern`) and predicates with nothing to measure: `tokenBudgetUnder` when usage was not captured, and the `widget*` predicates when the iteration recorded no render observations. `responseMatches` also rejects patterns with nested quantifiers (ReDoS guard) and messages over 100,000 characters.

#### Observations may never gate

Some kinds above are marked **Observation**. They are heuristics — a pattern
that can be right about what it saw and still wrong about what it means — so
they carry one rule everywhere: `role: "advisory"` is required, a gating one is
refused by the schema and by the backend, the authoring UI offers Warn and
Report only, and they never enter `allGatingScorersPassed`.

#### Missing evidence is an error, not a failure

The kinds that read tool results, per-call timings or the tool inventory report
`status: "error"` when the run did not capture what they need. An error row
carries no value, keeps its scorer in `unresolvedScorerIds`, and leaves its
stage `notMeasured`. A gate with an unscored row does not pass — but nothing is
attributed to the server for a measurement we could not take.

#### Example

```typescript theme={"theme":"css-variables"}
import type { Predicate } from "@mcpjam/sdk/predicates";

const predicates: Predicate[] = [
  // Tool was called with the right airline argument
  {
    type: "toolCalledWith",
    toolName: "book_flight",
    args: { args: { airline: "DL" } },
  },
  // Response mentions the booking confirmation
  { type: "responseContains", needle: "confirmed" },
  // No tool errors occurred
  { type: "noToolErrors" },
  // Token budget respected
  { type: "tokenBudgetUnder", tokens: 2000 },
];

// Pass predicates via the per-run override in the Inspector corpus,
// or use evaluatePredicates() directly in custom pipelines:
import { evaluatePredicates, allPredicatesPassed, buildIterationTranscript } from "@mcpjam/sdk/predicates";

const transcript = buildIterationTranscript({
  trace: myTrace,
  toolCalls: myToolCalls,
  usage: { totalTokens: 1200 },
});

const results = evaluatePredicates(transcript, predicates);
// `allPredicatesPassed`, not `results.every(...)`: the results array keeps
// failed ADVISORY rows so a reader can see them, and counting those as
// failures rejects a run over a finding that was never meant to gate.
const allPassed = allPredicatesPassed(results);
```

#### Predicate results in metadata

When the Inspector runner evaluates predicates, it persists one row per predicate to `testIteration.metadata.predicates`:

```typescript theme={"theme":"css-variables"}
type PredicateResult = {
  predicate: Predicate;   // The authored predicate (sensitive arg keys are redacted)
  passed: boolean;
  reason: string;         // Structured explanation, e.g. "tool "book_flight" called with matching args"
  scope?: { kind: "turn"; promptIndex: number }; // Absent = case-level; present = authored on one prompt turn
};
```

***

### EvalReportingError

Thrown by `reportEvalResults()` and (when `strict: true`) by `reportEvalResultsSafely()` when an upload fails. Extends `Error`.

| Property                         | Type                  | Description                                                                                                                                                                                                                                            |
| -------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `isBillingLimitReached`          | `boolean`             | The org's billing limit was reached. No results were filed.                                                                                                                                                                                            |
| `isReportingBackendIncompatible` | `boolean`             | The destination rejected a field this SDK sends — the reporting backend is older than the SDK's minimum contract. No results were filed, and retrying will not help. Upgrade the destination or point `baseUrl` at one that accepts declared case ids. |
| `statusCode`                     | `number \| undefined` | HTTP status code from the backend, if available.                                                                                                                                                                                                       |
| `attemptCount`                   | `number \| undefined` | Number of upload attempts made before giving up.                                                                                                                                                                                                       |
| `endpoint`                       | `string \| undefined` | The endpoint path that was called.                                                                                                                                                                                                                     |

```typescript theme={"theme":"css-variables"}
import { reportEvalResults, EvalReportingError } from "@mcpjam/sdk";

try {
  await reportEvalResults({ suiteName: "CI", results, baseUrl: myBaseUrl });
} catch (err) {
  if (err instanceof EvalReportingError) {
    if (err.isReportingBackendIncompatible) {
      // The destination predates @mcpjam/sdk 6 — upgrade it or switch baseUrl.
      console.error("Reporting backend needs upgrade:", err.message);
    } else if (err.isBillingLimitReached) {
      console.error("Billing limit reached");
    } else {
      throw err;
    }
  }
}
```

***

### ReportEvalResultsOutput

| Property           | Type                      | Description                                                                                                                           |
| ------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `suiteId`          | `string`                  | Created/matched suite ID                                                                                                              |
| `runId`            | `string`                  | Created run ID                                                                                                                        |
| `projectId`        | `string?`                 | Project the run landed in, echoed by the ingest response. Absent against a backend that predates the field, and when reporting fails. |
| `status`           | `"completed" \| "failed"` | Run status                                                                                                                            |
| `result`           | `"passed" \| "failed"`    | Pass/fail based on criteria                                                                                                           |
| `summary.total`    | `number`                  | Total iterations                                                                                                                      |
| `summary.passed`   | `number`                  | Passed iterations                                                                                                                     |
| `summary.failed`   | `number`                  | Failed iterations                                                                                                                     |
| `summary.passRate` | `number`                  | Pass rate (0.0 - 1.0)                                                                                                                 |

***

### The printed run URL

After a successful upload the SDK prints one line per run:

```text theme={"theme":"css-variables"}
[mcpjam/sdk] View run: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>
```

It is on by default and there is nothing to configure. Details worth knowing:

* **Once per run.** A chunked upload passes through several code paths
  (start, finalize) and still prints a single line. Each `EvalTest` is its
  own run, so a file with several tests prints one line each.
* **`?project=` may be absent.** The param carries whatever the backend
  echoed, falling back to a `project` you configured explicitly. The
  zero-config `"default"` sentinel is not a project ID, so it is omitted
  rather than sent — the app then opens the run in your active project.
* **Nothing prints when there is no run to link to.** A failed upload, or a
  non-strict reporter falling back to locally-computed counts, prints
  nothing.

***

## Related

* [Running Evals](/sdk/concepts/running-evals) - Conceptual guide
* [EvalTest Reference](/sdk/reference/eval-test) - EvalTest API
* [EvalSuite Reference](/sdk/reference/eval-suite) - EvalSuite API


# EvalSuite
Source: https://docs.mcpjam.com/sdk/reference/eval-suite

API reference for EvalSuite

The `EvalSuite` class groups multiple `EvalTest` instances and provides aggregate metrics across all tests.

## Import

```typescript theme={"theme":"css-variables"}
import { EvalSuite, EvalTest } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={"theme":"css-variables"}
new EvalSuite(options?: EvalSuiteConfig)
```

### Parameters

<ParamField type="EvalSuiteConfig">
  Configuration for the evaluation suite.
</ParamField>

### EvalSuiteConfig

| Property       | Type                                                        | Required | Description                                                                          |
| -------------- | ----------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------ |
| `name`         | `string`                                                    | No       | Name for the suite (defaults to `"EvalSuite"`)                                       |
| `mcpjam`       | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting)    | No       | Auto-save results to MCPJam when the suite completes                                 |
| `matchOptions` | [`EvalMatchOptions`](/sdk/reference/eval-test#matchoptions) | No       | Default matcher policy for every test in the suite that declares `expectedToolCalls` |

<Note>
  `matchOptions` is a **default**, not an override: a test that sets its own keeps it. This mirrors how the hosted product layers suite → case, so a suite behaves the same whether it runs from your test file or from the platform.
</Note>

<Note>
  When an API key is available — via `mcpjam.apiKey` or the `MCPJAM_API_KEY` environment variable — all test results are consolidated into a single run and saved to MCPJam after the suite completes. Individual `EvalTest` auto-saves are suppressed to avoid duplicate uploads. Set `mcpjam.enabled: false` to disable.
</Note>

### Example

```typescript theme={"theme":"css-variables"}
const suite = new EvalSuite({ name: "Math Operations" });
```

With results saved to MCPJam:

```typescript theme={"theme":"css-variables"}
const reportingSuite = new EvalSuite({
  name: "Math Operations",
  mcpjam: {
    suiteName: "Math Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});
```

***

## Methods

### add()

Adds a test to the suite.

```typescript theme={"theme":"css-variables"}
add(test: EvalTest): void
```

#### Parameters

| Parameter | Type       | Description     |
| --------- | ---------- | --------------- |
| `test`    | `EvalTest` | The test to add |

#### Example

```typescript theme={"theme":"css-variables"}
suite.add(new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 2 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  id: "c_multiplication",
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 5");
    return r.hasToolCall("multiply");
  },
}));
```

***

### run()

Runs all tests in the suite and returns aggregate results.

```typescript theme={"theme":"css-variables"}
run(executor: HostExecutor, options: EvalTestRunOptions): Promise<EvalSuiteResult>
```

#### Parameters

| Parameter  | Type                 | Description                                                      |
| ---------- | -------------------- | ---------------------------------------------------------------- |
| `executor` | `HostExecutor`       | The executor to test with (`HostRunner`, `HostRuntime`, or mock) |
| `options`  | `EvalTestRunOptions` | Run configuration (same as EvalTest)                             |

#### EvalTestRunOptions

| Property      | Type                                                     | Required | Default | Description                                                  |
| ------------- | -------------------------------------------------------- | -------- | ------- | ------------------------------------------------------------ |
| `iterations`  | `number`                                                 | Yes      | -       | Number of runs per test                                      |
| `concurrency` | `number`                                                 | No       | `5`     | Parallel runs per test                                       |
| `retries`     | `number`                                                 | No       | `0`     | Retry failed tests                                           |
| `timeoutMs`   | `number`                                                 | No       | `30000` | Timeout per test (ms)                                        |
| `onProgress`  | `ProgressCallback`                                       | No       | -       | Progress callback (tracks total across all tests)            |
| `onFailure`   | `(report: string) => void`                               | No       | -       | Called per test with a failure report if any iterations fail |
| `mcpjam`      | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting) | No       | -       | Override suite-level MCPJam config                           |

#### EvalSuiteResult

| Property               | Type                         | Description                       |
| ---------------------- | ---------------------------- | --------------------------------- |
| `tests`                | `Map<string, EvalRunResult>` | Per-test results                  |
| `aggregate.iterations` | `number`                     | Total iterations across all tests |
| `aggregate.successes`  | `number`                     | Total passes                      |
| `aggregate.failures`   | `number`                     | Total failures                    |
| `aggregate.accuracy`   | `number`                     | Overall accuracy (0.0 - 1.0)      |
| `aggregate.tokenUsage` | `{ total, perTest[] }`       | Token usage                       |
| `aggregate.latency`    | `{ e2e, llm, mcp }`          | Latency stats with p50/p95        |

#### Example

```typescript theme={"theme":"css-variables"}
const result = await suite.run(agent, {
  iterations: 30,
  concurrency: 5,
});

console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);
```

<Note>
  Tests within the suite run sequentially, but each test's iterations can run concurrently based on the `concurrency` setting.
</Note>

***

### accuracy()

Returns the aggregate accuracy across all tests.

```typescript theme={"theme":"css-variables"}
accuracy(): number
```

#### Returns

`number` - Average accuracy of all tests (0.0 - 1.0).

#### Example

```typescript theme={"theme":"css-variables"}
console.log(`Suite accuracy: ${(suite.accuracy() * 100).toFixed(1)}%`);
```

***

### get()

Retrieves a specific test by name.

```typescript theme={"theme":"css-variables"}
get(name: string): EvalTest | undefined
```

#### Parameters

| Parameter | Type     | Description   |
| --------- | -------- | ------------- |
| `name`    | `string` | The test name |

#### Returns

`EvalTest | undefined` - The test, or `undefined` if not found.

#### Example

```typescript theme={"theme":"css-variables"}
const addTest = suite.get("addition");
if (addTest) {
  console.log(`Addition: ${(addTest.accuracy() * 100).toFixed(1)}%`);
}
```

***

### getAll()

Returns all tests in the suite.

```typescript theme={"theme":"css-variables"}
getAll(): EvalTest[]
```

#### Returns

`EvalTest[]` - Array of all tests.

#### Example

```typescript theme={"theme":"css-variables"}
for (const test of suite.getAll()) {
  console.log(`${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}
```

***

### getName()

Returns the suite's name.

```typescript theme={"theme":"css-variables"}
getName(): string
```

***

### size()

Returns the number of tests in the suite.

```typescript theme={"theme":"css-variables"}
size(): number
```

***

### getResults()

Returns the full suite results from the last run.

```typescript theme={"theme":"css-variables"}
getResults(): EvalSuiteResult | null
```

***

### recall()

Aggregate recall across every test in the suite that declared `expectedToolCalls`.

```typescript theme={"theme":"css-variables"}
recall(): number
```

***

### precision()

Aggregate precision across every test in the suite that declared `expectedToolCalls`.

```typescript theme={"theme":"css-variables"}
precision(): number
```

<Warning>
  **Changed in 3.0.** These returned `accuracy()` — the suite's pass rate —
  under all three names. They now aggregate real tool-call counts and **throw**
  when no test in the suite declared `expectedToolCalls`. Tests without
  expectations are skipped in the aggregate rather than counted as perfect. See
  [`EvalTest.precision()`](/sdk/reference/eval-test#precision) for how the
  counts are derived.
</Warning>

***

### truePositiveRate()

Aggregate true positive rate (same as recall).

```typescript theme={"theme":"css-variables"}
truePositiveRate(): number
```

***

### unexpectedToolCallRate()

The fraction of expectation-bearing iterations across the suite that made at least one tool call nobody asked for.

```typescript theme={"theme":"css-variables"}
unexpectedToolCallRate(): number
```

***

### falsePositiveRate()

<Warning>
  **Deprecated in 3.0** — use [`unexpectedToolCallRate()`](#unexpectedtoolcallrate).
  It returned `failures / iterations`, which is the failure rate. Suites with no
  `expectedToolCalls` still get that legacy value; suites with expectations now
  delegate to `unexpectedToolCallRate()`.
</Warning>

```typescript theme={"theme":"css-variables"}
falsePositiveRate(): number
```

***

### averageTokenUse()

Returns average tokens per iteration across all tests.

```typescript theme={"theme":"css-variables"}
averageTokenUse(): number
```

***

## Properties

### name

The suite's name (via `getName()`).

```typescript theme={"theme":"css-variables"}
suite.getName() // "Math Operations"
```

***

## Complete Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalSuite, EvalTest } from "@mcpjam/sdk";

async function main() {
  // Setup
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-5",
    apiKey: process.env.ANTHROPIC_API_KEY,
    temperature: 0.1,
  });

  // Build suite (with auto-save to MCPJam)
  const suite = new EvalSuite({
    name: "Everything Server Tests",
    mcpjam: {
      suiteName: "Everything Server Eval",
      passCriteria: { minimumPassRate: 90 },
    },
  });

  suite.add(new EvalTest({
    id: "c_add",
    name: "add",
    test: async (a) => (await a.run("Add 2+3")).hasToolCall("add"),
  }));

  suite.add(new EvalTest({
    id: "c_echo",
    name: "echo",
    test: async (a) => (await a.run("Echo 'test'")).hasToolCall("echo"),
  }));

  suite.add(new EvalTest({
    id: "c_longrunningoperation",
    name: "longRunningOperation",
    test: async (a) => (await a.run("Run a long operation")).hasToolCall("longRunningOperation"),
  }));

  // Run
  console.log(`Running ${suite.getName()}...\n`);

  const result = await suite.run(agent, {
    iterations: 20,
    concurrency: 3,
    onProgress: (done, total) => {
      process.stdout.write(`\r  Progress: ${done}/${total}`);
    },
  });

  // Report
  console.log(`\n\nOverall: ${(suite.accuracy() * 100).toFixed(1)}%`);
  console.log(`Total iterations: ${result.aggregate.iterations}\n`);

  console.log("Per-test breakdown:");
  for (const test of suite.getAll()) {
    const pct = (test.accuracy() * 100).toFixed(1);
    console.log(`  ${test.getName()}: ${pct}%`);
  }

  // Access individual test
  const echoTest = suite.get("echo");
  if (echoTest) {
    console.log(`\nEcho test details:`);
    console.log(`  Precision: ${(echoTest.precision() * 100).toFixed(1)}%`);
    console.log(`  Recall: ${(echoTest.recall() * 100).toFixed(1)}%`);
    console.log(`  Avg tokens: ${echoTest.averageTokenUse()}`);
  }

  // Cleanup
  await manager.disconnectServer("everything");
}
```

***

## Patterns

### CI Quality Gate

```typescript theme={"theme":"css-variables"}
await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`❌ Suite accuracy ${(suite.accuracy() * 100).toFixed(1)}% below 90% threshold`);
  process.exit(1);
}

console.log("✅ All quality gates passed");
```

### Per-Test Thresholds

```typescript theme={"theme":"css-variables"}
await suite.run(agent, { iterations: 30 });

const criticalTests = ["createOrder", "processPayment"];
let failed = false;

for (const name of criticalTests) {
  const test = suite.get(name);
  if (test && test.accuracy() < 0.95) {
    console.error(`❌ Critical test "${name}" below 95%`);
    failed = true;
  }
}

if (failed) process.exit(1);
```

### Comparing Across Providers

```typescript theme={"theme":"css-variables"}
const providers = [
  { model: "anthropic/claude-sonnet-5", key: "ANTHROPIC_API_KEY" },
  { model: "openai/gpt-5.6-sol", key: "OPENAI_API_KEY" },
];

for (const { model, key } of providers) {
  const agent = new HostRunner({
    tools,
    model,
    apiKey: process.env[key],
  });

  await suite.run(agent, { iterations: 20 });
  console.log(`${model}: ${(suite.accuracy() * 100).toFixed(1)}%`);
}
```

***

## Related

* [Running Evals](/sdk/concepts/running-evals) - Conceptual guide
* [EvalTest Reference](/sdk/reference/eval-test) - Individual test API
* [Saving Eval Results](/sdk/reference/eval-reporting) - Save results to MCPJam
* [Testing Across Providers](/sdk/concepts/multi-provider) - Compare LLMs


# EvalTest
Source: https://docs.mcpjam.com/sdk/reference/eval-test

API reference for EvalTest

The `EvalTest` class runs a single test scenario multiple times and provides statistical metrics like accuracy, precision, and recall.

## Import

```typescript theme={"theme":"css-variables"}
import { EvalTest } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={"theme":"css-variables"}
new EvalTest(options: EvalTestConfig)
```

### Parameters

<ParamField type="EvalTestConfig">
  Configuration for the evaluation test.
</ParamField>

### EvalTestConfig

| Property            | Type                     | Required | Description                                                                                                                                                                                 |
| ------------------- | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                | `string`                 | Yes      | The case's identity. **Required from 5.0** — see below.                                                                                                                                     |
| `name`              | `string`                 | Yes      | Human-readable display name for the test                                                                                                                                                    |
| `test`              | `TestFunction`           | Yes      | The test function to run                                                                                                                                                                    |
| `expectedToolCalls` | `EvalExpectedToolCall[]` | No       | Tool calls the iteration must make. **Enforced locally from 3.0** — see below.                                                                                                              |
| `matchOptions`      | `EvalMatchOptions`       | No       | How `expectedToolCalls` is matched (order, arguments, extras)                                                                                                                               |
| `intent`            | `string`                 | No       | Analytics grouping label for this case. Forwarded with every result but never affects scoring or the verdict. Must be 1–64 characters, already trimmed (no leading or trailing whitespace). |
| `predicates`        | `Predicate[]`            | No       | Deterministic transcript checks that gate each iteration                                                                                                                                    |

<Warning>
  **New in 5.0: `id` is required.** A case's identity used to be its `name`, so
  renaming a test forked its history — the run page started a fresh series and
  the old one was orphaned. `id` separates the two: history joins on `id`, and
  `name` is free to change.

  Mint one **once** and commit the literal beside the test:

  ```typescript theme={"theme":"css-variables"}
  import { mintCaseId } from "@mcpjam/sdk/contract";
  console.log(mintCaseId()); // c_V1StGXR8Z5jdHi6Bmy — paste this in
  ```

  Any URL-safe string of 1–128 characters (`A-Z a-z 0-9 _ -`) is accepted, so an
  id you already have — a hosted case id, a ticket number, a slug you maintain —
  works as-is; the `c_` prefix is only a grep convenience. Do **not** call
  `mintCaseId()` inline: an id regenerated on every run is not an identity.
  Constructing an `EvalTest` without an `id` throws, and the error carries a
  freshly minted id to paste.

  **Migrating from `externalCaseId`:** If your test already declares
  `externalCaseId`, reuse that value verbatim as `id` — do not mint a new one.
  The hosted history for that case is keyed on `externalCaseId`, so setting
  `id` to the same value keeps the history intact. Setting `id` to a different
  value is a hard error — at construction from 6.0, and at ingest.

  If your `externalCaseId` is not a valid id (it was never charset-bound, so it
  may contain spaces, slashes or punctuation), no `id` can agree with it.
  Rename the external id itself to a conforming value and declare that same
  value in both fields.

  ```typescript theme={"theme":"css-variables"}
  // Before
  new EvalTest({ externalCaseId: "case_123", name: "refunds a duplicate charge", ... })

  // After — reuse the existing externalCaseId as id
  new EvalTest({ id: "case_123", externalCaseId: "case_123", name: "refunds a duplicate charge", ... })
  ```
</Warning>

<Warning>
  **New in 5.0: step objects reject unknown fields.** `prompt`, `toolCall`,
  `interact`, and `assert` steps — and their nested sub-objects such as
  `elementLocator` and widget assertions — are now **closed schemas**. A step
  carrying a field the contract does not declare fails to parse instead of
  silently dropping that field.

  Two things are deliberately left open:

  * `toolCallStep.arguments` — the tool's own argument object, whose keys come
    from the server's input schema.
  * A `Predicate` inside an `assert` step — predicates are a separate contract
    module with their own versioning.

  If you generate steps programmatically (for example, from an LLM or an
  importer), remove any fields that are not part of the step contract before
  passing them to the SDK.
</Warning>

<Warning>
  **Changed in 3.0.** `expectedToolCalls` used to be reporting metadata only:
  the local verdict came from your `test` function alone, while the dashboard
  recomputed the match — so the same run could show `accuracy() === 1` locally
  and fail in MCPJam. It is now enforced during the run, which means a test
  whose expectations were never actually checked can start failing on upgrade.

  If that happens, the expectation was wrong or over-specified. Fix it, relax it
  with `matchOptions` (e.g. `{ argumentMatching: "ignore" }`), or drop
  `expectedToolCalls` if it was only ever documentation. Tests that declare
  neither `expectedToolCalls` nor `predicates` behave exactly as before.
</Warning>

#### matchOptions

Layered suite → case, and validated when the test is constructed rather than mid-run.

| Option              | Values                                   | Default     | Meaning                                      |
| ------------------- | ---------------------------------------- | ----------- | -------------------------------------------- |
| `toolCallOrder`     | `"ignore"` \| `"strict"` \| `"superset"` | `"ignore"`  | Whether expected calls must appear in order  |
| `argumentMatching`  | `"partial"` \| `"exact"` \| `"ignore"`   | `"partial"` | How strictly arguments must match            |
| `maxExtraToolCalls` | `number` \| `null`                       | `null`      | Cap on unexpected calls (`null` = unlimited) |

#### predicates

Deterministic, state-based checks evaluated against the iteration transcript — same transcript, same verdict, which is what makes them usable as a CI gate. An iteration passes only if **every** predicate passes, independently of `failOnToolError`. Verdicts are reported under `metadata.predicates`, so the dashboard's check chips work for code-first runs too. See [eval reporting](/sdk/reference/eval-reporting#predicate-gate) for the full list of predicate types.

```typescript theme={"theme":"css-variables"}
const test = new EvalTest({
  id: "c_books_the_venue",
  name: "books the venue",
  expectedToolCalls: [{ toolName: "book_venue" }],
  matchOptions: { toolCallOrder: "strict" },
  predicates: [
    { type: "noToolErrors" },
    { type: "responseContains", needle: "confirmed" },
  ],
  test: async (executor) => {
    const r = await executor.run("Book the venue for Friday.");
    return r.hasToolCall("book_venue");
  },
});
```

The widget predicates (`widgetRendered`, `widgetRenderLatencyUnder`, `widgetNoConsoleErrors`) need render observations only a hosted run captures, so `EvalTest` rejects them at construction instead of failing every iteration.

### TestFunction Type

```typescript theme={"theme":"css-variables"}
type TestFunction = (executor: HostExecutor) => boolean | Promise<boolean>;
```

The test function receives a [`HostExecutor`](/sdk/reference/host-runner) (the interface implemented by both `HostRunner` and `HostRuntime`) and must return a `boolean`:

* `true` = test passed
* `false` = test failed

Both `HostRunner`, `HostRuntime`, and mock executors implement the `HostExecutor` interface, so you can use any of them for testing.

### Example

```typescript theme={"theme":"css-variables"}
const test = new EvalTest({
  id: "c_addition_accuracy",
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add");
  },
});
```

***

## Methods

### run()

Executes the test multiple times and returns detailed results.

```typescript theme={"theme":"css-variables"}
run(executor: HostExecutor, options: EvalTestRunOptions): Promise<EvalRunResult>
```

#### Parameters

| Parameter  | Type                 | Description                                                      |
| ---------- | -------------------- | ---------------------------------------------------------------- |
| `executor` | `HostExecutor`       | The executor to test with (`HostRunner`, `HostRuntime`, or mock) |
| `options`  | `EvalTestRunOptions` | Run configuration                                                |

#### EvalTestRunOptions

| Property      | Type                                                     | Required | Default | Description                                                                                                                                                                                        |
| ------------- | -------------------------------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iterations`  | `number`                                                 | Yes      | -       | Number of test runs                                                                                                                                                                                |
| `concurrency` | `number`                                                 | No       | `5`     | Parallel test runs                                                                                                                                                                                 |
| `retries`     | `number`                                                 | No       | `0`     | Retry failed tests                                                                                                                                                                                 |
| `timeoutMs`   | `number`                                                 | No       | `30000` | Per-iteration wall-clock timeout in ms. The active prompt is aborted at this deadline, then given a 1 second grace period to settle so partial tool calls and trace history can still be captured. |
| `onProgress`  | `ProgressCallback`                                       | No       | -       | Progress callback                                                                                                                                                                                  |
| `onFailure`   | `(report: string) => void`                               | No       | -       | Called with a failure report if any iterations fail                                                                                                                                                |
| `mcpjam`      | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting) | No       | -       | Auto-save results to MCPJam                                                                                                                                                                        |

<Note>
  Results are automatically saved to MCPJam after the run completes when an API
  key is available via `mcpjam.apiKey` or the `MCPJAM_API_KEY` environment
  variable. Set `mcpjam.enabled: false` to disable.
</Note>

#### ProgressCallback Type

```typescript theme={"theme":"css-variables"}
type ProgressCallback = (completed: number, total: number) => void;
```

#### Example

```typescript theme={"theme":"css-variables"}
await test.run(agent, {
  iterations: 30,
  concurrency: 5,
  retries: 2,
  timeoutMs: 30000,
  mcpjam: {
    suiteName: "Addition Eval",
    strict: false,
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
  onFailure: (report) => {
    console.error(report);
  },
});
```

***

### accuracy()

Returns the success rate (0.0 - 1.0).

```typescript theme={"theme":"css-variables"}
accuracy(): number
```

#### Returns

`number` - Proportion of tests that passed.

#### Example

```typescript theme={"theme":"css-variables"}
console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
```

***

### precision()

Returns the precision metric, micro-averaged over the run's tool-call matches.

```typescript theme={"theme":"css-variables"}
precision(): number
```

#### Returns

`number` — True positives / (True positives + False positives).

Counted per iteration from the expected/actual tool calls: a matched expectation is a true positive, an unexpected call is a false positive, a missing one is a false negative, and an argument mismatch counts as both.

<Warning>
  **Changed in 3.0.** `precision()`, `recall()` and `truePositiveRate()` all
  used to `return this.accuracy()` — three names for one number. They now
  compute real values, and **throw** when no test in the run declared
  `expectedToolCalls`, because there is nothing to compute them from. If you
  were reading `precision()` as a stand-in for accuracy, call
  [`accuracy()`](#accuracy) directly.
</Warning>

***

### recall()

Returns the recall metric, micro-averaged over the run's tool-call matches.

```typescript theme={"theme":"css-variables"}
recall(): number
```

#### Returns

`number` — True positives / (True positives + False negatives).

Throws when the run declared no `expectedToolCalls`.

***

### truePositiveRate()

Returns the true positive rate (same as recall).

```typescript theme={"theme":"css-variables"}
truePositiveRate(): number
```

***

### unexpectedToolCallRate()

The fraction of expectation-bearing iterations that made at least one tool call nobody asked for.

```typescript theme={"theme":"css-variables"}
unexpectedToolCallRate(): number
```

#### Returns

`number` — Iterations with an extra call / iterations that had expectations. `0` when the run declared no expectations.

***

### falsePositiveRate()

<Warning>
  **Deprecated in 3.0** — use [`unexpectedToolCallRate()`](#unexpectedtoolcallrate).
  The old implementation returned `failures / iterations`, which is the failure
  rate, not a false-positive rate. For runs with no `expectedToolCalls` it still
  returns that legacy value so existing dashboards do not change; for runs with
  expectations it now delegates to `unexpectedToolCallRate()`.
</Warning>

```typescript theme={"theme":"css-variables"}
falsePositiveRate(): number
```

***

### averageTokenUse()

Returns the average tokens used per iteration.

```typescript theme={"theme":"css-variables"}
averageTokenUse(): number
```

#### Returns

`number` - Mean token count.

#### Example

```typescript theme={"theme":"css-variables"}
console.log(`Avg tokens: ${test.averageTokenUse()}`);
```

***

### getResults()

Returns the full run result from the last run.

```typescript theme={"theme":"css-variables"}
getResults(): EvalRunResult | null
```

#### Returns

`EvalRunResult | null` - The run result, or `null` if `run()` hasn't been called.

#### EvalRunResult Type

| Property           | Type                | Description                                |
| ------------------ | ------------------- | ------------------------------------------ |
| `iterations`       | `number`            | Total iterations run                       |
| `successes`        | `number`            | Number that passed                         |
| `failures`         | `number`            | Number that failed                         |
| `results`          | `boolean[]`         | Pass/fail per iteration                    |
| `iterationDetails` | `IterationResult[]` | Detailed per-iteration results             |
| `tokenUsage`       | `object`            | Aggregate and per-iteration token usage    |
| `latency`          | `object`            | Latency stats (e2e, llm, mcp) with p50/p95 |

#### IterationResult Type

| Property     | Type                          | Description                          |
| ------------ | ----------------------------- | ------------------------------------ |
| `passed`     | `boolean`                     | Whether this iteration passed        |
| `latencies`  | `LatencyBreakdown[]`          | Latency per prompt in this iteration |
| `tokens`     | `{ total, input, output }`    | Token usage                          |
| `error`      | `string \| undefined`         | Error message if failed              |
| `retryCount` | `number \| undefined`         | Number of retries attempted          |
| `prompts`    | `PromptResult[] \| undefined` | Prompt results from this iteration   |

***

### getName()

Returns the test's name.

```typescript theme={"theme":"css-variables"}
getName(): string
```

***

### getConfig()

Returns the test's configuration.

```typescript theme={"theme":"css-variables"}
getConfig(): EvalTestConfig
```

***

### getAllIterations()

Returns all iteration details from the last run.

```typescript theme={"theme":"css-variables"}
getAllIterations(): IterationResult[]
```

***

### getFailedIterations()

Returns only the failed iterations from the last run.

```typescript theme={"theme":"css-variables"}
getFailedIterations(): IterationResult[]
```

#### Example

```typescript theme={"theme":"css-variables"}
const failures = test.getFailedIterations();
console.log(`${failures.length} failures`);

for (const fail of failures) {
  console.log(`  Error: ${fail.error}`);
}
```

***

### getSuccessfulIterations()

Returns only the successful iterations from the last run.

```typescript theme={"theme":"css-variables"}
getSuccessfulIterations(): IterationResult[]
```

***

### getFailureReport()

Returns a formatted failure report with traces from all failed iterations. Useful for debugging.

```typescript theme={"theme":"css-variables"}
getFailureReport(): string
```

#### Example

```typescript theme={"theme":"css-variables"}
await test.run(agent, { iterations: 30 });

if (test.accuracy() < 0.9) {
  console.error(test.getFailureReport());
}
```

***

## Properties

### name

The test's identifier (via `getName()`).

```typescript theme={"theme":"css-variables"}
test.getName(); // "addition-accuracy"
```

***

## Test Function Patterns

### Simple Tool Check

```typescript theme={"theme":"css-variables"}
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
};
```

### Argument Validation

```typescript theme={"theme":"css-variables"}
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
};
```

### Response Content

```typescript theme={"theme":"css-variables"}
test: async (agent) => {
  const result = await agent.run("What is 5 + 5?");
  return result.text.includes("10");
};
```

### Multiple Conditions

```typescript theme={"theme":"css-variables"}
test: async (agent) => {
  const result = await agent.run("Calculate 5 * 3");
  return (
    result.hasToolCall("multiply") &&
    !result.hasError() &&
    result.text.length > 0
  );
};
```

### Multi-Turn Conversation

```typescript theme={"theme":"css-variables"}
test: async (agent) => {
  const r1 = await agent.run("Create a project");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
};
```

### With Validators

```typescript theme={"theme":"css-variables"}
import { matchToolCallWithArgs } from "@mcpjam/sdk";

test: async (agent) => {
  const result = await agent.run("Add 2 and 3");
  return matchToolCallWithArgs("add", { a: 2, b: 3 }, result.getToolCalls());
};
```

***

## Complete Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner, EvalTest } from "@mcpjam/sdk";

async function main() {
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-5",
    apiKey: process.env.ANTHROPIC_API_KEY,
    temperature: 0.1,
  });

  const test = new EvalTest({
    id: "c_addition",
    name: "addition",
    test: async (agent) => {
      const r = await agent.run("Add 2 and 3");
      return r.hasToolCall("add");
    },
  });

  console.log("Running evaluation...");

  const result = await test.run(agent, {
    iterations: 30,
    concurrency: 5,
    mcpjam: { suiteName: "Addition Eval" },
    onProgress: (done, total) => {
      process.stdout.write(`\r${done}/${total}`);
    },
    onFailure: (report) => {
      console.error(report);
    },
  });

  console.log("\n\nResults:");
  console.log(`  Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
  console.log(`  Precision: ${(test.precision() * 100).toFixed(1)}%`);
  console.log(`  Recall: ${(test.recall() * 100).toFixed(1)}%`);
  console.log(`  Avg tokens: ${test.averageTokenUse()}`);
  console.log(`  Iterations: ${result.iterations}`);
  console.log(`  Successes: ${result.successes}`);

  await manager.disconnectServer("everything");
}
```

***

## Related

* [Running Evals](/sdk/concepts/running-evals) - Conceptual guide
* [EvalSuite Reference](/sdk/reference/eval-suite) - Group multiple tests
* [Saving Eval Results](/sdk/reference/eval-reporting) - Save results to MCPJam
* [Validators Reference](/sdk/reference/validators) - Assertion functions


# host-compat
Source: https://docs.mcpjam.com/sdk/reference/host-compat

API reference for @mcpjam/sdk/host-compat — derive server requirements, evaluate host compatibility, and scan widget bridge usage.

`@mcpjam/sdk/host-compat` is the shared host-compatibility engine. It answers "does this MCP server work on host X?" by joining a server's derived requirements against per-host capability profiles to produce `works` / `degraded` / `blocked` verdicts. The same engine powers the Inspector's [Compatibility destination](/inspector/compatibility), the CLI, and the public API.

## Import

```typescript theme={"theme":"css-variables"}
import {
  deriveServerRequirements,
  evaluateHostCompat,
  evaluateAllHosts,
  scanWidgetSource,
  scanWidgetMeta,
  detectHostCompatBridgeFromMeta,
  HostCompatBridge,
} from "@mcpjam/sdk/host-compat";
```

***

## deriveServerRequirements()

Derives what a server demands of a host from its tools list, optional widget-usage scan, and optional connection facts.

```typescript theme={"theme":"css-variables"}
deriveServerRequirements(
  toolsData?: HostCompatToolsInput | null,
  widgetUsage?: WidgetUsage,
  connectionFacts?: ConnectionFacts,
): ServerRequirements
```

### Parameters

| Parameter         | Type                           | Description                                                                                                                                                                              |
| ----------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `toolsData`       | `HostCompatToolsInput \| null` | Tools list from `ListToolsResult`, plus optional per-tool `_meta` metadata.                                                                                                              |
| `widgetUsage`     | `WidgetUsage`                  | Capability needs derived from scanning widget HTML (from `scanWidgetSource`). Pass `undefined` when HTML hasn't been fetched yet — the result will include an `unknownDimensions` entry. |
| `connectionFacts` | `ConnectionFacts`              | Protocol version negotiated at connect time.                                                                                                                                             |

### HostCompatToolsInput

| Property        | Type                                      | Description                                                                           |
| --------------- | ----------------------------------------- | ------------------------------------------------------------------------------------- |
| `tools`         | `HostCompatTool[]`                        | Array of tools. Each tool needs at minimum a `name` and optional `_meta`.             |
| `toolsMetadata` | `Record<string, Record<string, unknown>>` | Per-tool `_meta` map keyed by tool name. When present, wins over inline `tool._meta`. |

### Returns: ServerRequirements

| Property                 | Type                           | Description                                                                                                              |
| ------------------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `widgets.mcpAppsOnly`    | `string[]`                     | Tool names that declare only an MCP Apps widget template.                                                                |
| `widgets.openaiAppsOnly` | `string[]`                     | Tool names that declare only an OpenAI Apps SDK template.                                                                |
| `widgets.dual`           | `string[]`                     | Tool names that declare both templates.                                                                                  |
| `appOnlyWidgets`         | `string[]`                     | Widget tools with no text fallback (hidden from the model).                                                              |
| `hasWidgets`             | `boolean`                      | Whether any widget tools were found.                                                                                     |
| `widgetUsage`            | `WidgetUsage \| undefined`     | The passed-in widget usage scan, if any.                                                                                 |
| `connectionFacts`        | `ConnectionFacts \| undefined` | The passed-in connection facts, if any.                                                                                  |
| `unknownDimensions`      | `string[]`                     | Dimensions that couldn't be determined (e.g. widget HTML not yet fetched). Non-empty means the verdict may be `unknown`. |

### Example

```typescript theme={"theme":"css-variables"}
import { deriveServerRequirements } from "@mcpjam/sdk/host-compat";

const tools = await manager.listTools("my-server");
const requirements = deriveServerRequirements(
  { tools: tools.tools },
  undefined, // widgetUsage — not yet scanned
  { protocolVersion: "2025-11-25" },
);

console.log(requirements.hasWidgets);          // true / false
console.log(requirements.widgets.mcpAppsOnly); // ["draw_chart", ...]
console.log(requirements.unknownDimensions);   // ["widget capabilities (widget HTML not analyzed)"]
```

***

## evaluateHostCompat()

Evaluates a server's requirements against a single host profile and returns a `HostCompatReport`.

```typescript theme={"theme":"css-variables"}
evaluateHostCompat(
  requirements: ServerRequirements,
  profile: HostCompatProfile,
): HostCompatReport
```

### Parameters

| Parameter      | Type                 | Description                                                  |
| -------------- | -------------------- | ------------------------------------------------------------ |
| `requirements` | `ServerRequirements` | Output of `deriveServerRequirements`.                        |
| `profile`      | `HostCompatProfile`  | A host capability profile describing what the host supports. |

### Returns: HostCompatReport

| Property       | Type                | Description                                                                               |
| -------------- | ------------------- | ----------------------------------------------------------------------------------------- |
| `hostId`       | `string`            | Profile identifier (e.g. `"claude"`, `"chatgpt"`).                                        |
| `hostLabel`    | `string`            | Human-readable host name.                                                                 |
| `verdict`      | `CompatVerdict`     | `"works"` \| `"degraded"` \| `"blocked"` \| `"unknown"`                                   |
| `provenance`   | `CompatProvenance`  | Confidence level of the overall report.                                                   |
| `lanes.apps`   | `CompatLaneVerdict` | Apps-lane verdict (widget rendering + capability gaps).                                   |
| `lanes.server` | `CompatLaneVerdict` | Server-lane verdict (protocol version compatibility).                                     |
| `findings`     | `CompatFinding[]`   | Individual findings, each with a lane, severity, title, detail, and optional remediation. |

### CompatVerdict

| Value        | Meaning                                                |
| ------------ | ------------------------------------------------------ |
| `"works"`    | No blockers or degraded findings.                      |
| `"degraded"` | Functional but with capability gaps or text fallbacks. |
| `"blocked"`  | App-only tools are unusable on this host.              |
| `"unknown"`  | A required dimension couldn't be determined.           |

***

## evaluateAllHosts()

Evaluates a server against multiple host profiles in one call.

```typescript theme={"theme":"css-variables"}
evaluateAllHosts(
  toolsData: HostCompatToolsInput | null | undefined,
  profiles: HostCompatProfile[],
  options?: EvaluateAllHostsOptions,
): HostCompatEvaluation
```

### Parameters

| Parameter                 | Type                           | Description                                 |
| ------------------------- | ------------------------------ | ------------------------------------------- |
| `toolsData`               | `HostCompatToolsInput \| null` | Tools list input.                           |
| `profiles`                | `HostCompatProfile[]`          | Array of host profiles to evaluate against. |
| `options.widgetUsage`     | `WidgetUsage`                  | Widget capability scan results.             |
| `options.connectionFacts` | `ConnectionFacts`              | Protocol version from connect time.         |

### Returns: HostCompatEvaluation

| Property       | Type                 | Description                                              |
| -------------- | -------------------- | -------------------------------------------------------- |
| `requirements` | `ServerRequirements` | Derived server requirements.                             |
| `reports`      | `HostCompatReport[]` | One report per profile, in the same order as `profiles`. |

### Example

```typescript theme={"theme":"css-variables"}
import {
  evaluateAllHosts,
  type HostCompatProfile,
} from "@mcpjam/sdk/host-compat";

const claudeProfile: HostCompatProfile = {
  id: "claude",
  label: "Claude",
  provenance: "probe",
  rendersMcpApps: true,
  rendersOpenAiApps: false,
  supportedProtocolVersions: ["2025-11-25", "2026-07-28"],
  capabilities: {
    serverTools: true,
    openLinks: true,
    message: true,
    // ...
  },
};

const { requirements, reports } = evaluateAllHosts(
  { tools: toolsList },
  [claudeProfile],
  { connectionFacts: { protocolVersion: "2025-11-25" } },
);

for (const report of reports) {
  console.log(`${report.hostLabel}: ${report.verdict}`);
  for (const finding of report.findings) {
    console.log(`  [${finding.severity}] ${finding.title} — ${finding.detail}`);
  }
}
```

***

## scanWidgetSource()

Scans widget HTML source to detect which MCP Apps capabilities the widget uses.

```typescript theme={"theme":"css-variables"}
scanWidgetSource(html: string): WidgetUsage
```

### Parameters

| Parameter | Type     | Description                                                 |
| --------- | -------- | ----------------------------------------------------------- |
| `html`    | `string` | Raw HTML of the widget (fetched from the MCP resource URI). |

### Returns: WidgetUsage

A map from capability name to the list of tool names whose widgets use that capability. An empty object `{}` means the HTML was scanned and no capability usage was found — this is conclusive. `undefined` means the scan hasn't run yet.

```typescript theme={"theme":"css-variables"}
type WidgetUsage = Partial<Record<WidgetCapabilityNeed, string[]>>;

type WidgetCapabilityNeed =
  | "serverTools"
  | "serverResources"
  | "message"
  | "updateModelContext"
  | "openLinks"
  | "downloadFile"
  | "sandboxPermissions"
  | "cspFrameDomains"
  | "logging";
```

***

## scanWidgetMeta()

Scans a tool's `_meta` bag to detect widget bridge type and app-only status without fetching the HTML.

```typescript theme={"theme":"css-variables"}
scanWidgetMeta(meta: Record<string, unknown> | undefined): {
  bridge: HostCompatBridge;
  isAppOnly: boolean;
}
```

***

## detectHostCompatBridgeFromMeta()

Detects which widget bridge protocol a tool uses from its `_meta` bag.

```typescript theme={"theme":"css-variables"}
detectHostCompatBridgeFromMeta(
  meta: Record<string, unknown> | undefined,
): HostCompatBridge
```

### HostCompatBridge

```typescript theme={"theme":"css-variables"}
enum HostCompatBridge {
  NONE = "none",
  MCP_APPS = "mcp-apps",
  OPENAI_SDK = "openai-sdk",
  OPENAI_SDK_AND_MCP_APPS = "openai-sdk-and-mcp-apps",
}
```

| Value                     | Meaning                                                              |
| ------------------------- | -------------------------------------------------------------------- |
| `NONE`                    | Tool has no widget template.                                         |
| `MCP_APPS`                | Tool declares an MCP Apps widget (`_meta.ui.resourceUri`).           |
| `OPENAI_SDK`              | Tool declares an OpenAI Apps SDK template (`openai/outputTemplate`). |
| `OPENAI_SDK_AND_MCP_APPS` | Tool declares both templates.                                        |

***

## Types

### HostCompatProfile

Describes a host's capabilities for evaluation purposes. Supply your own profiles or use the Inspector's built-in catalog.

| Property                    | Type                                             | Description                                       |
| --------------------------- | ------------------------------------------------ | ------------------------------------------------- |
| `id`                        | `string`                                         | Unique identifier (e.g. `"claude"`).              |
| `label`                     | `string`                                         | Human-readable name.                              |
| `provenance`                | `CompatProvenance`                               | Confidence level of the profile data.             |
| `rendersMcpApps`            | `boolean`                                        | Whether the host renders MCP Apps widgets.        |
| `rendersOpenAiApps`         | `boolean`                                        | Whether the host renders OpenAI Apps SDK widgets. |
| `supportedProtocolVersions` | `string[]`                                       | MCP protocol versions the host advertises.        |
| `capabilities`              | `Partial<Record<WidgetCapabilityNeed, boolean>>` | Which widget capabilities the host supports.      |

### CompatProvenance

| Value          | Meaning                                                  |
| -------------- | -------------------------------------------------------- |
| `"probe"`      | Captured from a real host via DevTools or the inspector. |
| `"vendor-doc"` | Verified from vendor documentation.                      |
| `"observed"`   | Stamped by a live widget render run.                     |
| `"assumed"`    | Best-effort — unverified.                                |

### ConnectionFacts

| Property          | Type     | Description                                                            |
| ----------------- | -------- | ---------------------------------------------------------------------- |
| `protocolVersion` | `string` | MCP protocol version negotiated at connect time (e.g. `"2025-11-25"`). |

***

## Related

* [Compatibility destination](/inspector/compatibility) — the Inspector UI powered by this engine
* [Apps Conformance SDK](/sdk/reference/apps-conformance) — dynamic conformance testing
* [Protocol Conformance SDK](/sdk/reference/protocol-conformance) — MCP protocol checks


# HostRunner
Source: https://docs.mcpjam.com/sdk/reference/host-runner

API reference for HostRunner

The `HostRunner` class runs prompts with MCP tools enabled. It handles the multi-step prompt loop and returns rich result objects.

## Import

```typescript theme={"theme":"css-variables"}
import { HostRunner } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={"theme":"css-variables"}
new HostRunner(options: HostRunnerOptions)
```

### Parameters

<ParamField type="HostRunnerOptions">
  Configuration for the test agent.
</ParamField>

### HostRunnerOptions

| Property             | Type                             | Required | Default     | Description                                                                                                                                                                                                                                                                                                                                                            |
| -------------------- | -------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tools`              | `Record<string, Tool>`           | Yes      | -           | MCP tools from `manager.getTools()`                                                                                                                                                                                                                                                                                                                                    |
| `model`              | `string`                         | Yes      | -           | Model identifier in `provider/model` format                                                                                                                                                                                                                                                                                                                            |
| `apiKey`             | `string`                         | Yes      | -           | API key for the LLM provider                                                                                                                                                                                                                                                                                                                                           |
| `systemPrompt`       | `string`                         | No       | `undefined` | System prompt for the LLM                                                                                                                                                                                                                                                                                                                                              |
| `temperature`        | `number`                         | No       | `undefined` | Sampling temperature (0-2). If undefined, uses model default.                                                                                                                                                                                                                                                                                                          |
| `maxSteps`           | `number`                         | No       | `10`        | Maximum agentic loop iterations                                                                                                                                                                                                                                                                                                                                        |
| `customProviders`    | `Record<string, CustomProvider>` | No       | `undefined` | Custom LLM provider definitions                                                                                                                                                                                                                                                                                                                                        |
| `mcpClientManager`   | `MCPClientManager`               | No       | `undefined` | Same connected manager used for `connectToServer` / `getToolsForAiSdk`. **Required** to populate [`getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) on each [`PromptResult`](/sdk/reference/prompt-result) (see note below).                                                                                                              |
| `injectOpenAiCompat` | `boolean`                        | No       | `false`     | When `true`, injects the OpenAI Apps SDK `window.openai` shim into captured widget HTML snapshots. Enable this when testing widgets written against the OpenAI Apps SDK (ChatGPT / Copilot hosts). Leave it off for SEP-1865-native widgets (Claude, Cursor, Codex) — those hosts don't expose `window.openai` and snapshots should match what the live host produces. |

### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools: await manager.getTools(),
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
  systemPrompt: "You are a helpful assistant.",
  temperature: 0.3,
  maxSteps: 5,
});
```

<Info>
  **MCP Apps, traces, and `mcpClientManager`:** MCP App widgets are backed by HTML served from an MCP **resource** (`ui.resourceUri`), not by the tool’s JSON result alone. After each tool call, `HostRunner` uses the manager’s [`readResource`](/sdk/reference/mcp-client-manager#readresource) to fetch that HTML and attach **`widgetSnapshots`** to the [`PromptResult`](/sdk/reference/prompt-result). When you upload results to MCPJam, those snapshots are stored so the Evals trace viewer can **replay the widget offline**. If you omit `mcpClientManager`, prompts still run and tools still execute — but **`widgetSnapshots` stay empty** and uploaded traces will only contain messages and spans (no embedded app replay). Passing `manager` is also what enables **replay credential** capture for authenticated HTTP servers when reporting to MCPJam.
</Info>

***

## Methods

### run()

Sends a prompt to the LLM and returns the result.

```typescript theme={"theme":"css-variables"}
run(
  message: string,
  options?: PromptOptions
): Promise<PromptResult>
```

Use [`getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) on the returned result (or `toEvalResult()` when reporting) to access MCP App HTML captured during the run when `mcpClientManager` was set.

#### Parameters

| Parameter | Type            | Required | Description        |
| --------- | --------------- | -------- | ------------------ |
| `message` | `string`        | Yes      | The user prompt    |
| `options` | `PromptOptions` | No       | Additional options |

#### PromptOptions

| Property            | Type                                                                | Description                                                                                                                                                                           |
| ------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `context`           | `PromptResult \| PromptResult[]`                                    | Previous result(s) for multi-turn conversations                                                                                                                                       |
| `abortSignal`       | `AbortSignal`                                                       | Cancels the prompt runtime when aborted                                                                                                                                               |
| `stopWhen`          | `StopCondition<ToolSet> \| Array<StopCondition<ToolSet>>`           | Additional conditions for the multi-step prompt loop. Tools still execute normally. `HostRunner` always applies `stepCountIs(maxSteps)` as a safety guard.                            |
| `timeout`           | `number \| { totalMs?: number; stepMs?: number; chunkMs?: number }` | Bounds prompt runtime. `number` and `totalMs` cap the full prompt, `stepMs` caps each generation step, and `chunkMs` is accepted for parity but is mainly relevant to streaming APIs. |
| `timeoutMs`         | `number`                                                            | Shortcut for a total prompt timeout in milliseconds                                                                                                                                   |
| `stopAfterToolCall` | `string \| string[]`                                                | Short-circuits the named tool(s) with a stub result and stops after the step where they were called. Tool names and args are still captured in the `PromptResult`.                    |

#### Returns

`Promise<PromptResult>` - The result object with response and metadata.

#### Example

```typescript theme={"theme":"css-variables"}
import { hasToolCall } from "@mcpjam/sdk";

// Simple prompt
const result = await agent.run("Add 2 and 3");

// Multi-turn conversation
const r1 = await agent.run("Create a task called 'Test'");
const r2 = await agent.run("Mark it complete", { context: r1 });

// Multiple context items
const r3 = await agent.run("Show summary", { context: [r1, r2] });

// Stop the loop after the step where a tool is called
const r4 = await agent.run("Search for tasks", {
  stopWhen: hasToolCall("search_tasks"),
});
console.log(r4.hasToolCall("search_tasks"));

// Bound prompt runtime
const r5 = await agent.run("Run a long workflow", {
  timeout: { totalMs: 10_000, stepMs: 2_500 },
});

if (r5.hasError()) {
  console.error(r5.getError());
}

// Exit early after selecting a tool, without waiting for the real MCP call
const r6 = await agent.run("Search for tasks", {
  stopAfterToolCall: "search_tasks",
  timeoutMs: 5_000,
});
console.log(r6.getToolArguments("search_tasks"));
```

<Note>
  `run()` never throws exceptions. Errors are captured in the `PromptResult`.
  Check `result.hasError()` to detect failures.
</Note>

<Note>
  `stopAfterToolCall` is intended for evals that only care about tool selection
  and arguments. If the model emits multiple tool calls in the same step,
  non-target sibling tools may still execute before the loop stops.
</Note>

<Info>
  `timeout` bounds prompt runtime. The runtime creates an internal abort signal,
  so tools can stop early if their implementation respects the provided
  `abortSignal`. If a tool ignores that signal, its underlying work may continue
  briefly after the prompt returns an error result.
</Info>

***

## Model String Format

Models are specified as `provider/model`:

```typescript theme={"theme":"css-variables"}
// Anthropic
"anthropic/claude-sonnet-4-20250514";
"anthropic/claude-3-haiku-20240307";

// OpenAI
"openai/gpt-4o";
"openai/gpt-4o-mini";

// Google
"google/gemini-1.5-pro";
"google/gemini-1.5-flash";

// Azure
"azure/gpt-4o";

// Mistral
"mistral/mistral-large-latest";

// DeepSeek
"deepseek/deepseek-chat";

// Ollama (local)
"ollama/llama3";

// OpenRouter
"openrouter/anthropic/claude-3-opus";

// xAI
"xai/grok-beta";
```

***

## Custom Providers

Add custom OpenAI or Anthropic-compatible endpoints.

### CustomProvider Type

| Property             | Type                                            | Required | Description                      |
| -------------------- | ----------------------------------------------- | -------- | -------------------------------- |
| `name`               | `string`                                        | Yes      | Provider identifier              |
| `protocol`           | `"openai-compatible" \| "anthropic-compatible"` | Yes      | API protocol                     |
| `baseUrl`            | `string`                                        | Yes      | API endpoint URL                 |
| `modelIds`           | `string[]`                                      | Yes      | Available model IDs              |
| `useChatCompletions` | `boolean`                                       | No       | Use `/chat/completions` endpoint |
| `apiKeyEnvVar`       | `string`                                        | No       | Custom env var for API key       |

### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "my-litellm/gpt-4",
  apiKey: process.env.LITELLM_API_KEY,
  customProviders: {
    "my-litellm": {
      name: "my-litellm",
      protocol: "openai-compatible",
      baseUrl: "http://localhost:8000",
      modelIds: ["gpt-4", "gpt-3.5-turbo", "claude-3-sonnet"],
      useChatCompletions: true,
    },
  },
});
```

***

## Configuration Properties

### tools

The MCP tools available to the agent. Obtained from `MCPClientManager.getTools()`.

```typescript theme={"theme":"css-variables"}
const tools = await manager.getTools();
const agent = new HostRunner({ tools, ... });
```

### model

The LLM model identifier. Format: `provider/model-id`.

```typescript theme={"theme":"css-variables"}
model: "anthropic/claude-sonnet-4-20250514";
```

### apiKey

The API key for the LLM provider.

```typescript theme={"theme":"css-variables"}
apiKey: process.env.ANTHROPIC_API_KEY;
```

### systemPrompt

Optional system prompt to guide the LLM's behavior.

```typescript theme={"theme":"css-variables"}
systemPrompt: "You are a task management assistant. Be concise.";
```

### temperature

Controls response randomness. Range: 0.0 (deterministic) to 1.0 (creative).

```typescript theme={"theme":"css-variables"}
temperature: 0.1; // More deterministic, better for testing
temperature: 0.9; // More creative
```

### maxSteps

Maximum iterations of the agentic loop (prompt → tool → result → continue).

```typescript theme={"theme":"css-variables"}
maxSteps: 5; // Stop after 5 tool calls
```

<Warning>
  Setting `maxSteps` too low may prevent complex tasks from completing. Setting
  it too high may allow runaway loops.
</Warning>

### injectOpenAiCompat

Controls whether the OpenAI Apps SDK `window.openai` shim is injected into captured widget HTML snapshots. Defaults to `false`.

Set this to `true` when your widget is written against the OpenAI Apps SDK (i.e. it targets ChatGPT or Copilot). Leave it `false` for SEP-1865-native widgets targeting Claude, Cursor, Codex, or other hosts that don't expose `window.openai` — snapshots should match what the live host would have produced.

```typescript theme={"theme":"css-variables"}
// Testing an OpenAI Apps SDK widget (ChatGPT / Copilot host)
const agent = new HostRunner({
  tools,
  model: "openai/gpt-4o",
  apiKey: process.env.OPENAI_API_KEY,
  mcpClientManager: manager,
  injectOpenAiCompat: true,
});

// Testing a SEP-1865 widget (Claude / Cursor / Codex host) — default
const agent = new HostRunner({
  tools,
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
  mcpClientManager: manager,
  // injectOpenAiCompat defaults to false
});
```

<Note>
  This flag is stamped onto each widget snapshot (`injectedOpenAiCompat`) so the
  MCPJam trace viewer can replay the widget faithfully under a different host
  config without rewriting the cached bytes.
</Note>

***

## Control Multi-Step Loops with stopWhen

Use `stopWhen` to control whether the agent starts another step after the current step completes.

```typescript theme={"theme":"css-variables"}
import { hasToolCall } from "@mcpjam/sdk";

// Stop after the step where "search_tasks" is called
const result = await agent.run("Find my open tasks", {
  stopWhen: hasToolCall("search_tasks"),
});

expect(result.hasToolCall("search_tasks")).toBe(true);

// Stop after any of multiple conditions
const result2 = await agent.run("Do something", {
  stopWhen: [hasToolCall("tool_a"), hasToolCall("tool_b")],
});
```

<Info>
  `stopWhen` does not skip tool execution. It controls whether the prompt loop
  continues after the current step completes, and `HostRunner` also applies
  `stepCountIs(maxSteps)` as a safety guard.
</Info>

***

## Bound Prompt Runtime with timeout

Use `timeout` when you want to bound how long `HostRunner.run()` can run:

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Run a long workflow", {
  timeout: 10_000,
});

const result2 = await agent.run("Run a long workflow", {
  timeout: { totalMs: 10_000, stepMs: 2_500, chunkMs: 1_000 },
});
```

<Info>
  `chunkMs` is accepted for parity, but it is mainly useful for streaming APIs.
  For `HostRunner.run()`, `number`, `totalMs`, and `stepMs` are the main
  settings to focus on.
</Info>

***

## Complete Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner } from "@mcpjam/sdk";

async function main() {
  // Setup
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  // Create agent (pass manager when you need widget snapshots or MCPJam replay configs)
  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-4-20250514",
    apiKey: process.env.ANTHROPIC_API_KEY,
    systemPrompt: "You are a helpful assistant.",
    temperature: 0.3,
    maxSteps: 10,
    mcpClientManager: manager,
  });

  // Single prompt
  const r1 = await agent.run("What is 15 + 27?");
  console.log(r1.text);
  console.log("Tools:", r1.toolsCalled());

  // Multi-turn
  const r2 = await agent.run("Now multiply that by 2", { context: r1 });
  console.log(r2.text);

  // Error handling
  if (r2.hasError()) {
    console.error("Error:", r2.getError());
  }

  // Cleanup
  await manager.disconnectServer("everything");
}
```

***

## Related

* [Testing with LLMs](/sdk/concepts/testing-with-llms) - Conceptual guide
* [PromptResult Reference](/sdk/reference/prompt-result) - Result object API
* [LLM Providers Reference](/sdk/reference/llm-providers) - All providers


# LLM Providers
Source: https://docs.mcpjam.com/sdk/reference/llm-providers

API reference for LLM provider configuration

The SDK supports 10 built-in LLM providers and allows custom provider configuration for any OpenAI or Anthropic-compatible endpoint.

## Model String Format

All models use the format `provider/model-id`:

```typescript theme={"theme":"css-variables"}
"anthropic/claude-sonnet-4-20250514"
"openai/gpt-4o"
"google/gemini-1.5-pro"
```

***

## Built-in Providers

### Anthropic

| Property             | Value               |
| -------------------- | ------------------- |
| Prefix               | `anthropic/`        |
| Environment Variable | `ANTHROPIC_API_KEY` |

#### Models

| Model ID                     | Description                 |
| ---------------------------- | --------------------------- |
| `claude-sonnet-4-20250514`   | Latest Sonnet (recommended) |
| `claude-3-5-sonnet-20241022` | Claude 3.5 Sonnet           |
| `claude-3-haiku-20240307`    | Fast, efficient             |
| `claude-3-opus-20240229`     | Most capable                |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
});
```

***

### OpenAI

| Property             | Value            |
| -------------------- | ---------------- |
| Prefix               | `openai/`        |
| Environment Variable | `OPENAI_API_KEY` |

#### Models

| Model ID        | Description                     |
| --------------- | ------------------------------- |
| `gpt-4o`        | Latest GPT-4 Omni (recommended) |
| `gpt-4o-mini`   | Fast, cost-effective            |
| `gpt-4-turbo`   | GPT-4 Turbo                     |
| `gpt-4`         | Original GPT-4                  |
| `gpt-3.5-turbo` | Fast, economical                |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "openai/gpt-4o",
  apiKey: process.env.OPENAI_API_KEY,
});
```

***

### Azure OpenAI

| Property              | Value                                           |
| --------------------- | ----------------------------------------------- |
| Prefix                | `azure/`                                        |
| Environment Variables | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` |

#### Environment Variables

| Variable                   | Required | Description  |
| -------------------------- | -------- | ------------ |
| `AZURE_OPENAI_API_KEY`     | Yes      | API key      |
| `AZURE_OPENAI_ENDPOINT`    | Yes      | Endpoint URL |
| `AZURE_OPENAI_API_VERSION` | No       | API version  |

#### Example

```typescript theme={"theme":"css-variables"}
// Set environment variables first:
// AZURE_OPENAI_API_KEY=...
// AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com

const agent = new HostRunner({
  tools,
  model: "azure/gpt-4o",
  apiKey: process.env.AZURE_OPENAI_API_KEY,
});
```

***

### Amazon Bedrock

| Property              | Value                                    |
| --------------------- | ---------------------------------------- |
| Prefix                | `bedrock/`                               |
| Environment Variables | `AWS_BEARER_TOKEN_BEDROCK`, `AWS_REGION` |

Authentication uses a [Bedrock API key](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) (bearer token). The AWS region is read from `AWS_REGION` unless a full runtime endpoint is supplied via `baseUrls.bedrock` (e.g. `https://bedrock-runtime.us-east-1.amazonaws.com`).

#### Models

Use any Bedrock model or inference profile ID your AWS account has been granted access to:

| Model ID                                       | Description                              |
| ---------------------------------------------- | ---------------------------------------- |
| `us.anthropic.claude-sonnet-4-5-20250929-v1:0` | Claude Sonnet 4.5 (US inference profile) |
| `anthropic.claude-3-5-haiku-20241022-v1:0`     | Claude 3.5 Haiku                         |
| `us.amazon.nova-pro-v1:0`                      | Amazon Nova Pro                          |
| `meta.llama3-70b-instruct-v1:0`                | Meta Llama 3 70B                         |

#### Environment Variables

| Variable                   | Required | Description                                                     |
| -------------------------- | -------- | --------------------------------------------------------------- |
| `AWS_BEARER_TOKEN_BEDROCK` | Yes      | Bedrock API key (bearer token)                                  |
| `AWS_REGION`               | Yes      | AWS region (e.g. `us-east-1`), unless `baseUrls.bedrock` is set |

#### Example

```typescript theme={"theme":"css-variables"}
// Set environment variables first:
// AWS_BEARER_TOKEN_BEDROCK=...
// AWS_REGION=us-east-1

const agent = new HostRunner({
  tools,
  model: "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
  apiKey: process.env.AWS_BEARER_TOKEN_BEDROCK,
});
```

***

### Google (Gemini)

| Property             | Value                          |
| -------------------- | ------------------------------ |
| Prefix               | `google/`                      |
| Environment Variable | `GOOGLE_GENERATIVE_AI_API_KEY` |

#### Models

| Model ID           | Description                |
| ------------------ | -------------------------- |
| `gemini-1.5-pro`   | Most capable (recommended) |
| `gemini-1.5-flash` | Fast, efficient            |
| `gemini-pro`       | Original Gemini Pro        |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "google/gemini-1.5-pro",
  apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY,
});
```

***

### Mistral

| Property             | Value             |
| -------------------- | ----------------- |
| Prefix               | `mistral/`        |
| Environment Variable | `MISTRAL_API_KEY` |

#### Models

| Model ID                | Description      |
| ----------------------- | ---------------- |
| `mistral-large-latest`  | Most capable     |
| `mistral-medium-latest` | Balanced         |
| `mistral-small-latest`  | Fast, economical |
| `open-mistral-7b`       | Open source      |
| `open-mixtral-8x7b`     | Open source MoE  |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "mistral/mistral-large-latest",
  apiKey: process.env.MISTRAL_API_KEY,
});
```

***

### DeepSeek

| Property             | Value              |
| -------------------- | ------------------ |
| Prefix               | `deepseek/`        |
| Environment Variable | `DEEPSEEK_API_KEY` |

#### Models

| Model ID         | Description  |
| ---------------- | ------------ |
| `deepseek-chat`  | General chat |
| `deepseek-coder` | Code-focused |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "deepseek/deepseek-chat",
  apiKey: process.env.DEEPSEEK_API_KEY,
});
```

***

### Ollama (Local)

| Property    | Value                    |
| ----------- | ------------------------ |
| Prefix      | `ollama/`                |
| Default URL | `http://localhost:11434` |

#### Models

Any locally installed model:

* `llama3`
* `codellama`
* `mistral`
* `mixtral`
* etc.

<Note>
  Ensure Ollama is running locally before use. The API key parameter is required but not used, so pass any string.
</Note>

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "ollama/llama3",
  apiKey: "ollama", // Required but not used
});
```

***

### OpenRouter

| Property             | Value                |
| -------------------- | -------------------- |
| Prefix               | `openrouter/`        |
| Environment Variable | `OPENROUTER_API_KEY` |

Access many models through one API:

| Model ID                          | Description                  |
| --------------------------------- | ---------------------------- |
| `anthropic/claude-3-opus`         | Claude 3 Opus via OpenRouter |
| `openai/gpt-4-turbo`              | GPT-4 Turbo via OpenRouter   |
| `google/gemini-pro`               | Gemini via OpenRouter        |
| `meta-llama/llama-3-70b-instruct` | Llama 3 70B                  |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "openrouter/anthropic/claude-3-opus",
  apiKey: process.env.OPENROUTER_API_KEY,
});
```

***

### xAI (Grok)

| Property             | Value         |
| -------------------- | ------------- |
| Prefix               | `xai/`        |
| Environment Variable | `XAI_API_KEY` |

#### Models

| Model ID    | Description |
| ----------- | ----------- |
| `grok-beta` | Grok beta   |

#### Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "xai/grok-beta",
  apiKey: process.env.XAI_API_KEY,
});
```

***

## Custom Providers

Add your own OpenAI or Anthropic-compatible endpoints.

### CustomProvider Type

```typescript theme={"theme":"css-variables"}
interface CustomProvider {
  name: string;
  protocol: "openai-compatible" | "anthropic-compatible";
  baseUrl: string;
  modelIds: string[];
  useChatCompletions?: boolean;
  apiKeyEnvVar?: string;
}
```

### Properties

| Property             | Type       | Required | Description                                       |
| -------------------- | ---------- | -------- | ------------------------------------------------- |
| `name`               | `string`   | Yes      | Provider identifier (used in model string)        |
| `protocol`           | `string`   | Yes      | `"openai-compatible"` or `"anthropic-compatible"` |
| `baseUrl`            | `string`   | Yes      | API endpoint URL                                  |
| `modelIds`           | `string[]` | Yes      | Available model IDs                               |
| `useChatCompletions` | `boolean`  | No       | Use `/chat/completions` endpoint                  |
| `apiKeyEnvVar`       | `string`   | No       | Custom env var name for API key                   |

### OpenAI-Compatible Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "my-openai/gpt-4",
  apiKey: process.env.MY_API_KEY,
  customProviders: {
    "my-openai": {
      name: "my-openai",
      protocol: "openai-compatible",
      baseUrl: "https://api.my-provider.com/v1",
      modelIds: ["gpt-4", "gpt-3.5-turbo"],
    },
  },
});
```

### Anthropic-Compatible Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "my-anthropic/claude-3-sonnet",
  apiKey: process.env.MY_API_KEY,
  customProviders: {
    "my-anthropic": {
      name: "my-anthropic",
      protocol: "anthropic-compatible",
      baseUrl: "https://api.my-provider.com",
      modelIds: ["claude-3-sonnet", "claude-3-haiku"],
    },
  },
});
```

### LiteLLM Example

```typescript theme={"theme":"css-variables"}
const agent = new HostRunner({
  tools,
  model: "litellm/gpt-4",
  apiKey: process.env.LITELLM_API_KEY,
  customProviders: {
    litellm: {
      name: "litellm",
      protocol: "openai-compatible",
      baseUrl: "http://localhost:8000",
      modelIds: ["gpt-4", "claude-3-sonnet", "gemini-pro"],
      useChatCompletions: true, // Required for LiteLLM
    },
  },
});
```

***

## Helper Functions

### parseLLMString()

Parse a model string into provider and model ID.

```typescript theme={"theme":"css-variables"}
import { parseLLMString } from "@mcpjam/sdk";

parseLLMString(modelString: string): { provider: string; modelId: string }
```

#### Example

```typescript theme={"theme":"css-variables"}
const { provider, modelId } = parseLLMString("anthropic/claude-sonnet-4-20250514");
// provider: "anthropic"
// modelId: "claude-sonnet-4-20250514"
```

***

### createModelFromString()

Create a provider model instance directly.

```typescript theme={"theme":"css-variables"}
import { createModelFromString } from "@mcpjam/sdk";

createModelFromString(
  llmString: string,
  options: CreateModelOptions
): ProviderLanguageModel
```

#### Example

```typescript theme={"theme":"css-variables"}
const model = createModelFromString(
  "openai/gpt-4o",
  { apiKey: process.env.OPENAI_API_KEY }
);
```

***

### createCustomProvider()

Create a custom provider configuration object.

```typescript theme={"theme":"css-variables"}
import { createCustomProvider } from "@mcpjam/sdk";

createCustomProvider(config: CustomProvider): CustomProvider
```

#### Example

```typescript theme={"theme":"css-variables"}
const provider = createCustomProvider({
  name: "my-litellm",
  protocol: "openai-compatible",
  baseUrl: "http://localhost:8000",
  modelIds: ["gpt-4", "claude-3-sonnet"],
  useChatCompletions: true,
});
```

***

## Environment Variables Summary

| Provider       | Variable                                        |
| -------------- | ----------------------------------------------- |
| Anthropic      | `ANTHROPIC_API_KEY`                             |
| OpenAI         | `OPENAI_API_KEY`                                |
| Azure          | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` |
| Amazon Bedrock | `AWS_BEARER_TOKEN_BEDROCK`, `AWS_REGION`        |
| Google         | `GOOGLE_GENERATIVE_AI_API_KEY`                  |
| Mistral        | `MISTRAL_API_KEY`                               |
| DeepSeek       | `DEEPSEEK_API_KEY`                              |
| OpenRouter     | `OPENROUTER_API_KEY`                            |
| xAI            | `XAI_API_KEY`                                   |

***

## Related

* [Testing Across Providers](/sdk/concepts/multi-provider) - Conceptual guide
* [HostRunner Reference](/sdk/reference/host-runner) - Agent configuration


# MCPClientManager
Source: https://docs.mcpjam.com/sdk/reference/mcp-client-manager

API reference for MCPClientManager

The `MCPClientManager` class manages connections to one or more MCP servers and provides methods to interact with their tools, resources, and prompts.

## Import

```typescript theme={"theme":"css-variables"}
import { MCPClientManager } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={"theme":"css-variables"}
new MCPClientManager(servers: ServerConfig, options?: MCPClientManagerOptions)
```

### Parameters

<ParamField type="Record<string, StdioServerConfig | HttpServerConfig>">
  A map of server IDs to their configurations.
</ParamField>

<ParamField type="MCPClientManagerOptions">
  Optional manager-level settings. See [Manager options](#manager-options) below.
</ParamField>

### Server Configuration Types

#### StdioServerConfig

For subprocess-based MCP servers.

| Property                    | Type                                                            | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| --------------------------- | --------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `command`                   | `string`                                                        | Yes      | The command to run (e.g., `"node"`, `"python"`, `"npx"`)                                                                                                                                                                                                                                                                                                                                                                                     |
| `args`                      | `string[]`                                                      | No       | Command arguments                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `env`                       | `Record<string, string>`                                        | No       | Environment variables for the subprocess                                                                                                                                                                                                                                                                                                                                                                                                     |
| `clientInfo`                | `{ name?: string; version?: string } & Record<string, unknown>` | No       | Override the MCP `initialize.params.clientInfo` sent to this server. Useful for testing how servers respond to different client identities.                                                                                                                                                                                                                                                                                                  |
| `supportedProtocolVersions` | `string[]`                                                      | No       | Ordered list of MCP protocol versions to advertise. The first entry is proposed in `initialize.params.protocolVersion`; the full list is the accept-set.                                                                                                                                                                                                                                                                                     |
| `firstPageOnly`             | `boolean`                                                       | No       | When `true`, simulates a client that reads only the first page of paginated list results (`tools/list`, `resources/list`, `resources/templates/list`, `prompts/list`) and stops. Tools beyond page one are invisible to the client, and `Mcp-Param-*` header mirroring is limited to page-one tools. Applies on every transport (stdio, Streamable HTTP, SSE). Explicit-cursor requests (manual paging in the inspector) are left untouched. |

```typescript theme={"theme":"css-variables"}
{
  myServer: {
    command: "node",
    args: ["./server.js", "--port", "3000"],
    env: {
      API_KEY: process.env.API_KEY,
      DEBUG: "true",
    },
  },
}
```

#### HttpServerConfig

For remote MCP servers via SSE or Streamable HTTP.

| Property                    | Type                                                            | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| --------------------------- | --------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                       | `string`                                                        | Yes      | The server URL (e.g., `"https://mcp.example.com/sse"`)                                                                                                                                                                                                                                                                                                                                                                                       |
| `accessToken`               | `string`                                                        | No       | Static bearer token added as `Authorization: Bearer <token>`                                                                                                                                                                                                                                                                                                                                                                                 |
| `requestInit`               | `RequestInit`                                                   | No       | Fetch options including headers for authentication                                                                                                                                                                                                                                                                                                                                                                                           |
| `eventSourceInit`           | `EventSourceInit`                                               | No       | SSE-specific options                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `authProvider`              | `OAuthClientProvider`                                           | No       | Custom MCP SDK OAuth provider                                                                                                                                                                                                                                                                                                                                                                                                                |
| `refreshToken`              | `string`                                                        | No       | Refresh token used for non-interactive OAuth token exchange and automatic token refresh                                                                                                                                                                                                                                                                                                                                                      |
| `clientId`                  | `string`                                                        | No       | OAuth client ID. Required when `refreshToken` is set                                                                                                                                                                                                                                                                                                                                                                                         |
| `clientSecret`              | `string`                                                        | No       | OAuth client secret for confidential clients                                                                                                                                                                                                                                                                                                                                                                                                 |
| `reconnectionOptions`       | `StreamableHTTPClientTransportOptions["reconnectionOptions"]`   | No       | Streamable HTTP reconnection behavior                                                                                                                                                                                                                                                                                                                                                                                                        |
| `sessionId`                 | `string`                                                        | No       | Existing Streamable HTTP session ID                                                                                                                                                                                                                                                                                                                                                                                                          |
| `preferSSE`                 | `boolean`                                                       | No       | Forces SSE instead of Streamable HTTP                                                                                                                                                                                                                                                                                                                                                                                                        |
| `clientInfo`                | `{ name?: string; version?: string } & Record<string, unknown>` | No       | Override the MCP `initialize.params.clientInfo` sent to this server. Useful for testing how servers respond to different client identities.                                                                                                                                                                                                                                                                                                  |
| `supportedProtocolVersions` | `string[]`                                                      | No       | Ordered list of MCP protocol versions to advertise. The first entry is proposed in `initialize.params.protocolVersion`; the full list is the accept-set.                                                                                                                                                                                                                                                                                     |
| `onUnauthorized`            | `UnauthorizedRefreshHandler`                                    | No       | 401 recovery hook. Called once when an operation fails with HTTP 401; return a new `accessToken` and the SDK rebuilds the transport and retries. Not invoked for 403, or when `refreshToken`/`authProvider` is set.                                                                                                                                                                                                                          |
| `firstPageOnly`             | `boolean`                                                       | No       | When `true`, simulates a client that reads only the first page of paginated list results (`tools/list`, `resources/list`, `resources/templates/list`, `prompts/list`) and stops. Tools beyond page one are invisible to the client, and `Mcp-Param-*` header mirroring is limited to page-one tools. Applies on every transport (stdio, Streamable HTTP, SSE). Explicit-cursor requests (manual paging in the inspector) are left untouched. |

```typescript theme={"theme":"css-variables"}
{
  remoteServer: {
    url: "https://mcp.asana.com/sse",
    requestInit: {
      headers: {
        Authorization: "Bearer YOUR_TOKEN",
        "X-Custom-Header": "value",
      },
    },
  },
}
```

##### Refresh Token Example

```typescript theme={"theme":"css-variables"}
{
  remoteServer: {
    url: "https://mcp.example.com/mcp",
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
    clientSecret: process.env.MCP_CLIENT_SECRET,
  },
}
```

When `refreshToken` is provided, the SDK will exchange it for an access token during connection and automatically refresh tokens if the server challenges the current token.

<Note>
  `refreshToken` is mutually exclusive with `accessToken`, `authProvider`, and `requestInit.headers.Authorization`.
</Note>

##### Client Identity Example

Use `clientInfo` to pin the identity sent in MCP `initialize`. This is useful when testing how a server behaves for different clients (e.g., Claude, ChatGPT):

```typescript theme={"theme":"css-variables"}
{
  remoteServer: {
    url: "https://mcp.example.com/mcp",
    clientInfo: { name: "claude-ai", version: "0.1.0" },
    supportedProtocolVersions: ["2025-11-25", "2025-06-18"],
  },
}
```

The first entry in `supportedProtocolVersions` is proposed to the server; the full list is the accept-set. A server that negotiates any listed version is accepted.

##### onUnauthorized Example

Use `onUnauthorized` when you manage access tokens externally (e.g., from a secrets vault) and want the SDK to automatically recover from a 401 without a full reconnect flow:

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, UnauthorizedRefreshHandler } from "@mcpjam/sdk";

const refreshToken: UnauthorizedRefreshHandler = async ({ serverId }) => {
  // Fetch a fresh token from your own token store or backend
  const response = await fetch("https://auth.example.com/token/refresh", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.SERVICE_TOKEN}` },
    body: JSON.stringify({ serverId }),
  });
  const { accessToken } = await response.json();
  return { accessToken };
};

const manager = new MCPClientManager({
  myServer: {
    url: "https://mcp.example.com/mcp",
    accessToken: await getInitialToken(),
    onUnauthorized: refreshToken,
  },
});
```

When the server returns HTTP 401, the SDK calls `onUnauthorized` once, updates the access token, rebuilds the transport, and retries the operation. A second 401 after the retry surfaces the error to the caller.

<Note>
  `onUnauthorized` is only active for `accessToken`-based HTTP configs. It is ignored when `refreshToken` or `authProvider` is set, and it is never triggered by 403 responses.
</Note>

### Example

```typescript theme={"theme":"css-variables"}
const manager = new MCPClientManager({
  // STDIO server
  local: {
    command: "npx",
    args: ["-y", "@modelcontextprotocol/server-everything"],
  },
  // HTTP server
  remote: {
    url: "https://mcp.example.com/sse",
    requestInit: {
      headers: { Authorization: "Bearer token" },
    },
  },
});
```

***

## Manager options

The second argument to the constructor accepts an `MCPClientManagerOptions` object.

| Option                     | Type                       | Description                                                                                                                                                                                                                        |
| -------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `negotiationOutcomeLogger` | `NegotiationOutcomeLogger` | Optional callback invoked once per connection attempt with the negotiation outcome. Use it to forward telemetry to your own analytics pipeline. The manager guards it — a throw inside the callback never disturbs the connection. |
| `cacheEventLogger`         | `CacheEventLogger`         | Optional callback for response-cache hit events.                                                                                                                                                                                   |
| `rpcLogger`                | `RpcLogger`                | Optional callback for raw RPC request/response pairs.                                                                                                                                                                              |
| `retryPolicy`              | `RetryPolicy`              | Default retry policy for retryable manager operations.                                                                                                                                                                             |
| `lazyConnect`              | `boolean`                  | When `true`, defers the actual transport connection until the first RPC call.                                                                                                                                                      |

### negotiationOutcomeLogger

`negotiationOutcomeLogger` receives a `NegotiationOutcomeEvent` for every connection attempt:

| Field                       | Type                                 | Description                                                                                                                                                                                    |
| --------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serverId`                  | `string`                             | The server whose connection was attempted.                                                                                                                                                     |
| `transport`                 | `"http" \| "stdio"`                  | Transport the attempt used.                                                                                                                                                                    |
| `configuredMode`            | `"auto" \| "modern-pin" \| "legacy"` | The negotiation mode the client was asked to use. `"auto"` for unconfigured connections (always probed); `"modern-pin"` for an explicit 2026-07-28 pin; `"legacy"` for an explicit legacy pin. |
| `outcome`                   | `"connected" \| "failed"`            | Whether the connection established or failed.                                                                                                                                                  |
| `negotiatedEra`             | `"legacy" \| "modern"`               | *(connected only)* The era that was negotiated.                                                                                                                                                |
| `negotiatedProtocolVersion` | `string`                             | *(connected only)* The wire protocol version that was negotiated.                                                                                                                              |
| `failureClass`              | `string`                             | *(failed only)* Coarse, non-PII failure class — the error's `name` or `code`, e.g. `"UnauthorizedError"`, `"TypeError"`.                                                                       |

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, NegotiationOutcomeEvent } from "@mcpjam/sdk";

const manager = new MCPClientManager(
  {
    myServer: { command: "node", args: ["./server.js"] },
  },
  {
    negotiationOutcomeLogger: (event: NegotiationOutcomeEvent) => {
      console.log(
        `[${event.serverId}] ${event.transport} ${event.configuredMode} → ${event.outcome}`,
        event.negotiatedEra ?? event.failureClass,
      );
    },
  },
);
```

***

## Methods

### connectToServer()

Establishes a connection to a configured server.

```typescript theme={"theme":"css-variables"}
connectToServer(serverId: string): Promise<Client>
```

#### Parameters

| Parameter  | Type     | Description                               |
| ---------- | -------- | ----------------------------------------- |
| `serverId` | `string` | The server ID from the constructor config |

#### Returns

`Promise<Client>` - Resolves with the connected MCP client, rejects on failure.

#### Example

```typescript theme={"theme":"css-variables"}
await manager.connectToServer("myServer");
```

#### Throws

* Error if `serverId` is not in the configuration
* Error if connection fails (network, subprocess spawn, etc.)

***

### disconnectServer()

Closes the connection to a server and cleans up resources.

```typescript theme={"theme":"css-variables"}
disconnectServer(serverId: string): Promise<void>
```

#### Parameters

| Parameter  | Type     | Description                 |
| ---------- | -------- | --------------------------- |
| `serverId` | `string` | The server ID to disconnect |

#### Returns

`Promise<void>`

#### Example

```typescript theme={"theme":"css-variables"}
await manager.disconnectServer("myServer");
```

***

### pingServer()

Sends a liveness probe to check if a server is responsive. The probe is era-aware: on a modern (2026-07-28) connection it uses `server/discover`; on a legacy connection it uses `ping`.

```typescript theme={"theme":"css-variables"}
pingServer(serverId: string): Promise<EmptyResult>
```

#### Parameters

| Parameter  | Type     | Description           |
| ---------- | -------- | --------------------- |
| `serverId` | `string` | The server ID to ping |

#### Returns

`Promise<EmptyResult>` - Resolves with an empty result if the server is responsive.

#### Throws

* Error if the server is not connected
* Error if the probe fails

#### Example

```typescript theme={"theme":"css-variables"}
try {
  await manager.pingServer("myServer");
  console.log("Server is responsive");
} catch (error) {
  console.warn("Server not responding:", error.message);
}
```

***

### listTools()

Returns all tools available on a server.

```typescript theme={"theme":"css-variables"}
listTools(serverId: string): Promise<{ tools: Tool[] }>
```

#### Parameters

| Parameter  | Type     | Description   |
| ---------- | -------- | ------------- |
| `serverId` | `string` | The server ID |

#### Returns

`Promise<{ tools: Tool[] }>` - Tool list response from the server.

#### Tool Object

| Property      | Type     | Description                    |
| ------------- | -------- | ------------------------------ |
| `name`        | `string` | Tool identifier                |
| `description` | `string` | Human-readable description     |
| `inputSchema` | `object` | JSON Schema for tool arguments |

#### Example

```typescript theme={"theme":"css-variables"}
const tools = await manager.listTools("myServer");

for (const tool of tools.tools) {
  console.log(`${tool.name}: ${tool.description}`);
}
```

***

### executeTool()

Executes a tool and returns the result.

```typescript theme={"theme":"css-variables"}
executeTool(
  serverId: string,
  toolName: string,
  args: Record<string, unknown>
): Promise<unknown>
```

#### Parameters

| Parameter  | Type                      | Description                   |
| ---------- | ------------------------- | ----------------------------- |
| `serverId` | `string`                  | The server ID                 |
| `toolName` | `string`                  | Name of the tool to execute   |
| `args`     | `Record<string, unknown>` | Arguments to pass to the tool |

#### Returns

`Promise<unknown>` - The tool's return value (type depends on the tool).

#### Example

```typescript theme={"theme":"css-variables"}
const result = await manager.executeTool("myServer", "add", {
  a: 5,
  b: 3,
});
console.log(result); // 8
```

#### Throws

* Error if tool doesn't exist
* Error if arguments are invalid
* Error if tool execution fails

***

### listResources()

Returns all resources available on a server.

```typescript theme={"theme":"css-variables"}
listResources(serverId: string): Promise<Resource[]>
```

#### Parameters

| Parameter  | Type     | Description   |
| ---------- | -------- | ------------- |
| `serverId` | `string` | The server ID |

#### Returns

`Promise<Resource[]>` - Array of resource definitions.

#### Resource Object

| Property      | Type     | Description                |
| ------------- | -------- | -------------------------- |
| `name`        | `string` | Resource identifier        |
| `uri`         | `string` | Resource URI               |
| `description` | `string` | Human-readable description |
| `mimeType`    | `string` | Content type               |

#### Example

```typescript theme={"theme":"css-variables"}
const resources = await manager.listResources("myServer");

for (const resource of resources) {
  console.log(`${resource.name}: ${resource.uri}`);
}
```

***

### readResource()

Reads the content of a resource.

```typescript theme={"theme":"css-variables"}
readResource(
  serverId: string,
  params: { uri: string }
): Promise<ResourceContent>
```

#### Parameters

| Parameter    | Type     | Description              |
| ------------ | -------- | ------------------------ |
| `serverId`   | `string` | The server ID            |
| `params.uri` | `string` | The resource URI to read |

#### Returns

`Promise<ResourceContent>` - The resource content.

#### Example

```typescript theme={"theme":"css-variables"}
const content = await manager.readResource("myServer", {
  uri: "file://config.json",
});
console.log(content);
```

***

### listPrompts()

Returns all prompts available on a server.

```typescript theme={"theme":"css-variables"}
listPrompts(serverId: string): Promise<Prompt[]>
```

#### Parameters

| Parameter  | Type     | Description   |
| ---------- | -------- | ------------- |
| `serverId` | `string` | The server ID |

#### Returns

`Promise<Prompt[]>` - Array of prompt definitions.

#### Prompt Object

| Property      | Type               | Description                |
| ------------- | ------------------ | -------------------------- |
| `name`        | `string`           | Prompt identifier          |
| `description` | `string`           | Human-readable description |
| `arguments`   | `PromptArgument[]` | Expected arguments         |

#### Example

```typescript theme={"theme":"css-variables"}
const prompts = await manager.listPrompts("myServer");

for (const prompt of prompts) {
  console.log(`${prompt.name}: ${prompt.description}`);
}
```

***

### getPrompt()

Gets a prompt with optional arguments.

```typescript theme={"theme":"css-variables"}
getPrompt(
  serverId: string,
  params: { name: string; arguments?: Record<string, string> }
): Promise<PromptResult>
```

#### Parameters

| Parameter          | Type                     | Description               |
| ------------------ | ------------------------ | ------------------------- |
| `serverId`         | `string`                 | The server ID             |
| `params.name`      | `string`                 | The prompt name           |
| `params.arguments` | `Record<string, string>` | Optional prompt arguments |

#### Returns

`Promise<PromptResult>` - The prompt content with messages.

#### Example

```typescript theme={"theme":"css-variables"}
const prompt = await manager.getPrompt("myServer", {
  name: "summarize",
  arguments: { length: "short" },
});
console.log(prompt.messages);
```

***

### getTools()

Returns all tools from specified servers or all connected servers.

```typescript theme={"theme":"css-variables"}
getTools(serverIds?: string[]): Promise<Tool[]>
```

#### Parameters

| Parameter   | Type       | Description                                                                                              |
| ----------- | ---------- | -------------------------------------------------------------------------------------------------------- |
| `serverIds` | `string[]` | Optional. Array of server IDs to fetch tools from. If omitted, returns tools from all connected servers. |

#### Returns

`Promise<Tool[]>` - Array of tool definitions from all specified servers.

#### Tool Object

Each tool in the returned array includes:

| Property      | Type       | Description                                 |
| ------------- | ---------- | ------------------------------------------- |
| `name`        | `string`   | Tool identifier                             |
| `description` | `string`   | Human-readable description                  |
| `inputSchema` | `object`   | JSON Schema for tool arguments              |
| `execute`     | `function` | Function to execute the tool with arguments |

#### Example

```typescript theme={"theme":"css-variables"}
// Tools from specific servers
const tools = await manager.getTools(["myServer", "anotherServer"]);

// Tools from all connected servers
const allTools = await manager.getTools();

for (const tool of allTools) {
  console.log(`${tool.name}: ${tool.description}`);
}

// Use with HostRunner
const agent = new HostRunner({
  tools: await manager.getTools(),
  model: "anthropic/claude-sonnet-4-20250514",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

const result = await agent.run("Add 2 and 3");
console.log(result.text);
```

<Note>
  `getTools()` returns all tools including app-only ones. Tools with `_meta.ui.visibility = ["app"]` are filtered out by `getToolsForAiSdk()` per SEP-1865 before being passed to the model. These tools remain callable by apps via the MCP bridge.
</Note>

***

### getToolsForAiSdk()

Returns tools in Vercel AI SDK format, ready to pass directly to `generateText()` or `streamText()`. App-only tools are filtered out by default per SEP-1865.

```typescript theme={"theme":"css-variables"}
getToolsForAiSdk(
  serverIds?: string[] | string,
  options?: {
    schemas?: ToolSchemaOverrides | "automatic";
    needsApproval?: boolean;
    includeAppOnly?: boolean;
  }
): Promise<ToolSet>
```

#### Parameters

| Parameter                | Type                                 | Description                                                                                                                                                                                                                             |
| ------------------------ | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serverIds`              | `string[] \| string`                 | Optional. Server IDs to include. If omitted, uses all connected servers.                                                                                                                                                                |
| `options.schemas`        | `ToolSchemaOverrides \| "automatic"` | Optional. Control JSON schema conversion.                                                                                                                                                                                               |
| `options.needsApproval`  | `boolean`                            | Optional. When true, each tool call requires user approval before execution.                                                                                                                                                            |
| `options.includeAppOnly` | `boolean`                            | Optional. When true, includes tools with `_meta.ui.visibility = ["app"]` in the returned set. Defaults to `false` (spec-compliant). Use only when intentionally mirroring a host that does not implement SEP-1865 visibility filtering. |

#### Returns

`Promise<ToolSet>` - Tools in Vercel AI SDK format with execution wired up.

#### Example

```typescript theme={"theme":"css-variables"}
// Default: app-only tools are excluded (spec-compliant)
const tools = await manager.getToolsForAiSdk(["myServer"]);

// Include app-only tools (e.g. to mirror a non-filtering host)
const allTools = await manager.getToolsForAiSdk(["myServer"], {
  includeAppOnly: true,
});

// Use directly with Vercel AI SDK
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const response = await generateText({
  model: openai("gpt-4o"),
  tools: await manager.getToolsForAiSdk(),
  messages: [{ role: "user", content: "Add 5 and 7" }],
});
```

***

## Complete Example

```typescript theme={"theme":"css-variables"}
import { MCPClientManager, HostRunner } from "@mcpjam/sdk";

async function main() {
  const manager = new MCPClientManager({
    math: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
    asana: {
      url: "https://mcp.asana.com/sse",
      requestInit: {
        headers: { Authorization: `Bearer ${process.env.ASANA_TOKEN}` },
      },
    },
  });

  try {
    // Connect
    await manager.connectToServer("math");
    await manager.connectToServer("asana");

    // Direct tool execution
    const sum = await manager.executeTool("math", "add", { a: 10, b: 5 });
    console.log("Sum:", sum);

    // List capabilities
    const mathTools = await manager.listTools("math");
    const asanaTools = await manager.listTools("asana");
    console.log("Math tools:", mathTools.tools.length);
    console.log("Asana tools:", asanaTools.tools.length);

    // Create agent with all tools
    const agent = new HostRunner({
      tools: await manager.getTools(),
      model: "anthropic/claude-sonnet-4-20250514",
      apiKey: process.env.ANTHROPIC_API_KEY,
    });

    const result = await agent.run("Add 2 and 3");
    console.log(result.text);
  } finally {
    await manager.disconnectServer("math");
    await manager.disconnectServer("asana");
  }
}
```

## Related

* [Connecting to Servers](/sdk/concepts/connecting-servers) - Conceptual guide
* [HostRunner Reference](/sdk/reference/host-runner) - Use tools with LLMs


# OAuth Conformance SDK
Source: https://docs.mcpjam.com/sdk/reference/oauth-conformance

Programmatic OAuth conformance testing with OAuthConformanceTest and OAuthConformanceSuite

The OAuth Conformance SDK classes let you run the same flows the [CLI `oauth conformance`](/cli/oauth-conformance) command runs, but programmatically — for custom reporters, test frameworks, recording/replay via custom `fetchFn`, or Playwright-driven consent via `openUrl`.

<Note>
  For CLI usage (one-liners, CI recipes, troubleshooting), see [CLI: OAuth Conformance](/cli/oauth-conformance).
  This page covers the TypeScript SDK only.
</Note>

## Import

```typescript theme={"theme":"css-variables"}
import { OAuthConformanceTest, OAuthConformanceSuite } from "@mcpjam/sdk";
```

***

## Single flow

```typescript theme={"theme":"css-variables"}
const test = new OAuthConformanceTest({
  serverUrl: "https://your-server.com/mcp",
  protocolVersion: "2025-11-25",
  registrationStrategy: "dcr",
  auth: { mode: "headless" },
  verification: { listTools: true },
});

const result = await test.run();

console.log(result.passed);   // true
console.log(result.summary);  // "OAuth conformance passed for ..."
console.log(result.steps);    // Step-by-step results with HTTP traces
```

### OAuthConformanceConfig

| Property                 | Type                           | Required | Default                   | Description                                 |
| ------------------------ | ------------------------------ | -------- | ------------------------- | ------------------------------------------- |
| `serverUrl`              | `string`                       | Yes      |                           | MCP server URL                              |
| `protocolVersion`        | `string`                       | Yes      |                           | `2025-03-26`, `2025-06-18`, or `2025-11-25` |
| `registrationStrategy`   | `string`                       | Yes      |                           | `cimd`, `dcr`, or `preregistered`           |
| `auth`                   | `OAuthConformanceAuthConfig`   | No       | `{ mode: "interactive" }` | Auth mode config                            |
| `client`                 | `OAuthConformanceClientConfig` | No       | `{}`                      | Client credentials config                   |
| `verification`           | `OAuthVerificationConfig`      | No       | `{}`                      | Post-auth verification config               |
| `scopes`                 | `string`                       | No       |                           | Space-separated scope string                |
| `customHeaders`          | `Record<string, string>`       | No       |                           | Extra HTTP headers                          |
| `redirectUrl`            | `string`                       | No       | Auto-generated            | OAuth redirect URL                          |
| `fetchFn`                | `typeof fetch`                 | No       | `fetch`                   | Custom fetch for recording/replay           |
| `stepTimeout`            | `number`                       | No       | `30000`                   | Per-step timeout in ms                      |
| `oauthConformanceChecks` | `boolean`                      | No       | `false`                   | Run 6 additional negative checks post-flow  |

### OAuthConformanceAuthConfig

```typescript theme={"theme":"css-variables"}
type OAuthConformanceAuthConfig =
  | { mode: "interactive"; openUrl?: (url: string) => Promise<void> }
  | { mode: "headless" }
  | {
      mode: "client_credentials";
      clientId: string;
      clientSecret: string;
    };
```

* `interactive.openUrl` — override how the consent URL is opened. Default launches the system browser. Use for Playwright, custom launchers, or logging the URL.

### OAuthConformanceClientConfig

| Property                     | Type                                        | Description                                                                                    |
| ---------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `preregistered.clientId`     | `string`                                    | Required when `registrationStrategy === "preregistered"`                                       |
| `preregistered.clientSecret` | `string`                                    | Required for `preregistered` + `client_credentials`                                            |
| `dynamicRegistration`        | `Partial<OAuthDynamicRegistrationMetadata>` | Override the default DCR body (e.g. `client_name`, `logo_uri`, `scope`)                        |
| `clientIdMetadataUrl`        | `string`                                    | CIMD metadata URL. Defaults to `https://www.mcpjam.com/.well-known/oauth/client-metadata.json` |

### OAuthVerificationConfig

| Property    | Type                | Default | Description                                    |
| ----------- | ------------------- | ------- | ---------------------------------------------- |
| `listTools` | `boolean`           | `false` | Connect and call `tools/list` after auth       |
| `callTool`  | `{ name, params? }` |         | Also call the named tool (enables `listTools`) |
| `timeout`   | `number`            | `30000` | Verification timeout in ms                     |

***

## SDK-only features

These are available in the SDK but not exposed as CLI flags.

### Custom fetch (`fetchFn`)

Swap in a recording, mocking, or proxying fetch implementation:

```typescript theme={"theme":"css-variables"}
import { OAuthConformanceTest } from "@mcpjam/sdk";

const requests: Request[] = [];

const test = new OAuthConformanceTest({
  serverUrl: "https://your-server.com/mcp",
  protocolVersion: "2025-11-25",
  registrationStrategy: "dcr",
  fetchFn: async (url, init) => {
    requests.push(new Request(url, init));
    return fetch(url, init);
  },
});

const result = await test.run();
console.log(`Captured ${requests.length} HTTP requests`);
```

### DCR metadata overrides (`dynamicRegistration`)

Customize the client metadata sent during Dynamic Client Registration:

```typescript theme={"theme":"css-variables"}
const test = new OAuthConformanceTest({
  serverUrl: "https://your-server.com/mcp",
  protocolVersion: "2025-11-25",
  registrationStrategy: "dcr",
  client: {
    dynamicRegistration: {
      client_name: "My Custom Test Client",
      logo_uri: "https://example.com/logo.png",
      scope: "read:tools write:tools",
    },
  },
});
```

### Negative conformance checks (`oauthConformanceChecks`)

When enabled, six extra checks run after a successful flow:

1. **DCR redirect URI policy** — attempts dynamic client registration with a non-loopback `http://` redirect URI; expects rejection under the MCP authorization profile.
2. **Invalid client** — sends a token request with an unknown `client_id`; expects rejection.
3. **Invalid redirect at the authorization endpoint** — sends an authorization request with a mismatched `redirect_uri` and looks for rejection before the server redirects back to it. This step may be `skipped` if the server defers validation behind user interaction.
4. **Invalid token** — sends an authenticated MCP initialize request with an obviously invalid bearer token; expects HTTP 401 from the MCP server.
5. **Invalid redirect at the token endpoint** — sends a token request with a mismatched `redirect_uri` to look for redirect exact-match enforcement. This step may be `skipped` if the request is rejected for another reason before redirect validation is demonstrated.
6. **Token format** — validates the token response includes `access_token`, `token_type`, and expiration metadata.

```typescript theme={"theme":"css-variables"}
const test = new OAuthConformanceTest({
  serverUrl: "https://your-server.com/mcp",
  protocolVersion: "2025-11-25",
  registrationStrategy: "dcr",
  oauthConformanceChecks: true,
});

const result = await test.run();
// result.steps will include oauth_dcr_http_redirect_uri, oauth_invalid_client,
// oauth_invalid_authorize_redirect, oauth_invalid_token,
// oauth_invalid_redirect, and oauth_token_format
```

<Note>
  `oauthConformanceChecks` is also available via the CLI's `--conformance-checks` flag.
</Note>

***

## Suite

```typescript theme={"theme":"css-variables"}
const suite = new OAuthConformanceSuite({
  name: "My Server",
  serverUrl: "https://your-server.com/mcp",
  defaults: {
    auth: { mode: "headless" },
    verification: { listTools: true },
  },
  flows: [
    { protocolVersion: "2025-11-25", registrationStrategy: "cimd" },
    { protocolVersion: "2025-11-25", registrationStrategy: "dcr" },
  ],
});

const result = await suite.run();

console.log(result.passed);   // true
console.log(result.summary);  // "All 2 flows passed for ..."
for (const flow of result.results) {
  console.log(`${flow.passed ? "PASS" : "FAIL"} ${flow.label}`);
}
```

***

## Result types

### ConformanceResult

| Property               | Type                 | Description                                 |
| ---------------------- | -------------------- | ------------------------------------------- |
| `passed`               | `boolean`            | Whether the flow completed successfully     |
| `protocolVersion`      | `string`             | Protocol version tested                     |
| `registrationStrategy` | `string`             | Registration strategy tested                |
| `serverUrl`            | `string`             | Server URL tested                           |
| `steps`                | `StepResult[]`       | Step-by-step results                        |
| `summary`              | `string`             | Human-readable summary                      |
| `durationMs`           | `number`             | Total duration                              |
| `verification`         | `VerificationResult` | Post-auth verification results (if enabled) |

### StepResult

| Property           | Type                                | Description                                                                    |
| ------------------ | ----------------------------------- | ------------------------------------------------------------------------------ |
| `step`             | `string`                            | Step identifier (e.g. `"discovery"`, `"token_request"`, `"verify_list_tools"`) |
| `title`            | `string`                            | Human-readable step name                                                       |
| `summary`          | `string`                            | Short result summary                                                           |
| `status`           | `"passed" \| "failed" \| "skipped"` | Step outcome                                                                   |
| `durationMs`       | `number`                            | Step duration                                                                  |
| `logs`             | `InfoLogEntry[]`                    | Structured log entries                                                         |
| `http`             | `HttpHistoryEntry`                  | Primary HTTP exchange (if any)                                                 |
| `httpAttempts`     | `HttpHistoryEntry[]`                | All HTTP traces including retries                                              |
| `error`            | `{ message, details? }`             | Error info (if failed)                                                         |
| `teachableMoments` | `string[]`                          | Educational hints for this step                                                |

### VerificationResult

| Property               | Type      | Description                     |
| ---------------------- | --------- | ------------------------------- |
| `listTools.passed`     | `boolean` | Whether `tools/list` succeeded  |
| `listTools.toolCount`  | `number`  | Number of tools returned        |
| `listTools.durationMs` | `number`  | Duration                        |
| `listTools.error`      | `string`  | Error message if failed         |
| `callTool.passed`      | `boolean` | Whether the tool call succeeded |
| `callTool.toolName`    | `string`  | Name of the tool called         |
| `callTool.durationMs`  | `number`  | Duration                        |
| `callTool.error`       | `string`  | Error message if failed         |

### OAuthConformanceSuiteResult

| Property     | Type                                   | Description                  |
| ------------ | -------------------------------------- | ---------------------------- |
| `name`       | `string`                               | Suite name                   |
| `serverUrl`  | `string`                               | Shared server URL            |
| `passed`     | `boolean`                              | `true` iff every flow passed |
| `results`    | `Array<ConformanceResult & { label }>` | Per-flow results             |
| `summary`    | `string`                               | Human-readable suite summary |
| `durationMs` | `number`                               | Total suite duration         |

***

## Related

* [CLI: OAuth Conformance](/cli/oauth-conformance) — recipes, troubleshooting, CI integration
* [CLI: OAuth Login](/cli/oauth-login) — interactive authentication and debugging
* [MCPClientManager](/sdk/reference/mcp-client-manager) — connecting to MCP servers programmatically


# OAuth Client Emulation
Source: https://docs.mcpjam.com/sdk/reference/oauth-emulation

Run a headless OAuth preflight that emulates a real client's authentication ladder against an MCP server

Use `runEmulatedOAuthPreflight` to execute an emulated client's authentication ladder against a real MCP server, headlessly. The runner compiles an evidence-backed OAuth profile into ordered attempts, executes each one over the hardened OAuth networking path, and returns a structured result with three independent dimensions: `outcome`, `coverage`, and `comparison`.

<Note>
  `runEmulatedOAuthPreflight` is Node-only. The redirect-planning utilities
  (`planCompletionSafeRedirects`, `isInvalidRedirectUriRejection`) and their
  types are also exported from the browser-safe entry point.
</Note>

## Import

```typescript theme={"theme":"css-variables"}
import {
  runEmulatedOAuthPreflight,
  planCompletionSafeRedirects,
  isInvalidRedirectUriRejection,
} from "@mcpjam/sdk";
```

***

## Quick start

```typescript theme={"theme":"css-variables"}
import { deriveOAuthEmulation, runEmulatedOAuthPreflight } from "@mcpjam/sdk";

// 1. Compile a profile into machine knobs and an attempt ladder.
const emulation = deriveOAuthEmulation({
  profileVersion: 2,
  authModel: {
    status: "verified",
    value: ["oauth2-dcr"],
    source: "https://example.com/capture",
    capturedAt: "2026-01-01",
  },
});

// 2. Run the ladder headlessly against a real server.
const result = await runEmulatedOAuthPreflight({
  serverUrl: "https://your-server.com/mcp",
  emulation,
  callbackUrl: "http://127.0.0.1:41234/callback",
  completeAuthorization: async ({ authorizationUrl, callbackUrl }) => {
    // Drive consent headlessly (e.g. Playwright) and return the code.
    const code = await yourConsentDriver(authorizationUrl, callbackUrl);
    return { code };
  },
});

console.log(result.outcome);       // "completed"
console.log(result.credentials);   // { accessToken, clientId, ... }
console.log(result.sideEffects);   // { dcrRegistrations: 1, tokensIssued: 1 }
console.log(result.comparison);    // "not_compared" — always, until step 6
```

***

## `runEmulatedOAuthPreflight`

### `EmulatedOAuthPreflightConfig`

| Property                       | Type                                                                                      | Required | Default                      | Description                                                                                                                                                                                                                                                    |
| ------------------------------ | ----------------------------------------------------------------------------------------- | -------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serverUrl`                    | `string`                                                                                  | Yes      |                              | MCP server URL.                                                                                                                                                                                                                                                |
| `emulation`                    | `DerivedOAuthEmulation`                                                                   | Yes      |                              | Compiled profile from `deriveOAuthEmulation`.                                                                                                                                                                                                                  |
| `callbackUrl`                  | `string`                                                                                  | Yes      |                              | MCPJam-controlled callback. Authorization and token legs always use it; registration replays the captured list with it appended.                                                                                                                               |
| `serverName`                   | `string`                                                                                  | No       | `"Emulated OAuth Preflight"` | Display name used in log entries.                                                                                                                                                                                                                              |
| `clientIdMetadataUrl`          | `string`                                                                                  | No       | MCPJam default               | CIMD metadata document URL for CIMD attempts.                                                                                                                                                                                                                  |
| `preregistered`                | `{ clientId: string; clientSecret?: string }`                                             | No       |                              | Pre-registered credentials for `preregistered` strategy rungs.                                                                                                                                                                                                 |
| `staticCredential`             | `{ headerName: string; value: string }`                                                   | No       |                              | Static credential for `api-key` rungs. Absent means the rung is skipped and declared — the runner never invents a credential.                                                                                                                                  |
| `customHeaders`                | `Record<string, string>`                                                                  | No       |                              | Extra HTTP headers on every request.                                                                                                                                                                                                                           |
| `customScopes`                 | `string`                                                                                  | No       |                              | Space-separated scope string.                                                                                                                                                                                                                                  |
| `timeoutMs`                    | `number`                                                                                  | No       | `30000`                      | Total deadline for each outbound request.                                                                                                                                                                                                                      |
| `maxSteps`                     | `number`                                                                                  | No       | `40`                         | Maximum state-machine steps per OAuth attempt.                                                                                                                                                                                                                 |
| `completeAuthorization`        | `(input: { authorizationUrl: string; callbackUrl: string }) => Promise<{ code: string }>` | No       |                              | Headless consent handler. Absent means the run stops at the redirect and reports `stopped_at_redirect`.                                                                                                                                                        |
| `requestExecutor`              | `OAuthRequestExecutor`                                                                    | No       | Hardened default             | Override the outbound executor (tests, alternate transports). The default uses DNS pinning, total-deadline timeouts, body caps, redirect caps, and cross-origin credential stripping.                                                                          |
| `allowLoopbackMetadataFetch`   | `boolean`                                                                                 | No       | `false`                      | Allow fetching OAuth metadata from loopback addresses.                                                                                                                                                                                                         |
| `allowPrivateMetadataFetch`    | `boolean`                                                                                 | No       | `false`                      | Allow fetching OAuth metadata from any private address (loopback, RFC 1918, CGNAT, unique-local) and from hostnames that resolve to one. Set by local surfaces; supersedes `allowLoopbackMetadataFetch`. Link-local and cloud-metadata addresses stay refused. |
| `resourceIndicatorEnforcement` | `"warn" \| "reject" \| "reject-rfc9728"`                                                  | No       | `"warn"`                     | How to handle unexpected resource indicators. Defaults to `"warn"` so the run exposes server behavior rather than hiding it.                                                                                                                                   |

### `EmulatedOAuthPreflightOutcome`

| Value                    | Meaning                                                                                      |
| ------------------------ | -------------------------------------------------------------------------------------------- |
| `"completed"`            | Real token obtained AND a valid authenticated JSON-RPC response received.                    |
| `"stopped_at_redirect"`  | Ladder reached the human authorization leg; no `completeAuthorization` handler was supplied. |
| `"static_credential_ok"` | Server accepted the emulated client's static credential.                                     |
| `"unauthenticated_ok"`   | Server served an unauthenticated request (client models no auth).                            |
| `"blocked"`              | The ladder ran out of usable mechanisms.                                                     |
| `"error"`                | A transport or protocol error stopped the run.                                               |

### `EmulatedOAuthPreflightResult`

| Property           | Type                                                        | Description                                                                                               |
| ------------------ | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `serverUrl`        | `string`                                                    | The server URL tested.                                                                                    |
| `protocolVersion`  | `OAuthProtocolVersion`                                      | Protocol version derived from the profile.                                                                |
| `outcome`          | `EmulatedOAuthPreflightOutcome`                             | Overall run outcome.                                                                                      |
| `coverage`         | `OAuthEmulationCoverage`                                    | Per-field enforcement status, carried from the compiler.                                                  |
| `coverageSummary`  | `"complete" \| "partial"`                                   | `"complete"` only when every field is `"modeled"`.                                                        |
| `comparison`       | `"not_compared"`                                            | Always `"not_compared"` — golden-trace comparison is a later step. A run never implies parity on its own. |
| `attempts`         | `EmulatedAuthAttemptResult[]`                               | Per-rung results, in ladder order.                                                                        |
| `authorizationUrl` | `string`                                                    | Present when `outcome === "stopped_at_redirect"`.                                                         |
| `divergences`      | `OAuthEmulationDivergence[]`                                | Compile-time divergences plus everything this run declared.                                               |
| `sideEffects`      | `{ dcrRegistrations: number; tokensIssued: number }`        | What this run caused on the target. At most two DCR registrations per run.                                |
| `credentials`      | `{ accessToken?, refreshToken?, clientId?, clientSecret? }` | Secrets returned once, kept out of every diagnostic surface.                                              |
| `error`            | `{ message: string }`                                       | Present when `outcome === "error"`.                                                                       |

### `EmulatedAuthAttemptResult`

| Property               | Type                                                                          | Description                                                                |
| ---------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `kind`                 | `"oauth" \| "api-key" \| "none"`                                              | Which rung this result is for.                                             |
| `status`               | `"ok" \| "unauthorized" \| "skipped" \| "redirected" \| "blocked" \| "error"` | Rung outcome.                                                              |
| `registrationStrategy` | `EmulatedRegistrationPreference`                                              | Which registration strategy actually ran, for `oauth` attempts.            |
| `plan`                 | `ResolvedAuthorizationPlan`                                                   | The resolved authorization plan, for `oauth` attempts.                     |
| `httpHistory`          | `HttpHistoryEntry[]`                                                          | Redacted HTTP trace. Never carries tokens, codes, secrets, or PKCE values. |
| `detail`               | `string`                                                                      | Human-readable explanation of the outcome.                                 |

***

## Attempt ladder

The `authModel` field of an OAuth profile compiles into an ordered attempt ladder. The runner executes each rung in sequence, with these fallback rules:

* **Consecutive `oauth2-*` entries collapse into one OAuth attempt** carrying their relative order as a registration preference. A client that lists `["oauth2-cimd", "oauth2-dcr"]` performs a single OAuth dance preferring CIMD, not two separate dances.
* **`api-key` and `none` stay in position** as direct MCP probes, so a "static bearer first, OAuth on 401" client reproduces that sequence exactly.
* A static credential falls through to OAuth **only on a 401**. Any other status stops the ladder.
* An OAuth attempt falls through to the next strategy **only while nothing has been committed on the wire** — once a registration request has been sent or authorization has begun, failure stops the ladder.

```typescript theme={"theme":"css-variables"}
// Profile: try static credential first, fall back to DCR on 401.
const emulation = deriveOAuthEmulation({
  profileVersion: 2,
  authModel: {
    status: "verified",
    value: ["api-key", "oauth2-dcr"],
    source: "...",
    capturedAt: "2026-01-01",
  },
});

// result.attempts will be:
// [{ kind: "api-key", status: "unauthorized" }, { kind: "oauth", status: "ok" }]
```

When `authModel` has no evidence, the runner uses MCPJam's own AUTO precedence: `preregistered` → `cimd` → `dcr`, filtered to what the negotiated protocol version supports.

***

## Completion-safe redirects

The runner must replay the real client's registration body while still finishing the dance with a real token. It resolves this with `planCompletionSafeRedirects`:

* The registration body carries the captured `redirect_uris` **in captured order, with MCPJam's callback appended**. The appended entry is a declared divergence.
* The authorization and token legs **always** use MCPJam's callback, so the code comes back and can be exchanged.
* If the server rejects the registration with a structured RFC 7591 `invalid_redirect_uri` (exactly HTTP 400 with `{ "error": "invalid_redirect_uri" }`), the runner re-registers with MCPJam's callback alone. That is the second and final registration a run will ever perform.

### `planCompletionSafeRedirects`

```typescript theme={"theme":"css-variables"}
import { planCompletionSafeRedirects } from "@mcpjam/sdk";

const plan = planCompletionSafeRedirects({
  capturedRedirectUris: ["cursor://anysphere.cursor-retrieval/callback"],
  callbackUrl: "http://127.0.0.1:41234/callback",
});

// plan.registrationRedirectUris:
//   ["cursor://anysphere.cursor-retrieval/callback", "http://127.0.0.1:41234/callback"]
// plan.authorizationRedirectUri:
//   "http://127.0.0.1:41234/callback"
// plan.divergences:
//   [{ kind: "redirect-uri-appended", ... }]
```

**Input:**

| Property               | Type       | Description                                                           |
| ---------------------- | ---------- | --------------------------------------------------------------------- |
| `capturedRedirectUris` | `string[]` | Redirect URIs from the real client's registration, in captured order. |
| `callbackUrl`          | `string`   | MCPJam-controlled callback to append.                                 |

**Output (`CompletionSafeRedirectPlan`):**

| Property                   | Type                         | Description                                                     |
| -------------------------- | ---------------------------- | --------------------------------------------------------------- |
| `registrationRedirectUris` | `string[]`                   | `redirect_uris` for the registration body.                      |
| `authorizationRedirectUri` | `string`                     | Callback for the authorization and token legs. Always MCPJam's. |
| `divergences`              | `OAuthEmulationDivergence[]` | Declared differences from the captured registration.            |

### `isInvalidRedirectUriRejection`

Returns `true` only when a registration response is exactly HTTP 400 with `{ "error": "invalid_redirect_uri" }` in the body — the structured RFC 7591 error that authorizes a retry. Any other status or body shape returns `false`.

```typescript theme={"theme":"css-variables"}
import { isInvalidRedirectUriRejection } from "@mcpjam/sdk";

isInvalidRedirectUriRejection({ status: 400, body: { error: "invalid_redirect_uri" } }); // true
isInvalidRedirectUriRejection({ status: 400, body: { error: "invalid_client_metadata" } }); // false
isInvalidRedirectUriRejection({ status: 401, body: { error: "invalid_redirect_uri" } }); // false
```

***

## Diagnostics hygiene

Credentials are returned once, in `result.credentials`, and are kept out of every other surface:

* `result.attempts[*].httpHistory` is redacted — no tokens, codes, secrets, or PKCE values.
* A caller-supplied `staticCredential` is scrubbed by header name and value before the generic redactor runs, because no generic redactor can know an unconventional header name like `X-Acme-Tenant-Token`.
* `result.comparison` is always `"not_compared"` — a run never claims parity with a golden trace on its own.

***

## Related

* [OAuth Conformance SDK](/sdk/reference/oauth-conformance) — conformance testing with `OAuthConformanceTest`
* [CLI: OAuth Conformance](/cli/oauth-conformance) — CLI recipes and CI integration


# PromptResult
Source: https://docs.mcpjam.com/sdk/reference/prompt-result

API reference for PromptResult

The `PromptResult` class wraps every response from `HostRunner.run()`. It provides methods to inspect tool calls, performance metrics, errors, and conversation history.

## Import

```typescript theme={"theme":"css-variables"}
// PromptResult is returned by HostRunner.run()
// You don't import it directly

const result = await agent.run("...");
// result is a PromptResult
```

***

## Text Response

### text

The final text response from the LLM.

```typescript theme={"theme":"css-variables"}
readonly text: string
```

#### Value

`string` - The assistant's text response. Empty string if no text response.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("What is 2 + 2?");
console.log(result.text); // "2 + 2 equals 4."
```

***

## Tool Call Methods

### toolsCalled()

Returns the names of all tools that were called.

```typescript theme={"theme":"css-variables"}
toolsCalled(): string[]
```

#### Returns

`string[]` - Array of tool names in call order.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Add 2 and 3, then multiply by 4");
console.log(result.toolsCalled()); // ["add", "multiply"]
```

***

### hasToolCall()

Checks if a specific tool was called.

```typescript theme={"theme":"css-variables"}
hasToolCall(toolName: string): boolean
```

#### Parameters

| Parameter  | Type     | Description            |
| ---------- | -------- | ---------------------- |
| `toolName` | `string` | The tool name to check |

#### Returns

`boolean` - `true` if the tool was called.

#### Example

```typescript theme={"theme":"css-variables"}
if (result.hasToolCall("add")) {
  console.log("Addition was performed");
}
```

***

### getToolCalls()

Returns detailed information about all tool calls.

```typescript theme={"theme":"css-variables"}
getToolCalls(): ToolCall[]
```

#### Returns

`ToolCall[]` - Array of tool call objects.

#### ToolCall Type

| Property    | Type               | Description                  |
| ----------- | ------------------ | ---------------------------- |
| `toolName`  | `string`           | Name of the tool             |
| `arguments` | `Record<any, any>` | Arguments passed to the tool |

#### Example

```typescript theme={"theme":"css-variables"}
const calls = result.getToolCalls();

for (const call of calls) {
  console.log(`${call.toolName}(${JSON.stringify(call.arguments)})`);
}
// add({"a":2,"b":3})
// multiply({"a":5,"b":4})
```

***

### getToolArguments()

Returns the arguments passed to a specific tool.

```typescript theme={"theme":"css-variables"}
getToolArguments(toolName: string): Record<string, unknown> | undefined
```

#### Parameters

| Parameter  | Type     | Description   |
| ---------- | -------- | ------------- |
| `toolName` | `string` | The tool name |

#### Returns

`Record<string, unknown> | undefined` - The arguments, or `undefined` if tool wasn't called.

<Note>
  If the tool was called multiple times, returns arguments from the **first** call. Use `getToolCalls()` to access all calls.
</Note>

#### Example

```typescript theme={"theme":"css-variables"}
const args = result.getToolArguments("add");
console.log(args); // { a: 2, b: 3 }
```

***

## Error Handling

### hasError()

Checks if an error occurred.

```typescript theme={"theme":"css-variables"}
hasError(): boolean
```

#### Returns

`boolean` - `true` if an error occurred.

#### Example

```typescript theme={"theme":"css-variables"}
if (result.hasError()) {
  console.error("Something went wrong");
}
```

***

### getError()

Returns the error message if one occurred.

```typescript theme={"theme":"css-variables"}
getError(): string | undefined
```

#### Returns

`string | undefined` - The error message, or `undefined` if no error.

#### Example

```typescript theme={"theme":"css-variables"}
if (result.hasError()) {
  console.error("Error:", result.getError());
}
```

***

## Latency Metrics

### e2eLatencyMs()

Total wall-clock time for the prompt.

```typescript theme={"theme":"css-variables"}
e2eLatencyMs(): number
```

#### Returns

`number` - Milliseconds.

***

### llmLatencyMs()

Time spent waiting for LLM API responses.

```typescript theme={"theme":"css-variables"}
llmLatencyMs(): number
```

#### Returns

`number` - Milliseconds.

***

### mcpLatencyMs()

Time spent executing MCP tools.

```typescript theme={"theme":"css-variables"}
mcpLatencyMs(): number
```

#### Returns

`number` - Milliseconds.

***

### getLatency()

Returns the complete latency breakdown.

```typescript theme={"theme":"css-variables"}
getLatency(): LatencyBreakdown
```

#### Returns

```typescript theme={"theme":"css-variables"}
interface LatencyBreakdown {
  e2eMs: number;   // Total time
  llmMs: number;   // LLM API time
  mcpMs: number;   // Tool execution time
}
```

#### Example

```typescript theme={"theme":"css-variables"}
const latency = result.getLatency();

console.log(`Total: ${latency.e2eMs}ms`);
console.log(`LLM: ${latency.llmMs}ms`);
console.log(`Tools: ${latency.mcpMs}ms`);
```

***

## Token Usage

### totalTokens()

Total tokens used (input + output).

```typescript theme={"theme":"css-variables"}
totalTokens(): number
```

***

### inputTokens()

Tokens used for input (prompt + context).

```typescript theme={"theme":"css-variables"}
inputTokens(): number
```

***

### outputTokens()

Tokens generated by the LLM.

```typescript theme={"theme":"css-variables"}
outputTokens(): number
```

#### Example

```typescript theme={"theme":"css-variables"}
console.log(`Input: ${result.inputTokens()}`);
console.log(`Output: ${result.outputTokens()}`);
console.log(`Total: ${result.totalTokens()}`);
```

***

## Conversation History

### getMessages()

Returns the full conversation as a `CoreMessage[]` array.

```typescript theme={"theme":"css-variables"}
getMessages(): CoreMessage[]
```

#### Returns

`CoreMessage[]` - The full conversation message format used by the SDK.

#### Example

```typescript theme={"theme":"css-variables"}
const messages = result.getMessages();

for (const msg of messages) {
  console.log(`[${msg.role}]:`, msg.content);
}
```

***

### getUserMessages()

Returns only user messages.

```typescript theme={"theme":"css-variables"}
getUserMessages(): CoreMessage[]
```

***

### getAssistantMessages()

Returns only assistant messages.

```typescript theme={"theme":"css-variables"}
getAssistantMessages(): CoreMessage[]
```

***

### getToolMessages()

Returns only tool result messages.

```typescript theme={"theme":"css-variables"}
getToolMessages(): CoreMessage[]
```

#### Example

```typescript theme={"theme":"css-variables"}
const toolMsgs = result.getToolMessages();

for (const msg of toolMsgs) {
  console.log("Tool result:", msg.content);
}
```

***

## MCP App widget snapshots

### getWidgetSnapshots()

Returns HTML snapshot payloads captured during tool execution for **MCP Apps** (one entry per relevant tool call when replay data was collected).

```typescript theme={"theme":"css-variables"}
getWidgetSnapshots(): EvalWidgetSnapshotInput[]
```

#### Returns

`EvalWidgetSnapshotInput[]` — metadata plus inline HTML (before upload) or storage ids after reporting. Empty if no MCP App tools ran or if [`HostRunner`](/sdk/reference/host-runner) was created **without** [`mcpClientManager`](/sdk/reference/host-runner#parameters).

#### Why the manager matters

MCP App UIs reference HTML via **`ui.resourceUri`**. That HTML is fetched with MCP **`resources/read`**, which only the connected [`MCPClientManager`](/sdk/reference/mcp-client-manager) can perform. The model’s tool **result** is not a substitute for that document, so **`getWidgetSnapshots()` stays empty unless you pass the same `manager` into `new HostRunner({ …, mcpClientManager: manager })`**.

When you report results to MCPJam (for example via [`createEvalRunReporter`](/sdk/reference/eval-reporting#createevalrunreporter) or [`PromptResult.toEvalResult()`](/sdk/reference/prompt-result)), snapshots are uploaded and linked from the trace so the dashboard can **replay widgets offline**.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Open the chart widget");

const snapshots = result.getWidgetSnapshots();
console.log("Snapshots:", snapshots.length);
// After reporting, snapshots are merged into each eval result's trace on MCPJam
```

***

## toEvalResult()

Builds an [`EvalResultInput`](/sdk/reference/eval-reporting#evalresultinput) for MCPJam reporting (messages, spans, tool calls, tokens, optional widget snapshots).

```typescript theme={"theme":"css-variables"}
toEvalResult(
  options?: Partial<Omit<EvalResultInput, "actualToolCalls" | "tokens" | "trace">> & {
    failOnToolError?: boolean;
  },
): EvalResultInput
```

The **`passed`** field combines your explicit or inferred structural outcome with **tool execution** checks (see [Tool execution and `passed`](/sdk/reference/eval-reporting#tool-execution-and-passed)): by default, failures in the trace (MCP `isError`, errored tool spans, error-shaped tool-results) force `passed: false` even if you pass `passed: true` in `options`. Set **`failOnToolError: false`** in `options` to keep structural / explicit `passed` only.

See [Tool execution and `passed`](/sdk/reference/eval-reporting#tool-execution-and-passed) in the eval reporting reference.

***

## Using as Context

Pass `PromptResult` as context for multi-turn conversations.

```typescript theme={"theme":"css-variables"}
const r1 = await agent.run("Create a project");

// Single result as context
const r2 = await agent.run("Add a task", { context: r1 });

// Multiple results as context
const r3 = await agent.run("Show summary", { context: [r1, r2] });
```

***

## Complete Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Add 100 and 200");

// Response
console.log("Text:", result.text);

// Tool calls
console.log("Tools:", result.toolsCalled());
console.log("Has add?", result.hasToolCall("add"));
console.log("Add args:", result.getToolArguments("add"));

for (const call of result.getToolCalls()) {
  console.log(`  ${call.toolName}:`, call.arguments);
}

// Errors
if (result.hasError()) {
  console.error("Error:", result.getError());
}

// Performance
console.log("Latency:", result.e2eLatencyMs(), "ms");
console.log("  LLM:", result.llmLatencyMs(), "ms");
console.log("  Tools:", result.mcpLatencyMs(), "ms");

// Tokens
console.log("Tokens:", result.totalTokens());
console.log("  Input:", result.inputTokens());
console.log("  Output:", result.outputTokens());

// History
console.log("Messages:", result.getMessages().length);
```

***

## Related

* [Testing with LLMs](/sdk/concepts/testing-with-llms) - Conceptual guide
* [Validators Reference](/sdk/reference/validators) - Assert tool calls
* [HostRunner Reference](/sdk/reference/host-runner) - Agent configuration


# Protocol Conformance SDK
Source: https://docs.mcpjam.com/sdk/reference/protocol-conformance

Programmatic MCP protocol conformance testing with MCPConformanceTest and MCPConformanceSuite

The Protocol Conformance SDK lets you run the same checks as the CLI `protocol conformance` commands, but inside your own Jest or Vitest suites.

Use it when you want MCP protocol regressions to fail CI directly, or when you need JUnit XML and JSON artifacts from code rather than shell scripts.

<Note>
  Protocol conformance is currently HTTP-only. For local stdio-only servers, use
  [MCP Apps Conformance](/sdk/reference/apps-conformance),
  [unit tests](/sdk/reference/mcp-client-manager), or run your server through an
  HTTP bridge in CI.
</Note>

## Import

```typescript theme={"theme":"css-variables"}
import {
  MCPConformanceSuite,
  MCPConformanceTest,
  renderConformanceReportJUnitXml,
  renderConformanceReportJson,
  toConformanceReport,
} from "@mcpjam/sdk";
```

## Single run

```typescript theme={"theme":"css-variables"}
const test = new MCPConformanceTest({
  serverUrl: "https://your-server.com/mcp",
  checkTimeout: 15_000,
  checkIds: [
    "server-initialize",
    "ping",
    "tools-list",
    "prompts-list",
    "resources-list",
  ],
});

const result = await test.run();

console.log(result.passed);
console.log(result.summary);
console.log(result.checks);
```

## Suite

```typescript theme={"theme":"css-variables"}
import { writeFileSync } from "node:fs";

const suite = new MCPConformanceSuite({
  name: "Protocol CI",
  serverUrl: "https://your-server.com/mcp",
  defaults: {
    checkTimeout: 15_000,
  },
  runs: [
    {
      label: "core-surface",
      checkIds: [
        "server-initialize",
        "ping",
        "tools-list",
        "prompts-list",
        "resources-list",
      ],
    },
    {
      label: "security",
      checkIds: [
        "localhost-host-rebinding-rejected",
        "localhost-host-valid-accepted",
      ],
    },
  ],
});

const result = await suite.run();
const report = toConformanceReport(result);

writeFileSync(
  "protocol-conformance.junit.xml",
  renderConformanceReportJUnitXml(report),
);
writeFileSync(
  "protocol-conformance.report.json",
  JSON.stringify(renderConformanceReportJson(report), null, 2),
);
```

`MCPConformanceSuiteConfig` keeps the existing `serverUrl + defaults + runs` shape:

| Property    | Type                                                                           | Required | Description                                  |
| ----------- | ------------------------------------------------------------------------------ | -------- | -------------------------------------------- |
| `name`      | `string`                                                                       | No       | Suite label shown in summaries and JUnit XML |
| `serverUrl` | `string`                                                                       | Yes      | Shared MCP HTTP server URL                   |
| `defaults`  | `Partial<Omit<MCPConformanceConfig, "serverUrl">>`                             | No       | Shared defaults applied to every run         |
| `runs`      | `Array<Partial<Omit<MCPConformanceConfig, "serverUrl">> & { label?: string }>` | Yes      | Individual run labels and check selections   |

## MCPConformanceConfig

| Property        | Type                     | Required | Default        | Description                     |
| --------------- | ------------------------ | -------- | -------------- | ------------------------------- |
| `serverUrl`     | `string`                 | Yes      |                | MCP server URL                  |
| `accessToken`   | `string`                 | No       |                | Bearer access token             |
| `customHeaders` | `Record<string, string>` | No       |                | Extra HTTP headers              |
| `checkTimeout`  | `number`                 | No       | `15000`        | Per-check timeout in ms         |
| `categories`    | `MCPCheckCategory[]`     | No       | all categories | Restrict checks to categories   |
| `checkIds`      | `MCPCheckId[]`           | No       | all checks     | Restrict checks to specific ids |
| `fetchFn`       | `typeof fetch`           | No       | `fetch`        | Custom fetch implementation     |
| `clientName`    | `string`                 | No       | SDK default    | Custom MCP client name          |

## Available categories and check ids

Categories:

* `core`
* `protocol`
* `tools`
* `prompts`
* `resources`
* `security`
* `transport`

Check ids:

* `server-initialize`
* `ping`
* `logging-set-level`
* `completion-complete`
* `capabilities-consistent`
* `tools-list`
* `tools-input-schemas-valid`
* `tools-x-mcp-header-declarations-valid` (2026-07-28 only)
* `prompts-list`
* `resources-list`
* `protocol-invalid-method-error`
* `localhost-host-rebinding-rejected` — requires HTTP 403 when the run explicitly pins `protocolVersion` to `2025-11-25` or later; accepts any 4xx on earlier pins or when no version is pinned
* `localhost-host-valid-accepted`
* `server-sse-polling-session`
* `server-accepts-multiple-post-streams`
* `server-sse-streams-functional`
* `notification-post-accepted` (2025 revisions only)
* `get-stream-or-405` (2025 revisions only)
* `session-id-visible-ascii` (2025 revisions only)
* `post-response-content-type`

## Result types

`MCPConformanceTest.run()` returns an `MCPConformanceResult`:

| Property           | Type                                   | Description                                                                                       |
| ------------------ | -------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `passed`           | `boolean`                              | True only when `outcome` is `"passed"`. A check that could not run keeps this `false`.            |
| `outcome`          | `"passed" \| "failed" \| "incomplete"` | `"failed"` on any violation; `"incomplete"` when nothing failed but a selected check never ran.   |
| `incompleteReason` | `string \| undefined`                  | Present when `outcome` is `"incomplete"`: which checks did not run and what to change.            |
| `serverUrl`        | `string`                               | Server under test                                                                                 |
| `checks`           | `MCPCheckResult[]`                     | Individual check results                                                                          |
| `summary`          | `string`                               | Human-readable summary (`N/M checks passed, X failed, Y could not run, Z not applicable`)         |
| `durationMs`       | `number`                               | Total duration                                                                                    |
| `categorySummary`  | `Record<MCPCheckCategory, ...>`        | Per-category `total`, `passed`, `failed`, `skipped`, and `couldNotRun` counts                     |
| `readiness`        | `MCPReadinessWarning[]`                | Advisory observations that do not affect the verdict (see [Readiness advice](#readiness-advice)). |

Each `MCPCheckResult` includes a `skipReason` field whenever `status` is `"skipped"`:

* `"not-applicable"` — the check cannot apply to this server (era-gated check on the wrong protocol version, unadvertised capability, localhost-only requirement against a remote server). These never hold a run back.
* `"could-not-run"` — the check applies but was never exercised (broken session, missing `inputRequiredProbe`, subscription stream unavailable). These make the run `incomplete`.

`MCPConformanceSuite.run()` returns an `MCPConformanceSuiteResult`:

| Property     | Type                                              | Description                  |
| ------------ | ------------------------------------------------- | ---------------------------- |
| `name`       | `string`                                          | Suite name                   |
| `serverUrl`  | `string`                                          | Shared server URL            |
| `passed`     | `boolean`                                         | `true` iff every run passed  |
| `results`    | `Array<MCPConformanceResult & { label: string }>` | Per-run results              |
| `summary`    | `string`                                          | Human-readable suite summary |
| `durationMs` | `number`                                          | Total suite duration         |

## Outcomes: passed, failed, incomplete

A run reports one of three outcomes, and `passed` is `true` only for the first:

| `outcome`    | CLI exit code | Meaning                                                                                                      |
| ------------ | ------------- | ------------------------------------------------------------------------------------------------------------ |
| `passed`     | `0`           | Every selected check either ran and passed, or does not apply to this server.                                |
| `failed`     | `1`           | At least one check found a violation.                                                                        |
| `incomplete` | `3`           | Nothing failed, but at least one selected check could not be run, so the run does not establish conformance. |

An incomplete run carries a root `incompleteReason` naming the checks that did not run and what to change; non-quiet CLI runs also print it to stderr. For a suite, the exit code is the worst of all runs (failure outranks incomplete).

## Readiness advice

Every run also collects a `readiness` channel: advisory observations about server behavior that the spec does not mandate but that indicate potential issues. Readiness items never affect `outcome`, `passed`, or exit codes — they are informational only.

Each `MCPReadinessWarning` has:

| Property       | Type                                 | Description                                                |
| -------------- | ------------------------------------ | ---------------------------------------------------------- |
| `id`           | `string`                             | Stable identifier (e.g. `readiness-parse-error-handling`). |
| `title`        | `string`                             | Short human-readable label.                                |
| `specStrength` | `"MAY" \| "SHOULD" \| "RECOMMENDED"` | The spec strength of the underlying guidance.              |
| `message`      | `string`                             | Description of what was observed and why it matters.       |
| `details`      | `Record<string, unknown>`            | Structured details (status codes, headers, etc.).          |

The CLI prints readiness items as an `Advice` section on stderr after each `protocol conformance` or `conformance-suite` run. The section is suppressed by `--quiet` and does not appear in JSON or JUnit XML output.

Current readiness checks:

* `readiness-parse-error-handling` — fires when the server accepts an unparseable JSON body as success without returning a JSON-RPC `-32700` parse error. No MCP revision mandates a specific response to malformed input, so this is `MAY` advice rather than a check.
* `readiness-session-termination` — on 2025-era runs, fires when a session-termination `DELETE` returns a 5xx. On 2026-era runs, fires when `GET` or `DELETE` traffic from an older client is not answered with HTTP 405 (the 2026 backward-compat `SHOULD`).

## CI reporting

All three conformance domains share the same report helpers:

* `toConformanceReport(result)`
* `renderConformanceReportJUnitXml(report)`
* `renderConformanceReportJson(report)`

The CLI uses the same helpers internally, so `protocol conformance --format junit-xml` and `protocol conformance-suite --format junit-xml` emit the same XML that the SDK helpers produce for the same result.

## Related

* [CLI CI / CD](/cli/ci)
* [MCP Apps Conformance SDK](/sdk/reference/apps-conformance)
* [OAuth Conformance SDK](/sdk/reference/oauth-conformance)


# MCP Tasks Conformance SDK
Source: https://docs.mcpjam.com/sdk/reference/tasks-conformance

Programmatic MCP Tasks wire validation with MCPTasksConformanceTest

The MCP Tasks Conformance SDK lets you validate the MCP Tasks wire your server exposes: which wire the connection resolves to, whether declaration hygiene holds, and whether the server honours the observable parts of the contract.

Use it when you want the same checks as the CLI's [`tasks conformance`](/cli/tasks-conformance) command, but inside your own test runner or CI pipeline.

<Note>
  Tasks conformance provokes and then polls a real task, so it requires a
  persistent connection. It is not available in hosted mode.
</Note>

## Import

```typescript theme={"theme":"css-variables"}
import {
  MCPTasksConformanceTest,
  toConformanceReport,
  renderConformanceReportJUnitXml,
  renderConformanceReportJson,
} from "@mcpjam/sdk";
```

## Basic usage

```typescript theme={"theme":"css-variables"}
const test = new MCPTasksConformanceTest({
  url: "https://your-server.com/mcp",
  timeout: 30_000,
});

const result = await test.run();

console.log(result.passed);              // true
console.log(result.outcome);            // "passed" | "failed" | "incomplete"
console.log(result.summary);            // "8/8 checks passed, 0 failed, 0 could not run, 0 not applicable"
console.log(result.discovery.wire);     // "extension" | "legacy" | "none"
```

Pass a tool name on the extension wire. Auto-selection reads `execution.taskSupport`, which the 2026-07-28 `ToolSchema` strips, so without `toolName` the task-dependent checks cannot run and `run()` returns `outcome: "incomplete"` with `passed: false`:

```typescript theme={"theme":"css-variables"}
const test = new MCPTasksConformanceTest({
  url: "https://your-server.com/mcp",
  toolName: "long_job",
  timeout: 30_000,
});
```

For a stdio server:

```typescript theme={"theme":"css-variables"}
const test = new MCPTasksConformanceTest({
  command: "node",
  args: ["server.js"],
  timeout: 30_000,
});
```

## MCPTasksConformanceConfig

`MCPTasksConformanceConfig` extends the standard `MCPServerConfig`, so it accepts the same HTTP and stdio connection settings as `MCPClientManager`.

Additional properties:

| Property        | Type                      | Required | Default                               | Description                                                                                                                                                                                                                                            |
| --------------- | ------------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `checkIds`      | `MCPTasksCheckId[]`       | No       | all checks                            | Run only the selected checks.                                                                                                                                                                                                                          |
| `toolName`      | `string`                  | No       | auto-detected on the legacy wire only | Tool used to provoke a task. Required on the extension wire: auto-selection reads `execution.taskSupport`, which the 2026-07-28 `ToolSchema` strips. A name the server does not list is an error, not a silent miss — the run comes back `incomplete`. |
| `toolArguments` | `Record<string, unknown>` | No       | `{}`                                  | Arguments passed to the probe tool.                                                                                                                                                                                                                    |
| `pollTimeoutMs` | `number`                  | No       | `30000`                               | Upper bound on polling a created task to a terminal status.                                                                                                                                                                                            |

Example with a focused check set:

```typescript theme={"theme":"css-variables"}
const test = new MCPTasksConformanceTest({
  url: "https://your-server.com/mcp",
  toolName: "long_job",
  toolArguments: { seconds: 1 },
  pollTimeoutMs: 10_000,
  checkIds: ["tasks-wire-resolvable", "tasks-ttl-shape"],
});
```

## Check ids

| Check id                               | Category    | What it asserts                                                                                                                                                                                                                                                               |
| -------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tasks-wire-resolvable`                | `dispatch`  | The negotiated protocol version and capabilities resolve to exactly one tasks wire.                                                                                                                                                                                           |
| `tasks-declaration-hygiene`            | `dispatch`  | Outbound requests carry `params.task` only on the legacy wire and the extension declaration only on the extension wire.                                                                                                                                                       |
| `tasks-result-type-discipline`         | `creation`  | A task-eligible `tools/call` returns either a normal tool result or a flat `CreateTaskResult` with `resultType: "task"` and a non-empty `taskId`. The discriminator is required whether absent or wrong — it is the only signal that separates a task from a standard result. |
| `tasks-undeclared-creation-refused`    | `creation`  | On the extension wire, a `tools/call` that did not carry the extension declaration must not come back as a `CreateTaskResult`.                                                                                                                                                |
| `tasks-ttl-shape`                      | `lifecycle` | TTL and poll interval use the era-native shapes: `ttlMs`/`pollIntervalMs` on the extension, `ttl`/`pollInterval` on the legacy wire.                                                                                                                                          |
| `tasks-inline-result`                  | `lifecycle` | A completed extension task carries its result inline on `tasks/get`; a legacy task exposes it via `tasks/result`.                                                                                                                                                             |
| `tasks-mcp-name-routing`               | `lifecycle` | Over HTTP, `tasks/get` is sent with `Mcp-Name` set to the task id.                                                                                                                                                                                                            |
| `tasks-undeclared-capability-rejected` | `lifecycle` | `tasks/get`, `tasks/update`, `tasks/cancel` and a task-filtered `subscriptions/listen` sent WITHOUT the extension declaration must each be rejected with `-32021` (Missing Required Client Capability). Per-method outcomes are reported in the check's `details.probes`.     |

## Result shape

`run()` returns an `MCPTasksConformanceResult`.

| Property           | Type                                                   | Description                                                                                     |
| ------------------ | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| `passed`           | `boolean`                                              | True only when `outcome` is `"passed"`. A check that could not run keeps this `false`.          |
| `outcome`          | `"passed" \| "failed" \| "incomplete"`                 | `"failed"` on any violation; `"incomplete"` when nothing failed but a selected check never ran. |
| `incompleteReason` | `string \| undefined`                                  | Present when `outcome` is `"incomplete"`: which checks did not run and what to change.          |
| `target`           | `string`                                               | URL or stdio command under test.                                                                |
| `checks`           | `MCPTasksCheckResult[]`                                | Individual check results.                                                                       |
| `summary`          | `string`                                               | Human-readable summary.                                                                         |
| `durationMs`       | `number`                                               | Total duration.                                                                                 |
| `categorySummary`  | `Record<"dispatch" \| "creation" \| "lifecycle", ...>` | Per-category `total`, `passed`, `failed`, `skipped`, and `couldNotRun` counts.                  |
| `discovery`        | object                                                 | Wire, protocol version, and tool counts discovered during the run.                              |

The `discovery` object includes:

| Property               | Type                                | Description                                  |
| ---------------------- | ----------------------------------- | -------------------------------------------- |
| `wire`                 | `"extension" \| "legacy" \| "none"` | Resolved tasks wire.                         |
| `protocolVersion`      | `string \| undefined`               | Negotiated MCP protocol version.             |
| `toolCount`            | `number`                            | Total tools listed.                          |
| `taskCapableToolCount` | `number`                            | Tools with `execution.taskSupport` metadata. |
| `probedTool`           | `string \| undefined`               | Name of the tool used to provoke a task.     |
| `createdTaskId`        | `string \| undefined`               | Task id returned by the probe tool call.     |

Each `MCPTasksCheckResult` includes:

* `id`
* `category`
* `title`
* `description`
* `status` — `"passed"`, `"failed"`, or `"skipped"`
* `skipReason` — set whenever `status` is `"skipped"`: `"not-applicable"` (the check cannot apply to this server, so it does not hold the run back) or `"could-not-run"` (the check applies but was never exercised, which makes the run `incomplete`)
* `durationMs`
* optional `details`
* optional `warnings`
* optional `error`

## CI reporting

Use the shared reporting helpers to produce JUnit XML or JSON artifacts:

```typescript theme={"theme":"css-variables"}
import { writeFileSync } from "node:fs";

const test = new MCPTasksConformanceTest({
  url: "https://your-server.com/mcp",
  toolName: "long_job",
});

const result = await test.run();
const report = toConformanceReport(result);

writeFileSync(
  "tasks-conformance.junit.xml",
  renderConformanceReportJUnitXml(report),
);
writeFileSync(
  "tasks-conformance.report.json",
  JSON.stringify(renderConformanceReportJson(report), null, 2),
);
```

`toConformanceReport` accepts protocol, OAuth, apps, and tasks results and normalizes them into a single report shape, so the SDK and CLI JUnit output are byte-identical for the same result.

## Notes

* Declaration hygiene is asserted against captured outbound JSON-RPC bytes, not re-derived from intent.
* The `tasks-undeclared-creation-refused` and `tasks-undeclared-capability-rejected` checks apply to the extension wire only and are skipped on the legacy wire.
* `tasks-undeclared-capability-rejected` needs a live task to probe with, and its probes run after every check that reads that task so a wrongly-accepted `tasks/update` or `tasks/cancel` cannot corrupt the rest of the run. A server with no `subscriptions/listen` (`-32601`) gets that sub-probe reported as a warning rather than a failure.
* The `tasks-mcp-name-routing` check applies to HTTP transports only and is skipped for stdio servers.
* Checks that require a created task report `skipReason: "could-not-run"` when no probe tool resolves, when the named tool is not listed, when the probed call produced no task, or when the task never became readable. Any of those makes the run `incomplete`; a skip can never add up to a pass.
* CLI exit codes: `0` for `passed`, `1` for `failed`, `3` for `incomplete` (`2` stays reserved for usage errors).


# Validators
Source: https://docs.mcpjam.com/sdk/reference/validators

API reference for validator functions

The SDK provides 9 boolean validator functions for asserting tool call behavior in tests, plus `evaluateToolCalls` — a richer, configurable matcher that returns structured results.

The matcher's verdict is filed at the **Selection** link of the [user-value chain](/sdk/concepts/user-value-chain), alongside the predicates that measure which tool the model reached for.

## Import

```typescript theme={"theme":"css-variables"}
import {
  matchToolCalls,
  matchToolCallsSubset,
  matchAnyToolCall,
  matchToolCallCount,
  matchNoToolCalls,
  matchToolCallWithArgs,
  matchToolCallWithPartialArgs,
  matchToolArgument,
  matchToolArgumentWith,
  // Structured matcher
  evaluateToolCalls,
} from "@mcpjam/sdk";
```

The matchers module is also available as a standalone browser-safe entry point:

```typescript theme={"theme":"css-variables"}
import { evaluateToolCalls } from "@mcpjam/sdk/matchers";
```

***

## Tool Name Validators

### matchToolCalls()

Exact match - all tools in exact order.

```typescript theme={"theme":"css-variables"}
matchToolCalls(expected: string[], actual: string[]): boolean
```

#### Parameters

| Parameter  | Type       | Description                                     |
| ---------- | ---------- | ----------------------------------------------- |
| `expected` | `string[]` | Expected tool names in order                    |
| `actual`   | `string[]` | Actual tool names (from `result.toolsCalled()`) |

#### Returns

`boolean` - `true` if exact match.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Add 2+3, then multiply by 4");
// toolsCalled() = ["add", "multiply"]

matchToolCalls(["add", "multiply"], result.toolsCalled()); // true
matchToolCalls(["multiply", "add"], result.toolsCalled()); // false (wrong order)
matchToolCalls(["add"], result.toolsCalled());             // false (missing multiply)
```

***

### matchToolCallsSubset()

Checks if all expected tools were called (any order, extras allowed).

```typescript theme={"theme":"css-variables"}
matchToolCallsSubset(expected: string[], actual: string[]): boolean
```

#### Parameters

| Parameter  | Type       | Description                |
| ---------- | ---------- | -------------------------- |
| `expected` | `string[]` | Tools that must be present |
| `actual`   | `string[]` | Actual tool names          |

#### Returns

`boolean` - `true` if all expected tools are present.

#### Example

```typescript theme={"theme":"css-variables"}
// toolsCalled() = ["add", "multiply"]

matchToolCallsSubset(["add"], result.toolsCalled());             // true
matchToolCallsSubset(["multiply", "add"], result.toolsCalled()); // true (order doesn't matter)
matchToolCallsSubset(["divide"], result.toolsCalled());          // false
```

***

### matchAnyToolCall()

Checks if at least one expected tool was called.

```typescript theme={"theme":"css-variables"}
matchAnyToolCall(expected: string[], actual: string[]): boolean
```

#### Parameters

| Parameter  | Type       | Description                         |
| ---------- | ---------- | ----------------------------------- |
| `expected` | `string[]` | Any of these tools should be called |
| `actual`   | `string[]` | Actual tool names                   |

#### Returns

`boolean` - `true` if at least one match.

#### Example

```typescript theme={"theme":"css-variables"}
// toolsCalled() = ["add"]

matchAnyToolCall(["add", "subtract"], result.toolsCalled()); // true
matchAnyToolCall(["multiply", "divide"], result.toolsCalled()); // false
```

***

### matchToolCallCount()

Checks if a tool was called exactly N times.

```typescript theme={"theme":"css-variables"}
matchToolCallCount(
  toolName: string,
  actual: string[],
  count: number
): boolean
```

#### Parameters

| Parameter  | Type       | Description       |
| ---------- | ---------- | ----------------- |
| `toolName` | `string`   | Tool to count     |
| `actual`   | `string[]` | Actual tool names |
| `count`    | `number`   | Expected count    |

#### Returns

`boolean` - `true` if count matches.

#### Example

```typescript theme={"theme":"css-variables"}
// toolsCalled() = ["add", "add", "add"]

matchToolCallCount("add", result.toolsCalled(), 3); // true
matchToolCallCount("add", result.toolsCalled(), 2); // false
```

***

### matchNoToolCalls()

Checks that no tools were called.

```typescript theme={"theme":"css-variables"}
matchNoToolCalls(actual: string[]): boolean
```

#### Parameters

| Parameter | Type       | Description       |
| --------- | ---------- | ----------------- |
| `actual`  | `string[]` | Actual tool names |

#### Returns

`boolean` - `true` if empty.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("What is the capital of France?");
matchNoToolCalls(result.toolsCalled()); // true (if no tools called)
```

***

## Argument Validators

### matchToolCallWithArgs()

Checks if a tool was called with exactly the specified arguments.

```typescript theme={"theme":"css-variables"}
matchToolCallWithArgs(
  toolName: string,
  expectedArgs: Record<string, unknown>,
  toolCalls: ToolCall[]
): boolean
```

#### Parameters

| Parameter      | Type                      | Description                      |
| -------------- | ------------------------- | -------------------------------- |
| `toolName`     | `string`                  | Tool to check                    |
| `expectedArgs` | `Record<string, unknown>` | Expected arguments (exact match) |
| `toolCalls`    | `ToolCall[]`              | From `result.getToolCalls()`     |

#### Returns

`boolean` - `true` if exact argument match.

#### Example

```typescript theme={"theme":"css-variables"}
const result = await agent.run("Add 2 and 3");

matchToolCallWithArgs("add", { a: 2, b: 3 }, result.getToolCalls()); // true
matchToolCallWithArgs("add", { a: 2 }, result.getToolCalls());       // false (missing b)
matchToolCallWithArgs("add", { a: 3, b: 2 }, result.getToolCalls()); // false (wrong values)
```

***

### matchToolCallWithPartialArgs()

Checks if a tool was called with arguments containing the expected subset.

```typescript theme={"theme":"css-variables"}
matchToolCallWithPartialArgs(
  toolName: string,
  expectedArgs: Record<string, unknown>,
  toolCalls: ToolCall[]
): boolean
```

#### Parameters

| Parameter      | Type                      | Description                  |
| -------------- | ------------------------- | ---------------------------- |
| `toolName`     | `string`                  | Tool to check                |
| `expectedArgs` | `Record<string, unknown>` | Arguments to match (subset)  |
| `toolCalls`    | `ToolCall[]`              | From `result.getToolCalls()` |

#### Returns

`boolean` - `true` if all expected args present with matching values.

#### Example

```typescript theme={"theme":"css-variables"}
// Tool called with { name: "Task", priority: "high", project: "default" }

matchToolCallWithPartialArgs("createTask", { name: "Task" }, calls);     // true
matchToolCallWithPartialArgs("createTask", { priority: "high" }, calls); // true
matchToolCallWithPartialArgs("createTask", { status: "open" }, calls);   // false
```

***

### matchToolArgument()

Checks if a specific argument has the expected value.

```typescript theme={"theme":"css-variables"}
matchToolArgument(
  toolName: string,
  argName: string,
  expectedValue: unknown,
  toolCalls: ToolCall[]
): boolean
```

#### Parameters

| Parameter       | Type         | Description                  |
| --------------- | ------------ | ---------------------------- |
| `toolName`      | `string`     | Tool to check                |
| `argName`       | `string`     | Argument name                |
| `expectedValue` | `unknown`    | Expected value               |
| `toolCalls`     | `ToolCall[]` | From `result.getToolCalls()` |

#### Returns

`boolean` - `true` if argument matches.

#### Example

```typescript theme={"theme":"css-variables"}
matchToolArgument("add", "a", 5, result.getToolCalls());  // true if a=5
matchToolArgument("add", "b", 10, result.getToolCalls()); // true if b=10
```

***

### matchToolArgumentWith()

Checks if an argument passes a custom predicate.

```typescript theme={"theme":"css-variables"}
matchToolArgumentWith(
  toolName: string,
  argName: string,
  predicate: (value: unknown) => boolean,
  toolCalls: ToolCall[]
): boolean
```

#### Parameters

| Parameter   | Type                          | Description                  |
| ----------- | ----------------------------- | ---------------------------- |
| `toolName`  | `string`                      | Tool to check                |
| `argName`   | `string`                      | Argument name                |
| `predicate` | `(value: unknown) => boolean` | Custom validation function   |
| `toolCalls` | `ToolCall[]`                  | From `result.getToolCalls()` |

#### Returns

`boolean` - `true` if predicate returns `true`.

#### Example

```typescript theme={"theme":"css-variables"}
// Type check
matchToolArgumentWith(
  "echo",
  "message",
  (v) => typeof v === "string",
  result.getToolCalls()
);

// Content check
matchToolArgumentWith(
  "echo",
  "message",
  (v) => String(v).includes("hello"),
  result.getToolCalls()
);

// Range check
matchToolArgumentWith(
  "add",
  "a",
  (v) => typeof v === "number" && v > 0 && v < 100,
  result.getToolCalls()
);

// Email validation
matchToolArgumentWith(
  "sendEmail",
  "to",
  (v) => /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(String(v)),
  result.getToolCalls()
);
```

***

## Usage with Test Frameworks

### Jest / Vitest

```typescript theme={"theme":"css-variables"}
import { describe, it, expect } from "vitest";
import { matchToolCalls, matchToolCallWithArgs } from "@mcpjam/sdk";

describe("Calculator", () => {
  it("calls add for addition", async () => {
    const result = await agent.run("Add 5 and 3");
    expect(matchToolCalls(["add"], result.toolsCalled())).toBe(true);
  });

  it("passes correct arguments", async () => {
    const result = await agent.run("Add 10 and 20");
    expect(
      matchToolCallWithArgs("add", { a: 10, b: 20 }, result.getToolCalls())
    ).toBe(true);
  });
});
```

### With EvalTest

```typescript theme={"theme":"css-variables"}
import { EvalTest, matchToolCallsSubset } from "@mcpjam/sdk";

const test = new EvalTest({
  id: "c_multi_step",
  name: "multi-step",
  test: async (agent) => {
    const result = await agent.run("Add 2+3, then multiply by 4");
    return matchToolCallsSubset(["add", "multiply"], result.toolsCalled());
  },
});
```

***

## evaluateToolCalls()

A configurable matcher that returns a structured result instead of a plain boolean. Use this when you need to know *why* a match failed (missing tools, argument mismatches, wrong order, extra calls) or when you want to control matching behavior per test case.

```typescript theme={"theme":"css-variables"}
evaluateToolCalls(
  expected: EvalToolCall[],
  actual: EvalToolCall[],
  options?: EvalMatchOptions & { isNegativeTest?: boolean }
): EvalToolCallMatchResult
```

### Parameters

| Parameter  | Type               | Description                                                        |
| ---------- | ------------------ | ------------------------------------------------------------------ |
| `expected` | `EvalToolCall[]`   | Expected tool calls (each has `toolName` and optional `arguments`) |
| `actual`   | `EvalToolCall[]`   | Actual tool calls from the trace                                   |
| `options`  | `EvalMatchOptions` | Matching behavior overrides (see below)                            |

### EvalMatchOptions

| Option                | Type                               | Default     | Description                                                                                                                                                                           |
| --------------------- | ---------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `toolCallOrder`       | `"ignore" \| "strict"`             | `"ignore"`  | `"strict"` requires matched calls to appear in the same relative order as `expected`.                                                                                                 |
| `allowExtraToolCalls` | `boolean`                          | `true`      | When `false`, any actual call beyond what's expected fails the test.                                                                                                                  |
| `argumentMatching`    | `"partial" \| "exact" \| "ignore"` | `"partial"` | `"partial"` checks only expected keys and supports type placeholders (`"string"`, `"number"`, `"any"`, etc.). `"exact"` requires deep equality. `"ignore"` skips argument comparison. |

### EvalToolCallMatchResult

| Property             | Type                       | Description                                                                                     |
| -------------------- | -------------------------- | ----------------------------------------------------------------------------------------------- |
| `passed`             | `boolean`                  | Whether the match succeeded.                                                                    |
| `missing`            | `EvalToolCall[]`           | Expected calls that were not found in actual.                                                   |
| `extra`              | `EvalToolCall[]`           | Actual calls beyond what was expected.                                                          |
| `outOfOrder`         | `EvalOutOfOrderToolCall[]` | Matched calls that appeared in the wrong order (only populated when `toolCallOrder: "strict"`). |
| `argumentMismatches` | `EvalArgumentMismatch[]`   | Calls matched by name but with incompatible arguments.                                          |

### Example

```typescript theme={"theme":"css-variables"}
import { evaluateToolCalls } from "@mcpjam/sdk";

const result = await agent.run("Create a task called 'Deploy' with high priority");

const match = evaluateToolCalls(
  [{ toolName: "createTask", arguments: { priority: "high" } }],
  result.getToolCalls(),
  { argumentMatching: "partial" },
);

if (!match.passed) {
  console.log("Missing:", match.missing);
  console.log("Arg mismatches:", match.argumentMismatches);
}
```

### Argument type placeholders

When `argumentMatching` is `"partial"` (the default), string values in `expected` arguments are interpreted as type checks:

| Placeholder | Matches                      |
| ----------- | ---------------------------- |
| `"any"`     | Any non-undefined value      |
| `"string"`  | `typeof value === "string"`  |
| `"number"`  | `typeof value === "number"`  |
| `"boolean"` | `typeof value === "boolean"` |
| `"object"`  | Non-null, non-array object   |
| `"array"`   | Array                        |
| `"null"`    | `null`                       |

```typescript theme={"theme":"css-variables"}
evaluateToolCalls(
  [{ toolName: "sendEmail", arguments: { to: "string", subject: "any" } }],
  result.getToolCalls(),
);
```

### Layered match options

`resolveMatchOptions` merges suite → case → run-override layers on top of defaults. Import it from `@mcpjam/sdk/matchers` when building custom eval pipelines:

```typescript theme={"theme":"css-variables"}
import { resolveMatchOptions } from "@mcpjam/sdk/matchers";

const resolved = resolveMatchOptions(
  { allowExtraToolCalls: false },   // suite-level
  { argumentMatching: "exact" },    // case-level
  undefined,                        // run-level override
);
// { toolCallOrder: "ignore", allowExtraToolCalls: false, argumentMatching: "exact" }
```

***

## Quick Reference

| Validator                      | Use Case                                   |
| ------------------------------ | ------------------------------------------ |
| `evaluateToolCalls`            | Structured match with detailed diagnostics |
| `matchToolCalls`               | Exact sequence of tools                    |
| `matchToolCallsSubset`         | Required tools (any order)                 |
| `matchAnyToolCall`             | At least one of these tools                |
| `matchToolCallCount`           | Tool called N times                        |
| `matchNoToolCalls`             | No tools called                            |
| `matchToolCallWithArgs`        | Exact argument match                       |
| `matchToolCallWithPartialArgs` | Subset of arguments                        |
| `matchToolArgument`            | Single argument value                      |
| `matchToolArgumentWith`        | Custom validation                          |

***

## Related

* [Testing with LLMs](/sdk/concepts/testing-with-llms) - Conceptual guide
* [PromptResult Reference](/sdk/reference/prompt-result) - Get tool calls
* [EvalTest Reference](/sdk/reference/eval-test) - Use in evaluations
* [Saving Eval Results](/sdk/reference/eval-reporting) - Per-result `matchOptions` on `EvalResultInput`


# Common MCP Errors
Source: https://docs.mcpjam.com/troubleshooting/common-errors

Solutions to common MCP server connection and configuration issues

<Info>
  Looking for a specific error code? See the [error code reference](/troubleshooting/error-codes) — every ErrorCard in the inspector deep-links to a section there.
</Info>

This guide covers common errors you might encounter when working with MCP servers and how to resolve them.

## Connection Issues

### Diagnose a connection failure

Most "failed to connect" / `fetch failed` / `ECONNREFUSED` / `MCP error -32000: Connection closed` / `MCP error -32001: Request timed out` reports come down to one of five root causes. Work through them in order — each step takes seconds.

1. **Is the server actually running?**

   Open a second terminal and run the same command you'd ask Inspector to run. If it exits or prints a stack trace, fix that first — Inspector can only attach to a server that stays alive.

   ```bash theme={"theme":"css-variables"}
   # For STDIO servers — run the same command Inspector would run
   python /Users/yourname/project/server.py
   # or: uv run --directory /Users/yourname/project mcp.py
   # or: npx -y @modelcontextprotocol/server-everything

   # For HTTP servers
   curl -i http://localhost:PORT/mcp
   ```

   A clean stdio process should sit waiting on stdin; a healthy HTTP server should respond with something other than `Connection refused` (the exact code varies — common responses are `405 Method Not Allowed` for `GET`, `406 Not Acceptable` if Accept headers are missing, or `400 Bad Request` if the server expects a session header).

2. **Right port, right path?**

   `ECONNREFUSED 127.0.0.1:5555` means **nothing is listening on that port** — typo, wrong port, or the server is bound to `127.0.0.1` only while you're calling it via a different hostname (or vice versa). `SSE error: Non-200 status code (404)` means the port is correct but the path is wrong: most MCP servers mount under `/mcp` or `/sse`, not `/`.

3. **Right transport selected?**

   In the server card, the transport dropdown must match what your server speaks. STDIO servers won't respond to HTTP attempts and vice versa. If you used a `url:` field in `config.json`, Inspector picks HTTP; if you used `command:` + `args:`, Inspector picks STDIO.

4. **Right auth setting?**

   New servers default to **Auto**, which connects without credentials and escalates to OAuth on 401. For local development servers that have no auth, pick **No Authentication** to skip the escalation prompt. If you get `401`, see [Getting 401 Unauthorized errors](#getting-401-unauthorized-errors) below.

5. **Did the server actually finish booting before the handshake?**

   `MCP error -32001: Request timed out` during `initialize` means the server didn't respond to the handshake in time — usually because it's still loading models, opening a database, or waiting on an upstream. Raise the **Request timeout** in **Clients → your client → Request timeout** (default 30,000ms), or reduce the server's startup work.

If all five check out and you still see a failure, copy the full error from the JSON-RPC log into a [GitHub issue](https://github.com/MCPJam/inspector/issues) or [Discord](https://discord.gg/JEnDtz8X6z) thread.

### `(intermediate value).subtle is undefined`

The browser's Web Crypto API (`crypto.subtle`) is only available in **secure contexts** — that means HTTPS, or `http://localhost` / `http://127.0.0.1`. If you're hitting Inspector over a non-localhost HTTP URL (for example, a LAN IP like `http://192.168.1.10:6274`), the browser disables `crypto.subtle` and OAuth/PKCE flows fail.

**Fix:** open Inspector at `http://localhost:6274` or `http://127.0.0.1:6274`, or put HTTPS in front of it. The hosted app and the desktop app are unaffected.

### Server fails to start or connect

* **Use absolute paths**: Specify the full path to your server
  * ✅ Good: `/Users/yourname/project/server.py`
  * ❌ Bad: `./server.py` or `server.py`

* **For `uv` projects**: Use the `--directory` flag with absolute path

  ```bash theme={"theme":"css-variables"}
  uv run --directory /Users/username/weather-mcp mcp.py
  ```

* **Test your command first**: Run the server command manually in a terminal to verify it works before adding it to Inspector

* **For HTTP servers**: Verify the URL is accessible
  * Test with `curl` or a browser first

### "Invalid host header" when connecting to an HTTP server

This response comes from **your MCP server**, not from MCPJam Inspector. The server's HTTP layer is rejecting the request because the `Host` header isn't on its allowlist. Most common culprits:

* **Next.js / webpack-dev-server** — both block unknown hosts by default in dev mode. Add your host to `allowedHosts` (webpack-dev-server) or `experimental.allowedDevOrigins` (Next.js).
* **FastAPI / Starlette (Python)** — `TrustedHostMiddleware` rejects hosts not in `allowed_hosts`. Add the host the inspector is calling from, or temporarily set `allowed_hosts=["*"]` for local testing.
* **Reverse proxies (nginx, Caddy)** — check the `server_name` / `host_match` config.

**Quick check:** test the same URL with `curl http://localhost:PORT/mcp` — if curl gets the same error, it's definitely server-side, not inspector.

## Authentication Issues

### Server fails to connect with authentication errors

**If your server is failing to connect, first verify you've selected the correct authentication option:**

* **Auto** (default) - Connects without credentials first; if the server returns 401, MCPJam prompts you to continue with OAuth. Use this when you're unsure what the server requires.
* **No Authentication** - Use this for local or development servers that don't require authentication
* **Bearer Token** - Use this if you already have an API token from the server
* **OAuth 2.0** - Use this for servers that require OAuth authorization flow

### "resolves to a private/reserved IP address" during OAuth discovery

The **web app at app.mcpjam.com runs on our servers**, so it can only reach
authorization servers that are reachable from the public internet. A local
authorization server — on `localhost`, on your LAN, or on a hostname like
`auth.local` that resolves to `127.0.0.1` — is not, and discovery is refused
with a message naming the address it resolved to.

**Solution:** use the local inspector for a local OAuth setup:

```bash theme={"theme":"css-variables"}
npx @mcpjam/inspector
```

The local inspector reaches loopback, LAN, CGNAT, and unique-local addresses,
including hostnames that resolve to them, because it runs on your machine. No
flag or configuration is needed. The same applies to the CLI, which reaches
private targets by default and takes `--https-only` when you want to reproduce
the hosted policy.

Link-local and cloud-metadata addresses (`169.254.169.254` and friends) are
refused everywhere, including locally. If you see the message naming a
"link-local or cloud-metadata host", something is pointing discovery at an
instance-metadata endpoint rather than at an authorization server.

### Getting 401 Unauthorized errors

If you're seeing **401 Unauthorized** errors when trying to connect to your server, this typically means the server requires authentication that hasn't been set up yet.

**Solution:**

* **If you're using Auto (the default):** MCPJam will prompt "Continue with OAuth?" when it receives a 401. Accept the prompt to start the OAuth flow.
* **If you're using No Authentication:** Change your authentication type to **Auto** or **OAuth 2.0** in the server configuration and go through the OAuth authorization flow.
* Use the OAuth Debugger to troubleshoot any issues with the OAuth process.

<Info>
  The OAuth Debugger helps you step through each part of the authorization
  process and identify where things might be going wrong.
</Info>

### OAuth connection failures

When an OAuth flow fails, Inspector surfaces the failure directly in the connection UI so you can diagnose the problem without leaving the Servers tab.

**Connection card** — a banner above the error message identifies the exact OAuth step that failed (for example, "OAuth failed during Token Request").

**Server detail panel** — open the server's detail view to see the full **Last OAuth Trace**, which includes:

* Each OAuth step with its status (pending / success / error) and a short message
* The specific error for any failed step
* An **HTTP History** log of every request and response made during the flow, with sensitive values automatically redacted

Use the trace to pinpoint whether the failure occurred during discovery, client registration, PKCE generation, the authorization redirect, or the token exchange, then cross-reference with the OAuth Debugger for a guided walkthrough of the same steps.

### Toggle shows a re-authentication error for an OAuth server

The enable/connect toggle on a server card or detail modal reuses existing credentials only. If your tokens are expired or missing, the toggle will report a re-authentication error rather than opening the OAuth consent screen.

**Solution:** Use the **Reconnect** button on the server card or detail modal to start a full OAuth flow and obtain new credentials. The toggle is intentionally non-interactive for OAuth to avoid unexpected browser redirects.

### Common Configuration Mistakes

* Missing required environment variables
* Incorrect transport type selection
* Typos in server URLs or commands
* Missing file permissions or executable flags

## Chat Model Limit

### "MCPJam model limit reached" warning

MCPJam provides free frontier models for chat. When you hit the daily usage limit, the chat surface shows a warning:

> **MCPJam model limit reached.** Add your own API key under LLM Providers in Settings to continue now, or try again in *N* minutes.

This is an expected quota state, not an error with your MCP server.

**To continue immediately:**

1. Go to **Settings → LLM Providers**
2. Add an API key for any supported provider (OpenAI, Anthropic, Gemini, etc.)
3. Select that provider's model in the chat model picker

**To wait for the limit to reset:** The warning includes the retry window. Free quota resets daily.

## Desktop app startup failures

### App bounces in the dock and disappears (macOS) or closes immediately

If the desktop app fails to start, it now shows a recovery dialog instead of quitting silently. The dialog offers three options:

* **Reset app data and quit** — removes cached data (`Cache`, `Code Cache`, `GPUCache`, `Local Storage`) from the app's user data folder and relaunches. Use this when the app crashes consistently after an update or when local data appears corrupted.
* **Open logs folder** — opens the folder containing `electron-log` output, then quits. Use this to inspect the error before deciding whether to reset.
* **Quit** — exits without making any changes.

### Port 6274 is already in use

The desktop app defaults to port **6274**. If another process holds that port, the app automatically tries the next available port up to port **6283**. A warning is written to the app logs when a fallback port is used. No action is required — the app starts normally on the fallback port.

If you need to free port 6274 manually, find and stop the process occupying it:

```bash theme={"theme":"css-variables"}
# macOS / Linux
lsof -ti :6274 | xargs kill -9

# Windows (PowerShell)
Get-Process -Id (Get-NetTCPConnection -LocalPort 6274).OwningProcess | Stop-Process
```

### App crashes after an automatic update

The desktop app automatically clears stale GPU and HTTP caches (`Cache`, `Code Cache`, `GPUCache`) the first time it launches after a version change. This handles most renderer or GPU process crashes caused by cached data from a previous build. If the app still fails to start after an update, use the **Reset app data and quit** option in the recovery dialog.

## Getting Help

If you're still experiencing issues:

1. **Check server documentation**
   * Review the MCP server's specific documentation
   * Look for known issues or troubleshooting guides

2. **Community support**
   * Join our [Discord](https://discord.gg/JEnDtz8X6z) for community help
   * Check [GitHub Issues](https://github.com/MCPJam/inspector/issues) for similar problems


# Error code reference
Source: https://docs.mcpjam.com/troubleshooting/error-codes

What every MCPJam Inspector error code means, the likely causes, and the next step to fix it.

This page is the source of truth for the error messages MCPJam Inspector surfaces. Every error rendered in the inspector deep-links to one of the sections below.

If you arrived here from an ErrorCard's "Learn more" link, the page should already be scrolled to the right section. Otherwise, use the index below.

## JSON-RPC codes

These come from the MCP transport itself. Numeric codes are defined by the [JSON-RPC 2.0 spec](https://www.jsonrpc.org/specification#error_object) plus a handful of MCP-specific extensions.

| Code     | Anchor                                                                      |
| -------- | --------------------------------------------------------------------------- |
| `-32700` | [Parse error](#parse-error)                                                 |
| `-32600` | [Invalid request](#invalid-request)                                         |
| `-32601` | [Method not found](#method-not-found)                                       |
| `-32602` | [Invalid params](#invalid-params)                                           |
| `-32603` | [Internal error](#internal-error)                                           |
| `-32000` | [Connection closed](#connection-closed)                                     |
| `-32001` | [Request timed out](#request-timeout) / [Header mismatch](#header-mismatch) |
| `-32004` | [Unsupported protocol version](#unsupported-protocol-version)               |
| `-32042` | [URL elicitation required](#url-elicitation-required)                       |

### Parse error

`-32700 — jsonrpc/parse_error`

The MCP wire contained JSON the inspector could not parse as JSON-RPC.

**Likely causes**

* The server emitted invalid JSON on the response channel.
* A proxy or middleware mangled the response body.
* Server emitted log output on stdout instead of stderr (STDIO transport).
* The inspector sent malformed JSON and the server returned a parse error.

**Next steps**

* Check the server's stdout/stderr for unintended log output.
* Use the inspector's Traffic Log to inspect the raw response.

### Invalid request

`-32600 — jsonrpc/invalid_request`

The server rejected the payload as not a well-formed JSON-RPC request.

**Likely causes**

* Client sent a request shape the server's MCP runtime does not accept.
* Outdated server SDK that disagrees with the protocol version advertised.

**Next steps**

* Verify the negotiated MCP protocol version.
* Update the server's MCP SDK.

### Method not found

`-32601 — jsonrpc/method_not_found`

The server does not implement the JSON-RPC method that was called.

**Likely causes**

* The server hasn't implemented the requested MCP method.
* Client and server are on incompatible MCP protocol versions.
* The capability you expected was not advertised in `initialize`.

**Next steps**

* Check the server's advertised capabilities in the connection info modal.
* Confirm the protocol version negotiated at `initialize`.

### Invalid params

`-32602 — jsonrpc/invalid_params`

The server rejected the request parameters.

**Likely causes**

* Required field is missing from the call.
* Field type does not match the tool's input schema.
* Server-side validator is stricter than the published schema.

**Next steps**

* Re-check the tool's input schema in the Tools tab.
* Compare your call payload against the schema in the inspector.

### Internal error

`-32603 — jsonrpc/internal_error`

The server hit an unexpected error while handling the request.

**Likely causes**

* Unhandled exception inside the server's tool/resource/prompt handler.
* Downstream dependency (database, API) failed during the call.

**Next steps**

* Check the server's logs around the time of the error.
* Retry the request once the server is healthy.

### Connection closed

`-32000 — jsonrpc/connection_closed`

The underlying transport closed before the response could be delivered.

**Likely causes**

* STDIO server process exited or crashed mid-request.
* HTTP server dropped the streaming connection.
* Network blip between the inspector and the server.

**Next steps**

* Restart the server and reconnect.
* Check the server logs for a crash or exit message.

**Inspector-specific:** the inspector synthesizes this code when an active transport goes away mid-request. If you see it on every call, the server is probably exiting immediately after startup — check for a fatal log.

### Request timeout

`-32001 — jsonrpc/request_timeout`

The server did not respond within the configured request timeout.

**Likely causes**

* Server is overloaded or stuck on the operation.
* Long-running tool call exceeds the inspector's per-request timeout.
* Network latency between client and server.

**Next steps**

* Increase the per-server request timeout in the Servers tab.
* Use MCP `tasks/*` for operations that legitimately run long.

### Header mismatch

`-32001 — jsonrpc/header_mismatch`

The server rejected the request because an HTTP header disagreed with the JSON-RPC body — for example, `Mcp-Protocol-Version` does not match the version negotiated at `initialize`, or `Mcp-Name` does not match the tool name in the request body.

**Likely causes**

* Server is enforcing a different protocol version than the one negotiated at `initialize`.
* A proxy stripped or rewrote an `Mcp-*` header.
* A routing header (`Mcp-Name`, `Mcp-Method`) was set to a value that disagrees with the request body.

**Next steps**

* Expand the failing JSON-RPC frame in the traffic log. The collapsed **HTTP headers** row beneath the JSON body names the disagreeing header (for example, `mcp-name disagrees with the body`) without requiring you to open a separate view.
* If the inline headers section is absent, open the **HTTP** source filter in the traffic log to find the matching exchange and inspect its headers directly.
* Verify the server's protocol-version pinning in the connection settings.
* If you set an explicit protocol version per server, ensure it matches what the server advertises.

> The `-32001` code is overloaded between `request_timeout` and `header_mismatch`. The inspector disambiguates by message; check the raw message in the ErrorCard's details panel.

### Unsupported protocol version

`-32004 — jsonrpc/unsupported_protocol_version`

The server does not support any protocol version this inspector offered.

**Likely causes**

* Server pinned to a newer MCP draft your inspector build does not understand.
* Server pinned to a legacy version this build dropped support for.

**Next steps**

* Update the inspector to a newer build.
* Check the supported versions list in the server's `initialize` response.

### URL elicitation required

`-32042 — jsonrpc/url_elicitation_required`

The server needs the user to visit an external URL to complete the operation.

**Likely causes**

* Server requested a URL elicitation (OAuth, payment, confirmation).
* Operation cannot proceed until the user opens the URL in a browser.

**Next steps**

* Open the elicited URL and complete the flow.
* Re-issue the request after the external step completes.

***

## Transport errors

These come from the OS or the HTTP stack. Most of them are recoverable.

### ECONNREFUSED

`transport/econnrefused`

Nothing is listening on the host and port the server URL points at.

**Likely causes**

* Server isn't running.
* Port number is wrong in the server URL.
* Server is bound to a different interface (e.g. only `127.0.0.1` but you're connecting via the LAN IP).

**Next steps**

* Start the server.
* Double-check the URL's host and port.
* For Docker/containers, confirm the port is published to your host.

### ECONNRESET

`transport/econnreset`

The remote side closed the TCP connection abruptly.

**Likely causes**

* Server process crashed mid-request.
* Intermediate proxy or load balancer dropped the connection.
* Server hit an OS-level resource limit.

**Next steps**

* Inspect the server logs for a crash.
* Retry the request.

### ETIMEDOUT

`transport/etimedout`

The OS-level TCP connection attempt did not complete in time.

**Likely causes**

* Wrong host/port in the server URL.
* Firewall is silently dropping packets.
* Server is overloaded and never accepted the connection.

**Next steps**

* Verify the URL is reachable from your machine (e.g. `curl`).
* Check firewall / VPN rules.

### ENOTFOUND

`transport/enotfound`

DNS lookup failed for the server's hostname.

**Likely causes**

* Hostname is misspelled in the URL.
* DNS resolver is misconfigured.
* You're offline.

**Next steps**

* Confirm the hostname in the URL is correct.
* Try resolving the host with `nslookup` or `dig`.

### EAI again

`transport/eai_again`

DNS resolution failed with a transient error.

**Likely causes**

* Local DNS resolver is overloaded or restarting.
* Upstream DNS server is briefly unavailable.

**Next steps**

* Wait a few seconds and retry.
* Switch to a different DNS resolver if this persists.

### Undici transport error

`transport/undici`

The underlying HTTP client (undici / fetch) reported a low-level transport failure. Common subcodes: `UND_ERR_SOCKET`, `UND_ERR_CONNECT_TIMEOUT`, `UND_ERR_HEADERS_TIMEOUT`.

**Likely causes**

* Server closed the connection mid-response.
* TLS handshake failed.
* Socket-level error during streaming.

**Next steps**

* Inspect the Traffic Log for the failed request.
* Verify the server's TLS certificate is valid.

### Fetch failed

`transport/fetch_failed`

The HTTP request never produced a response.

**Likely causes**

* Server is unreachable (offline, wrong URL, blocked by firewall).
* TLS handshake failed (self-signed cert, expired cert).
* Mixed-content block (HTTPS page calling HTTP endpoint in browser).

**Next steps**

* Open the URL in a browser to confirm it loads.
* If self-signed, install the certificate or switch to a trusted one.

### Socket hang up

`transport/socket_hang_up`

The server closed the connection without sending a response.

**Likely causes**

* Server crashed or restarted during the request.
* Reverse proxy timed the request out.

**Next steps**

* Retry the request.
* Check server-side logs for the crash.

***

## Auth & OAuth

For the difference between the three "API key" concepts (LLM provider keys, MCPJam API keys, Playground BYOK), see [API keys](/reference/api-keys).

### Unauthorized 401

`auth/http_401`

The server requires authentication that wasn't provided or is no longer valid.

**Likely causes**

* Missing or expired bearer token.
* OAuth access token expired and refresh failed.
* Server changed its required authentication scheme.

**Next steps**

* Re-authenticate using the **Reconnect** button on the server card.
* If using OAuth, run through the OAuth flow again from Servers.

### Forbidden 403

`auth/http_403`

You authenticated successfully but lack permission for the operation.

**Likely causes**

* OAuth scopes granted don't cover the requested operation.
* Server-side ACL blocks this account.

**Next steps**

* Re-run OAuth and request the additional scopes if the server allows.
* Ask the server admin to grant the necessary permissions.

### OAuth refresh failed

`auth/oauth_refresh_failed`

An expired OAuth access token could not be refreshed.

**Likely causes**

* Refresh token was revoked.
* Refresh token expired.
* Server returned `invalid_grant` to the refresh attempt.

**Next steps**

* Click **Reconnect** on the server card to run a fresh OAuth flow.

### Missing bearer

`auth/missing_bearer`

The API call did not include the required `Authorization: Bearer ...` header.

**Likely causes**

* Inspector session expired.
* Sign-in token failed to attach to the request.

**Next steps**

* Refresh the page and sign in again.

### OAuth invalid grant

`oauth/invalid_grant`

The OAuth server rejected the authorization code or refresh token.

**Likely causes**

* Authorization code was already redeemed.
* Refresh token was revoked.
* Authorization code expired (typical lifetime \~60s).

**Next steps**

* Start the OAuth flow again from the server card.

### OAuth invalid client

`oauth/invalid_client`

The OAuth server does not recognize the client credentials.

**Likely causes**

* Client was deleted on the authorization server.
* Dynamic registration cache is stale.
* `client_id` was rotated server-side.

**Next steps**

* Re-register the client (Reconnect from the server card triggers DCR if supported).

### OAuth redirect mismatch

`oauth/redirect_mismatch`

The redirect URI in the request does not match the one registered with the OAuth server.

**Likely causes**

* Authorization server requires the inspector's callback URL to be registered explicitly.
* Server's allow-list is wrong.

**Next steps**

* Add the inspector's callback URL to the OAuth server's allowed redirects.
* Verify the inspector's base URL hasn't changed.

### OAuth well-known unreachable

`oauth/well_known_unreachable`

The OAuth `.well-known` discovery endpoint could not be fetched.

**Likely causes**

* Authorization server is down.
* Wrong issuer URL.
* CORS blocks the discovery request from the browser.

**Next steps**

* Confirm the issuer URL in the server config.
* Open the `.well-known/openid-configuration` (or `oauth-authorization-server`) URL in a browser.

***

## Inspector-specific

These come from the inspector's own SDK runtime.

### Not yet supported in stateless

`sdk/not_yet_supported_in_stateless`

The inspector's stateless HTTP transport does not yet implement this MCP operation.

**Likely causes**

* Operation requires a server-initiated channel (subscriptions, MRTR) the stateless preview transport hasn't wired up yet.

**Next steps**

* Switch the server to the legacy stateful transport in its protocol-mode toggle.

### Stateless requires HTTP

`sdk/stateless_requires_http`

Stateless mode can only be used with an HTTP-transport server, not STDIO.

**Likely causes**

* You enabled the stateless protocol toggle on a stdio server.

**Next steps**

* Disable the stateless toggle for stdio servers.

### Protocol version pin unsupported

`sdk/protocol_version_pin_unsupported`

This connection is pinned to a specific MCP protocol version that the server does not offer.

**Likely causes**

* The client profile you're using pins a protocol version this server hasn't adopted — for example, a client set to **Latest (2026-07-28)** against a server that still speaks a 2025 revision.
* A per-server protocol override is pinned to a version the server dropped or never shipped.

**Next steps**

* Click **Change protocol version** in the error banner or toast to open the client's **MCP Protocol** tab, then set the version to **Automatic** to negotiate whatever the server supports.
* Or pick the version the server advertises — `server/discover` (modern) or the `initialize` response (legacy) lists the supported versions.

### Paginated tool / header discovery unsupported

`sdk/paginated_tool_header_discovery_unsupported`

Paginated tools discovery cannot run alongside per-request header overrides on this transport.

**Likely causes**

* Conflicting combination of progressive tool discovery + per-server header overrides.

**Next steps**

* Disable progressive tool discovery for this server, or move headers into the server config.

***

## Provider errors

Errors from the LLM provider running behind the playground.

### Provider invalid tool name

`provider/invalid_tool_name`

An LLM provider rejected a tool name (Anthropic's strict tool-name validator is the most common source).

**Likely causes**

* Tool name contains characters or length the provider does not allow.
* Two attached servers expose tools whose namespaced names collide after sanitization.

**Next steps**

* Rename the offending tool on the server.
* Detach one of the colliding servers from the chat surface.

### Provider auth error

`provider/auth_error`

Your LLM provider rejected the API key for this request.

**Likely causes**

* Key is missing.
* Key is invalid or revoked.
* Key is for a different environment (project, region).

**Next steps**

* Add or update your API key under **Settings → LLM Providers**.
* Verify the key in the provider's dashboard.

### Provider quota

`provider/quota`

Your LLM provider rejected the request because you hit a rate limit or quota. This is a limit on your own provider key — no MCPJam purchase lifts it.

**Likely causes**

* Per-minute rate limit exceeded.
* Daily or monthly quota exhausted.
* Free tier limits hit.

**Next steps**

* Wait for the limit window to reset, then retry.
* Switch the host to a different model or provider.
* Upgrade your plan on the provider's own dashboard.

> **MCPJam account limit vs. provider throttle** — If the session pane shows a billing or credit message instead of this card, the limit is on your MCPJam account, not your provider key. Adding credits or switching to BYOK resolves that case; the steps above do not apply.

### MCPJam model limit reached

`provider/mcpjam_limit_daily` · `provider/mcpjam_limit_monthly` · `provider/mcpjam_limit`

MCPJam refused the call before it reached a provider, because the account's own
allowance is spent. Which allowance depends on the plan: free accounts share a
daily bucket per organization, Team plans spend a monthly per-seat allowance
that renews with the billing period. The error card names the one you hit.

**Next steps**

* Top up credits or upgrade the plan — the limit dialog offers both.
* Wait for the daily reset, or for the billing period to renew.
* Add your own API key under **Settings → LLM Providers** for chat.

<Warning>
  Bring-your-own-key covers chat. Swarm persona generation and persona-driver
  turns are always billed to MCPJam and have no BYOK path, so adding a key does
  not lift this limit for a swarm.
</Warning>

***

## Environment errors

These codes come from the environment resolution service. They appear in the Playground, the Environment Tools pane, and on scenario shared links when a project environment cannot be resolved to a usable set of servers. The error card offers an action where one exists — for example, **Open Servers** for `ENV_NO_SERVERS` — rather than a generic retry that would fail identically.

When a visitor opens a scenario shared link and the scenario's environment is archived, they see **"This link has been archived"** instead of a generic failure message. For other unresolvable environment states they see **"This link isn't available right now"**. Signing in cannot change either state — the environment itself must be restored or reconfigured by the scenario owner.

### No servers

`ENV_NO_SERVERS`

This environment has no servers.

**Likely causes**

* The client this environment points at has no servers connected.
* The attached server group is empty.
* Every server contributed by a pinned plugin has been removed.

**Next steps**

* Connect a server to the client, or attach a server group.
* Check the environment's pinned plugins if it relied on one for servers.

### Archived

`ENV_ARCHIVED`

This environment is archived.

**Likely causes**

* Someone archived it after this was configured.

**Next steps**

* Restore it from the Environments list, or pick a different environment.

### Client missing

`ENV_HOST_MISSING`

This environment's client no longer exists.

**Likely causes**

* The client was deleted after the environment was created.

**Next steps**

* Point the environment at a different client, or recreate it.

### Server group missing

`ENV_ATTACHMENT_MISSING`

This environment's server group is gone.

**Likely causes**

* The attached server group was deleted.

**Next steps**

* Attach a different server group, or clear the attachment.

***

## Unknown error

`internal/unknown`

An error occurred that the inspector could not classify.

**Likely causes**

* Unhandled error path.
* New error class the inspector hasn't been taught about yet.

**Next steps**

* Open the details panel of the ErrorCard and copy the raw message.
* File an issue with the raw message so we can add it to the catalog.