HostRunner class runs prompts with MCP tools enabled. It handles the multi-step prompt loop and returns rich result objects.
Import
Constructor
Parameters
HostRunnerOptions
required
Configuration for the test agent.
HostRunnerOptions
Example
MCP Apps, traces, and
mcpClientManager: MCP App widgets are backed by HTML served from an MCP resource (ui.resourceUri), not by the tool’s JSON result alone. After each tool call, HostRunner uses the manager’s readResource to fetch that HTML and attach widgetSnapshots to the PromptResult. When you upload results to MCPJam, those snapshots are stored so the Evals trace viewer can replay the widget offline. If you omit mcpClientManager, prompts still run and tools still execute — but widgetSnapshots stay empty and uploaded traces will only contain messages and spans (no embedded app replay). Passing manager is also what enables replay credential capture for authenticated HTTP servers when reporting to MCPJam.Methods
run()
Sends a prompt to the LLM and returns the result.getWidgetSnapshots() on the returned result (or toEvalResult() when reporting) to access MCP App HTML captured during the run when mcpClientManager was set.
Parameters
PromptOptions
Returns
Promise<PromptResult> - The result object with response and metadata.
Example
run() never throws exceptions. Errors are captured in the PromptResult.
Check result.hasError() to detect failures.stopAfterToolCall is intended for evals that only care about tool selection
and arguments. If the model emits multiple tool calls in the same step,
non-target sibling tools may still execute before the loop stops.timeout bounds prompt runtime. The runtime creates an internal abort signal,
so tools can stop early if their implementation respects the provided
abortSignal. If a tool ignores that signal, its underlying work may continue
briefly after the prompt returns an error result.Model String Format
Models are specified asprovider/model:
Custom Providers
Add custom OpenAI or Anthropic-compatible endpoints.CustomProvider Type
Example
Configuration Properties
tools
The MCP tools available to the agent. Obtained fromMCPClientManager.getTools().
model
The LLM model identifier. Format:provider/model-id.
apiKey
The API key for the LLM provider.systemPrompt
Optional system prompt to guide the LLM’s behavior. An empty string ("") is treated the same as omitting the field — both fall back to "You are a helpful assistant.". This matters for saved MCPJam clients, which store a missing prompt as "".
temperature
Controls response randomness. Range: 0.0 (deterministic) to 1.0 (creative).reasoningEffort
Reasoning effort for the model. The runner maps it through the same table the hosted runner uses (reasoningEffortProviderOptions) into the provider’s own
providerOptions — OpenAI reasoningEffort, Anthropic effort (with adaptive
thinking on models that need it), Google thinkingConfig.thinkingLevel — and
also works for mcpjam/… models, which speak Anthropic’s and OpenAI’s own wire
formats.
- Refused, never dropped. A model or provider with no effort control the
installed AI SDK provider exposes (for example
openai/gpt-4o) makes the constructor throwcapability_missing, before any request is made.withOptionscarries the effort with the runner and re-checks it against a newmodel. - Replaces the temperature. While an effort is set the runner does not send a temperature, and the recorded context reports what was sent.
- Saved clients.
runWithClientapplies the effort on the saved client’s hosted model selection (settings.reasoningEffort) — and its savedsettings.temperature— instead of discarding them with the selection. A host snapshot carries no model selection, so passreasoningEffortyourself when you build a runner from aHost. - Which levels a model accepts:
supportedReasoningEffortson the model catalog (GET /v1/models) for hosted models, andsupportedReasoningEfforts()from@mcpjam/sdk/host-configfor provider models.
maxSteps
Maximum iterations of the agentic loop (prompt → tool → result → continue).injectOpenAiCompat
Controls whether the OpenAI Apps SDKwindow.openai shim is injected into captured widget HTML snapshots. Defaults to false.
Set this to true when your widget is written against the OpenAI Apps SDK (i.e. it targets ChatGPT or Copilot). Leave it false for SEP-1865-native widgets targeting Claude, Cursor, Codex, or other hosts that don’t expose window.openai — snapshots should match what the live host would have produced.
This flag is stamped onto each widget snapshot (
injectedOpenAiCompat) so the
MCPJam trace viewer can replay the widget faithfully under a different host
config without rewriting the cached bytes.Control Multi-Step Loops with stopWhen
UsestopWhen to control whether the agent starts another step after the current step completes.
stopWhen does not skip tool execution. It controls whether the prompt loop
continues after the current step completes, and HostRunner also applies
stepCountIs(maxSteps) as a safety guard.Bound Prompt Runtime with timeout
Usetimeout when you want to bound how long HostRunner.run() can run:
chunkMs is accepted for parity, but it is mainly useful for streaming APIs.
For HostRunner.run(), number, totalMs, and stepMs are the main
settings to focus on.Complete Example
Related
- Testing with LLMs - Conceptual guide
- PromptResult Reference - Result object API
- LLM Providers Reference - All providers

