HostRunner class runs prompts with MCP tools enabled. It handles the multi-step prompt loop and returns rich result objects.
Import
Constructor
Parameters
Configuration for the test agent.
HostRunnerOptions
Example
MCP Apps, traces, and
mcpClientManager: MCP App widgets are backed by HTML served from an MCP resource (ui.resourceUri), not by the tool’s JSON result alone. After each tool call, HostRunner uses the manager’s readResource to fetch that HTML and attach widgetSnapshots to the PromptResult. When you upload results to MCPJam, those snapshots are stored so the Evals trace viewer can replay the widget offline. If you omit mcpClientManager, prompts still run and tools still execute — but widgetSnapshots stay empty and uploaded traces will only contain messages and spans (no embedded app replay). Passing manager is also what enables replay credential capture for authenticated HTTP servers when reporting to MCPJam.Methods
run()
Sends a prompt to the LLM and returns the result.getWidgetSnapshots() on the returned result (or toEvalResult() when reporting) to access MCP App HTML captured during the run when mcpClientManager was set.
Parameters
PromptOptions
Returns
Promise<PromptResult> - The result object with response and metadata.
Example
run() never throws exceptions. Errors are captured in the PromptResult.
Check result.hasError() to detect failures.stopAfterToolCall is intended for evals that only care about tool selection
and arguments. If the model emits multiple tool calls in the same step,
non-target sibling tools may still execute before the loop stops.timeout bounds prompt runtime. The runtime creates an internal abort signal,
so tools can stop early if their implementation respects the provided
abortSignal. If a tool ignores that signal, its underlying work may continue
briefly after the prompt returns an error result.Model String Format
Models are specified asprovider/model:
Custom Providers
Add custom OpenAI or Anthropic-compatible endpoints.CustomProvider Type
Example
Configuration Properties
tools
The MCP tools available to the agent. Obtained fromMCPClientManager.getTools().
model
The LLM model identifier. Format:provider/model-id.
apiKey
The API key for the LLM provider.systemPrompt
Optional system prompt to guide the LLM’s behavior.temperature
Controls response randomness. Range: 0.0 (deterministic) to 1.0 (creative).maxSteps
Maximum iterations of the agentic loop (prompt → tool → result → continue).injectOpenAiCompat
Controls whether the OpenAI Apps SDKwindow.openai shim is injected into captured widget HTML snapshots. Defaults to false.
Set this to true when your widget is written against the OpenAI Apps SDK (i.e. it targets ChatGPT or Copilot). Leave it false for SEP-1865-native widgets targeting Claude, Cursor, Codex, or other hosts that don’t expose window.openai — snapshots should match what the live host would have produced.
This flag is stamped onto each widget snapshot (
injectedOpenAiCompat) so the
MCPJam trace viewer can replay the widget faithfully under a different host
config without rewriting the cached bytes.Control Multi-Step Loops with stopWhen
UsestopWhen to control whether the agent starts another step after the current step completes.
stopWhen does not skip tool execution. It controls whether the prompt loop
continues after the current step completes, and HostRunner also applies
stepCountIs(maxSteps) as a safety guard.Bound Prompt Runtime with timeout
Usetimeout when you want to bound how long HostRunner.run() can run:
chunkMs is accepted for parity, but it is mainly useful for streaming APIs.
For HostRunner.run(), number, totalMs, and stepMs are the main
settings to focus on.Complete Example
Related
- Testing with LLMs - Conceptual guide
- PromptResult Reference - Result object API
- LLM Providers Reference - All providers

