> ## Documentation Index > Fetch the complete documentation index at: https://docs.mcpjam.com/llms.txt > Use this file to discover all available pages before exploring further. # Create several eval cases > Adds up to 100 cases to the suite in one call — the bulk form of the single create, and the way to import a suite or convert a repo's test files without a round trip per case. Every entry needs a `title` and a non-empty `steps` array. Cases are validated together and reported individually: a `failed` entry does not roll back its committed siblings, so the response is `201` even when some entries were refused — read `failed` rather than branching on the status alone. Send an `Idempotency-Key` header to make a retry land on the same rows. ## OpenAPI ````yaml /reference/openapi.json post /projects/{projectId}/eval-suites/{suiteId}/cases/batch openapi: 3.1.0 info: title: MCPJam API version: 1.0.0-preview description: >- Programmatic access to MCP servers saved in your MCPJam projects — live diagnostics (validate, inspect, export) and operations: call tools, render prompts, run eval suites asynchronously and poll their results, and import OAuth tokens. **The API is in preview**: the surface may change while we finish the design. Error `code` values are stable; error `message` strings are not. Write clients that ignore unknown response fields. contact: name: MCPJam url: https://github.com/MCPJam/inspector/issues servers: - url: https://app.mcpjam.com/api/v1 description: Hosted MCPJam security: - bearerAuth: [] tags: - name: Hosts description: >- Project hosts: named model + capability profiles you run chats and eval suites against. - name: Environments description: >- Project environments: named, live-editable execution bundles (one host, an optional standalone server group, optionally pinned skills and plugin versions) that eval suites and journeys run against. Distinct from Sandbox images, which are Computer base images. Reads require project membership; every write requires project admin. - name: Plugins description: >- Agent Plugins imported into a project — read-only inventory and version detail. - name: Sandbox images description: >- Custom Computer images: a digest-pinned Dockerfile built into an immutable image your project's computers boot from. - name: Server diagnostics description: Connect-level health checks against a saved MCP server. - name: Primitives description: 'The server''s MCP primitives: tools, prompts, and resources.' - name: Export description: Full-server snapshots for diffing and CI. - name: Execution description: 'Run the server''s primitives: call tools, render prompts.' - name: Eval runs description: >- Asynchronous eval suite runs: create with 202, poll status, iterations, and traces. - name: Server connections description: >- Connect an MCP server URL to a project, authorizing in a browser when the server requires it. - name: OAuth description: 'Bring-your-own OAuth: import externally obtained tokens for a server.' - name: Scenarios description: >- Read-only access to the scenarios published from a project: listing, settings, attached servers, and share links. - name: Catalog description: >- Discover the resources the other routes operate on: your account, projects, servers, eval suites, and chat sessions. - name: Tunnels description: >- Relay tunnels that expose local MCP servers through a public URL, registered as first-class project servers (the `mcpjam tunnel` CLI flow). - name: Agent description: >- Headless agent turns over the public API: send a message history, the server runs one assistant turn with project-scoped workspace tools (eval reads + suite creation) on a pinned hosted model, and returns the reply plus created-resource references. - name: Swarms description: >- Personas, journeys and swarm containers — the authoring half of Swarms — plus the model-backed generation that drafts them. - name: Swarm runs description: >- Launching journeys and reading what they produced. Launching SPENDS — see the per-operation notes. - name: Swarm insights description: >- What a swarm run revealed. The scorecard and findings are deterministic and free; requesting wave insights runs models and draws on your shared daily ledger. - name: User testing description: >- Publishing an environment for real visitors, and controlling who can reach it. Several of these NARROW access and take effect immediately. paths: /projects/{projectId}/eval-suites/{suiteId}/cases/batch: post: tags: - Eval runs summary: Create several eval cases description: >- Adds up to 100 cases to the suite in one call — the bulk form of the single create, and the way to import a suite or convert a repo's test files without a round trip per case. Every entry needs a `title` and a non-empty `steps` array. Cases are validated together and reported individually: a `failed` entry does not roll back its committed siblings, so the response is `201` even when some entries were refused — read `failed` rather than branching on the status alone. Send an `Idempotency-Key` header to make a retry land on the same rows. operationId: createEvalCases parameters: - $ref: '#/components/parameters/projectId' - $ref: '#/components/parameters/suiteId' - name: Idempotency-Key in: header required: false description: >- Makes the write retry-safe. A repeat of the same request under the same key lands on the rows the first attempt authored instead of creating a second copy; those entries come back with `replayed: true`. schema: type: string maxLength: 256 requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalCasesBatchCreateRequest' example: cases: - id: c_lists_files title: lists files steps: - id: s1 kind: prompt prompt: List the files in the project root. - id: s2 kind: assert assertion: type: toolCalledWith toolName: list_files args: args: {} - title: reads a file steps: - id: s1 kind: prompt prompt: Read README.md. - id: s2 kind: assert assertion: type: toolCalledWith toolName: read_file args: args: {} responses: '201': description: Per-case outcomes. Some entries may have failed; see `failed`. content: application/json: schema: $ref: '#/components/schemas/EvalCasesBatchResult' '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '502': $ref: '#/components/responses/ServerUnreachable' components: parameters: projectId: name: projectId in: path required: true description: ID of the hosted project that contains the server. schema: type: string suiteId: name: suiteId in: path required: true description: Eval suite ID, as returned by `POST /eval-runs`. schema: type: string schemas: EvalCasesBatchCreateRequest: type: object required: - cases description: >- Author several cases in one call. Each entry takes the same fields as a single create. The 100-case cap is deliberately smaller than a suite file's 500-case limit, so a maximal file uploads in several calls. properties: cases: type: array minItems: 1 maxItems: 100 items: $ref: '#/components/schemas/EvalCaseCreateRequest' duplicatePolicy: type: string enum: - block - warn - create_anyway default: block description: >- What to do with a case whose definition already matches one in the suite. An unrecognized value coerces to `block` and reports the coercion in the response's `duplicatePolicy` rather than failing the call. overrideReason: type: string minLength: 1 description: >- Required by `warn` and `create_anyway`. Recorded on the case's revision. EvalCasesBatchResult: type: object required: - created - failed - duplicatePolicy description: >- A partial outcome by design: the cases in `created` were written even when `failed` is non-empty. Both arrays carry the `index` of the request entry they describe. properties: created: type: array items: type: object required: - index - id - title - replayed properties: index: type: integer description: Position in the request's `cases` array. id: type: string description: Platform case id — the path parameter for the per-case routes. declaredId: type: string description: >- The effective declared id. On a replay this is the stored case's, not the one just sent. title: type: string replayed: type: boolean description: >- True when an idempotent retry landed on an already-authored case; nothing new was written. warnings: type: array items: $ref: '#/components/schemas/EvalCaseWarning' failed: type: array items: type: object required: - index - code - message properties: index: type: integer title: type: string declaredId: type: string code: type: string description: Stable machine-readable reason. enum: - INVALID_ITEM - INVALID_CASE_ID - DUPLICATE_CASE_ID - DUPLICATE_IDEMPOTENCY_KEY - IDEMPOTENCY_CONFLICT - DUPLICATE_CONTENT - OVERRIDE_REASON_REQUIRED - INVALID_CASE - UNSTORABLE_ITEM - ITEM_TOO_LARGE message: type: string duplicatePolicy: type: object required: - effectivePolicy - coerced description: What policy actually applied, so a coercion is never silent. properties: requestedPolicy: type: string effectivePolicy: type: string coerced: type: boolean warnings: type: array items: $ref: '#/components/schemas/EvalCaseWarning' EvalCaseCreateRequest: type: object required: - title - steps description: >- Create one case. `title` and a non-empty `steps` array are both required — a case persisted without executable steps could never run. properties: title: type: string minLength: 1 id: type: string pattern: ^[A-Za-z0-9_-]{1,128}$ description: >- Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the `id` on the response, which is the case's platform row id. steps: type: array minItems: 1 items: $ref: '#/components/schemas/EvalTestStep' description: REPLACES the case's test definition wholesale when provided. expectedOutput: type: string iterations: type: integer minimum: 1 maximum: 10 isNegative: type: boolean scenario: type: string models: type: array items: type: object required: - model properties: model: type: string provider: type: string description: Derived from a `provider/model` id when omitted. matchOptions: type: - object - 'null' description: '`null` clears the case override and inherits the suite''s.' checks: type: - object - 'null' description: '`null` clears the case override.' properties: mode: type: string enum: - inherit - replace - extend list: type: array items: type: object EvalCaseWarning: type: object required: - code - message description: >- Names something that would otherwise have to be inferred from silence. A warning never changes what was written. properties: code: type: string message: type: string Error: type: object required: - code - message properties: code: type: string description: >- Stable, machine-readable error code. New codes may be added over time; treat unknown codes as non-retryable failures unless the HTTP status says otherwise. enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - CONFLICT - VALIDATION_ERROR - RATE_LIMITED - FEATURE_NOT_SUPPORTED - SERVER_UNREACHABLE - TIMEOUT - OAUTH_REQUIRED - INTERNAL_ERROR message: type: string description: >- Human-readable description. May change between releases — don't match on it. details: type: object description: Optional, unstructured context bag. additionalProperties: true EvalTestStep: type: object description: >- One authored test step (the unified test model). `kind` discriminates: `prompt` is a user message (model turn); `toolCall` is a deterministic, model-free tool call; `interact` is one pure widget action; `assert` is an assertion (a `Predicate` like `toolCalledWith` / `widgetRendered`, or a DOM `WidgetAssertion`). required: - id - kind properties: id: type: string minLength: 1 kind: type: string enum: - prompt - toolCall - interact - assert prompt: type: string description: 'User message (`kind: prompt`).' serverName: type: string description: 'Server that owns the tool (`kind: toolCall`).' toolName: type: string description: 'Tool name (`kind: toolCall` / `interact`).' arguments: type: object description: 'Tool-call arguments (`kind: toolCall`).' additionalProperties: true action: type: object description: 'Widget action (`kind: interact`).' additionalProperties: true assertion: type: object description: 'Predicate or widget assertion (`kind: assert`).' additionalProperties: true additionalProperties: true responses: ValidationError: description: Malformed body or parameters. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: VALIDATION_ERROR message: Invalid JSON body Unauthorized: description: >- Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. content: application/json: schema: $ref: '#/components/schemas/Error' examples: badKey: summary: Invalid or revoked key value: code: UNAUTHORIZED message: Invalid API key oauthRequired: summary: Target server needs an OAuth grant value: code: OAUTH_REQUIRED message: Server requires OAuth authorization Forbidden: description: Key is valid but not allowed to do this. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: FORBIDDEN message: You do not have access to this project NotFound: description: Unknown project, server, or resource. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: NOT_FOUND message: Server not found RateLimited: description: >- Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. headers: Retry-After: description: Seconds to wait before retrying. schema: type: integer content: application/json: schema: $ref: '#/components/schemas/Error' example: code: RATE_LIMITED message: API key rate limit exceeded. Slow down and retry. InternalError: description: Something failed on MCPJam's side. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: INTERNAL_ERROR message: Unexpected internal error ServerUnreachable: description: Could not connect to the target MCP server. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: SERVER_UNREACHABLE message: Failed to connect to server securitySchemes: bearerAuth: type: http scheme: bearer description: >- MCPJam API key (`sk_…`). Create one at [Settings → API keys](https://app.mcpjam.com/settings/api-keys). Guest sessions cannot use the API, and API keys cannot manage other API keys. ````