Run mcpjam in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.
GitHub Actions
Authentication
There are three ways to authenticate in CI, depending on your server setup.
Option 1: Headless OAuth login
Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.
Secrets needed:
Option 2: Refresh token
Best when you already have a refresh token from a previous oauth login. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.
Secrets needed:
To get a refresh token, run mcpjam oauth login locally with --format json and grab .credentials.refreshToken from the output.
Option 3: Static API key
Best when your server uses a non-expiring API key instead of OAuth.
Secrets needed:
Option 4: No auth
Some servers don’t require authentication at all.
Secrets needed:
Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).
Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.
See OAuth Conformance for details on the config file format.
Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.
Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.
Single-run protocol conformance, oauth conformance, and apps conformance also accept --reporter junit-xml when you only need one target/check selection instead of a suite config file.
GitLab CI
The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and .gitlab-ci.yml syntax.
Authentication
Headless OAuth login
Refresh token
Static API key
Snapshot your tool surface before and after a deploy to catch breaking changes.
See OAuth Conformance for details on the config file format.
Evals in CI
There are two ways to wire MCPJam evals into a pipeline: trigger a hosted eval run with the CLI, or run evals locally with the SDK and upload the results. Both authenticate with an MCPJam API key (sk_…) from Settings → API keys.
Trigger a hosted eval suite
mcpjam cloud eval run starts an asynchronous run of a suite that lives in your MCPJam project. Without --wait, it prints a launch receipt and returns immediately. In CI, add --wait and --out to write a structured JSON report after every launched run reaches a terminal state.
Secrets needed:
In human format, eval run prints a View: line after the payload so you can open the run directly from the terminal:
This line is only emitted in human format — --format json output is unchanged, so scripts that parse the JSON stream are unaffected.
Use --wait-timeout <ms> to replace the 10-minute default. --out defaults to the structured JSON format; add --reporter junit-xml to write JUnit XML instead, or --reporter html for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When --reporter is present, the same report is also written to stdout.
eval run --wait sets a verdict-based exit code. 0 pass, 1 a completed run’s verdict failed (the ONLY condition that produces 1), 2 usage error or an invalid suite file, 3 auth failed (no credential, or the platform rejected it, at launch or mid-wait), 4 a connection/setup failure this CLI itself observed before evaluation ran (or a local --out write failure), 5 no valid verdict — inconclusive, a null/unrecognized result, a run status of failed/cancelled/timed-out, or a wait that hit its deadline. A multi-target launch merges these worst-of across every waited run, in the order 1 > 3 > 4 > 5 > 0.Retry guidance: 4 and 5 mean infrastructure, or an absence of observation — nothing here says the server is wrong, so retrying the CI job is reasonable. But a bare re-run is not automatically safe: eval run --suite only dedupes against an in-flight or already-completed launch when you pass a stable --idempotency-key, and exit 5 can mean the run is still running (a wait that hit its deadline) — without that key, a retry can start a second paid run alongside the first rather than resuming it. Pass --idempotency-key (or poll/resume the run id already in the receipt) before retrying on 4 or 5. 3 means fix the credential first; it poisons every other observation in the same launch. Never retry blindly on 1 — that code is reserved for a run the platform actually graded as failed.Either eval run --wait or eval gate already fails the job on its own — the eval gate step above adds a configurable pass/fail policy (thresholds, per-scorer gates) and baseline comparison on top of the same verdict, so keep it when you want more than “did this run’s own verdict pass”.
eval gate sets a verdict-based exit code, and writes its report before doing so: 0 passed or waived, 1 an eval verdict failed, 2 usage error, 3 incomplete or non-gateable. Infrastructure conditions never map to 1, so retrying on 3 is safe. This is a different, four-code contract from eval run --wait above — gate’s 3 means “incomplete”, not the six-code scheme’s 3 (“auth failed”), and the two are deliberately not unified (see the CLI reference for why). eval status also prints a View: line in human format, identical to the one eval run prints.Waiving a gate
A run whose gate failed can be overridden by an authorized user until an expiry
they name, so a release is not blocked while a known regression is being fixed:
eval gate then exits 0 and reports the outcome as waived. It is not
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a <skipped> element, so it neither fails the build nor renders as
a clean green row).
Only a real verdict failure is waivable. A cancelled run, a --wait timeout, or
a network failure still exits 3 with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.
Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform’s answer. mcpjam cloud eval gate unwaive --run "$RUN_ID" ends one early.
The waiver reason is stored unredacted and readable by anyone who can see
the suite, for as long as the suite exists. Never put secrets, tokens, or
customer data in it.
Decision summary
eval run --wait, eval status, eval gate and eval compare all read one versioned object — the run decision summary — and every output format restates it. In --format human it is printed as a block; in --format json and in --out/--reporter artifacts it rides on the report as decisionSummary, verbatim.
Four things about it are worth knowing before you script against it.
The counts carry the population they count. measurementUnit is caseVariant under verdict policy v2 — one case under one provider/model, with repetitions as trials inside it — and trial on a legacy percent-threshold run. A 3-case suite with 5 repetitions is legitimately “3” under one unit and “15” under the other, so a count quoted without its unit is not a fact.
The summary explains the verdict; it never re-decides it. Under policy v2 the run’s own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on decision. The per-trial diagnostics sit underneath that: a case can pass with a failing trial in it, so tallying the diagnostics gives a different answer than the platform reached.
notEstablished is not a failure. It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. undecided.reason says which. It is also not inconclusive, which is a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.
A page of diagnostics says whether it is the whole story. diagnostics.complete is true only when the listed trials are the run’s entire non-passing set, and scannedIterations says how many were examined — so an empty list from a complete page (“nothing failed”) is distinguishable from an empty list from a partial one (“we did not look”).
Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage’s row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.
eval status prints the block only when a terminal run did not pass, and eval gate and eval compare write theirs to stderr alongside the gate report. --format json is unaffected in every case: it stays exactly one parseable document. If the summary cannot be fetched, the block is omitted rather than failing the command.
Hosted runs execute LLM iterations on the platform and consume your organization’s credits or configured provider keys. See the cloud eval command reference for the full surface, including cloud eval judge (request LLM-as-judge grading on a finished run), cloud eval validate (offline suite-file validation), cloud eval export (write a hosted suite to a local file), cloud eval checks list/connect (GitHub Checks integration), and more.
Upload SDK eval results
If you instead run evals inside your own CI job with @mcpjam/sdk (EvalTest / EvalSuite), set MCPJAM_API_KEY and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):
See Save Results to MCPJam for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON). 
