GitHub Checks is configured in the hosted app, at Settings → Integrations →
GitHub. Connecting a repository needs an organization admin.The pull-request comment described below is rolling out: it is enabled per
deployment by an operator, and until it is on for yours a connected repository
gets the check run and no comment. The per-repository toggle is visible before
then and records your choice; nothing is posted until the rollout reaches you.
mcpjam.yaml
MCPJam can usually work out how to build and start a server on its own. When it
cannot — or when you would rather say it explicitly than let detection guess —
commit an mcpjam.yaml at the root of the repository.
A committed file is authoritative: MCPJam runs what it says and never falls back
to detection. A file that is present but invalid fails the check rather than
being ignored, because ignoring it would run a server the author never declared.
mcpjam.yaml
Fields
Unknown keys inside
checks: are rejected — inside the authoritative
section a stray key is a typo that would otherwise silently change what runs.
Unknown keys at the top level are ignored, so the file can grow new sections
without breaking older checks. The file itself is capped at 32 KB.
env
checks.env is a flat mapping of at most 20 entries. Names must be uppercase
environment-variable names (^[A-Z][A-Z0-9_]*$, up to 64 characters) and values
must be strings of up to 1024 characters — quote anything that looks like a
number, a boolean, or an empty value, or the file is rejected rather than
coerced.
Binding and ports
The port is what has to match; the bind address is not. MCPJam reaches your server through the sandbox’s own host bridge, from inside the box, so a server bound to loopback is reachable. What fails is listening on a port other than the onechecks.port declares, or a process that exits after startup — both are
reported as an unhealthy server, and the check body says so.
If your framework needs a HOST (or equivalent) variable set to bind the way you
want, set it in checks.env like any other boot flag. Only add it if your
framework actually needs it — it is not something MCPJam requires.
The pull-request comment
Alongside the check run, MCPJam posts one comment per pull request and updates that same comment in place on every later push. It is posted for every conclusion, including a pass — so the comment is always current, and there is never a thread of stale verdicts to scroll past. The comment carries:- The verdict line — what the check concluded on this head commit.
- What the result is not. A check that could not produce a verdict says so in the comment rather than leaving you to infer it from a colour.
- “What to fix” — one line per failing case, saying what to go and change. Each line names the case and the one thing to look at; it does not claim to know why the case failed.
- A collapsed JSON block, described below, for coding agents.
The machine-readable JSON block
The last section of the comment is a collapsed, fenced JSON block. It exists so a coding agent working on the pull request can read the result without parsing prose, and it is stable enough to be worth parsing. It is a bounded projection of MCPJam’s eval decision contract — not the whole contract, and not a dump of the run. It carries:
Two things to know when reading it:
- It is a projection, and it says so. Anything the projection drops is named
in
omittedFields, so a consumer can tell “MCPJam did not include this” from “MCPJam measured nothing here”. Do not infer an absent field’s value. - The first failed stage is a location, not a diagnosis. It says where the chain stopped — connection, discovery, selection, the tool call, the response, or the user’s request being satisfied — and nothing about why. Treating it as a cause is how a change lands in the wrong file.
Review comments on removed tool names
When a failing case asserts a tool that was never called, MCPJam looks for that tool name in the pull request’s own diff. If the diff removes a line containing the name in quotes, and the name does not come back on any added line, MCPJam leaves a single review comment on that removed line. That comment is a location, not a cause. It says: the case expected this tool, this line is where the name stopped existing, start here. It does not claim the removal is a mistake — renaming or deleting a tool on purpose is an ordinary thing for a pull request to do, and when it is, the fix is to update the case in MCPJam rather than to put the line back. MCPJam never requests changes. Review comments are posted as plain comments, so they never sit as a blocking review on your pull request.What appears in a comment, and what never does
Comments and review comments are public on a public repository. What MCPJam writes there is deliberately bounded. What can appear:- fixed MCPJam copy — the same sentences on every repository;
- the head SHA, the check-run id, and the pull-request number;
- links back into MCPJam;
- eval case titles, as you wrote them;
- tool names;
- stage reasons and counts;
- the same clamped build and server output the check run already shows.
- organization, project, or installation identifiers;
- model or provider names;
- prompts;
- tool-call arguments;
- credentials — evidence strings are credential-redacted before they are rendered.