Run mcpjam in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.
GitHub Actions
Authentication
There are three ways to authenticate in CI, depending on your server setup.
Option 1: Headless OAuth login
Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.
Secrets needed:
Option 2: Refresh token
Best when you already have a refresh token from a previous oauth login. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.
Secrets needed:
To get a refresh token, run mcpjam oauth login locally with --format json and grab .credentials.refreshToken from the output.
Option 3: Static API key
Best when your server uses a non-expiring API key instead of OAuth.
Secrets needed:
Option 4: No auth
Some servers don’t require authentication at all.
Secrets needed:
Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).
Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.
See OAuth Conformance for details on the config file format.
Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.
Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.
Single-run protocol conformance, oauth conformance, and apps conformance also accept --reporter junit-xml when you only need one target/check selection instead of a suite config file.
GitLab CI
The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and .gitlab-ci.yml syntax.
Authentication
Headless OAuth login
Refresh token
Static API key
Snapshot your tool surface before and after a deploy to catch breaking changes.
See OAuth Conformance for details on the config file format.
Evals in CI
There are two ways to wire MCPJam evals into a pipeline: trigger a hosted eval run with the CLI, or run evals locally with the SDK and upload the results. Both authenticate with an MCPJam API key (sk_…) from Settings → API keys.
Trigger a hosted eval suite
mcpjam cloud eval run starts an asynchronous run of a suite that lives in your MCPJam project. Without --wait, it prints a launch receipt and returns immediately. In CI, add --wait and --out to write a structured JSON report after every launched run reaches a terminal state.
Secrets needed:
In human format, eval run prints a View: line after the payload so you can open the run directly from the terminal:
This line is only emitted in human format — --format json output is unchanged, so scripts that parse the JSON stream are unaffected.
Use --wait-timeout <ms> to replace the 10-minute default. --out defaults to the structured JSON format; add --reporter junit-xml to write JUnit XML instead. When --reporter is present, the same report is also written to stdout.
eval run --wait exits 0 even when the evals failed. Waiting for a run is not the same as judging it: --wait keeps eval run’s existing exit codes, where 1 means a run never launched, not that a run came back red. The report it writes records the verdict faithfully (passed: false, and a JUnit failures count above zero), but the process still exits 0.So the eval gate step above is what fails the job — do not drop it and rely on the first step’s exit code. If you only want the artifact and not the gate, assert on the verdict yourself: [ "$(jq -r '.passed' eval-report.json)" = "true" ].
eval gate sets a verdict-based exit code, and writes its report before doing so: 0 passed, 1 an eval verdict failed, 2 usage error, 3 incomplete or non-gateable. Infrastructure conditions never map to 1, so retrying on 3 is safe. eval status also prints a View: line in human format, identical to the one eval run prints.
eval status runs in --format human and the run has finished with a failed result, it prints a decision summary block to stdout after the status payload. The block lists the overall pass rate and, for each failed case, the first stage that failed, the failure category, any recorded evidence (span IDs, prompt indexes, predicate reasons), and a suggested next action:
The block is only emitted in --format human — --format json output is byte-identical to before. If the iteration data cannot be fetched (for example, a network error), the block is silently omitted rather than failing the status read.
eval gate and eval compare also emit a decision summary in --format human, written to stderr alongside their existing gate report.
Hosted runs execute LLM iterations on the platform and consume your organization’s credits or configured provider keys. See the cloud eval command reference for the full surface, including cloud eval judge (request LLM-as-judge grading on a finished run), cloud eval validate (offline suite-file validation), cloud eval export (write a hosted suite to a local file), cloud eval checks list/connect (GitHub Checks integration), and more.
Upload SDK eval results
If you instead run evals inside your own CI job with @mcpjam/sdk (EvalTest / EvalSuite), set MCPJAM_API_KEY and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):
See Save Results to MCPJam for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON). 
