> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-4053-1786992934526.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CI / CD

> Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run `mcpjam` in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

## GitHub Actions

### Authentication

There are three ways to authenticate in CI, depending on your server setup.

#### Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json
```

#### Option 2: Refresh token

Best when you already have a refresh token from a previous `oauth login`. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

**Secrets needed:**

| Secret              | Description                                         |
| ------------------- | --------------------------------------------------- |
| `MCP_SERVER_URL`    | Your MCP server URL                                 |
| `MCP_REFRESH_TOKEN` | OAuth refresh token from a previous login           |
| `MCP_CLIENT_ID`     | OAuth client ID (required with refresh tokens)      |
| `MCP_CLIENT_SECRET` | OAuth client secret (if the client is confidential) |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json
```

<Tip>
  To get a refresh token, run `mcpjam oauth login` locally with `--format json` and grab `.credentials.refreshToken` from the output.
</Tip>

#### Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |
| `MCP_API_KEY`    | Static API key      |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json
```

#### Option 4: No auth

Some servers don't require authentication at all.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

```yaml theme={"theme":"css-variables"}
      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)
```

### OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

```yaml theme={"theme":"css-variables"}
      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

### Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

```yaml theme={"theme":"css-variables"}
      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml
```

### MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

```yaml theme={"theme":"css-variables"}
      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml
```

Single-run `protocol conformance`, `oauth conformance`, and `apps conformance` also accept `--reporter junit-xml` when you only need one target/check selection instead of a suite config file.

***

## GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and `.gitlab-ci.yml` syntax.

### Authentication

#### Headless OAuth login

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Refresh token

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Static API key

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

```yaml theme={"theme":"css-variables"}
mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json
```

### OAuth conformance suite

```yaml theme={"theme":"css-variables"}
mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

***

## Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a **hosted eval run** with the CLI, or run evals **locally with the SDK** and upload the results. Both authenticate with an MCPJam API key (`sk_…`) from **Settings → API keys**.

### Trigger a hosted eval suite

`mcpjam cloud eval run` starts an asynchronous run of a suite that lives in your MCPJam project. Without `--wait`, it prints a launch receipt and returns immediately. In CI, add `--wait` and `--out` to write a structured JSON report after every launched run reaches a terminal state.

**Secrets needed:**

| Secret           | Description             |
| ---------------- | ----------------------- |
| `MCPJAM_API_KEY` | MCPJam API key (`sk_…`) |

```yaml theme={"theme":"css-variables"}
      - name: Run hosted eval
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest cloud eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --wait \
            --out eval-report.json \
            --format json > eval-result.json
          echo "Completed run $(jq -r '.runs[0].id' eval-result.json)"

      - name: Gate and write JUnit
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runs[0].id' eval-result.json)
          npx -y @mcpjam/cli@latest cloud eval gate \
            --run "$RUN_ID" \
            --project "My project" \
            --wait \
            --min-pass-rate-percent 100 \
            --reporter junit-xml \
            --out eval-report.xml

      - name: Upload eval reports
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: hosted-eval
          path: |
            eval-report.json
            eval-report.xml
```

In human format, `eval run` prints a `View:` line after the payload so you can open the run directly from the terminal:

```text theme={"theme":"css-variables"}
View: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>
```

This line is only emitted in human format — `--format json` output is unchanged, so scripts that parse the JSON stream are unaffected.

Use `--wait-timeout <ms>` to replace the 10-minute default. `--out` defaults to the structured JSON format; add `--reporter junit-xml` to write JUnit XML instead. When `--reporter` is present, the same report is also written to stdout.

<Warning>
  **`eval run --wait` exits 0 even when the evals failed.** Waiting for a run is not the same as judging it: `--wait` keeps `eval run`'s existing exit codes, where `1` means a run never launched, not that a run came back red. The report it writes records the verdict faithfully (`passed: false`, and a JUnit `failures` count above zero), but the process still exits 0.

  So the `eval gate` step above is what fails the job — do not drop it and rely on the first step's exit code. If you only want the artifact and not the gate, assert on the verdict yourself: `[ "$(jq -r '.passed' eval-report.json)" = "true" ]`.
</Warning>

<Note>
  `eval gate` sets a verdict-based exit code, and writes its report before doing so: `0` passed, `1` an eval verdict failed, `2` usage error, `3` incomplete or non-gateable. Infrastructure conditions never map to `1`, so retrying on `3` is safe. `eval status` also prints a `View:` line in human format, identical to the one `eval run` prints.
</Note>

#### Decision summary in human format

When `eval status` runs in `--format human` and the run has finished with a failed result, it prints a decision summary block to stdout after the status payload. The block lists the overall pass rate and, for each failed case, the first stage that failed, the failure category, any recorded evidence (span IDs, prompt indexes, predicate reasons), and a suggested next action:

```text theme={"theme":"css-variables"}
Decision summary: failed — 0/3 cases passed (0%)
  Fetch order (iteration-1, iteration 1)
    first failed stage call
    failure category arguments
    expected tool calls: fetch_order
    observed failure: server rejected arguments
    evidence: span ids span-abc; reasons wrong argument type
    next action: review the authored arguments against the tool input schema
  Setup failure (iteration-2, iteration 2)
    no first failed stage — did not reach the server's stages
    failure category setup
    next action: check the server connection and environment configuration
```

The block is only emitted in `--format human` — `--format json` output is byte-identical to before. If the iteration data cannot be fetched (for example, a network error), the block is silently omitted rather than failing the status read.

`eval gate` and `eval compare` also emit a decision summary in `--format human`, written to stderr alongside their existing gate report.

Hosted runs execute LLM iterations on the platform and consume your organization's credits or configured provider keys. See the [`cloud eval` command reference](/cli/reference#cloud-eval-commands) for the full surface, including `cloud eval judge` (request LLM-as-judge grading on a finished run), `cloud eval validate` (offline suite-file validation), `cloud eval export` (write a hosted suite to a local file), `cloud eval checks list/connect` (GitHub Checks integration), and more.

### Upload SDK eval results

If you instead run evals inside your own CI job with [`@mcpjam/sdk`](/sdk/concepts/running-evals) (`EvalTest` / `EvalSuite`), set `MCPJAM_API_KEY` and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

```yaml theme={"theme":"css-variables"}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/
```

See [Save Results to MCPJam](/sdk/concepts/saving-results) for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).