CI integration

StackNext regression checks fit into a normal CI job without a hosted StackNext service. The CI runner is the machine: the collector binds to loopback, the Playwright browser runs there, and run, baseline and comparison files stay in the workspace. The example uses GitHub Actions, but the same steps work in any runner that can run Node.js, Playwright and the app.

The default is report-only. A job fails only when stacknext compare is given a matching --fail-on option.

What the job needs

Node.js 20.9 or newer.
pnpm or npm with the repository’s lockfile.
The app built and able to start in the runner.
Playwright browsers installed (pnpm exec playwright install --with-deps chromium).
One Playwright worker.
A baseline whose recorded environment matches the candidate runner.
At least three verified samples per scenario.

Do not run captured Playwright tests in parallel. Correlation is verified per injected trace with one worker; a second worker can make the sample set incomplete.

GitHub Actions example

This example assumes the app directory is apps/my-app, the Playwright config starts the app through webServer, and a baseline file is available from a previous baseline job or artifact. Adjust paths and package manager commands.

name: runtime-regression

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: runtime-regression-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  regression:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    env:
      CI: 'true'
      NEXT_TELEMETRY_DISABLED: '1'
    defaults:
      run:
        working-directory: apps/my-app
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v4
        with:
          version: 12.16.1

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm

      - name: Install dependencies
        working-directory: .
        run: pnpm install --frozen-lockfile

      - name: Install Playwright Chromium
        run: pnpm exec playwright install --with-deps chromium

      - name: Build
        working-directory: .
        run: pnpm build

      # Restore a baseline captured on the same OS/Node/Next/Playwright setup.
      - name: Restore baseline
        run: |
          mkdir -p .stacknext/baselines
          cp ci-baselines/base-0123456789abcdef.json .stacknext/baselines/

      - name: Capture candidate
        run: |
          set -o pipefail
          pnpm exec stacknext capture \
            --name "ci candidate" \
            --quiet --json \
            -- pnpm exec playwright test --workers=1 --repeat-each=3 \
            > candidate.json

      - name: Compare with the baseline
        run: |
          CANDIDATE_ID=$(jq -r .run.id candidate.json)
          pnpm exec stacknext compare \
            base-0123456789abcdef "$CANDIDATE_ID" \
            --fail-on regression \
            --json > comparison.json

      - name: Upload regression records
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: stacknext-regression
          path: |
            apps/my-app/candidate.json
            apps/my-app/comparison.json
            apps/my-app/.stacknext/runs/
            apps/my-app/.stacknext/baselines/
            apps/my-app/.stacknext/comparisons/
          if-no-files-found: ignore

--fail-on regression makes the compare step exit 1 only for a regressed comparison. Use --fail-on inconclusive to fail when evidence is insufficient, or --fail-on any to fail on either. The capture step exits non-zero when the child test command fails, and exits 3 when the child succeeds but telemetry is incomplete. Do not ignore that exit code.

Where the baseline comes from

The baseline must be produced by an earlier capture and saved with stacknext baseline save. Pick one approach:

Same-runner baseline job

Run both the baseline and candidate captures in one workflow on the same runner type. This is the most reliable option because Node.js, Next.js, Playwright, OS and database versions are identical.

Check out the known-good revision in a separate directory or worktree.
Install, build and start that revision.
Capture it and save a baseline.
Check out the candidate revision in the primary workspace.
Capture the candidate and compare.

Keep ports, app names and collector processes separate between the two revisions. Do not compare runs from two matrix entries.

Stored baseline artifact

Save a baseline from the main branch once, upload the JSON as an artifact, and download it into later PR jobs. Downloading from a different workflow run needs the source run-id and a token with actions: read:

permissions:
  contents: read
  actions: read

# In the baseline job:
      - name: Upload baseline artifact
        uses: actions/upload-artifact@v4
        with:
          name: stacknext-baseline
          path: apps/my-app/.stacknext/baselines/base-*.json
          if-no-files-found: error

# In a later candidate job:
      - name: Download baseline artifact
        uses: actions/download-artifact@v4
        with:
          name: stacknext-baseline
          path: apps/my-app/.stacknext/baselines
          run-id: ${{ vars.STACKNEXT_BASELINE_RUN_ID }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

Re-capture the baseline whenever Node.js, Next.js, Playwright, the database setup or the instrumentation flags change. A stale artifact is not a stable reference.

Reviewed baseline file

A team can keep a reviewed baseline JSON in a directory outside .stacknext, for example ci-baselines/, and copy it into the workspace at the start of the job. Review it before committing; it contains redacted route, span, query, error and environment data. Never commit .stacknext runtime state or tokens.

Capture wiring

App started by Playwright `webServer`

stacknext capture starts an ephemeral collector and passes the run id, collector URL and token to the child. Playwright’s webServer inherits the environment and starts the app:

export default defineConfig({
  testDir: './e2e',
  fullyParallel: false,
  workers: 1,
  retries: 0,
  webServer: {
    command: 'pnpm dev',
    url: 'http://127.0.0.1:4402',
    reuseExistingServer: !process.env.CI,
  },
  use: {
    baseURL: process.env.STACKNEXT_APP_URL ?? 'http://127.0.0.1:4402',
    trace: 'off',
    screenshot: 'off',
    video: 'off',
  },
})

Then:

pnpm exec stacknext capture \
  --name "ci candidate" --quiet --json \
  -- pnpm exec playwright test --workers=1 --repeat-each=3 \
  > candidate.json

If the runner is slow to start the dev server, raise the fixture timeout in the child environment:

STACKNEXT_CAPTURE_TIMEOUT_MS=30000 \
STACKNEXT_CAPTURE_SETTLE_MS=1000 \
pnpm exec stacknext capture --name "ci candidate" --quiet --json \
  -- pnpm exec playwright test --workers=1 --repeat-each=3 \
  > candidate.json

App started by `stacknext dev`

If the workflow starts the app separately, run stacknext dev in the app directory and wait for both the collector and the app before capturing. Capture attaches to the live collector when .stacknext/run.json is valid.

pnpm exec stacknext dev --quiet > stacknext-dev.log 2>&1 &
echo $! > stacknext-dev.pid

for attempt in $(seq 1 60); do
  curl -fsS http://127.0.0.1:4400/api/ping > /dev/null && break
  sleep 1
done

# Warm route modules so one-time spans do not appear only on one side.
for path in / /projects/1 /api/n-plus-one; do
  curl -fsS -o /dev/null "http://127.0.0.1:4402$path"
done

Always stop the process with SIGINT or SIGTERM in a cleanup step so it closes the collector and removes .stacknext/run.json:

      - name: Stop StackNext
        if: always()
        run: |
          if [ -f stacknext-dev.pid ]; then
            kill -TERM "$(cat stacknext-dev.pid)" || true
          fi

Check ports 4400-4425 before the capture step if the runner reuses processes.

Thresholds and environment

Keep the defaults unless you have measured your own noise. The comparison requires three verified samples per side by default; duration regressions must cross both 20% and 25 ms. See comparison methodology for the exact ordering. Pin the runner image and versions. The comparison records Node.js, Next.js, platform, architecture, runtime, enabled adapters, instrumentation flags, SDK version, Playwright version and project name, and defaults to inconclusive when they differ. Do not build a baseline on macOS and compare it with a Linux candidate unless you explicitly accept --ignore-environment and its warning.

Artifacts and privacy

Upload the capture JSON, .stacknext/runs/run-*.json, .stacknext/baselines/base-*.json, .stacknext/comparisons/cmp-*.json and the Playwright report if your project produces one. The records contain redacted scenario titles and file paths, normalized routes and SQL shapes, bounded span/error evidence, timings and environment facts. They do not contain request/response bodies, headers, cookies, URL query values, SQL bind parameters, screenshots, video or Playwright trace archives. They are still local debugging artifacts; review them before sharing artifacts outside the team. Never upload .stacknext/run.json or any file containing the collector bearer token.

Concurrency and cleanup

Use one capture per collector.
Use one Playwright worker for captured runs.
Give parallel CI jobs separate ports or separate ephemeral collectors.
Always stop stacknext dev with a signal and check for orphaned next dev processes.
Set a job timeout.
Keep CI=true so Playwright does not reuse a developer’s server.

What CI does not do

It does not upload telemetry to StackNext or any hosted service.
It does not run an LLM or ask an AI provider to classify a regression.
It does not take screenshots, record video or read Playwright trace archives.
It does not prove statistical significance or causality.
It does not make an inconclusive comparison a pass.
It does not automatically delete old baselines or comparisons.

Overview

Getting started

Guides

Integrations

AI agents

Reference

Troubleshooting

Security and privacy

What the job needs

GitHub Actions example

Where the baseline comes from

Same-runner baseline job

Stored baseline artifact

Reviewed baseline file

Capture wiring

App started by Playwright `webServer`

App started by `stacknext dev`

Thresholds and environment

Artifacts and privacy

Concurrency and cleanup

What CI does not do

​What the job needs

​GitHub Actions example

​Where the baseline comes from

​Same-runner baseline job

​Stored baseline artifact

​Reviewed baseline file

​Capture wiring

​App started by Playwright webServer

​App started by stacknext dev

​Thresholds and environment

​Artifacts and privacy

​Concurrency and cleanup

​What CI does not do

​Related

What the job needs

GitHub Actions example

Where the baseline comes from

Same-runner baseline job

Stored baseline artifact

Reviewed baseline file

Capture wiring

App started by Playwright `webServer`

App started by `stacknext dev`

Thresholds and environment

Artifacts and privacy

Concurrency and cleanup

What CI does not do

Related