GitLab CI

Run Autonomy from GitLab CI with a preview URL, an iOS Simulator app, or an Android APK and gate the pipeline on the verdict.

Trigger Autonomy after a deployment or mobile build, then keep the GitLab job open until every selected test case has a valid passing verdict. Copy the shared script and the configuration for the platform you ship.

Workflow placement

Put Autonomy after the deployment job that discovers the final preview URL, or after the job that builds the mobile artifact. The web example below accepts an already deployed URL and checks that it responds. It does not deploy your application.

Each YAML example is a complete alternative .gitlab-ci.yml. When adding one to an existing pipeline, retain your build and deployment jobs, merge its stages, and point needs at the job that produces the URL or artifact. Run only trusted code on a protected default branch.

Required secrets

In Settings → CI/CD → Variables, add these as Variable values, not File values, with environment scope *. Protect the variables and disable variable-reference expansion. Mark the API key Masked and hidden. Protect the default branch before running the examples.

AUTONOMY_API_KEY: an organization API key beginning with aut_, created in Autonomy Settings → API Keys. It must belong to the organization containing the plan. Organization keys currently include all API scopes.
AUTONOMY_API_URL: copy CONVEX_SITE_URL or NEXT_PUBLIC_CONVEX_SITE_URL from the deployment matching your API key. Use its exact Convex HTTP URL, including any region in the hostname, without /api or a trailing path. Do not use the dashboard URL or .convex.cloud. A dev key requires its matching dev deployment URL.
AUTONOMY_TEST_PLAN_ID: the ID of a real plan from Test Plans, containing at least one web test case. Copy the ID from the plan's URL. A Test Cases ID is not interchangeable with a plan ID.
PREVIEW_URL: the ready HTTPS preview URL for the web example. It must be reachable from the Autonomy runner, not localhost or a service available only inside the CI job.
For iOS, add AUTONOMY_IOS_TEST_PLAN_ID and AUTONOMY_IOS_BUNDLE_ID. For Android, add AUTONOMY_ANDROID_TEST_PLAN_ID and AUTONOMY_ANDROID_PACKAGE_NAME. Each plan must contain cases for that platform; identifiers must match the built app.
Have an Autonomy runner available for the requested platform and enough run credits. For these explicit standalone targets, use cases that do not reference runtime values from a saved Environment.

The examples set AUTONOMY_REQUEST_ID from CI_JOB_ID, so a retried GitLab job gets a fresh request. Branch and commit metadata come from GitLab's predefined variables. See Test Plans for arranging the selected cases.

Web preview URL

Create .ci/ in the repository root, save the following script as .ci/autonomy.sh, and commit it with your pipeline file. Every recipe on this page uses this exact script. It needs Bash, jq, and curl 7.76 or later; the container jobs install them.

.ci/autonomy.sh

#!/usr/bin/env bash
set -euo pipefail
set +x
: "${AUTONOMY_API_URL:?Set the API base URL, without /api}"
: "${AUTONOMY_API_KEY:?Set an aut_ organization API key}"
: "${AUTONOMY_TEST_PLAN_ID:?Set the Test Plans ID for this platform}"
: "${AUTONOMY_REQUEST_ID:?Set a unique CI job/attempt ID}"
platform="${1:-web}"
api="${AUTONOMY_API_URL%/}"
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

post() {
  curl --fail-with-body --silent --show-error \
    --connect-timeout 15 --max-time 90 \
    -X POST "$api$1" \
    -H "Authorization: Bearer $AUTONOMY_API_KEY" \
    -H 'Content-Type: application/json' --data-binary "$2"
}

case "$platform" in
  web)
    : "${PREVIEW_URL:?Set the reachable preview URL after deployment}"
    targets=$(jq -n --arg url "$PREVIEW_URL" '{web:{baseUrl:$url}}')
    ;;
  ios|android)
    : "${ARTIFACT_PATH:?Set the path to the app archive or APK}"
    : "${APP_IDENTIFIER:?Set the bundle ID or Android package name}"
    test -s "$ARTIFACT_PATH"
    file=$(basename "$ARTIFACT_PATH")
    bytes=$(wc -c < "$ARTIFACT_PATH" | tr -d '[:space:]')
    content_type=application/zip
    if [ "$platform" = android ]; then
      content_type=application/vnd.android.package-archive
    fi
    upload=$(post /api/artifacts/upload-url "$(jq -n \
      --arg platform "$platform" --arg fileName "$file" \
      --arg contentType "$content_type" --argjson declaredBytes "$bytes" \
      '{platform:$platform,fileName:$fileName,contentType:$contentType,declaredBytes:$declaredBytes}')")
    url=$(jq -er '.uploadUrl | strings | select(length > 0)' <<< "$upload")
    storage=$(jq -er '.storageId | strings | select(length > 0)' <<< "$upload")
    intent=$(jq -er '.intentId | strings | select(length > 0)' <<< "$upload")
    jq -e '.method == "PUT"' <<< "$upload" > /dev/null
    curl --fail-with-body --silent --show-error \
      --connect-timeout 15 --max-time 600 -X PUT "$url" \
      -H "Content-Type: $content_type" --data-binary "@$ARTIFACT_PATH"
    scan=$(post /api/artifacts/scan "$(jq -n \
      --arg storageId "$storage" --arg intentId "$intent" \
      '{storageId:$storageId,intentId:$intentId}')")
    if ! jq -e '.status == "clean"' <<< "$scan" > /dev/null; then
      echo 'Artifact scan did not pass; refusing to trigger.' >&2
      exit 1
    fi
    targets=$(jq -n --arg platform "$platform" --arg storageId "$storage" \
      --arg fileName "$file" --arg identifier "$APP_IDENTIFIER" \
      '{($platform):({storageId:$storageId,sourceMode:"upload",fileName:$fileName} +
        (if $platform == "ios" then {bundleId:$identifier} else {packageName:$identifier} end))}')
    ;;
  *) echo 'Usage: bash .ci/autonomy.sh web|ios|android' >&2; exit 2 ;;
esac

body=$(jq -n --arg testPlanId "$AUTONOMY_TEST_PLAN_ID" \
  --arg platform "$platform" --argjson targets "$targets" \
  --arg branch "${AUTONOMY_BRANCH:-manual}" \
  --arg commitSha "${AUTONOMY_COMMIT_SHA:-}" \
  --arg key "$AUTONOMY_REQUEST_ID:$platform" \
  '{testPlanId:$testPlanId,platforms:[$platform],targets:$targets,
    branch:$branch,deployment:{branch:$branch,commitSha:$commitSha},idempotencyKey:$key}')
post /api/v1/run.trigger "$body" > "$work/trigger.json"
jq -e '.runIds | type == "array" and length > 0' "$work/trigger.json" > /dev/null
jq -er '.runIds[] | strings | select(length > 0)' "$work/trigger.json" > "$work/run-ids"
printf 'Queued Autonomy runs:\n'
cat "$work/run-ids"

deadline=$((SECONDS + ${AUTONOMY_WAIT_SECONDS:-1200}))
while IFS= read -r run_id; do
  while :; do
    if (( SECONDS >= deadline )); then
      echo "Timed out waiting for $run_id; failing the CI job." >&2
      exit 1
    fi
    result=$(post /api/v1/run.get "$(jq -n --arg runId "$run_id" '{runId:$runId}')")
    status=$(jq -er '.status' <<< "$result")
    case "$status" in
      queued|running|retrying) sleep 10 ;;
      passing|failed|unverified|invalid|canceled)
        printf '%s: %s\n' "$run_id" "$status"
        if ! jq -e '.verdict == "passed" and .validity == "valid"' <<< "$result" > /dev/null; then
          echo "Autonomy gate failed for $run_id." >&2
          exit 1
        fi
        break ;;
      *) echo "Unexpected run status: $status" >&2; exit 1 ;;
    esac
  done
done < "$work/run-ids"
echo 'Every Autonomy run passed.'

Save the following as .gitlab-ci.yml. Enable GitLab-hosted Linux runners for your project; the web jobs and iOS upload job explicitly select saas-linux-small-amd64. On GitLab Self-Managed, replace that tag in every Linux container job with the tag of your own Linux Docker executor runner. Start with Build → Pipelines → New pipeline on the protected default branch. The log prints Queued Autonomy runs: and the run IDs after the API accepts the request.

.gitlab-ci.yml

stages: [prepare, qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

default:
  tags: [saas-linux-small-amd64]
  image: debian:stable-slim
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends bash curl jq ca-certificates

preview_ready:
  stage: prepare
  script:
    - |
      : "${PREVIEW_URL:?Set the deployed preview URL in CI/CD Variables}"
      curl --fail --silent --show-error --location \
        --connect-timeout 15 --max-time 60 "$PREVIEW_URL" > /dev/null
      printf 'PREVIEW_URL=%s\n' "$PREVIEW_URL" > preview.env
  artifacts:
    reports:
      dotenv: preview.env
    expire_in: 1 day

autonomy_web:
  stage: qa
  timeout: 30m
  needs:
    - job: preview_ready
      artifacts: true
  script:
    - |
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh web

For a URL created inside your existing deployment job, move the printf and artifacts:reports:dotenv entries into that job, then change needs to its name. Remove the fixed PREVIEW_URL project variable so it does not override the generated dotenv value. Keep credentials out of preview.env; GitLab stores it as a downloadable artifact.

Mobile artifact handoff

Use the same .ci/autonomy.sh. The helper requests a signed upload URL, uploads the file, requires a clean scan, and supplies the resulting storage ID to the run. Select only the platform whose build is present in your repository.

iOS

Use a dedicated, protected, self-hosted macOS runner with the macos tag, the Bash shell executor, Xcode and the iOS Simulator SDK installed. Disable Run untagged jobs for this runner and do not give it the Linux runner tag. GitLab's hosted macOS runners require Premium or Ultimate, apart from eligible open-source programs. A standard Free project needs its own Mac. Linux cannot perform this Xcode build.

This native example assumes ios/MyApp.xcodeproj, a shared MyApp scheme, and a MyApp.app product. Replace all three with your project's names, and perform any project-specific dependency installation before xcodebuild. For a workspace project, use -workspace ios/MyApp.xcworkspace in place of -project. It builds a Simulator .app, not a device .ipa.

The macOS job archives the app with executable permissions and symlinks intact before GitLab transports the ZIP. needs:artifacts downloads that ZIP in the Linux job, which handles uploading and waiting.

.gitlab-ci.yml

stages: [build, qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

build_ios:
  stage: build
  tags: [macos]
  script:
    - |
      set -euo pipefail
      xcodebuild \
        -project ios/MyApp.xcodeproj \
        -scheme MyApp \
        -configuration Release \
        -sdk iphonesimulator \
        -destination 'generic/platform=iOS Simulator' \
        -derivedDataPath ios/build \
        CODE_SIGNING_ALLOWED=NO \
        build
      mkdir -p .build
      (cd ios/build/Build/Products/Release-iphonesimulator && \
        zip -qry -y "$CI_PROJECT_DIR/.build/MyApp.app.zip" MyApp.app)
  artifacts:
    paths:
      - .build/MyApp.app.zip
    expire_in: 1 day

autonomy_ios:
  stage: qa
  tags: [saas-linux-small-amd64]
  image: debian:stable-slim
  timeout: 30m
  needs:
    - job: build_ios
      artifacts: true
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends bash curl jq ca-certificates
  script:
    - |
      export AUTONOMY_TEST_PLAN_ID="$AUTONOMY_IOS_TEST_PLAN_ID"
      export APP_IDENTIFIER="$AUTONOMY_IOS_BUNDLE_ID"
      export ARTIFACT_PATH=".build/MyApp.app.zip"
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh ios

Android

Register a dedicated, protected Linux runner with the android-linux tag and Bash shell executor. Disable Run untagged jobs for this runner and do not give it the Linux container runner tag. Install the JDK required by your project, the Android command-line tools, your compile SDK and build-tools versions, Bash, jq, curl 7.76 or later, and trusted CA certificates on that machine. Set ANDROID_HOME, accept the SDK licenses, and commit an executable android/gradlew with its wrapper files. The shell executor uses the host's tools; it does not install an Android SDK for you.

This example builds the app module's debug APK and uploads it in the same Linux job. Adjust the Gradle task and ARTIFACT_PATH together if your project uses flavors or another module.

.gitlab-ci.yml

stages: [qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

autonomy_android:
  stage: qa
  tags: [android-linux]
  timeout: 60m
  script:
    - |
      set -euo pipefail
      (cd android && ./gradlew --no-daemon assembleDebug)
      export AUTONOMY_TEST_PLAN_ID="$AUTONOMY_ANDROID_TEST_PLAN_ID"
      export APP_IDENTIFIER="$AUTONOMY_ANDROID_PACKAGE_NAME"
      export ARTIFACT_PATH="android/app/build/outputs/apk/debug/app-debug.apk"
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh android

Gate the pipeline on the verdict

The helper checks every ID returned by run.trigger through run.get. It keeps waiting while a run is queued, running, or retrying, and succeeds only when every terminal result has verdict: "passed" and validity: "valid". A queued run alone does not pass the pipeline. Failed, invalid, unverified, canceled, unexpected, or timed-out results fail the job.

The default wait budget is 1,200 seconds across all returned runs. Set AUTONOMY_WAIT_SECONDS to change it, and keep the GitLab job and runner timeouts longer than that budget plus build and upload time. Keep allow_failure disabled. Canceling or timing out the GitLab job stops polling; it does not cancel an already queued Autonomy run.

Branch and event strategy

These configurations allow pushes, schedules and manually started pipelines only on the protected default branch. Merge request pipelines and fork code do not receive the key. Keep short smoke plans on ordinary deployments; use a schedule or manual pipeline with a broader plan for release checks.

To check a merge request preview, run the trusted default-branch configuration manually with the reviewed preview URL. Do not enable protected-variable access for untrusted merge request code. If you widen the branch rules, protect those branches and review their pipeline changes first.

Vendor references

GitLab CI/CD YAML reference: stages, workflow rules, runner tags, timeouts and needs.
CI/CD variables: variable values, masking and protected branches.
Job artifacts and dotenv reports: passing a ZIP or URL between jobs.
Predefined variables: job IDs, branch names and commit SHAs.
Hosted Linux runners: the saas-linux-small-amd64 tag and container execution.
Hosted macOS runners and shell executor: runner availability and host prerequisites.

On this page