GitLab CI

Start Autonomy runs from GitLab CI for a preview URL or an Android APK, and fail the pipeline unless every run passes.

Start Autonomy after a deployment or mobile build, then keep the GitLab job open until every run has a result. The job passes only when every run is Passed. Copy the shared script and the configuration for the platform you ship.

Workflow placement

Put Autonomy after the deployment job that finds the final preview URL, or after the job that builds the mobile app. The web example below takes an already deployed URL and checks that it responds. It doesn't deploy your app.

Each YAML example is a complete alternative .gitlab-ci.yml. When adding one to an existing pipeline, keep your build and deployment jobs, merge its stages, and point needs at the job that produces the URL or the build. Run only trusted code on a protected default branch.

Required secrets

In Settings → CI/CD → Variables, add these as Variable values, not File values, with environment scope *. Protect the variables and turn off variable-reference expansion. Mark the API key Masked and hidden. Protect the default branch before running the examples.

AUTONOMY_API_KEY: an organization API key starting with aut_. Owners and admins create keys in Organization → API keys. A key can reach every project in the organization.
AUTONOMY_TEST_PLAN_ID: the test plan to run. Open it in the project's Test plans: the ID is the last part of the page URL, /p/<project>/plans/<id>. The plan needs at least one test case for the platform. To run one test case instead, set AUTONOMY_TEST_CASE_ID (from /p/<project>/tests/<id>) and leave the plan ID unset. Set exactly one of the two.
PREVIEW_URL: the ready HTTPS preview URL for the web example. Autonomy must reach it from the internet: not localhost, and not a service that only exists inside the CI job.
For iOS, add AUTONOMY_IOS_TEST_PLAN_ID and AUTONOMY_IOS_BUNDLE_ID. For Android, add AUTONOMY_ANDROID_TEST_PLAN_ID and AUTONOMY_ANDROID_PACKAGE_NAME. Each plan needs test cases for that platform, and the identifiers must match the built app.
Optional: AUTONOMY_ENVIRONMENT_SLUG runs against a saved environment of the same project, with its targets and environment values, instead of this job's URL or build. The helper then skips the upload. AUTONOMY_WAIT_SECONDS changes the total wait time.

Without an environment, a run has no environment values: a step that uses {{env.NAME}} ends as Couldn't start. Set AUTONOMY_ENVIRONMENT_SLUG, or use test cases that don't need environment values.

The examples set AUTONOMY_REQUEST_ID from CI_JOB_ID, so a retried GitLab job gets a new request. Branch and commit come from GitLab's predefined variables. See Test plans for grouping test cases.

Web preview URL

Create .ci/ in the repository root, save this script as .ci/autonomy.sh, and commit it with your pipeline file. Every recipe on this page uses this same script. It needs Bash, jq and curl 7.76 or later; the container jobs install them.

.ci/autonomy.sh

#!/usr/bin/env bash
# Usage: bash .ci/autonomy.sh web|ios|android
# Starts Autonomy runs for this CI job and exits 0 only if every run passed.
set -euo pipefail
set +x
: "${AUTONOMY_API_KEY:?Set an aut_ organization API key}"
: "${AUTONOMY_REQUEST_ID:?Set an ID that is unique for this CI job attempt}"
platform="${1:?Usage: bash .ci/autonomy.sh web|ios|android}"
api="${AUTONOMY_API_URL:-https://api.autonomyqa.com}"
api="${api%/}"

post() {
  curl --fail-with-body --silent --show-error \
    --connect-timeout 15 --max-time 90 \
    -X POST "$api$1" \
    -H "Authorization: Bearer $AUTONOMY_API_KEY" \
    -H 'Content-Type: application/json' --data-binary "$2"
}

# 1. What to run: a test plan or one test case.
if [ -n "${AUTONOMY_TEST_PLAN_ID:-}" ] && [ -z "${AUTONOMY_TEST_CASE_ID:-}" ]; then
  what=$(jq -n --arg id "$AUTONOMY_TEST_PLAN_ID" '{testPlanId:$id}')
elif [ -n "${AUTONOMY_TEST_CASE_ID:-}" ] && [ -z "${AUTONOMY_TEST_PLAN_ID:-}" ]; then
  what=$(jq -n --arg id "$AUTONOMY_TEST_CASE_ID" '{testCaseId:$id}')
else
  echo 'Set exactly one of AUTONOMY_TEST_PLAN_ID or AUTONOMY_TEST_CASE_ID.' >&2
  exit 2
fi

# 2. Where to run: a saved environment, or this job's preview URL or build.
if [ -n "${AUTONOMY_ENVIRONMENT_SLUG:-}" ]; then
  where=$(jq -n --arg slug "$AUTONOMY_ENVIRONMENT_SLUG" '{environmentSlug:$slug}')
else
  case "$platform" in
    web)
      : "${PREVIEW_URL:?Set PREVIEW_URL, or AUTONOMY_ENVIRONMENT_SLUG}"
      where=$(jq -n --arg url "$PREVIEW_URL" '{targets:{web:{baseUrl:$url}}}')
      ;;
    ios|android)
      : "${ARTIFACT_PATH:?Set the path to the zipped .app, .apk or .aab}"
      : "${APP_IDENTIFIER:?Set the iOS bundle ID or Android package name}"
      file=$(basename "$ARTIFACT_PATH")
      bytes=$(wc -c < "$ARTIFACT_PATH" | tr -d '[:space:]')
      if [ "$platform" = ios ]; then
        type=application/zip
      else
        type=application/vnd.android.package-archive
      fi
      upload=$(post /api/artifacts/upload-url "$(jq -n \
        --arg platform "$platform" --arg fileName "$file" \
        --arg contentType "$type" --argjson declaredBytes "$bytes" \
        '{platform:$platform,fileName:$fileName,contentType:$contentType,declaredBytes:$declaredBytes}')")
      storage=$(jq -er '.storageId' <<< "$upload")
      intent=$(jq -er '.intentId' <<< "$upload")
      curl --fail-with-body --silent --show-error \
        --connect-timeout 15 --max-time 600 \
        -X PUT "$(jq -er '.uploadUrl' <<< "$upload")" \
        -H "Content-Type: $type" --data-binary "@$ARTIFACT_PATH"
      scan=$(post /api/artifacts/scan "$(jq -n \
        --arg storageId "$storage" --arg intentId "$intent" \
        '{storageId:$storageId,intentId:$intentId}')")
      if [ "$(jq -r '.status' <<< "$scan")" != clean ]; then
        echo 'The build did not pass the safety check, so no run was started:' >&2
        jq -r '.findings[]?.message' <<< "$scan" >&2
        exit 1
      fi
      where=$(jq -n --arg p "$platform" --arg storageId "$storage" \
        --arg fileName "$file" --arg id "$APP_IDENTIFIER" \
        '{targets:{($p):({storageId:$storageId,sourceMode:"upload",fileName:$fileName}
          + (if $p == "ios" then {bundleId:$id} else {packageName:$id} end))}}')
      ;;
    *) echo 'Usage: bash .ci/autonomy.sh web|ios|android' >&2; exit 2 ;;
  esac
fi

# 3. Start the runs. A plan starts one run per test case.
#    The request ID's prefix (github, gitlab, ...) names the CI provider.
body=$(jq -n --argjson what "$what" --argjson where "$where" \
  --arg platform "$platform" \
  --arg branch "${AUTONOMY_BRANCH:-}" --arg pr "${AUTONOMY_PR:-}" \
  --arg commitSha "${AUTONOMY_COMMIT_SHA:-}" \
  --arg provider "${AUTONOMY_REQUEST_ID%%:*}" \
  --arg key "$AUTONOMY_REQUEST_ID:$platform" \
  '$what + $where + ({
     platforms: [$platform],
     branch: $branch,
     pr: $pr,
     deployment: ({provider: $provider, branch: $branch, commitSha: $commitSha}
       | with_entries(select(.value != ""))),
     idempotencyKey: $key
   } | with_entries(select(.value != "")))')
run_ids=$(post /api/v1/run.trigger "$body" | jq -er '.runIds[]')
printf 'Started Autonomy runs:\n%s\n' "$run_ids"

# 4. Follow each run until it ends. run.get waits up to 25 s for a change;
#    sending the last cursor back makes each call wait for something new.
deadline=$((SECONDS + ${AUTONOMY_WAIT_SECONDS:-1200}))
all_passed=true
for run_id in $run_ids; do
  cursor=null
  while :; do
    left=$((deadline - SECONDS))
    if [ "$left" -le 0 ]; then
      echo "Stopped waiting for run $run_id after AUTONOMY_WAIT_SECONDS. It keeps going in Autonomy." >&2
      exit 1
    fi
    wait_ms=$(( left < 25 ? left * 1000 : 25000 ))
    run=$(post /api/v1/run.get "$(jq -n --arg runId "$run_id" \
      --argjson cursor "$cursor" --argjson waitMs "$wait_ms" \
      '{runId:$runId,cursor:$cursor,waitMs:$waitMs}')")
    cursor=$(jq '.cursor' <<< "$run")
    status=$(jq -r '.status' <<< "$run")
    [ "$status" = queued ] || [ "$status" = running ] || break
  done

  name="$(jq -r '.testCaseName' <<< "$run") ($run_id)"
  case "$status" in
    passed)
      echo "Passed: $name" ;;
    failed)
      echo "Failed: $name. Open the run in Autonomy to see which step failed." >&2
      all_passed=false ;;
    couldnt_start)
      echo "Couldn't start: $name. The environment couldn't be tested, so this says nothing about your app. Open the run for the reason, fix it and run again." >&2
      all_passed=false ;;
    cancelled)
      echo "Cancelled: $name." >&2
      all_passed=false ;;
    *)
      echo "Unexpected status $status: $name." >&2
      all_passed=false ;;
  esac
done

if [ "$all_passed" != true ]; then
  echo 'Not every Autonomy run passed.' >&2
  exit 1
fi
echo 'Every Autonomy run passed.'

Save this as .gitlab-ci.yml. Turn on GitLab-hosted Linux runners for your project; the web job and the iOS upload job select saas-linux-small-amd64. On GitLab Self-Managed, replace that tag in every Linux container job with the tag of your own Linux Docker executor runner. Start with Build → Pipelines → New pipeline on the protected default branch. The log prints Started Autonomy runs: and the run IDs once Autonomy accepts the request.

.gitlab-ci.yml

stages: [prepare, qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

default:
  tags: [saas-linux-small-amd64]
  image: debian:stable-slim
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends bash curl jq ca-certificates

preview_ready:
  stage: prepare
  script:
    - |
      : "${PREVIEW_URL:?Set the deployed preview URL in CI/CD Variables}"
      curl --fail --silent --show-error --location \
        --connect-timeout 15 --max-time 60 "$PREVIEW_URL" > /dev/null
      printf 'PREVIEW_URL=%s\n' "$PREVIEW_URL" > preview.env
  artifacts:
    reports:
      dotenv: preview.env
    expire_in: 1 day

autonomy_web:
  stage: qa
  timeout: 30m
  needs:
    - job: preview_ready
      artifacts: true
  script:
    - |
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh web

For a URL created inside your existing deployment job, move the printf and artifacts:reports:dotenv entries into that job, then change needs to its name. Remove the fixed PREVIEW_URL project variable so it doesn't override the generated value. Keep credentials out of preview.env; GitLab stores it as a downloadable artifact.

Mobile artifact handoff

Use the same .ci/autonomy.sh. For iOS and Android it asks for an upload URL, uploads the build, waits for the safety check to come back clean and starts the runs with the uploaded build. Use only the platform whose build is in your repository.

iOS

iOS testing is coming soon. Keep this for when it's available. Until then, Autonomy doesn't start runs on iOS.

Use a dedicated, protected, self-hosted macOS runner with the macos tag, the Bash shell executor, Xcode and the iOS Simulator SDK. Turn off Run untagged jobs for this runner and don't give it the Linux runner tag. GitLab's hosted macOS runners need Premium or Ultimate, apart from eligible open-source programs, so a Free project needs its own Mac. Linux can't do this Xcode build.

This example assumes ios/MyApp.xcodeproj, a shared MyApp scheme and a MyApp.app product. Replace all three with your own names, and install your project's dependencies before xcodebuild. For a workspace, use -workspace ios/MyApp.xcworkspace instead of -project. It builds a simulator .app, not a device .ipa.

The macOS job zips the app with permissions and symlinks intact before GitLab moves it. needs:artifacts downloads the zip in the Linux job, which uploads it and waits for the results.

.gitlab-ci.yml

stages: [build, qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

build_ios:
  stage: build
  tags: [macos]
  script:
    - |
      set -euo pipefail
      xcodebuild \
        -project ios/MyApp.xcodeproj \
        -scheme MyApp \
        -configuration Release \
        -sdk iphonesimulator \
        -destination 'generic/platform=iOS Simulator' \
        -derivedDataPath ios/build \
        CODE_SIGNING_ALLOWED=NO \
        build
      mkdir -p .build
      (cd ios/build/Build/Products/Release-iphonesimulator && \
        zip -qry "$CI_PROJECT_DIR/.build/MyApp.app.zip" MyApp.app)
  artifacts:
    paths:
      - .build/MyApp.app.zip
    expire_in: 1 day

autonomy_ios:
  stage: qa
  tags: [saas-linux-small-amd64]
  image: debian:stable-slim
  timeout: 30m
  needs:
    - job: build_ios
      artifacts: true
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends bash curl jq ca-certificates
  script:
    - |
      export AUTONOMY_TEST_PLAN_ID="$AUTONOMY_IOS_TEST_PLAN_ID"
      export APP_IDENTIFIER="$AUTONOMY_IOS_BUNDLE_ID"
      export ARTIFACT_PATH=".build/MyApp.app.zip"
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh ios

Android

Register a dedicated, protected Linux runner with the android-linux tag and the Bash shell executor. Turn off Run untagged jobs for this runner and don't give it the Linux container runner tag. On that machine, install the JDK your project needs, the Android command-line tools, your compile SDK and build-tools versions, Bash, jq, curl 7.76 or later and trusted CA certificates. Set ANDROID_HOME, accept the SDK licenses, and commit an executable android/gradlew with its wrapper files. The shell executor uses the host's tools; it doesn't install an Android SDK for you.

This example builds the app module's debug APK and uploads it in the same job. Change the Gradle task and ARTIFACT_PATH together if you use flavors or another module. An .aab works too.

.gitlab-ci.yml

stages: [qa]

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
    - when: never

autonomy_android:
  stage: qa
  tags: [android-linux]
  timeout: 60m
  script:
    - |
      set -euo pipefail
      (cd android && ./gradlew --no-daemon assembleDebug)
      export AUTONOMY_TEST_PLAN_ID="$AUTONOMY_ANDROID_TEST_PLAN_ID"
      export APP_IDENTIFIER="$AUTONOMY_ANDROID_PACKAGE_NAME"
      export ARTIFACT_PATH="android/app/build/outputs/apk/debug/app-debug.apk"
      export AUTONOMY_REQUEST_ID="gitlab:$CI_PROJECT_ID:$CI_JOB_ID"
      export AUTONOMY_BRANCH="$CI_COMMIT_REF_NAME"
      export AUTONOMY_COMMIT_SHA="$CI_COMMIT_SHA"
      bash .ci/autonomy.sh android

Gate the pipeline on the result

The helper starts the runs with POST /api/v1/run.trigger and prints their IDs. A test plan starts one run per test case. It then follows each run with POST /api/v1/run.get, sending back the last cursor with waitMs: 25000, so each call waits until the run changes. A response with timedOut: true only means nothing changed yet. See Runs and evidence.

Run result	The job	What the log says
Passed	Passes, if every other run passed too. A flaky pass counts as passed.	`Passed`
Failed	Fails.	Open the run to see which step failed and why, including a Runner problem.
Couldn't start	Fails.	The environment couldn't be tested, for example the build didn't install or a value was missing. This says nothing about your app; fix the cause and run again.
Cancelled	Fails.	`Cancelled`

API errors, a build that fails the safety check and running out of wait time also fail the job. AUTONOMY_WAIT_SECONDS is the total wait time for all runs, 1,200 seconds by default. Keep the CI job's time limit longer than that plus build and upload time. Stopping the CI job doesn't cancel runs that already started.

AUTONOMY_REQUEST_ID, with the platform added, becomes the idempotencyKey of run.trigger: if the same job attempt sends the request again, it gets the same runs back instead of new ones. Give every job attempt its own ID. The part before the first colon is sent as the deployment provider, and the branch and commit go with the runs. The full request and response shapes are in Operations.

Keep allow_failure off on the Autonomy jobs.

Branch and event strategy

These configurations allow pushes, schedules and manually started pipelines only on the protected default branch. Merge request pipelines and fork code don't receive the key. Keep short smoke plans on ordinary deployments; use a schedule or a manual pipeline with a broader plan for release checks.

To check a merge request preview, run the trusted default-branch configuration by hand with the reviewed preview URL. Don't give untrusted merge request code access to protected variables. If you widen the branch rules, protect those branches and review their pipeline changes first.

Vendor references

GitLab CI/CD YAML reference: stages, workflow rules, runner tags, timeouts and needs.
CI/CD variables: variable values, masking and protected branches.
Job artifacts and dotenv reports: passing a zip or a URL between jobs.
Predefined variables: job IDs, branch names and commit SHAs.
Hosted Linux runners: the saas-linux-small-amd64 tag and container execution.
Hosted macOS runners and shell executor: runner availability and host prerequisites.

On this page