GitHub App

The Autonomy GitHub App connects a repository to a project. It runs the project's pull request plan on every pull request, turns an @autonomy comment into runs, and reports back as a GitHub check.

For runs started from a workflow file, see GitHub Actions. For setting up the pull request plan, see Pull request testing.

Set up

1. Open Organization → Integrations → GitHub and connect GitHub. Owners and admins only.
2. Install the App for your GitHub organization or account, and pick the repositories.
3. Grant the permissions it asks for: Checks (read and write), Pull requests (read and write), Contents (read), Deployments (read).
4. For each repository, choose its project. A repository's runs, test plans and environments come from that project. Moving a repository to another project clears its defaults.
5. Comment @autonomy help on a pull request to check the App is listening.

Ask for a run in a comment

Mention @autonomy, or start a line with /autonomy, then say what to run:

@autonomy test the checkout flow against the Vercel preview
@autonomy run plans "Signup", "Checkout" on staging and preview
@autonomy test this PR using https://deploy-preview-128.example.com
@autonomy rerun only the failed scenarios
@autonomy help

Autonomy reads the comment with a fixed grammar first. When that can't tell which test cases you meant, it asks AI to choose, but only from the test cases and environments that exist in the repository's project.

Explicit overrides

The verbs test, run, rerun, qa and check read naturally and are ignored, and so are on, in, against and using before an environment name. browser, chrome and desktop mean web; iphone and ipad mean iOS; mobile means iOS and Android.

preview targets the pull request's latest preview deployment. rerun … failed reruns only what didn't pass last time.

How the target is chosen

1. An explicit URL in the comment.
2. A named environment in the comment.
3. The pull request's preview deployment, when you asked for preview.
4. An environment whose branch matcher matches the pull request's branch.
5. The repository default.
6. The project's default environment.
7. The test case's own default environment.

What you get back

A comment run keeps one comment on the pull request up to date: the command it understood, who asked, and a table of test case, target, platforms and result, with a link to each run. Automatic runs on every commit report only through the check.

Every request also has one check, named Autonomy QA, for example Autonomy QA: 2 passed, 1 failed of 3. It concludes as described in Pull request testing: a failed run makes it failure, a run that couldn't start makes it neutral. A comment Autonomy couldn't run also concludes neutral, so a typo never blocks a merge.

The check has Re-run failed and Re-run all buttons.

Limits

• Only collaborators can start runs. The commenter must be an owner, member or collaborator on the repository. Bots are ignored.
• AI can't invent targets. It chooses from the project's test cases and environments, and any URL must appear in your comment.
• Production is protected. An environment that looks like production is only used when you typed its name.
• Targets must be public HTTPS. Loopback addresses, private IPs, .local hosts and URLs with credentials are refused.
• Unclear requests are refused. If Autonomy isn't sure what you meant, it replies with a suggestion instead of guessing.
• At most 8 runs per request. A command that would start more is refused, not cut short.

Start runs from CI or your own tools

With an organization API key, call run.trigger with exactly one of testCaseId or testPlanId, and either a saved environmentSlug or explicit targets. Mutations carry their idempotency key in the JSON body.

1{2  "testPlanId": "...",3  "environmentSlug": "preview",4  "branch": "feature/checkout",5  "pr": "123",6  "idempotencyKey": "github:123456789:1"7}

1{2  "runIds": [3    "..."4  ],5  "planGroupRunId": "...",6  "environment": {7    "slug": "preview",8    "name": "Preview"9  }10}

Follow each run with run.get until its status is final. See Runs and evidence.

Troubleshooting

Nothing happens when I comment

Check that the App is installed on the repository, that the repository has a project, that you're an owner, member or collaborator, and that the mention is @autonomy or a line starting with /autonomy.

The comment says it couldn't find a test case

Name it exactly with plans:"Exact Name", or set the project's pull request plan so a bare command has something to run. Only the repository's project is searched.

The run used the wrong environment

Walk the list in "How the target is chosen". An explicit url: or env: always wins.