GitHub App

The Autonomy GitHub App turns a pull request comment into a run. Mention @autonomy with what you want tested and Autonomy resolves the test cases, the target, and the platforms, then reports back in place — one comment it keeps updating, and one check run.

This is the conversational surface. For pipeline-triggered runs from a workflow file, see GitHub Actions. The two coexist: use Actions for the runs that should happen on every push, and the App for the ones a reviewer asks for.

Setup

1. In the dashboard, open Settings → Integrations and connect GitHub.
2. Authorize the App for the organization or account, and select the repositories.
3. Grant the permissions it requests: Checks (read and write), Pull requests (read and write), Contents (read), Deployments (read).
4. Optional but recommended — under GitHub Auto-Trigger Defaults, set a default target per repository. A repository can default to one test case or one test plan, not both.
5. Comment @autonomy help on any pull request to confirm the App is listening.

Invoking a run

Mention @autonomy, or start a line with /autonomy, followed by what to run:

@autonomy test the checkout flow against the Vercel preview
@autonomy run plans "Signup", "Checkout" on staging and preview
@autonomy test this PR using https://deploy-preview-128.example.com
@autonomy rerun only the failed scenarios
@autonomy help

Autonomy parses the comment with a deterministic grammar first. When the grammar cannot identify which test cases you meant, it falls back to a language model that must choose from the test cases and environments that actually exist in your workspace.

Explicit overrides

Anything the grammar recognises directly bypasses the model:

Optional leading verbs — test, run, rerun, qa, check — read naturally and are ignored by the parser. So do the prepositions on, in, against, and using before an environment name.

Platform words are aliased: browser, chrome, firefox, safari, and desktop all mean web; iphone and ipad mean iOS; mobile means iOS and Android.

Two phrases are special. preview targets the pull request's latest preview deployment. rerun … failed or only … failed re-runs only the scenarios that failed last time.

How the target is chosen

When several sources could supply a target, Autonomy applies this precedence:

1. An explicit URL in the command.
2. A named environment in the command.
3. The pull request's preview deployment, when preview was requested.
4. An environment whose branch matcher matches the PR head branch.
5. The repository default.
6. The workspace default.
7. The test case's own default target.

What you get back

One pull request comment, created when the command is accepted and patched in place as the run progresses — never a new thread per update. It contains the command it understood, who asked, an estimated credit cost, and a matrix table of test case × target × platforms × status, with a deep link to each run's evidence. When the case has API assurance configured, the assurance result is appended.

One check run, named Autonomy QA: <summary> — for example Autonomy QA: 2 passed, 1 failed of 3. It moves through queued → in_progress → completed and concludes as:

• success — everything passed.
• failure — at least one run failed, or execution errored.
• timed_out — a run exceeded its execution timeout.
• cancelled — the runs were cancelled.
• neutral — the command was rejected: unparseable, ambiguous, over the fan-out limit, or from an unauthorized commenter.

Rejections conclude neutral on purpose. A malformed comment should not block a merge. Real failures conclude failure and will block a merge under branch protection.

The check run carries two action buttons: Rerun failed scenarios and Rerun all plans. GitHub's native re-run control works too.

Limits and guardrails

The natural-language path is deliberately fenced in:

• Only collaborators can trigger runs. The commenter must be an owner, member, or collaborator on the repository. Comments from bots are ignored entirely.
• The model cannot invent targets. It selects from the candidate test cases and environments passed to it, and any URL it emits must appear literally in your comment.
• Production is protected. An environment whose name looks like production is never selected unless you typed it yourself.
• Targets must be public HTTPS. Loopback addresses, private IP ranges, .local hosts, and URLs with embedded credentials are rejected.
• Low confidence is refused. If the model is not reasonably sure what you meant, Autonomy replies with a syntax suggestion rather than guessing.
• Comment text is untrusted. It is passed to the model as data, never as instructions.
• Fan-out is capped at 8 runs per request. A command that expands past that is rejected rather than truncated.

Opening a pull request does not by itself start a run. A run begins when someone comments, or when a configured trigger fires.

Triggering from CI or your own tooling

The same aggregate request pipeline is available over the REST API with an organization API key that has the autonomy:runs:write scope.

1{2  "request": "test the checkout flow against https://preview.example.com"3}

1{2  "requestId": "req_...",3  "status": "queued"4}

You can send the same natural-language request string, or a structured body with planIds, environmentSlugs, urls, and platforms. Pass an Idempotency-Key header so a retried delivery does not double-run.

Poll the result:

Troubleshooting

Nothing happens when I comment

Check that the App is installed on that repository, that you are an owner, member, or collaborator, and that the mention is @autonomy or a line starting with /autonomy. Comment @autonomy help to confirm the App is receiving events.

The comment says it could not identify a test case

Name it explicitly with plans:"Exact Name", or set a repository default target under GitHub Auto-Trigger Defaults so a bare command has something to run.

The run targeted the wrong environment

Walk the precedence list above. An explicit url: or env: in the comment always wins; if you did not supply one, a branch matcher or a repository default probably did.

The check is red but nothing is broken

Look at the conclusion. neutral means the command was rejected, not that the product failed — read the comment body for the reason. Only failure and timed_out indicate a real run outcome.