Pull request testing

Pull request testing runs your journeys before code is merged. When a pull request is opened or updated, Autonomy waits for its preview deployment, runs the project's pull request plan against it, and reports the result as a GitHub check.

Use it for the journeys reviewers care about but can't check from build logs: signup, checkout, onboarding, permissions, receipts.

Three ways to start a run

Set up

1. In Organization → Integrations → GitHub, connect GitHub and choose the project for the repository. Admins only.
2. Open the project's Settings → Integrations. Under Pull requests, pick the test plan and turn on Run a test plan on every pull request.
3. Open a pull request. The check appears as soon as the commit is pushed.

Draft pull requests and pull requests from forks don't run automatically. When a newer commit is pushed, Autonomy runs the newest one.

How a pull request run works

1. A pull request is opened, updated, reopened or marked ready for review.
2. Autonomy waits up to 10 minutes for the preview deployment of that commit. The check says "Waiting for the preview deployment of this commit." meanwhile.
3. The plan runs against the preview. If no preview appears, it runs against the repository's environment instead, and the check says so.
4. The check updates as runs finish, with a link to each run.

Automatic runs report only through the check. Runs you ask for in a comment also keep one comment on the pull request up to date.

The check

The check is named Autonomy QA and concludes as:

A failed run fails the check, whether its outcome wasn't reached or the agent couldn't show that it was. The check gives each run's reason, such as "the step ran out of time". A run with a Runner problem fails it too, after Autonomy has already run it again once; run the check again later.

A run that couldn't start leaves the check neutral: the environment couldn't be tested, so there's nothing to judge.

The check has Re-run failed and Re-run all buttons. GitHub's own re-run works too.

Fail the check on issues

Issues never change a run's result, so by default a passing run passes the check even if it reported an issue. To make bugs block the merge, turn on Fail the check on issues under Pull requests in the project's settings. Off by default.

When it's on, an open Bug that a run of the check found or reopened turns a passing check to failure: "Failed: a bug was found". Risks, notes, and bugs you resolved or ignored don't count. See Issues.

Protected previews

If your previews are protected, store the bypass token or preview password as a secret value of the environment the runs use. Don't put bypass values in test cases or pull request comments.

Troubleshooting

No check appears

Check that the GitHub App is installed on the repository, that the repository has a project in Organization → Integrations → GitHub, and that the project has a pull request plan with the switch on. Draft and fork pull requests don't run automatically.

The runs went to the repository's environment, not the preview

No preview deployment for that commit was reported to GitHub within 10 minutes. Check that your host reports preview deployments to GitHub and builds one for each commit.

The check failed but every run passed

Fail the check on issues is on, and a run found a bug. Open the run's Issues found.