Pull Request Testing
Run Autonomy against preview deployments and publish one evidence comment back to GitHub.
Pull request testing runs Autonomy before code is merged. When a preview deployment is ready, Autonomy runs the selected test case against that preview URL and writes the result back to GitHub as one comment with evidence links, plus a check run.
The goal is not to replace every CI check. Use pull request runs for the product journeys reviewers care about but cannot verify from build logs alone: signup, checkout, onboarding, settings changes, permissions, receipts, and integration handoffs.
Two ways to trigger
| Trigger | Use when |
|---|---|
A comment — @autonomy test the checkout flow against the preview | A reviewer wants a run on demand, or wants to vary the target or the cases. See GitHub App. |
| Your pipeline — a workflow step calling the Autonomy API after the preview URL is known | The run should happen on every push without anyone asking. See GitHub Actions. |
Both produce the same evidence and both report through the same comment and check run.
Prerequisites
Before enabling PR runs
- Install the Autonomy GitHub App for the repository.
- Choose the test case or test plan that should run on pull requests.
- Make the preview deployment URL available to Autonomy.
- Run the case once manually before treating the result as merge-blocking.
Setup
- Connect the repository in Autonomy.
- Install or confirm the GitHub App installation.
- Connect the deployment provider or expose the preview URL from CI.
- Set the repository default target — one test case or one test plan — under Settings → Integrations → GitHub Auto-Trigger Defaults, so a bare command has something to run.
- Open a pull request, comment
@autonomy test this PR against the preview, and verify that one Autonomy comment appears with the run status and evidence link.
How it works
- A pull request is opened or updated.
- Your deployment provider creates a preview environment.
- Autonomy receives or is given the preview URL, or resolves it from the pull request.
- The selected test case or plan runs against that URL.
- Autonomy posts one GitHub comment with status, evidence, and the highest-signal failure details, and one check run named
Autonomy QA: …. - On later pushes, the existing comment is updated instead of creating a new thread.
Opening a pull request does not by itself start a run. A run begins when someone comments or when your pipeline calls the API.
Protected previews
If preview deployments are protected, store the bypass token or preview password in the provider secret store. Do not place bypass values in test cases, MDX docs, or pull request comments.
Troubleshooting
No comment appears
Check that the GitHub App is installed for the repository, that the commenter is an owner, member, or collaborator, that the preview URL was available, that the API key secret is present, and that the case can run manually against the same URL.
The run targets production
Move the Autonomy step after preview URL discovery and confirm the URL passed to Autonomy is the deployment URL for the pull request, not the default production domain. In a comment, an explicit url: or env: always wins over any default.
The preview shows an auth wall
Add the provider-specific bypass token or preview password to the environment used by the run, then retry the same pull request.
The check is red but the product is fine
A neutral conclusion means the command was rejected — unparseable, ambiguous, or unauthorized — not that the run failed. Only failure and timed_out reflect an actual run outcome.