Autonomy Docs

Pull Request Testing

Run Autonomy against preview deployments and publish one evidence comment back to GitHub.

Pull request testing runs Autonomy before code is merged. When a preview deployment is ready, Autonomy runs the selected plan against that preview URL and writes the result back to GitHub as one comment with evidence links.

The goal is not to replace every CI check. Use pull request runs for the product journeys reviewers care about but cannot verify from build logs alone: signup, checkout, onboarding, settings changes, permissions, receipts, and integration handoffs.

Prerequisites

Before enabling PR runs

  • Install the Autonomy GitHub App for the repository.
  • Choose the plan or plan group that should run on pull requests.
  • Make the preview deployment URL available to Autonomy.
  • Run the plan once manually before treating the result as merge-blocking.

Setup

  1. Connect the repository in Autonomy.
  2. Install or confirm the GitHub App installation.
  3. Connect the deployment provider or expose the preview URL from CI.
  4. Select a short smoke plan for the first PR run.
  5. Open a pull request and verify that one Autonomy comment appears with the run status and evidence link.

How it works

  1. A pull request is opened or updated.
  2. Your deployment provider creates a preview environment.
  3. Autonomy receives or is given the preview URL.
  4. The selected plan runs against that URL.
  5. Autonomy posts one GitHub comment with status, evidence, and the highest-signal failure details.
  6. On later pushes, the existing comment is updated instead of creating a new thread.

Protected previews

If preview deployments are protected, store the bypass token or preview password in the provider secret store. Do not place bypass values in plans, MDX docs, or pull request comments.

Troubleshooting

No comment appears

Check that the GitHub App is installed for the repository, the preview URL was available, the API key secret is present, and the plan can run manually against the same URL.

The run targets production

Move the Autonomy step after preview URL discovery and confirm the URL passed to Autonomy is the deployment URL for the pull request, not the default production domain.

The preview shows an auth wall

Add the provider-specific bypass token or preview password to the environment used by the run, then retry the same pull request.

Run Evidence

Understand the evidence Autonomy captures for every run.

API and Network

Capture direct requests, assertions, waterfall evidence, and HAR exports.

On this page

PrerequisitesSetupHow it worksProtected previewsTroubleshootingNo comment appearsThe run targets productionThe preview shows an auth wall