Netlify

The Netlify integration watches your Netlify deploys. When a deploy is ready, Autonomy runs a project's test case or test plan against that exact deploy and, if a GitHub repository is linked, reports the result as a GitHub check on the commit.

What it does

• Starts runs when a deploy is ready, filtered by deploy type (deploy preview, production, branch deploy) and branch.
• Tests the deploy's own permalink, such as https://<deploy-id>--site.netlify.app, so a newer push can't change what the run sees.
• Posts an Autonomy QA GitHub check through a linked repository. Netlify has no check surface of its own.
• Waits for deploys from forked pull requests until a Netlify team owner approves them.

Set up

1. Open Organization → Integrations → Netlify and click Connect. Approve the access in Netlify.
2. Click Link site and pick the Netlify site to test.
3. Under Project, choose the Autonomy project whose tests these deploys run. Changing it later clears the defaults below.
4. Under Default test case or plan, choose what runs on each deploy. Only the project's test cases and test plans are listed.
5. Under Default environment, choose the project's environment. Runs use its values and sign-ins, with the deploy URL as the web target. Without an environment, deploys are skipped.
6. Under Trigger on, choose the deploy types that start a run, usually Preview. Add Branch filters to limit it to some branches.
7. Turn on Post a GitHub check via linked repository and choose the GitHub repository for checks.
8. Open a pull request and check that a run starts when the deploy preview is ready.

How a run starts

When a deploy succeeds, Autonomy:

1. Finds the linked site and its Autonomy project. Nothing runs if that project is archived.
2. Maps the deploy type: Netlify's deploy-preview is Preview, production is Production, and branch-deploy is Branch. It checks it against Trigger on, and the branch against Branch filters.
3. Starts the default test case or test plan in the default environment, against the deploy permalink.

Each deploy runs once. A webhook Netlify sends twice doesn't start a second run.

A deploy from a forked pull request is held by Netlify until a team owner approves it. Autonomy shows it as pending, and runs it once the approved deploy succeeds.

The check

With a linked repository, the GitHub check is named after what ran, for example Autonomy QA: Checkout, and shows a link to each run. It concludes as:

A failed run fails the check, whether its outcome wasn't reached or the agent couldn't show that it was. The summary gives the reason.

To make bugs fail the check too, turn on Fail the check on issues in the project's Settings → Integrations → Pull requests. It's off by default. When it's on, an open Bug that one of the check's runs found or reopened makes the check fail: "Failed: a bug was found". See Pull request testing.

Without a linked repository, results appear only in Autonomy. The link's settings warn you when checks are on but no repository is chosen.

Password-protected sites

If the site or its deploys are password protected, paste the password into Preview access secret on the linked site. Autonomy uses it for every run of that site. It is stored as a secret and never shown again.

Several sites

Link each Netlify site on its own. Each link has its own Autonomy project, test plan, environment, trigger types and branch filters. Several sites can share one repository for checks.

Troubleshooting

No run starts when a deploy preview is ready

Open the linked site in Autonomy and look at the recent deploys. A skipped deploy says why:

• project disabled: the link is turned off.
• deployment not ready: the deploy is still building or failed.
• context <x> not enabled: the deploy type isn't in Trigger on.
• branch <x> filtered: the branch doesn't match Branch filters.
• no default test target: choose a Default test case or plan.
• "Choose an environment for this deploy link in Autonomy.": choose a Default environment.
• project archived, or a default that is archived or in another project: restore the project or choose new defaults.
• no preview URL: Netlify sent no URL for the deploy.

A deploy stays pending

It's a deploy from a forked pull request. Approve it in Netlify; the run starts once it succeeds.

The run can't reach the deploy

The site is password protected. Add the password as the Preview access secret.

No check appears on the pull request

Check that Post a GitHub check via linked repository is on, that a GitHub repository for checks is chosen, and that the Autonomy GitHub App is installed on it.