Autonomy uses a small set of terms everywhere: in the dashboard, in notifications, in the API and in AI assistants. This page defines them.

Projects

A project is one product you test, such as a web app, a mobile app or an API. Everything you test lives in a project: its test cases, test plans, environments, sign-ins, issues, digests, memory and files.

Your organization sits above its projects. Members, billing, run credit and integrations belong to the organization and are shared by every project. See Projects.

Environments

An environment is one place a project runs, such as Production, Staging or a preview deployment. It has one platform target for each platform you test: a URL for web and API, and an app build for Android. iOS testing is coming soon.

Every run happens in an environment. The same test case runs against any environment of its project, because the environment supplies the URL, the build and the values.

Environment values are named values used in steps as {{env.NAME}}. Each run fills them in from the environment it runs in. Mark a value as a secret for passwords and tokens: nobody can view it again after you save it, and only runs receive it. See Environments.

Test cases and steps

A test case is one journey you want to prove works, written as a short list of steps. Each step is an outcome in plain words: something a person could check when the step is done.

A new grower account exists for a fresh email.
The profile shows Maize as the farming preference.
The receipt email arrives with the order total.

Steps don't say where to click. The agent works out how to reach each outcome on the live product, the same way a careful tester would, and shows everything it did.

A step can carry attachments: things the agent uses to reach the outcome.

Attachment	What it gives the agent
Account	A saved sign-in to use.
Inbox	A fresh inbox for the run, to receive codes, links and receipts.
Phone	A phone number that can receive SMS codes.
File	A file to upload.
Screenshot	A screenshot of the outcome to keep as evidence.
Data	A value to use, a value to generate (such as a strong password), or a hidden value.

Attachments are test data unless you mark them Exact. If the product refuses a test-data value only because it's taken, the agent uses a fresh value of the same kind and says so. An Exact value is used as written: if the product refuses it, that refusal is what the step is judged on. See Authoring.

Test plans

A test plan is a named selection of test cases in one project. Running a plan starts one run per test case, grouped together. A test case doesn't need a plan, and the same test case can be in several plans. See Test plans and schedules.

Runs

A run is one execution of a test case in one environment. It's the only source of evidence in Autonomy: nothing is claimed about your product that a run didn't see.

The agent works through the steps in order. Under each step you see its Step Actions: every action the agent took to reach the outcome, as it happens. Values the run made or received, such as a generated password or a code from an email, are shown as they were. Only values you saved as secrets are filled by reference: the agent never sees them, and they never appear in evidence.

Each step ends with a Final check: a fresh look at the product at the end of the step. A step's result must agree with its Final check, and without fresh evidence a step is never marked passed.

Run results

Every step gets a result: passed or failed. The run's result follows from its steps.

Result	Meaning
Queued	Waiting for a free runner.
Running	In progress. Before the first step, the run shows Getting ready.
Passed	Every step reached its outcome, and each Final check agrees. flaky means a step needed a retry.
Failed	A step didn't reach its outcome, or the agent couldn't show that it did: for example, the step ran out of time or an account couldn't sign in. Each failed step says why, in plain words.
Couldn't start	The environment couldn't be tested, so no step ran: for example, the app didn't open or blocked the test browser as a bot. It isn't billed.
Cancelled	Someone stopped the run.

A run is Failed if any step failed, and Passed when every step passed. A step only passes when fresh evidence shows its outcome, so a step the agent couldn't prove is Failed, with its reason. Pass rate leaves out Couldn't start runs.

Runner problem. If a run fails because something broke on Autonomy's side, such as the AI model being unavailable, the runner crashing or its browser or device stopping, Autonomy runs it again once, automatically. If the second attempt breaks the same way, the run is Failed with a Runner problem tag. Runner-problem attempts aren't billed.

See Runs and results for what a run page shows.

Issues

An issue is a problem in your product, reported by a person or found by Autonomy. Its kind is Bug (something is broken) or Risk (a gap or concern; nothing is broken yet).

Issues are kept apart from results. A passing run can still report an issue, such as a console error or a failed request it saw along the way, and an issue never changes a step's or a run's result. You can turn an issue into a ticket in Linear or Jira. See Issues.

Memory

Memory is what the agent knows about how to reach things in your project: where a page moved, which email is slow, what to wait for. Runs suggest memories, and people can add them. Memory never adds steps, records bugs or changes a result. Product knowledge is a briefing you write for the agent, like one for a new tester. See Memory.

Where runs execute

Runs execute on Autonomy's runners: the machines that host the browser, device or HTTP client a run uses. Every run starts signed out, with a fresh browser or app session. Enterprise plans can also run on their own machines. See Runners.

A note on naming

The dashboard, the API and these docs say test case for the journey that runs and test plan for a selection of test cases. A few API identifiers still carry older names; where that matters, the page says so.

Concepts