Autonomy uses a small set of terms across web, mobile, API, and message checks. The same model applies whether a run starts from the dashboard, a pull request, Slack, an integration, or an AI assistant.

Test cases

A test case is the unit of definition and execution: one named journey with an ordered list of steps. It is what QA practice calls a test case, and it is what actually runs.

Steps describe intent — Navigate, Type, Tap/Click, Scroll, Wait, Assert, Visual Compare, Validate Network — and the agent resolves them against the live product at run time. A case can cover browser steps, mobile sessions, direct API checks, inbox waits, SMS checks, and visual comparisons.

You author a case by writing the steps, describing the flow in plain language, or recording it once. See Authoring.

Test plans

A test plan is a named selection of test cases — many-to-many, like a playlist. Cases are standalone; a plan references them, so the same case can belong to a smoke plan and a release plan without being duplicated.

A case does not need a plan. Running a single case is a first-class workflow. See Test Plans.

Runs

A run is one execution of one test case against one target environment. It has a status, timeline, artifacts, trace data, generated checks, and a review link.

The run is the atomic unit of evidence — everything downstream cites it. Running a plan produces one run per member case, grouped under a plan run whose status is derived from its members rather than stored separately.

Runs are what you attach to pull requests, Slack messages, release notes, and incident follow-up.

Environments

An environment tells Autonomy where to run and with what configuration: a preview URL, staging app, production smoke surface, mobile artifact, or a private target reachable through your infrastructure. Environments are managed at Environments in the dashboard and selected when you author a case.

Environment values are referenced from steps as {{env.VAR_NAME}} rather than pasted in, so the same case runs against every target you point it at.

Use explicit environment names. Reviewers need to know whether evidence came from preview, staging, production, or a private execution environment.

Evidence

Evidence is the review surface for a run. It connects live view, replay video, screenshots, API responses, network timing, inbox and SMS messages, logs, generated checks, visual comparisons, and trace pivots into one timeline.

Evidence should answer three questions quickly:

What did Autonomy try to do?
Where did the journey pass or fail?
Which artifact proves the result?

Checks

Checks are the assertions Autonomy evaluates during or after execution. Some come from the test case. Others are generated from observed behavior, network responses, messages, or visual state.

Contract checks are a distinct class: when a test case has an OpenAPI document attached, response validation is deterministic and cannot be overridden by a model-authored assertion. See API Assurance.

Execution environments

Autonomy can execute against public preview URLs, staging apps, uploaded mobile artifacts, and private environments. Use the simplest reachable environment for the first run, then move private or compliance-sensitive flows into your own infrastructure when needed.

A note on naming

The dashboard, API surface, and these docs say test case for the executable unit and test plan for a selection of cases. Some older API fields and identifiers still use plan for what is now a case; where that matters, the page says so.

Concepts