Datadog

Connect Autonomy to Datadog through its OTLP/HTTP trace intake. Each run produces an autonomy.run span, nested autonomy.step spans, and autonomy.action.* children. The integration is available now through Settings → Observability for runs executed by runner-labs.

Before you start

The ingest credential is a Datadog API key. Application keys and Autonomy aut_ keys do not authenticate to Datadog. Keep it in the Headers field, where Autonomy stores it encrypted and never returns it to the dashboard.

Choose your Datadog endpoint

Enter the base URL matching your Datadog site in OTLP Endpoint. Autonomy adds /v1/traces and sends JSON over HTTPS. Datadog supports this http/json encoding.

These URLs follow Datadog's current OTLP trace intake site selector. A key for one site does not authenticate to another site's intake.

Connect and verify

1. Open Settings → Observability in Autonomy and select Datadog.

2. Paste your site's base URL into OTLP Endpoint. Enter it explicitly, even if the placeholder shows the right address.

3. Enter this value in Headers, replacing YOUR_DATADOG_API_KEY with your API key:

   dd-api-key=YOUR_DATADOG_API_KEY

4. Click Send Test Span. A successful HTTP response confirms the intake accepted the request; it does not prove the span is searchable yet.

5. In Datadog, open APM → Trace Explorer, select the recent time window, and search service:autonomy-runner-labs. Find the autonomy.test_connection span.

6. Enable Trace Export and click Save Configuration. Start a new test case run so its runner receives the saved configuration.

7. Once the run finishes, search for its ID with the query below. Replace RUN_ID with the ID from the Autonomy run URL. Open the trace and confirm the run and step spans arrived.

service:autonomy-runner-labs @test.run_id:"RUN_ID"

For Datadog APM trace metrics, optionally append ,compute_stats=true to Headers. Direct ingestion does not compute those metrics by default. Save the complete header value, including dd-api-key, when changing or rotating it.

Inspect a failed step in APM

Run a test case with a known failing assertion in a test environment. Set Trace Explorer's time range to that run and search:

service:autonomy-runner-labs status:error @test.status:failed @test.step_index:*

The failed autonomy.step has OTLP error status (code: 2). Inspect test.step_action, test.step_target, autonomy.recovery_count, and autonomy.diagnosis.category when present. Open its trace waterfall to see the action timings. These are APM spans, not a separate Datadog test-management integration.

The run and step spans contain autonomy.run_url; open that URL to return to the run's evidence in Autonomy. Action spans contain autonomy.action.success and inherit their place in the trace, but do not carry the run URL themselves. Test case names (test.plan_name) are on the run span. Model details, prompts, screenshots, and raw error messages are not exported by this integration.

Datadog may map span names to its operation and resource fields. Use the exported attributes to filter reliably; Datadog's trace search syntax explains attribute queries.

Pivot to an application trace

Autonomy creates a separate trace ID for each run. It does not inject that ID or a traceparent header into your web or mobile application. The Autonomy trace and the application's request trace are therefore separate, and a run ID search does not find production requests automatically.

1. Instrument the application and its backend services to send their own traces to Datadog. They must record the relevant service, environment, and request resource; request or order identifiers help narrow a match.

2. From the failed Autonomy step, note its time window and target. Use autonomy.run_url to inspect the run evidence for an application request ID or another identifier, if your application exposes one.

3. Open a second Trace Explorer tab at the same time window. Search the application service and environment, then narrow by the actual resource or request identifier. For example, adapt these values to your application's tags:

   service:checkout-api env:production resource_name:"POST /orders"

4. Open the candidate request trace and confirm its identifiers and timing match before attributing the failure. If the run targeted a preview, use that preview's environment; a production trace at a similar time alone is not a match.

This is manual correlation. Without application instrumentation or distinguishing request evidence, the integration cannot identify a unique production trace.

Troubleshooting

• HTTP 401 or 403: check the Datadog site and API key. Use dd-api-key, not an application key or an Autonomy key.
• HTTP 404: copy the base URL from the table. The resulting request path must be /v1/traces.
• Test span arrives, run spans do not: confirm Trace Export is enabled and saved, the run started afterward, and it executed on runner-labs. Self-hosted runners can override the service name with OTEL_SERVICE_NAME.
• Spans disappear from searches: check the selected time window and Datadog's trace retention settings. An accepted export does not guarantee every span is indexed for later searches.
• A run succeeds despite a Datadog outage: trace export is best effort and does not change the run verdict. Use the Autonomy run result to gate CI.

For self-hosted environment variables and credential removal, see Observability (OTLP).