Runs, retries, and backfills

Start a run

You need runs.trigger and an existing deployment or registered executable version. After deploying a flow:

curl --fail-with-body -X POST "$DAGY_API_URL/runs" \
  -H "Authorization: Bearer $DAGY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "deployment":"orders-develop",
    "parameters":{"region":"eu"}
  }'

Illustrative response, with optional metadata omitted:

{
  "run_id":"c8d39336-f467-4a5c-b804-e832929d1b38",
  "run_slug":"sample-run",
  "flow_name":"orders_daily",
  "flow_version":"1",
  "status":"QUEUED",
  "environment":"develop",
  "execution_mode":"nano",
  "trigger_source":"manual"
}

Save the returned run_id. Submission normally returns 200; it is asynchronous by default (execute: false). Poll the run to confirm success. Do not assume a timeout means nothing started or blindly repeat a submission.

To address a version directly, replace deployment with flow_name and flow_version. Both are required together. Supply environment for environment-dependent runs. Deployment-based submissions inherit the deployment's environment and runtime tier unless explicitly overridden. If you supply a deployment and explicit flow fields, the deployment selects the flow/version.

execution_mode can override the tier for one run: nano, micro, small, medium, large, or xlarge. Use the configured runtime options described in Execution. Avoid the low-level executor override unless the workspace explicitly requires it.

execute: true requests synchronous execution and can keep the HTTP request open for the duration of the work. Use the default asynchronous path for application integrations.

Inspect runs and logs

These operations require runs.read. Run lists have no continuation cursor; use explicit bounds and time/flow filters for manageable responses. Live log pages cap the requested limit at 10,000. Archived logs may return all events without a token. Continue while a token is present, even if one page is empty.

curl --fail-with-body "$DAGY_API_URL/runs/$RUN_ID" \
  -H "Authorization: Bearer $DAGY_TOKEN"

curl --fail-with-body "$DAGY_API_URL/runs/$RUN_ID/logs?limit=500" \
  -H "Authorization: Bearer $DAGY_TOKEN"

Run metadata includes the triggering source/user and, when available, start/end times and an error message. Inspect individual task attempts to find the first failing operation. Logs contain event-specific fields such as event_type, message, and timestamp; treat optional fields defensively. Empty logs do not prove that work succeeded.

Run lifecycle

Treat unfamiliar future statuses as unresolved until your client knows how to handle them. The local executor also uses task-level states such as skipped; do not confuse task and run status fields.

Cancel a run

POST /runs/{run_id}/cancel requires runs.cancel (owner/admin). Cancellation is accepted for queued or running runs. It may stop the runtime immediately or take effect cooperatively between tasks.

curl --fail-with-body -X POST "$DAGY_API_URL/runs/$RUN_ID/cancel" \
  -H "Authorization: Bearer $DAGY_TOKEN"

The response includes status such as cancellation_requested or cancelled, run_id, cancelled_by, and cancelled_at. Repeating the request for an already requested/canceled run is supported. Other completed states return 400.

Continue checking the run until cancellation is confirmed. Cancellation does not reverse writes, sent notifications, or other effects already performed by tasks.

Retry, backfill, and compare

Use full retry to repeat a run, partial retry to reuse successful task outputs where supported, and backfill to materialize a historical time window. All can repeat effects in external systems. Design destination writes with stable business keys or deduplication.

Retry a run

Both require runs.trigger; neither takes a parameter override body. To change inputs, submit a new ordinary run. They retain the source environment and requested runtime tier. The source run remains available for comparison.

{
  "run_id":"NEW_RUN_ID",
  "parent_run_id":"SOURCE_RUN_ID",
  "flow_name":"orders_daily",
  "flow_version":"1",
  "status":"QUEUED",
  "partial":true,
  "reused_task_count":2,
  "skip_task_ids":["extract-1","normalize-1"]
}

This is an illustrative partial-retry response. Reuse fields describe the plan, not a guarantee that every output will be reused. Reuse depends on the runtime and availability of source task outputs. Unsupported reuse or missing usable output can cause re-execution; when no source tasks succeeded, the plan becomes a full retry. Inspect the new task records and logs to confirm actual behavior.

Run parameter keys beginning with __dagy_ are reserved for orchestration. Do not use them as application inputs or manufacture retry hints yourself.

Backfill a time window

A backfill creates one run for each logical occurrence in an inclusive from–to window. It is useful for loading historical partitions or recovering missed processing without changing the live schedule.

The flow should have a cron/interval schedule referencing the intended version and deployment. An explicit interval_seconds or cron_expression can override the cadence for this job. Without a stored schedule, a cadence-only request falls back to version 1; create the schedule first when you need another version, deployment, or timezone. Backfill creation does not have an explicit version or schedule-ID selector.

Preview first:

curl --fail-with-body -X POST "$DAGY_API_URL/flows/orders_daily/backfill" \
  -H "Authorization: Bearer $DAGY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "from":"2026-08-01T00:00:00Z",
    "to":"2026-08-03T00:00:00Z",
    "interval_seconds":86400,
    "max_concurrency":2,
    "parameters":{"region":"eu"},
    "environment":"develop",
    "dry_run":true
  }'

The interval example produces three occurrences, including both endpoints. A dry run returns status: "DRY_RUN", dry_run: true, total, and intervals with index, logical_epoch, logical_at, and no run IDs. It creates no job or runs. Review the dates/count, then repeat with dry_run: false to start the job.

Creating requires runs.trigger; scoped API keys also need flows.write (or the compatibility admin scope) because this operation belongs to the flow route family. Backfills must be enabled for the workspace; actual job operations return 503 when unavailable. A plan is limited to 10,000 occurrences. Narrow the window or increase the cadence to stay within the bound. max_concurrency defaults to 1; use a positive value appropriate for the destination and compute budget.

Each run receives your supplied parameters plus these defaults:

Make your flow accept these parameters or **kwargs. A signature that rejects them can fail; some hosted paths can fall back to the packaged default graph after a parameter-binding error. That fallback can run the wrong partition rather than applying the requested dates. Validate parameter handling in the selected runtime before a real backfill. data_interval_end is not supplied. If you provide these keys yourself, your values are preserved for every occurrence, which can unintentionally process the same partition repeatedly. The job uses parameters from the backfill request; include any required values rather than assuming the live schedule's parameter map is copied.

For the orders_daily example, use a flow that explicitly accepts the injected fields and passes the logical date to a task:

from datetime import datetime
from dagy import flow, task

@task
def select_order_partition(logical_start: str, region: str) -> dict:
    if not logical_start:
        raise ValueError("A logical processing date is required")
    day = datetime.fromisoformat(logical_start.replace("Z", "+00:00")).date().isoformat()
    # Use this partition in your source query and destination idempotency key.
    return {"region": region, "partition": day, "key": f"{region}/{day}"}

@flow(name="orders_daily")
def orders_daily(
    region: str = "eu",
    data_interval_start: str = "",
    logical_date: str = "",
    ds: str = "",
):
    return select_order_partition(data_interval_start or logical_date or ds, region)

Save it as a module, build and deploy it, and update the flow's schedule to the resulting version before backfilling. Test locally with orders_daily.run_local(region="eu", data_interval_start="2026-08-01T00:00:00Z", logical_date="2026-08-01T00:00:00Z", ds="2026-08-01"). The task selects partition 2026-08-01 and key eu/2026-08-01. Replace the fixture result with real source/destination operations only after confirming each backfill interval selects a different intended partition.

Monitor and stop a backfill

The job launches an initial batch and advances as previous runs finish. Monitor dispatched, completed, failed, and interval statuses. A job can reach COMPLETED while some constituent runs failed; inspect failed before declaring the data complete. Progress refreshes periodically rather than every time you read the job.

Canceling a backfill does not cancel runs already dispatched. Cancel those run IDs separately if necessary, with runs.cancel permission.

Compare two runs

curl --fail-with-body --get "$DAGY_API_URL/runs/compare" \
  -H "Authorization: Bearer $DAGY_TOKEN" \
  --data-urlencode "run_a=$RUN_A" \
  --data-urlencode "run_b=$RUN_B"

This requires runs.read for both accessible runs. The result compares status, flow/version, duration, parameters, and task durations/statuses. A positive duration delta means run B took longer. Task comparisons group by task name and use a latest attempt; repeated invocations of the same task name may be combined. Missing times produce null, not zero. Cost fields can be null; use usage endpoints for available aggregate estimates.

Common problems