Execution

Runtime Tiers

Dagy offers six runtime tiers to match your workflow's resource and duration needs. The system automatically routes each flow run to the appropriate tier. You can also override the tier explicitly with the execution_mode parameter at deploy time or the task level.

Tier	Description	Resources	Max Duration
nano	All tasks in one invocation	Up to 10 min runtime	10 minutes
micro	Each task runs independently	Up to 10 min per task	10 minutes per task
small	Dedicated compute	0.5 vCPU / 1 GB RAM	Unlimited
medium	Dedicated compute	1 vCPU / 2 GB RAM	Unlimited
large	Dedicated compute	2 vCPU / 4 GB RAM	Unlimited
xlarge	Dedicated compute	4 vCPU / 12 GB RAM	Unlimited

The nano and micro tiers leverage managed serverless execution optimized for speed and cost. The small through xlarge tiers provision dedicated compute resources on ECS Fargate for resource-intensive or long-running workflows.

Execution Backends (Internal)

Dagy implements runtime tiers using several internal backends. The mapping is automatic and transparent to users.

Backend	Tier(s)	Purpose
Local	Development	Local machine execution during development and testing
Lambda	nano / micro	Serverless execution of Flow Executor
Step Functions	Step Functions meta-orchestrator	Workflow orchestration (optional)
ECS Fargate	small / medium / large / xlarge	Dedicated compute containers

LocalExecutor

LocalExecutor runs tasks in-process on a single machine. It supports:

• Dependency resolution via a ready-check loop + ThreadPoolExecutor
• Configurable concurrency via max_workers
• Retry / backoff with optional jitter and per-attempt delays
• Hard timeouts enforced via a single-threaded ThreadPoolExecutor
• Task-level lifecycle hooks (on_running, on_completion, on_failure, on_retry)
• Artifacts stored under ~/.dagy/artifacts/<run_id>/<task_run_id>/
• Run metadata persisted in DuckDB at ~/.dagy/dagy.duckdb

Scheduling algorithm

pending = all task_run_ids
while pending or inflight:
  ready = [id for id in pending where all depends_on are in statuses]
  for each ready task (up to max_workers):
    if any dep failed → mark SKIPPED
    else → submit to thread pool
  wait(FIRST_COMPLETED) → update statuses, store artifacts
  if pending and nothing scheduled → DependencyCycleError

fail_fast=True (default): on the first FAILED future, cancels all inflight futures and raises TaskFailedError.

Serverless Tiers (nano / micro)

The nano and micro runtime tiers execute flows inside AWS Lambda with a dedicated Flow Executor environment. The Flow Executor bootstraps a full runtime environment, downloads dependency packages from S3, extracts the flow artifact, and prepares a temporary workspace with all dependencies on sys.path.

nano: In-Process Execution

The nano tier runs the entire flow DAG within a single Lambda invocation. Tasks are executed in topological order using a ThreadPoolExecutor. This tier is suitable for flows with fast tasks that complete well within the 10-minute runtime limit.

• All tasks in one invocation
• Fastest execution path with minimal orchestration overhead
• Best for tightly-coupled, fast workflows
• Configure with execution_mode="nano" at deploy time

micro: Task-Isolated Execution

The micro tier runs each task in its own Lambda invocation. An orchestrator Lambda walks the DAG in topological order, invoking the Flow Executor Lambda once per task. Task outputs are persisted to S3 so downstream tasks can load their upstream dependencies. This tier provides stronger isolation between tasks and is suitable for flows where individual tasks may be resource-intensive or where per-task retry and observability are important.

• Each task runs independently
• Stronger isolation and per-task observability
• Better resource efficiency for uneven task loads
• Safe to retry at the task level
• Configure with execution_mode="micro" at deploy time

Runtime State (Internal)

Each flow run carries a FlowRunState that includes the run ID, execution mode, artifact and dependency S3 URIs, resolved parameters, and the serialized flow spec. For micro mode, a target_task_run_id field directs each Lambda invocation to execute a single specific task.

Task State Persistence (Internal)

Task outputs are stored in S3 at s3://<bucket>/task-state/<run_id>/<task_name>/output.json. The TaskStateStore provides methods for saving and loading task state, enabling cross-invocation data passing in micro mode and providing durable state for retries and observability in both modes.

Small payloads (under ~300 KB) are also stored inline in DynamoDB for fast access. Large payloads are stored exclusively in S3 with a URI reference in the DynamoDB record.

Idempotent Execution (Internal)

In micro mode, the orchestrator checks whether a task has already completed (via DynamoDB status or S3 output existence) before invoking a new Lambda. This makes micro execution idempotent and safe to retry at the orchestrator level.

Workspace Bootstrap (Internal)

The Flow Executor Lambda follows this bootstrap sequence:

1. Create a temporary workspace directory
2. Download and extract dependency packages from S3 (supports .zip, .whl, .tar.gz)
3. Add extracted packages to sys.path
4. Download and extract the flow artifact from S3
5. Load the flow module and resolve the FlowSpec
6. Execute the flow (full DAG or single task depending on mode)
7. Clean up the temporary workspace

Configuration (Internal)

The Flow Executor backend is activated when the DAGY_FLOW_EXECUTOR_FUNCTION environment variable is set to the Lambda function name or ARN. It also requires DAGY_ARTIFACTS_BUCKET to be set.

Rich Task Run State (Internal)

The Flow Executor tracks detailed per-task state including input payloads, output payloads, resolved parameters, stack traces on failure, retry counts, artifact references, and S3 URIs for output and state data. This information is persisted to DynamoDB and available via the task runs API.

Dedicated Compute Tiers (small / medium / large / xlarge)

The dedicated compute tiers (small, medium, large, xlarge) launch an ECS Fargate container to run the flow with guaranteed resources.

• Suitable for flows requiring significant memory, CPU, or long runtime
• Supports custom Docker images and VPC networking
• Automatically selected when resource or duration requirements exceed serverless limits
• Configure with execution_mode="small" (or "medium", "large", "xlarge") at deploy time
• Returns an external_id (Fargate task ID); status is polled until terminal state

Step Functions (Internal)

The Step Functions backend optionally maps the FlowSpec to an AWS Step Functions state machine for complex orchestration scenarios.

• Available as an advanced orchestration backend
• Suitable for flows up to 1 year in duration
• Supports native parallel branches and retries
• May be automatically selected for very large or complex workflows

Automatic Tier Selection

The runtime tier is selected automatically using a priority-based system:

1. Explicit execution_mode override on the task/deployment
2. Deployment-level default execution_mode
3. Flow-level default execution_mode
4. Automatic routing rules based on resource/duration requirements
5. Default: nano

Automatic routing rules:

• Estimated memory > 1 GB or duration > 10 min → micro
• Estimated memory > 2 GB or duration > 1 hour → small
• Estimated memory > 4 GB or duration > 2 hours → medium
• Estimated memory > 8 GB or duration > 6 hours → large
• Estimated memory > 12 GB → xlarge
• Otherwise → nano

You can lock a deployment to a specific tier with execution_mode="medium" (or any tier name) at deploy time. This mapping automatically routes to the appropriate underlying backend and infrastructure.

Updating Settings After Deployment

Runtime tier (execution_mode), schedule, tags, and dependency packages can be changed on a live deployment without redeploying the artifact. Use the Flow Settings dialog in the web UI or the PUT /deployments/{name}/settings API endpoint. Changes apply to new runs only; in-progress runs are not affected. See the deployment guide for details.

Local vs remote

Use local execution (run_local() or dagy run path/to/flow.py:fn) during development. It produces the same artifacts and metadata as remote execution, making it easy to debug before deploying.

Remote execution adds scheduling, distributed retries, centralized CloudWatch logs, and usage metering. Deploy with flow.deploy() or the dagy build + dagy deploy CLI workflow.