Operations

Runbook: ECS Worker Failures

**Applies to:** the Fargate execution backend. Runs execute as tasks in cluster `dagy-ecs-cluster-<env>`, from the `dagy-worker-<env>` ECR image, logging to `/ecs/dagy-worker-<env>` (one stream per `run_id`). Tasks use `dagy-ecs-exec-role-<env>` (pull image + inject secrets) and `dagy-ecs-task-role-<env>` (the container's runtime data access).

Severity: Sev-2 (runs fail/stall; retries and the reconciler mitigate).

Symptoms

Runs stuck in QUEUED/RUNNING or flipping to FAILED right after launch.
Tasks stopping with a non-zero exit or a STOPPED (CannotPull…) reason.
Reconciler marking runs as zombies (see below).

Diagnosis

export ENV=develop REGION=us-east-1
CLUSTER=dagy-ecs-cluster-$ENV

List recently stopped tasks and their stop reason (the single most useful signal — pull failures, OOM, and role errors all show here):

for ARN in $(aws ecs list-tasks --region "$REGION" --cluster "$CLUSTER" \
    --desired-status STOPPED --query 'taskArns[:10]' --output text); do
  aws ecs describe-tasks --region "$REGION" --cluster "$CLUSTER" --tasks "$ARN" \
    --query 'tasks[0].{stopped:stoppedReason,exit:containers[0].exitCode,reason:containers[0].reason}'
done

Common stop reasons and what they mean:

Stop reason	Cause	Fix
`CannotPullContainerError`	ECR image missing / exec-role can't pull	Confirm `dagy-worker-<env>:latest` exists; check `dagy-ecs-exec-role-<env>`.
`ResourceInitializationError … secretsmanager`	secret ARN missing or exec-role lacks access	Check the `dagy/*` secret exists and role policy.
`OutOfMemoryError` / exit 137	task memory too low	Raise task memory.
Non-zero app exit	user code / node failure	Read the run log stream.

Read the run's log stream (run_id = stream name):

aws logs tail "/ecs/dagy-worker-$ENV" --region "$REGION" --since 1h \
  --log-stream-names "<run_id>"

Networking / capacity — CannotPull or ENI errors can also be egress. The NAT-free topology launches Fargate with a public IP. Confirm the selected subnet is public, has a default route to the internet gateway, and the task security group permits outbound HTTPS.
Custom worker metrics are emitted under the Dagy/ECS CloudWatch namespace by the task role's cloudwatch:PutMetricData grant.

Remediation

Image/pull → republish the worker image (infrastructure/publish_worker_image.sh) and confirm the latest tag.
Secrets → create/repair the dagy/* Secrets Manager entry; the exec-role grants secretsmanager:GetSecretValue on dagy/* only.
OOM → bump memory in the task definition and redeploy.
Zombie runs → the dagy-ecs-reconciler-<env> Lambda runs every rate(5 minutes) and reconciles runs whose ECS task died without a terminal update. If a run is wedged, wait one reconciler tick, then retry the run (POST /runs/{run_id}/retry) or re-launch.

Escalation

Page L2 if a class of runs fails consistently (image, IAM, or network) or the reconciler itself is erroring (lambda-error-spike).
Page L3 if worker failures correlate with a suspected cross-tenant data issue — note that S3 access is scoped by application path prefixes, not per-org IAM, so tenant isolation depends on org_id keys (see ECS Fargate Execution).