Back to docs
Operations
Runbooks
Operational playbooks for the top Dagy failure modes. Each runbook follows the same shape: **Symptoms → Diagnosis → Remediation → Escalation**.
Conventions
- Environment suffix. Every AWS resource name is env-suffixed by the CDK
NameFactory(infrastructure/dagy_stack.py). Examples below use thedevelopenvironment (e.g.dagy-api-develop). Substitute your environment (develop,staging,production) for<env>everywhere. - Lambda alias. All Lambdas are invoked through the
live-<env>alias, not$LATEST. When you target a function by ARN, use the alias qualifier. - Region. Commands assume
AWS_REGIONis set (e.g.us-east-1). Bucket names embed a compacted region, e.g.dagy-artifacts-<account>-useast1-<env>.
Resource name quick reference (<env> = your environment)
| Component | Name pattern |
|---|---|
| Control-plane API Lambda | dagy-api-<env> (alias live-<env>) |
| Scheduler Lambda | dagy-scheduler-<env> |
| Flow-executor Lambda | dagy-flow-executor-<env> |
| Executor Lambda | dagy-executor-<env> |
| DAG launcher Lambda | dagy-dag-launcher-<env> |
| ECS reconciler Lambda | dagy-ecs-reconciler-<env> |
| API log group | /aws/lambda/dagy-api-<env> |
| Scheduler log group | /aws/lambda/dagy-scheduler-<env> |
| API Gateway access logs | /aws/apigateway/dagy-api-gateway-<env> |
| ECS worker log group | /ecs/dagy-worker-<env> |
| Events queue | dagy-events-queue-<env> |
| Events DLQ | dagy-events-dlq-<env> |
| DLQ-depth alarm | dagy-events-dlq-depth-<env> |
| ECS cluster | dagy-ecs-cluster-<env> |
| ECR worker repo | dagy-worker-<env> |
| Scheduler EventBridge rule | dagy-scheduler-tick-<env> (rate(1 minute)) |
| Reconciler EventBridge rule | dagy-ecs-reconciler-tick-<env> (rate(5 minutes)) |
| Artifacts bucket | dagy-artifacts-<account>-<region>-<env> |
| Attachments bucket | dagy-attachments-<account>-<region>-<env> |
| Traces bucket | dagy-traces-<account>-<region>-<env> |
| Backup vault (staging/prod) | dagy-backup-vault-<env> |
| Backup plan (staging/prod) | dagy-backup-plan-<env> |
| Backup export bucket (staging/prod) | dagy-backup-export-<account>-<region>-<env> |
| Secrets Manager prefix | dagy/* |
| Metrics namespaces | Dagy/Runs, Dagy/AI, Dagy/ECS |
Runbook index
| # | Incident | Runbook |
|---|---|---|
| 1 | Control-plane API outage / 5xx | runbooks/api-outage.md |
| 2 | DynamoDB throttling | runbooks/dynamodb-throttling.md |
| 3 | Lambda error spike | runbooks/lambda-error-spike.md |
| 4 | SQS DLQ backlog | runbooks/sqs-dlq-backlog.md |
| 5 | ECS worker failures | runbooks/ecs-worker-failures.md |
| 6 | Scheduler stall | runbooks/scheduler-stall.md |
| 7 | Billing / credit incident | runbooks/billing-credit-incident.md |
| 8 | Secrets-key rotation | runbooks/secrets-key-rotation.md |
| 9 | Disaster recovery (data loss / region loss) | runbooks/disaster-recovery.md |
| 10 | DynamoDB restore (PITR / backup / export) | runbooks/dynamodb-restore.md |
See also:
- Disaster Recovery (RPO/RTO) — RPO/RTO targets and the DR mechanisms the DR runbooks execute against.
- SLOs & Error Budgets — targets and alarm mapping these runbooks defend.
- Scheduler Troubleshooting — deeper scheduler diagnostics.
- Incident Response — exception-trace forensics.
- ECS Operations — worker task lifecycle.
On-call escalation ladder
- L1 — On-call engineer. Owns triage using these runbooks (first 15 min).
- L2 — Service owner (platform team). Paged if an SLO is breaching its error budget or an incident exceeds 30 minutes.
- L3 — Engineering lead. Paged for data-loss risk, security incidents, or a Sev-1 (full API outage) exceeding 60 minutes.
Open a customer-facing status update once a Sev-1/Sev-2 is confirmed.