Runbook: Lambda Error Spike

Severity: Sev-1 for the API or scheduler; Sev-2 for data-plane executors (runs retry / reconcile).

Symptoms

• CloudWatch Errors metric spiking for one or more functions.
• Downstream effects: 5xx (api), late/missing schedules (scheduler), stuck or failed runs (executors/launcher), zombie tasks not cleaned up (reconciler).
• New exceptions in the corresponding log group.

Diagnosis

export ENV=develop REGION=us-east-1

1. Which function is erroring? Sweep all six:

   for F in api scheduler flow-executor executor dag-launcher ecs-reconciler; do
     SUM=$(aws cloudwatch get-metric-statistics --region "$REGION" \
       --namespace AWS/Lambda --metric-name Errors \
       --dimensions Name=FunctionName,Value=dagy-$F-$ENV \
       --start-time "$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
       --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
       --period 1800 --statistics Sum \
       --query 'Datapoints[0].Sum' --output text)
     echo "dagy-$F-$ENV errors(30m)=$SUM"
   done
   

2. Pull the stack traces for the offending function:

   aws logs tail "/aws/lambda/dagy-scheduler-$ENV" --region "$REGION" \
     --since 30m --filter-pattern '?ERROR ?Traceback ?Exception'
   

3. Distinguish error type:

   • Throttles > 0 → concurrency limit (see api-outage for the reserved-concurrency knob).
   • Duration near the configured timeout → hung dependency; check /health/detailed and the DynamoDB runbook.
   • IAM/AccessDenied in traces → a role lost a permission (recent infra change); compare against infrastructure/dagy_stack.py.

4. Correlate with a recent deploy:

   aws lambda get-alias --region "$REGION" \
     --function-name dagy-scheduler-$ENV --name live-$ENV \
     --query 'FunctionVersion'
   

Remediation

• Bad deploy → roll the alias back to the last-good version:

  aws lambda update-alias --region "$REGION" \
    --function-name dagy-<fn>-$ENV --name live-$ENV \
    --function-version <last-good>
  

• AccessDenied → the fix is an IAM policy change in the CDK stack; redeploy the stack. Do not hand-edit the role in the console (drift).
• Scheduler/reconciler run on EventBridge ticks and are idempotent — a transient spike self-heals on the next tick (rate(1 minute) for the scheduler, rate(5 minutes) for the reconciler). Failed tick deliveries land in the events DLQ (see sqs-dlq-backlog).
• Poison event (executor loops on one bad record) → the event source has a max-receive-count of 5, after which the record is dead-lettered; drain/replay per the DLQ runbook.

Escalation

• Page L2 if the spike doesn't clear within one tick interval after remediation, or if the API/scheduler is affected.
• Page L3 for repeated crash-looping that risks run data integrity.