Back to docs
Operations

Runbook: Scheduler Stall

**Applies to:** `dagy-scheduler-<env>` Lambda, driven by EventBridge rule `dagy-scheduler-tick-<env>` (`rate(1 minute)`), reading `dagy-schedules-<env>` and enqueuing to `dagy-events-queue-<env>`. The scheduler uses `croniter` for next-run computation and a **conditional write** on `next_run_epoch` so a double-tick cannot fire the same schedule twice.

Severity: Sev-2 (schedules fire late/not at all; defends the scheduler punctuality SLO — 95% of schedules fire within 60s of due).

Symptoms

  • next_run_epoch in the past for many schedules but no new runs.
  • Scheduler-punctuality SLO burning error budget (see slos.md).
  • Scheduler Lambda has no recent invocations, or Errors > 0.

Diagnosis

export ENV=develop REGION=us-east-1

  1. Is the tick rule enabled and firing?

    aws events describe-rule --region "$REGION" --name dagy-scheduler-tick-$ENV \
  --query '{State:State,Schedule:ScheduleExpression}'

aws cloudwatch get-metric-statistics --region "$REGION" \
  --namespace AWS/Lambda --metric-name Invocations \
  --dimensions Name=FunctionName,Value=dagy-scheduler-$ENV \
  --start-time "$(date -u -d '15 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 60 --statistics Sum

    Expect roughly one invocation per minute.

  2. Scheduler errors:

    aws logs tail "/aws/lambda/dagy-scheduler-$ENV" --region "$REGION" \
  --since 15m --filter-pattern '?ERROR ?Traceback ?ConditionalCheckFailed'

    Occasional ConditionalCheckFailedException is normal (two ticks racing for the same due schedule; the conditional write correctly lets only one win). A flood of them alongside no runs is not.

  3. Manually invoke one tick to see the counters:

    aws lambda invoke --region "$REGION" --function-name dagy-scheduler-$ENV \
  --payload '{"dagy_event_type":"scheduler_tick","max_due":100}' \
  /tmp/tick.json && cat /tmp/tick.json

    Look for status=processed, checked, triggered, failures.

  4. Are due runs draining downstream? If the scheduler enqueues but runs stay QUEUED, the problem is the launcher/worker, not the scheduler — see ecs-worker-failures and the run-launch-latency SLO.

  5. Failed tick deliveries are dead-lettered — check the DLQ (sqs-dlq-backlog).

Remediation

  • Rule disabled → aws events enable-rule --name dagy-scheduler-tick-$ENV.
  • Lambda erroring / bad deploy → roll the alias back (lambda-error-spike).
  • Backlog after an outage → catchup policy governs replay: catchup_policy=none skips the backlog and recomputes from now; catchup_policy=all replays from the last due timestamp. Temporarily raise max_due in the tick payload to drain a large backlog, and disable noisy schedules (enabled=false) to reduce blast radius.
  • Cron/timezone surprises → validate the expression and timezone; croniter interprets fields in the schedule's local zone.

Deep-dive scenarios (missed ticks, backlog growth, catchup semantics) are in Scheduler Troubleshooting.

Escalation

  • Page L2 if the punctuality SLO is breaching or schedules stay stalled after re-enabling the rule / rolling back.
  • Page L3 for suspected duplicate-fire (would indicate the conditional-write guard regressed) — that is a correctness bug, not just a stall.