Runbook: SQS DLQ Backlog

Severity: Sev-2 (events are being dropped/delayed but the core control plane is up).

Symptoms

• Alarm dagy-events-dlq-depth-<env> in ALARM.
• Notifications/lineage/side-effects that ride the event bus not firing.
• ApproximateNumberOfMessagesVisible > 0 on the DLQ.

Diagnosis

export ENV=develop REGION=us-east-1
DLQ_URL=$(aws sqs get-queue-url --region "$REGION" --queue-name dagy-events-dlq-$ENV --query QueueUrl --output text)
MAIN_URL=$(aws sqs get-queue-url --region "$REGION" --queue-name dagy-events-queue-$ENV --query QueueUrl --output text)

1. How deep is the DLQ?

   aws sqs get-queue-attributes --region "$REGION" --queue-url "$DLQ_URL" \
     --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
   

2. Confirm the alarm state:

   aws cloudwatch describe-alarms --region "$REGION" \
     --alarm-names dagy-events-dlq-depth-$ENV \
     --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'
   

3. Peek (without deleting) at a sample message to learn why it failed:

   aws sqs receive-message --region "$REGION" --queue-url "$DLQ_URL" \
     --max-number-of-messages 5 --visibility-timeout 0 \
     --message-attribute-names All --attribute-names All
   

   • EventBridge-sourced messages carry the rule name — a scheduler/reconciler tick that keeps failing points at scheduler-stall or lambda-error-spike.
   • Application events carry the event type/payload — correlate with the consumer logs in /aws/lambda/dagy-api-<env> (the SQS mapping targets the API alias).

4. Find the consumer error:

   aws logs tail "/aws/lambda/dagy-api-$ENV" --region "$REGION" \
     --since 1h --filter-pattern '?"events" ?ERROR ?Traceback'
   

Remediation

1. Fix the root cause first (bad deploy → roll back the consumer alias; bad IAM → redeploy stack; poison payload → patch the handler). Replaying before fixing just re-fills the DLQ.
2. Replay DLQ messages back onto the main queue once the consumer is healthy:

   # Drain a batch from the DLQ and re-enqueue to the main queue
   while true; do
     MSG=$(aws sqs receive-message --region "$REGION" --queue-url "$DLQ_URL" \
       --max-number-of-messages 10 --wait-time-seconds 2)
     [ -z "$MSG" ] && break
     echo "$MSG" | jq -c '.Messages[]?' | while read -r m; do
       BODY=$(echo "$m" | jq -r '.Body'); RH=$(echo "$m" | jq -r '.ReceiptHandle')
       aws sqs send-message --region "$REGION" --queue-url "$MAIN_URL" --message-body "$BODY"
       aws sqs delete-message --region "$REGION" --queue-url "$DLQ_URL" --receipt-handle "$RH"
     done
     echo "$MSG" | jq -e '.Messages | length > 0' >/dev/null || break
   done
   

   (Prefer an SQS redrive policy / start-message-move-task in the console for large backlogs.)
3. Truly poison messages that can never succeed → purge only after capturing them for forensics.

Escalation

• Page L2 if the DLQ keeps refilling after a replay (root cause not fixed) or depth is growing.
• Page L3 if dropped events imply customer-visible data loss (e.g. missed billing/usage events).