Runbook: Control-Plane API Outage / 5xx

Severity: Sev-1 if /health is down or 5xx rate is sustained; Sev-2 for partial/intermittent 5xx.

Symptoms

• GET /health returns non-200 or times out.
• Elevated 5xx from API Gateway; users report the dashboard/API failing.
• API Gateway 5XXError metric climbing; Lambda Errors/Throttles climbing.
• Availability SLO (99.9%) burning error budget (see slos.md).

Diagnosis

Set ENV and REGION first:

export ENV=develop REGION=us-east-1

1. Is the API itself up?

   curl -fsS "$DAGY_PUBLIC_API_URL/health" || echo "HEALTH DOWN"
   

2. Recent API Lambda errors and throttles (last 30 min):

   aws cloudwatch get-metric-statistics --region "$REGION" \
     --namespace AWS/Lambda --metric-name Errors \
     --dimensions Name=FunctionName,Value=dagy-api-$ENV \
     --start-time "$(date -u -v-30M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
     --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --period 300 --statistics Sum
   
   aws cloudwatch get-metric-statistics --region "$REGION" \
     --namespace AWS/Lambda --metric-name Throttles \
     --dimensions Name=FunctionName,Value=dagy-api-$ENV \
     --start-time "$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
     --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --period 300 --statistics Sum
   

3. API Gateway 5xx:

   aws cloudwatch get-metric-statistics --region "$REGION" \
     --namespace AWS/ApiGateway --metric-name 5XXError \
     --dimensions Name=ApiName,Value=dagy-api-gateway-$ENV \
     --start-time "$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
     --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --period 300 --statistics Sum
   

4. Tail the API logs for the actual exception:

   aws logs tail "/aws/lambda/dagy-api-$ENV" --region "$REGION" \
     --since 30m --filter-pattern '?ERROR ?Traceback ?"status_code=5"'
   

5. Confirm the alias points where you expect (bad deploy is a common cause):

   aws lambda get-alias --region "$REGION" \
     --function-name dagy-api-$ENV --name live-$ENV
   

6. Dependency health — run the authenticated readiness probe, which actively pings DynamoDB/S3/SQS with per-dependency latency:

   curl -fsS -H "Authorization: Bearer $TOKEN" \
     "$DAGY_PUBLIC_API_URL/health/detailed" | jq .
   

   A red dynamodb/s3/sqs component points you at the corresponding runbook.

Remediation

• Bad deploy → roll the alias back to the previous version:

  aws lambda update-alias --region "$REGION" \
    --function-name dagy-api-$ENV --name live-$ENV \
    --function-version <last-known-good-version>
  

• Throttling (Throttles > 0) → inspect the regional Lambda concurrency quota and current account usage. Reduce competing load or request a quota increase through AWS Service Quotas.
• Dependency-driven → follow the pointed-to runbook (DynamoDB throttling, etc.).
• Malicious/abusive traffic → inspect API Gateway access logs and gateway throttling metrics. API Gateway v2 HTTP APIs cannot be direct regional WAF association targets; add a WAF-protected CloudFront distribution or migrate to a REST API if WAF filtering is required.

Escalation

• Page L2 (platform team) if 5xx persists > 15 min after rollback, or the availability error budget is exhausted.
• Page L3 (eng lead) for a full outage > 60 min or suspected data-plane compromise. Post a status-page update for any confirmed Sev-1.