Runbook: DynamoDB Throttling

Severity: Sev-2 (usually manifests as API 5xx or stuck runs before it is Sev-1).

Symptoms

• ProvisionedThroughputExceededException / ThrottlingException in API, scheduler, launcher, or worker logs.
• Runs stuck in QUEUED, schedules firing late, or 5xx from write-heavy endpoints.
• CloudWatch ThrottledRequests / ReadThrottleEvents / WriteThrottleEvents climbing on a table or one of its GSIs.

Diagnosis

export ENV=develop REGION=us-east-1

1. Find which tables are throttling (scan the usual hot tables):

   for T in flows runs task-runs schedules rate-limits usage-events; do
     echo "== dagy-$T-$ENV =="
     aws cloudwatch get-metric-statistics --region "$REGION" \
       --namespace AWS/DynamoDB --metric-name ThrottledRequests \
       --dimensions Name=TableName,Value=dagy-$T-$ENV \
       --start-time "$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
       --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
       --period 300 --statistics Sum
   done
   

2. Read vs write throttles on the hottest table:

   aws cloudwatch get-metric-statistics --region "$REGION" \
     --namespace AWS/DynamoDB --metric-name WriteThrottleEvents \
     --dimensions Name=TableName,Value=dagy-runs-$ENV \
     --start-time "$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)" \
     --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --period 300 --statistics Sum
   

3. Check the table's billing mode / capacity:

   aws dynamodb describe-table --region "$REGION" \
     --table-name dagy-runs-$ENV \
     --query 'Table.{Billing:BillingModeSummary.BillingMode,RCU:ProvisionedThroughput.ReadCapacityUnits,WCU:ProvisionedThroughput.WriteCapacityUnits,GSIs:GlobalSecondaryIndexes[].IndexName}'
   

4. GSI hot-partition check — throttles on an index name (not the base table) point at a low-cardinality GSI partition key. Inspect the same metric with Name=GlobalSecondaryIndexName,Value=<gsi> added.

Remediation

• On-demand tables rarely throttle steadily; a burst throttle self-heals as the table adapts. Add jittered client retry (the readiness probe uses max_attempts=0 deliberately, but application clients should retry).
• Provisioned tables → raise capacity or switch to on-demand:

  aws dynamodb update-table --region "$REGION" \
    --table-name dagy-runs-$ENV \
    --billing-mode PAY_PER_REQUEST
  

• Hot partition → the fix is a key-design change (more partition-key cardinality), not a capacity bump. File a follow-up; the migrations runner (python -m dagy_api.persistence.migrations) is the tool for a safe repartition.
• Runaway writer (e.g. a backfill dispatching too fast) → pause it: POST /backfills/{id}/cancel, or throttle the scheduler tick.

Escalation

• Page L2 if throttling persists after capacity change or if it is caused by a hot partition needing a schema migration.
• Note: all tables have PITR enabled, so if a bad migration/backfill caused the load, point-in-time restore is available as a last resort — coordinate with L3 before restoring (it creates a new table).