Runbook: DynamoDB Restore

Severity: Sev-1 (data-loss risk). Page L3 before any destructive step.

Parent: Runbook: Disaster Recovery · Targets: DR (RPO/RTO).

Every restore below is non-destructive: it lands in a NEW table, you verify, then you repoint the service. You never restore in place.

export ENV=production REGION=us-east-1 ACCOUNT=<account-id>

Table name reference (<env> = your environment)

All 33 tables are protected by PITR + deletion protection and (in staging/production) by the AWS Backup plan. The commonly-restored ones:

Table	Purpose
dagy-flows-<env>	Flow definitions (org-partitioned)
dagy-deployments-<env>	Deployments
dagy-schedules-<env>	Schedules
dagy-runs-<env> / dagy-task-runs-<env>	Run + task-run state
dagy-users-<env> / dagy-organizations-<env> / dagy-memberships-<env>	Identity
dagy-secrets-<env>	Envelope-encrypted secrets
dagy-audit-logs-<env>	Audit trail
dagy-schema-migrations-<env>	Applied-migration ledger

Enumerate them all:

aws dynamodb list-tables --region "$REGION" \
  --query "TableNames[?ends_with(@, '-$ENV')]" --output table

1 · Restore a table from PITR (point-in-time)

Fastest path for an accidental delete / corruption within the last 35 days.

1. Confirm PITR is on and read the restorable window:

   aws dynamodb describe-continuous-backups --region "$REGION" \
     --table-name dagy-runs-$ENV \
     --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription'
   # -> {EarliestRestorableDateTime, LatestRestorableDateTime, PITR Status: ENABLED}
   

2. Restore to a scratch table at a known-good timestamp (just before the bad write). Use --restore-date-time (UTC) or --use-latest-restorable-time:

   aws dynamodb restore-table-to-point-in-time --region "$REGION" \
     --source-table-name dagy-runs-$ENV \
     --target-table-name dagy-runs-$ENV-restore \
     --restore-date-time 2026-07-12T09:30:00Z
   

3. Wait until the restore is ACTIVE (large tables take minutes to hours):

   aws dynamodb describe-table --region "$REGION" \
     --table-name dagy-runs-$ENV-restore \
     --query 'Table.{Status:TableStatus,Items:ItemCount,Size:TableSizeBytes}'
   

4. Verify — spot-check known keys and compare item counts to expectation:

   aws dynamodb get-item --region "$REGION" \
     --table-name dagy-runs-$ENV-restore \
     --key '{"run_id":{"S":"<known-run-id>"}}'
   

Restored tables come back with PITR disabled; if you promote a restored table to be the live table, re-enable PITR on it:

aws dynamodb update-continuous-backups --region "$REGION" \
  --table-name dagy-runs-$ENV-restore \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

2 · Restore from the AWS Backup vault

Use when the loss is older than the PITR window, or the live table was replaced/wiped. Requires backup_enabled (staging/production). Full flow in the DR runbook §3:

# Find recovery points for the table.
aws backup list-recovery-points-by-backup-vault --region "$REGION" \
  --backup-vault-name dagy-backup-vault-$ENV \
  --by-resource-type DynamoDB \
  --query 'RecoveryPoints[].{Arn:RecoveryPointArn,Created:CreationDate}' --output table

# Start a restore to a new table.
aws backup start-restore-job --region "$REGION" \
  --recovery-point-arn "<recovery-point-arn>" \
  --iam-role-arn "arn:aws:iam::$ACCOUNT:role/service-role/AWSBackupDefaultServiceRole" \
  --resource-type DynamoDB \
  --metadata '{"TargetTableName":"dagy-runs-'$ENV'-bkprestore"}'

3 · Export a table to S3 (portable / Athena-queryable copy)

The stack provisions a versioned export bucket dagy-backup-export-$ACCOUNT-<region>-$ENV and a scoped export role dagy-backup-export-role-$ENV. Trigger an on-demand PITR export (useful for forensic diffing without touching the live table):

REGION_NODASH=$(echo "$REGION" | tr -d '-')
aws dynamodb export-table-to-point-in-time --region "$REGION" \
  --table-arn arn:aws:dynamodb:$REGION:$ACCOUNT:table/dagy-runs-$ENV \
  --s3-bucket dagy-backup-export-$ACCOUNT-$REGION_NODASH-$ENV \
  --s3-prefix exports/dagy-runs \
  --export-format DYNAMODB_JSON \
  --export-time 2026-07-12T09:30:00Z

# Track it.
aws dynamodb list-exports --region "$REGION" \
  --table-arn arn:aws:dynamodb:$REGION:$ACCOUNT:table/dagy-runs-$ENV
aws dynamodb describe-export --region "$REGION" --export-arn "<export-arn>"

The export can be queried in place with Athena, or re-imported into a fresh table with import-table for a full rebuild.

Repointing the service

Once a restored/verified scratch table holds the good data, make the service use it. Two options — coordinate with L3:

• Env-var swap (cleanest, reversible): point the relevant DAGY_* env var (e.g. DAGY_RUNS) at the restored table and redeploy the stack. The runtime reads every table name from env, so no code change is needed. Revert by swapping back.
• Copy items back: for a partial corruption, copy only the affected items from the restored table into the original with batch-write-item (or a small script). Preferred when most of the live table is intact.

To fully replace a table you must first clear deletion protection — a deliberate, logged, L3-gated action:

aws dynamodb update-table --region "$REGION" \
  --table-name dagy-runs-$ENV \
  --no-deletion-protection-enabled

S3 object-version recovery

dagy-artifacts-$ACCOUNT-<region>-$ENV and dagy-traces-$ACCOUNT-<region>-$ENV are versioned — a deleted/overwritten object is recoverable to RPO 0.

1. List the versions (and delete markers) of the affected key:

   aws s3api list-object-versions --region "$REGION" \
     --bucket dagy-artifacts-$ACCOUNT-<region>-$ENV \
     --prefix dagy/specs/<flow>/spec.json \
     --query '{Versions:Versions[].{V:VersionId,Latest:IsLatest,Modified:LastModified}, DeleteMarkers:DeleteMarkers[].{V:VersionId,Latest:IsLatest}}'
   

2. Overwritten object — copy the good version back over the current one:

   aws s3api copy-object --region "$REGION" \
     --bucket dagy-artifacts-$ACCOUNT-<region>-$ENV \
     --key dagy/specs/<flow>/spec.json \
     --copy-source "dagy-artifacts-$ACCOUNT-<region>-$ENV/dagy/specs/<flow>/spec.json?versionId=<good-version-id>"
   

3. Deleted object — remove the delete marker so the prior version becomes current again:

   aws s3api delete-object --region "$REGION" \
     --bucket dagy-artifacts-$ACCOUNT-<region>-$ENV \
     --key dagy/specs/<flow>/spec.json \
     --version-id <delete-marker-version-id>
   

Cleanup

Scratch tables (-restore, -bkprestore) cost money and count against table limits. After the incident is fully closed, delete them (they have PITR off and no deletion protection unless you enabled it):

aws dynamodb delete-table --region "$REGION" --table-name dagy-runs-$ENV-restore

Escalation

• L3 gates every destructive step (disable deletion protection, delete a table, repoint production).
• Note the recovered RPO/RTO in the incident timeline and compare to the DR targets.