Back to docs
Operations

Runbook: Disaster Recovery

**Applies to:** loss or corruption of Dagy's durable state — a deleted/corrupted DynamoDB table, a bad migration/backfill, deleted S3 objects, an AZ failure, or a whole-region impairment.

Severity: Sev-1 (data-loss risk). Page L3 before any destructive restore.

Read first: Disaster Recovery (RPO/RTO) for the targets and the mechanisms. This runbook is the decision tree + procedures. Table-level commands live in the DynamoDB restore runbook.

Conventions

  • Env-suffix every name: substitute develop / staging / production for <env>. Set export ENV=production REGION=us-east-1 before running commands.
  • PITR restores always create a NEW table. You never restore in place — you restore to a scratch table, verify, then repoint the service.
  • Lambdas run behind the live-<env> alias; the runtime reads table names from the DAGY_* env vars on dagy-api-<env> (and the other functions).

Decision tree

Is a single AZ down?            → §1 (transparent; verify, don't restore)
Is one table gone/corrupted?    → §2 PITR restore  (RPO ~5 min, fastest)
Was the loss > PITR window,
  or a table replaced/wiped?    → §3 AWS Backup vault restore (RPO ~24 h)
Are S3 objects deleted/overwritten? → §4 S3 object-version recovery
Is the whole region impaired?   → §5 region-impairment posture

§1 — Single-AZ failure (no restore needed)

Multi-AZ is transparent. Confirm the service is self-healing rather than restoring anything.

# Confirm the VPC exposes public ECS subnets in at least two AZs.
aws ec2 describe-subnets --region "$REGION" \
  --filters Name=tag:Name,Values='*Public*' Name=tag:environment,Values="$ENV" \
  --query 'Subnets[].{Id:SubnetId,AZ:AvailabilityZone,PublicIp:MapPublicIpOnLaunch}'

# API health through the gateway (should stay green as Lambda reschedules).
curl -fsS "https://<api-domain>/health" || echo "API degraded"

There are no NAT gateways in any environment. Lambdas use AWS service networking. Fargate tasks use public subnets and public IPs; verify the surviving subnet still has a default route to the VPC internet gateway.

§2 — DynamoDB PITR restore (primary path)

Fastest recovery for an accidental delete / corruption within the last 35 days. Full commands: DynamoDB restore runbook. Summary:

  1. Restore the affected table to a scratch table at a known-good timestamp: 
    aws dynamodb restore-table-to-point-in-time --region "$REGION" \
  --source-table-name dagy-runs-$ENV \
  --target-table-name dagy-runs-$ENV-restore \
  --restore-date-time 2026-07-12T09:30:00Z
  2. Verify the scratch table's item count / spot-check known keys.
  3. Repoint the service at the restored data — the clean approach is to swap env vars (e.g. point DAGY_RUNS at dagy-runs-$ENV-restore) and redeploy, or copy the restored items back into the original table. Coordinate the repoint with L3; see restore runbook §Repoint.

Deletion protection blocks delete-table. To fully replace a table you must first update-table --no-deletion-protection-enabled, which is a deliberate, logged, L3-gated action.

§3 — Restore from the AWS Backup vault

Use this when the PITR window can't help (loss older than 35 days, or the live table was replaced/wiped). Requires backup_enabled (staging/production).

  1. Find the latest recovery point for the table in the vault: 
    aws backup list-recovery-points-by-backup-vault --region "$REGION" \
  --backup-vault-name dagy-backup-vault-$ENV \
  --by-resource-type DynamoDB \
  --query 'RecoveryPoints[].{Arn:RecoveryPointArn,Created:CreationDate,Status:Status}' \
  --output table
  2. Start a restore to a new table (DynamoDB restores land as a new table name in the metadata): 
    aws backup start-restore-job --region "$REGION" \
  --recovery-point-arn "<recovery-point-arn>" \
  --iam-role-arn "arn:aws:iam::<account>:role/dagy-backup-vault-$ENV" \
  --resource-type DynamoDB \
  --metadata '{"TargetTableName":"dagy-runs-'$ENV'-bkprestore"}'
    Use the backup service-linked role (AWSBackupDefaultServiceRole) or the vault role shown by describe-backup-vault if you scoped a custom one.
  3. Watch the job, then verify and repoint exactly as in §2. 
    aws backup describe-restore-job --region "$REGION" --restore-job-id "<job-id>" \
  --query '{Status:Status,Pct:PercentDone,Created:CreatedResourceArn}'

The production vault denies recovery-point deletion — this is intentional; never try to "clean up" recovery points during an incident.

§4 — S3 object-version recovery

dagy-artifacts-* and dagy-traces-* are versioned, so a deleted/overwritten object is recoverable to RPO 0. See DynamoDB restore runbook §S3 for the list/copy/delete-marker-removal commands.

§5 — Whole-region impairment

There is no hot standby region by default. Posture depends on whether cross-region replication was enabled.

If enable_cross_region_replication was ON (production toggle):

  1. dagy-artifacts-dr-<account>-<dest-region>-$ENV and the traces replica in the destination region hold up-to-date object copies. Confirm replication status: 
    aws s3api head-object --region "$REGION" \
  --bucket dagy-artifacts-<account>-<region>-$ENV --key <key> \
  --query 'ReplicationStatus'
  2. Deploy the CDK stack into the destination region (cdk deploy --context environment=$ENV with region: set to the DR region in <env>.yml), then restore DynamoDB from the latest AWS Backup copy (backup vaults are regional; use a cross-region backup copy if one was configured, otherwise the last S3 export).
  3. Cut DNS / the public API domain over to the DR-region API endpoint.

If CRR was OFF (default):

  • Rebuild is from the latest AWS Backup recovery points and S3 exports, which is an hours-scale RTO with up-to-24 h RPO. Deploy the stack in the DR region, restore tables from backup (§3), and re-hydrate S3 from the export bucket / any available copy. This is the honest limitation documented in dr.md → Next DR steps.

Post-incident

  • File the timeline: what was lost, which mechanism recovered it, wall-clock RTO vs target, and actual RPO (data-loss window).
  • If PITR/backup couldn't meet the target, open a follow-up to close the gap (e.g. enable CRR, add cross-region backup copy, shorten backfill blast radius).
  • Leave scratch/-restore tables in place until the incident is fully closed, then delete them (they cost money and count against table limits).

Escalation

  • L3 before any destructive step (disabling deletion protection, deleting a table, repointing production env vars).
  • Page L3 immediately for any suspected data loss, corruption from a migration/backfill, or a region-scope event.