Disaster Recovery (RPO / RTO)

• Runbook: Disaster Recovery — the decision tree and step-by-step recovery procedures.
• Runbook: DynamoDB Restore — the exact aws CLI commands to restore a table (PITR, AWS Backup, S3 export) and recover S3 object versions.

What DR covers

Dagy's durable state lives in two AWS services:

Store	What it holds
DynamoDB (33 tables)	Flows, deployments, schedules, runs/task-runs, users/orgs/memberships, api-keys, secrets (envelope-encrypted), audit logs, lineage/quality/catalog, the migration ledger, etc.
S3	dagy-artifacts-* (flow specs, dep-packages, run state/logs), dagy-traces-* (exception forensics), dagy-attachments-* (30-day-lifecycle uploads), dagy-frontend-* (static site).

Compute (Lambda, ECS, API Gateway, the VPC) is stateless and reproducible from the CDK stack — a full-region rebuild is a cdk deploy plus data restore, so compute does not have its own RPO. The RPO/RTO targets below are about the data stores.

Targets

Scenario	RPO (max data loss)	RTO (max downtime)	Primary mechanism
Accidental table delete / bad migration / corruption	≈ 5 min (PITR)	< 1 h (restore-to-new-table + repoint)	DynamoDB PITR
S3 object overwrite / delete	0 (all writes versioned)	< 15 min	S3 versioning
Loss of a single Availability Zone	0	< 5 min (auto)	Multi-AZ public-subnet Fargate placement (max_azs >= 2)
Account-level corruption / ransomware of live tables	≤ 24 h (daily AWS Backup)	< 4 h	AWS Backup vault (KMS-encrypted, deletion-protected)
Whole-region impairment	≤ 24 h for S3 (if CRR on), else last export/backup	hours (rebuild in DR region)	S3 cross-region replication (opt-in) + AWS Backup copy

These are the binding production targets. develop runs a leaner posture (see the config table below) and is explicitly not covered by AWS Backup or cross-region replication.

Mechanisms and how they map to the targets

1. DynamoDB Point-in-Time Recovery (PITR) — RPO ≈ 5 min

Every table is created with PointInTimeRecoverySpecification.PointInTimeRecoveryEnabled = true and DeletionProtectionEnabled = true (infrastructure/dagy_stack.py, the all_tables loop). PITR gives continuous per-second restore to any point in the trailing 35 days; effective RPO is ~5 minutes because DynamoDB's restore granularity and the last-write lag bound it there. Restores always create a new table — see the DynamoDB restore runbook. Deletion protection means a table can't be dropped out from under the running service.

2. S3 versioning — RPO 0 for durable objects

dagy-artifacts-*, dagy-traces-*, and dagy-frontend-* are versioned. An overwrite or delete creates a new version / delete-marker; the prior bytes are never destroyed, so object-level RPO is zero. Recover a prior version by listing ListObjectVersions and copying the desired VersionId back (see the restore runbook). dagy-attachments-* is intentionally not versioned (30-day lifecycle, KMS-encrypted) because uploads are ephemeral working data.

3. Multi-AZ, NAT-free VPC — AZ loss is transparent

The stack-managed VPC spans max_azs Availability Zones (minimum 2, enforced in-code). Fargate tasks launch in public subnets with public IPs and a security group that has no ingress rules. Lambdas run outside the VPC and use AWS Lambda service networking. Losing a single AZ removes at most half of the available Fargate placement; Lambda and Fargate reschedule into a surviving AZ, so an AZ failure remains a transparent, RPO-0 event.

4. AWS Backup — daily, independent, immutable copy (RPO ≤ 24 h)

When backup_enabled: true (staging + production), the stack provisions:

• a KMS-encrypted backup vault dagy-backup-vault-<env> (in production, recovery-point deletion is denied by the vault access policy so DR copies can't be wiped);
• a daily backup plan dagy-backup-plan-<env> that snapshots all 33 tables at 05:00 UTC, warm-stores them, transitions long-retention copies to cold (Glacier) storage, and expires them at backup_retention_days (35 in staging, 120 in production);
• a versioned S3 export bucket dagy-backup-export-<account>-<region>-<env> (lifecycle → Glacier after 30 days → expiry) that dynamodb:ExportTableToPointInTime writes a portable, Athena-queryable mirror of table data into.

AWS Backup is the second, independent copy: it survives a table replacement, a bad migration that PITR's window couldn't catch in time, and is deletion-protected against a compromised operator. Restore from the vault via the DR runbook.

5. S3 cross-region replication — whole-region survival (opt-in)

enable_cross_region_replication (default off) is the toggle for whole-region-loss durability. When on, dagy-artifacts-* and dagy-traces-* replicate every object (including delete markers) to a same-account bucket in replication_region (default us-west-2) via a scoped S3 replication role. The destination replica buckets (dagy-artifacts-dr-<account>-<region>-<env> / dagy-traces-dr-<account>-<region>-<env>) must be pre-created in the destination region (CRR requires the destination to exist and be versioned) — this is the one manual pre-req; the CDK emits their names as stack outputs.

Per-environment DR posture

Configured in infrastructure/<env>.yml (dagy: block) and wired through infrastructure/app.py → StackConfig.

Setting	develop	staging	production
max_azs	2	2	2
NAT gateways	0	0	0
DynamoDB PITR + deletion protection	✅	✅	✅
S3 versioning (artifacts/traces/frontend)	✅	✅	✅
backup_enabled (AWS Backup)	❌	✅	✅
backup_retention_days	35	35	120
enable_cross_region_replication	❌	❌	❌ (toggle-ready)

develop deliberately runs without AWS Backup and CRR to stay cheap; PITR + S3 versioning still make it recoverable from operator error.

Testing the DR plan

DR mechanisms are exercised, not assumed:

• Automated (CI): tests/infrastructure/test_dagy_stack.py asserts PITR + deletion protection on every table, artifacts/traces versioning, the backup vault/plan/selection + export bucket when enabled, max_azs configurability, and the CRR configuration when enabled.
• Manual game-day (quarterly, staging): follow the DR runbook end-to-end — restore a table from PITR to a scratch table, restore one recovery point from the backup vault, and recover a deleted S3 object version. Record the wall-clock time and compare to the RTO targets above.

Next DR steps (honest gaps)

• CRR is provisioned but off by default. Turning it on requires pre-creating the destination-region replica buckets (versioned). Until a whole-region-loss RPO is mandated, the DR posture for region impairment is "rebuild in the DR region from the latest AWS Backup / S3 export", which is hours, not minutes.
• DynamoDB global tables (active-active multi-region) are the next step beyond CRR if a near-zero cross-region RPO for the control plane is ever required; not implemented today.