Runbook: Secrets-Key Rotation

Severity: planned change (Sev-2 if rotating due to suspected key compromise).

When to rotate

• Suspected exposure of DAGY_SECRETS_KEY (compromise → treat as Sev-1/2).
• Routine rotation policy.

Important: Fernet here has no key-version envelope — a single active key encrypts and decrypts everything. Simply swapping the env var to a new key makes all existing ciphertext undecryptable. Rotation therefore requires re-encrypting every stored secret. Plan a maintenance window.

Diagnosis / pre-checks

export ENV=develop REGION=us-east-1

1. Confirm current secrets decrypt (the app must have the current key set):

   curl -fsS -H "Authorization: Bearer $TOKEN" \
     "$DAGY_PUBLIC_API_URL/secrets" | jq '.[].secret_name'
   # Reading a secret value exercises decrypt with the current key.
   

2. Inventory scope — how many secrets across how many orgs:

   aws dynamodb scan --region "$REGION" --table-name dagy-secrets-$ENV \
     --select COUNT
   

3. Generate a new Fernet key (keep it secret):

   uv run python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
   

Remediation (rotation procedure)

Because there is no dual-key window in code, perform a re-encrypt migration with both keys available to the migration process:

1. Freeze writes to secrets for the window (announce; optionally disable the secrets write endpoints or coordinate with tenants).

2. Snapshot for safety — the table has PITR; note the pre-rotation timestamp so you can restore if the migration fails:

   aws dynamodb describe-continuous-backups --region "$REGION" \
     --table-name dagy-secrets-$ENV \
     --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription'
   

3. Re-encrypt every row: read with the OLD key → decrypt → encrypt with the NEW key → write back. Run this as a one-off migration with both keys in the environment (OLD as DAGY_SECRETS_KEY for the read/decrypt, NEW for the re-encrypt). The DynamoDB migration runner (python -m dagy_api.persistence.migrations) is the sanctioned framework for a versioned, resumable backfill of this shape — add a migration that performs the re-encrypt so it is auditable and idempotent, then:

   python -m dagy_api.persistence.migrations status --env $ENV
   python -m dagy_api.persistence.migrations run    --env $ENV --dry-run
   python -m dagy_api.persistence.migrations run    --env $ENV
   

4. Flip the live key — update DAGY_SECRETS_KEY to the NEW key on the API Lambda and redeploy (the value comes from the stack config / secret store, not a hand-edit).

5. Verify — read several secrets across multiple orgs; they must decrypt with only the NEW key present.

6. Retire the OLD key — remove it from all locations and rotate any place it was stored.

Escalation

• Page L2 to schedule/own the maintenance window and the migration.
• Page L3 for a suspected key compromise: in addition to rotation, rotate every downstream credential those user secrets protected (they must be assumed leaked), and open a security incident.