Runbook: Billing / Credit Incident

Severity: Sev-2 (revenue / cost-safety impact, not availability). Escalate to Sev-1 if unmetered spend is actively accruing.

Symptoms

• AI turns rejected with "insufficient credits" for orgs that should have balance, or (worse) turns succeeding with no metering.
• Stripe webhook failing → subscription state not updating after payment; POST /billing/webhook returning 4xx/5xx.
• Usage/billing drift → invoice/usage mismatch, duplicate charges.
• Credit reservation errors in AI logs.

Diagnosis

export ENV=develop REGION=us-east-1

1. Stripe webhook health — the endpoint verifies the signature with STRIPE_WEBHOOK_SECRET and records each event once via a conditional put (attribute_not_exists) against dagy-subscriptions-<env>, so retries are idempotent. Check for failures:

   aws logs tail "/aws/lambda/dagy-api-$ENV" --region "$REGION" --since 1h \
     --filter-pattern '?"/billing/webhook" ?"stripe" ?"signature"'
   

   • Signature verification failed → STRIPE_WEBHOOK_SECRET is wrong/rotated, or a proxy mutated the body. Fix the secret; Stripe will auto-retry.
   • already processed / conditional-check messages are normal dedup.

2. Credit reservation behavior — reservation is fail-closed by design: ConditionalCheckFailedException means genuine insufficient balance (deny is correct); any other error also denies (to avoid free, unmetered Bedrock spend) unless DAGY_AI_STUDIO_ALLOW_UNMETERED is explicitly set.

   aws logs tail "/aws/lambda/dagy-api-$ENV" --region "$REGION" --since 30m \
     --filter-pattern '?"Credit reservation" ?"insufficient balance" ?"fail-closed"'
   

   • A spike of "fail-closed" (non-conditional) errors → the org/credits table is erroring; check the DynamoDB runbook. Turns are being denied, not leaking spend.
   • If you see "ALLOW_UNMETERED set — allowing turn", unmetered spend is accruing — treat as Sev-1 and unset that flag.

3. Stuck reservations — a crash between reserve_credits() and deduct_and_record() can leave credits_reserved elevated. Inspect the org:

   aws dynamodb get-item --region "$REGION" \
     --table-name dagy-organizations-$ENV \
     --key '{"org_id":{"S":"<org_id>"}}' \
     --query 'Item.{balance:credits_balance,reserved:credits_reserved}'
   

4. Usage drift — compare raw events vs aggregates for an org over a window using GET /usage / GET /usage/aggregates and reconcile against Stripe.

Remediation

• Webhook secret → set the correct STRIPE_WEBHOOK_SECRET; Stripe retries failed events automatically for 72h (or replay from the Stripe dashboard).
• Unmetered flag on in production → unset DAGY_AI_STUDIO_ALLOW_UNMETERED and redeploy immediately; audit spend during the exposure window.
• Stuck credits_reserved → after confirming no in-flight turn, release the held amount back to credits_balance (settle path). Do NOT hand-edit balances without an audit-log entry — all mutations are audited.
• Usage drift → prefer re-deriving aggregates from dagy-usage-events-<env> (source of truth) rather than editing aggregates directly.

Escalation

• Page L2 for any subscription-state or usage-drift issue affecting a customer's plan or invoice.
• Page L3 immediately for confirmed unmetered spend, incorrect charges, or credit-balance corruption — these are money-affecting and may need customer comms and a Stripe-side correction.