Service Level Objectives (SLOs) & Error Budgets

Summary

SLO	Target	Measurement window	Error budget
API availability	99.9% of requests succeed (non-5xx)	30-day rolling	0.1% ≈ 43m 12s/month of allowed error
API latency (p95)	p95 request latency < 500 ms	30-day rolling	≤ 5% of requests may exceed 500 ms
Run-launch latency (p95)	p95 QUEUED → RUNNING < 30 s	7-day rolling	≤ 5% of runs may exceed 30 s
Scheduler punctuality	95% of schedules fire within 60 s of due	7-day rolling	≤ 5% of fires may be late
Async event delivery	DLQ depth stays at 0 (no dropped events)	continuous	any DLQ message consumes budget

An SLO is a promise; the error budget is the amount of failure allowed before we stop shipping risk and prioritize reliability. When a budget is exhausted, freeze non-critical changes to that surface until it recovers.

What each SLO measures and how

1. API availability — 99.9%

• Definition: fraction of API requests that return a non-5xx response.
• Signal: API Gateway 5XXError and Count on dagy-api-gateway-<env>, and Lambda Errors on dagy-api-<env>. Availability = 1 − 5XXError / Count.
• Budget: 0.1%/month ≈ 43m 12s of full unavailability (or an equivalent spread of partial errors).
• Excluded from the denominator: unauthenticated/health probes to /health.
• Runbook: API outage.

2. API latency (p95) — < 500 ms

• Definition: 95th-percentile server-side request latency.
• Signal: API Gateway Latency/IntegrationLatency (p95 statistic) on dagy-api-gateway-<env>; Lambda Duration (p95) on dagy-api-<env>.

3. Run-launch latency (p95) — < 30 s (QUEUED → RUNNING)

• Definition: time from a run entering QUEUED to it reaching RUNNING.
• Signal: derived from run records (dagy-runs-<env> timestamps). The run_started counter and run_duration_seconds metric are emitted under the Dagy/Runs CloudWatch namespace by dagy_api.metrics.emit_run_metrics; a QUEUED-age metric should be added alongside them (see "Gaps" below).
• Common cause of breach: launcher/worker capacity or ECR pull latency — ECS worker failures.

4. Scheduler punctuality — 95% within 60 s

• Definition: fraction of due schedules that fire within 60 s of next_run_epoch.
• Signal: the scheduler runs on dagy-scheduler-tick-<env> at rate(1 minute); punctuality is bounded by tick cadence + processing time. Measure firing lag from dagy-schedules-<env> (next_run_epoch vs actual trigger time) and the scheduler Lambda Duration/Errors on dagy-scheduler-<env>.
• Runbook: Scheduler stall.

5. Async event delivery — DLQ depth 0

• Definition: no events are dropped; failed deliveries must not accumulate.
• Signal + alarm (exists today): the CDK alarm dagy-events-dlq-depth-<env> fires when ApproximateNumberOfMessagesVisible on dagy-events-dlq-<env> is ≥ 1 (Maximum, 5-min period, 1 evaluation period, treatMissingData=notBreaching).
• Runbook: SQS DLQ backlog.

Alarm mapping

SLO	CloudWatch alarm / metric	Status in CDK
Async event delivery	dagy-events-dlq-depth-<env> (SQS ApproximateNumberOfMessagesVisible ≥ 1)	Implemented in infrastructure/dagy_stack.py.
API availability	API Gateway 5XXError / Lambda Errors on dagy-api-<env>	Metrics emitted by AWS; dedicated SLO alarm to be added (see Gaps).
API latency (p95)	API Gateway Latency p95 on dagy-api-gateway-<env>	Metric available; alarm to be added.
Run-launch latency	run_started / run_duration_seconds in Dagy/Runs	App metrics emitted; QUEUED-age metric + alarm to be added.
Scheduler punctuality	dagy-scheduler-<env> Duration/Errors; schedule-lag metric	Lambda metrics available; lag metric + alarm to be added.
AI turn cost/latency	ai_turn, ai_turn_latency_seconds, ai_credits in Dagy/AI	App metrics emitted (informational, no SLO alarm).
ECS worker	Dagy/ECS namespace + task stop reasons	App/ECS metrics; alarm to be added.

CloudWatch metric namespaces emitted by the app (see src/dagy_api/metrics.py): Dagy/Runs (run lifecycle), Dagy/AI (AI turns), Dagy/ECS (worker, via the task role's scoped PutMetricData).

Gaps / follow-up (honesty note)

Today the CDK stack ships one SLO-defending alarm — the events DLQ-depth alarm. The remaining SLOs are measurable from metrics that already exist (AWS service metrics + the Dagy/Runs/Dagy/AI/Dagy/ECS custom metrics) but do not yet have dedicated CloudWatch alarms or a burn-rate dashboard. To fully close this SLO framework:

1. Add CloudWatch alarms for API 5XXError ratio, API p95 Latency, scheduler error/lag, and run-launch latency, named with the NameFactory pattern (e.g. dagy-api-availability-<env>, dagy-api-latency-p95-<env>, dagy-scheduler-lag-<env>, dagy-run-launch-latency-<env>).
2. Emit a run_queued_age_seconds metric at the QUEUED→RUNNING transition and a schedule_fire_lag_seconds metric at trigger time so latency/punctuality SLOs are measured directly rather than derived.
3. Wire multi-window burn-rate alerts (fast + slow) into a notification channel.

These are infra/metric follow-ups (owned by the infra + observability tracks); this page is the source of truth for the targets they must enforce.