Architecture

Data Model

Dagy organizes platform data into domain-specific entities. This document describes the core entities, their relationships, and access patterns.

Entity Overview

Dagy manages 21 entity types organized by domain:

#	Entity	Purpose
1	Flow	Flow definitions (immutable per version)
2	Deployment	Named deployments binding a flow version
3	Run	Flow run executions
4	TaskRun	Individual task executions within a run
5	Schedule	Cron, interval, and one-time schedules
6	User	User accounts
7	AccessToken	Authentication tokens (with TTL expiry)
8	AccessLog	Access audit trail
9	Organization	Organizations/tenants
10	Membership	Org memberships and roles
11	ApiKey	API keys (hashed at rest)
12	DagDraft	Visual builder drafts
13	UsageEvent	Raw usage events
14	UsageAggregate	Aggregated usage counters
15	Subscription	Billing subscriptions
16	AuditLog	Mutation audit trail
17	Secret	Encrypted secrets
18	NotificationChannel	Alert channels
19	AlertRule	Alert rules
20	Environment	Environments (dev/staging/prod)
21	Sensor	Event sensors/triggers

Multi-Tenancy

All entities that store user-facing data include an org_id attribute. The API resolves org_id from the authenticated identity and filters all queries by it, providing data isolation between organizations.

Org-scoped entities: Flow, Deployment, Run, TaskRun, Schedule, DagDraft, UsageEvent, UsageAggregate, Subscription, AuditLog, Secret, NotificationChannel, AlertRule, Environment, Sensor, ApiKey.

Core Execution Entities

Flow

Stores flow definitions (immutable per version).

Field	Type	Description
flow_name	String	Flow identifier (unique key with flow_version)
flow_version	String	Version string (e.g. `"1.0.0"`, `"latest"`)
org_id	String	Organization ID
namespace	String	Hierarchical grouping path (e.g. `data/ingestion`)
tags	Map	Arbitrary key-value metadata (e.g. `{"team": "data-eng"}`)
flow_spec_json	String	Serialized FlowSpec (tasks, edges, outputs)
artifact_s3_uri	String	S3 path to ZIP artifact
status	String	ACTIVE, INACTIVE
deployment_name	String	Associated deployment
default_executor	String	Preferred backend
deployment_version	Number	Auto-incrementing deployment counter
code_hash	String	SHA-256 of flow spec + source code (for change detection)
created_at	String	ISO timestamp
updated_at	String	ISO timestamp

Deployment

Named deployment binding a flow version to a deployment name with optional schedule.

Field	Type	Description
deployment_name	String	Deployment identifier (primary key)
org_id	String	Organization ID
flow_name	String	Reference to Flow
flow_version	String	Pinned flow version
schedule	String	Optional cron expression
default_executor	String	Preferred backend
tags	Map	Arbitrary key-value metadata
status	String	ACTIVE, PAUSED

Run

Tracks individual flow run executions.

Field	Type	Description
run_id	String	UUID (primary key)
org_id	String	Organization ID
run_slug	String	Human-friendly identifier
flow_name	String	Flow being executed
flow_version	String	Flow version
deployment_name	String	Source deployment
status	String	QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED
parameters	Map	Runtime parameters
executor	String	Backend used (lambda, step-functions, ecs)
external_id	String	Backend-specific ID (ARN, task ID)
started_at	String	ISO timestamp
completed_at	String	ISO timestamp
created_at	String	ISO timestamp

TaskRun

Tracks individual task executions within a run.

Field	Type	Description
run_id	String	Parent run ID
task_run_id	String	Unique task run ID
task_name	String	Task name
status	String	PENDING, RUNNING, SUCCEEDED, FAILED, SKIPPED
attempt	Number	Current attempt number
executor	String	Backend that executed this task
started_at	String	ISO timestamp
completed_at	String	ISO timestamp
error_message	String	Failure details
output_ref	String	TaskOutput reference

Schedule

Defines recurring or one-time schedules for flows.

Field	Type	Description
flow_name	String	Associated flow
schedule_id	String	Schedule identifier
org_id	String	Organization ID
mode	String	cron, interval, one_time, manual
enabled	Boolean	Whether schedule is active
cron_expression	String	Cron expression (for cron mode)
interval_seconds	Number	Interval (for interval mode)
timezone	String	IANA timezone
catchup_policy	String	none, all
parameters	Map	Default parameters for triggered runs
next_run_epoch	Number	Next due time (epoch seconds)
next_run_at	String	ISO timestamp
last_triggered_at	String	ISO timestamp
start_at	String	Schedule start boundary
end_at	String	Schedule end boundary

Enterprise Entities

AuditLog

Immutable audit trail of all mutations across the platform.

Field	Type	Description
org_id	String	Organization ID
event_time	String	ISO timestamp + UUID suffix for uniqueness
resource_type	String	flow, secret, member, schedule, channel, etc.
resource_id	String	ID of the affected resource
action	String	create, update, delete
actor_email	String	Who performed the action
before_json	String	State before the change
after_json	String	State after the change
ip_address	String	Client IP address

Secret

Fernet-encrypted secrets with per-environment scoping.

Field	Type	Description
org_id	String	Organization ID
secret_name	String	Secret identifier
encrypted_value	String	Base64-encoded Fernet-encrypted value
environment	String	Optional environment scope (null = org-wide)
created_by	String	Creator's email
created_at	String	ISO timestamp
updated_at	String	ISO timestamp

Environment

Named environments for deployment promotion workflows.

Field	Type	Description
org_id	String	Organization ID
env_name	String	Environment name (dev, staging, production)
default_executor	String	Preferred backend for this environment
config_json	String	Environment-specific configuration
is_protected	Boolean	If true, cannot be deleted
created_at	String	ISO timestamp
updated_at	String	ISO timestamp

Sensor

Event-driven triggers that automatically start flows.

Field	Type	Description
org_id	String	Organization ID
sensor_id	String	Sensor identifier
name	String	Display name
sensor_type	String	s3, webhook, polling
flow_name	String	Flow to trigger
flow_params_json	String	Parameters to pass
config_json	String	Type-specific configuration
enabled	Boolean	Whether sensor is active
webhook_token	String	Auto-generated token (webhook type only)
last_triggered_at	String	ISO timestamp
created_at	String	ISO timestamp

Relationships

Organization (1) ---- (*) Membership ---- (1) User
     |
     +-- (*) Flow ---- (*) Deployment ---- (*) Run ---- (*) TaskRun
     |                       |
     |                       +-- (*) Schedule
     +-- (*) ApiKey
     +-- (*) DagDraft
     +-- (*) Secret
     +-- (*) Environment
     +-- (*) NotificationChannel
     +-- (*) AlertRule
     +-- (*) Sensor
     +-- (*) AuditLog
     +-- (*) UsageEvent
     +-- (1) UsageAggregate (per period)
     +-- (1) Subscription

Access Patterns

Pattern	Entity	Key Fields
Get flow by name + version	Flow	flow_name, flow_version
List flows for org	Flow	org_id filter
Get runs for flow	Run	flow_name
Get due schedules	Schedule	enabled=true, next_run_epoch <= now
List members of org	Membership	org_id
Get audit trail for resource	AuditLog	resource_type + resource_id
List secrets for environment	Secret	org_id, environment filter
Find sensor by webhook token	Sensor	webhook_token filter