Athena Pipelines Crate

crates/athena-pipelines is the main ownership seam for Athena pipeline definitions, persistence adapters, and request-domain helpers. The crate avoids Actix and AppState, but it now owns both portable DTO/domain logic and the pipeline-specific SQL adapters that work directly against sqlx::PgPool.

What Lives Here Today

• Source, transform, and sink pipeline DTOs.
• Pipeline request payloads and template summaries.
• Persisted template DTOs and record-to-runtime conversion helpers.
• Database-backed template store SQL helpers.
• Database-backed template catalog loaders and /pipelines/templates response contract.
• Pipeline step-log SQL ensure/insert helpers.
• Request-independent execution runner behind fetch/insert runtime traits.
• YAML registry loading for named pipelines.
• Validation for required source/sink pipeline fields.
• Template resolution and inline-override merging.
• Template-name normalization, boot-template shadow checks, and merged template summary policy for YAML plus runtime-managed templates.
• Gateway fetch-body construction for pipeline source stages.
• Row alias rewriting for transformed pipeline output.
• Pipeline execution response contracts shared by /pipelines and /pipelines/simulate.

What Still Stays In athena_rs

• Actix route handlers for /pipelines and /pipelines/templates.
• Gateway auth enforcement and request logging.
• Pool resolution, async spawning, and HTTP error translation around crate calls.
• Concrete source fetch execution through the gateway runtime.
• Concrete sink writes through the Postgres/Supabase/Scylla runtime.

Current Extraction Rule

Use athena-pipelines when code is about pipeline-owned contracts, pipeline data persistence, or request-domain transformation and can run without HttpRequest or AppState.

Keep code in athena_rs when it needs:

• Actix types
• runtime state
• auth or request logging
• actual fetch or insert execution

Near-Term Direction

The next pipeline slices should keep shrinking the route module around the same boundary:

1. Keep moving portable definition parsing and transformation helpers first.
2. Leave thin adapters in src/api/pipelines/* for HTTP/runtime concerns.
3. Keep template policy in the crate when it only decides names, precedence, and summary shaping instead of touching database pools or Actix.
4. Keep pipeline-owned SQL adapters in the crate while leaving pool lookup, runtime orchestration, and route auth in athena_rs.
5. Keep the execution core in the crate when it can target runtime traits instead of HttpRequest or AppState.
6. Move longer-running orchestration only after a dedicated worker/runtime crate owns the pipeline execution shell.