Workflow Composition

Prodigy enables building complex workflows from reusable components through imports, inheritance, templates, and parameters.

Overview

Workflow composition features: - Imports: Include external workflow definitions - Inheritance: Extend base workflows with extends - Templates: Reusable workflow templates with parameters - Parameters: Type-safe workflow parameterization - Sub-workflows: Nested workflow execution with result passing

graph TD
    Base[Base Workflow<br/>base-ci.yml] --> Child[Child Workflow<br/>specialized-ci.yml]

    Import1[Common Steps<br/>common/ci-steps.yml] --> Main[Main Workflow<br/>main.yml]
    Import2[Deploy Steps<br/>common/deploy.yml] --> Main

    Template[Template<br/>rust-ci-template] --> Instance1[Instance 1<br/>project-a-ci]
    Template --> Instance2[Instance 2<br/>project-b-ci]

    Main --> Sub1[Sub-workflow<br/>build]
    Main --> Sub2[Sub-workflow<br/>deploy]

    style Base fill:#e1f5ff
    style Template fill:#fff3e0
    style Main fill:#f3e5f5
    style Sub1 fill:#e8f5e9
    style Sub2 fill:#e8f5e9

Figure: Workflow composition architecture showing imports, inheritance, templates, and sub-workflows.

Imports

Import workflow definitions from external files:

Best Practice

Use imports to share common steps across multiple workflows. This reduces duplication and ensures consistency across your CI/CD pipelines.

# main-workflow.yml
name: main-workflow

imports:
  - path: "common/ci-steps.yml"    # (1)!
    alias: ci                        # (2)!
  - path: "common/deploy-steps.yml"
    alias: deploy

- shell: "cargo build"
- workflow: ci.test-suite           # (3)!
- workflow: deploy.to-staging       # (4)!

1. Path relative to workflow file location
2. Alias for referencing imported workflows
3. Execute imported workflow using alias
4. Access nested workflows with dot notation

Import Syntax

imports:
  - path: "relative/path/to/workflow.yml"
    alias: workflow_name  # Optional alias for referencing

Import Caching

Imported workflows are cached in memory to avoid reloading the same file multiple times during composition. This improves performance when multiple workflows reference the same imports.

# Source: src/cook/workflow/composition/composer.rs:661-698
# Both workflows below share the same cached import
name: workflow-1
imports:
  - path: "common/steps.yml"
    alias: common

---
name: workflow-2
imports:
  - path: "common/steps.yml"  # Loaded from cache
    alias: common

Inheritance

Extend base workflows to customize behavior:

# base-workflow.yml
name: base-ci
env:
  RUST_BACKTRACE: "1"

- shell: "cargo build"
- shell: "cargo test"

---
# specialized-workflow.yml
name: specialized-ci
extends: "base-workflow.yml"

env:
  EXTRA_FLAG: "true"

# Adds additional steps to base workflow
- shell: "cargo clippy"
- shell: "cargo doc"

Override Behavior

• Environment variables are merged (child overrides parent)
• Steps from parent executed first
• Child can override specific fields

Base Workflow Resolution

When using extends, base workflows are searched in priority order:

# Source: src/cook/workflow/composition/composer.rs:625-642
# Search order:
# 1. bases/{name}.yml
# 2. templates/{name}.yml
# 3. workflows/{name}.yml
# 4. {name}.yml (current directory)

name: my-workflow
extends: "common-steps"  # Searches in order above

This allows organizing base workflows in dedicated directories while maintaining backward compatibility with workflows in the project root.

flowchart TD
    Start[extends: common-steps] --> Check1{bases/<br/>common-steps.yml?}
    Check1 -->|Found| Load1[Load Base Workflow]
    Check1 -->|Not Found| Check2{templates/<br/>common-steps.yml?}

    Check2 -->|Found| Load2[Load Base Workflow]
    Check2 -->|Not Found| Check3{workflows/<br/>common-steps.yml?}

    Check3 -->|Found| Load3[Load Base Workflow]
    Check3 -->|Not Found| Check4{common-steps.yml<br/>current dir?}

    Check4 -->|Found| Load4[Load Base Workflow]
    Check4 -->|Not Found| Error[Error: Base Not Found]

    Load1 --> Merge[Merge with Child]
    Load2 --> Merge
    Load3 --> Merge
    Load4 --> Merge

    Merge --> Result[Composed Workflow]

    style Check1 fill:#e1f5ff
    style Check2 fill:#e1f5ff
    style Check3 fill:#e1f5ff
    style Check4 fill:#e1f5ff
    style Error fill:#ffebee
    style Result fill:#e8f5e9

Figure: Base workflow resolution flow showing priority search order.

Templates

Create reusable workflow templates:

# Source: workflows/example-template.yml
# template: rust-ci-template
name: rust-ci
description: "Standard Rust CI workflow"

parameters:
  target:
    type: string
    required: true
    description: "Build target"
  coverage:
    type: boolean
    default: false
    description: "Run coverage analysis"

- shell: "cargo build --target ${target}"
- shell: "cargo test --target ${target}"

- shell: "cargo tarpaulin"
  when: "${coverage} == true"

Template Detection

Workflow files using composition features (template, imports, extends, workflows, or parameters keywords) are automatically detected by Prodigy during workflow parsing via the is_composable_workflow() function and composed before execution.

Source: src/cook/workflow/composer_integration.rs:44-50

Using Templates

name: my-project-ci
template: "rust-ci-template"

parameters:
  target: "x86_64-unknown-linux-gnu"
  coverage: true

Template Storage

Templates are searched in priority order:

1. Global (~/.prodigy/templates/): Shared across all repositories
2. Project-local (.prodigy/templates/): Repository-specific templates
3. Legacy (templates/): Older project-local templates

Automatic Directory Creation

Template directories are automatically created by the template registry if they don't exist, so you can start using templates immediately.

Source: src/cook/workflow/composer_integration.rs:93-136

Global TemplatesProject TemplatesLegacy Location

# Shared across all projects
~/.prodigy/templates/
├── rust-ci.yml
├── python-ci.yml
└── deploy-k8s.yml

# Repository-specific templates
.prodigy/templates/
├── custom-ci.yml
└── integration-tests.yml

# Older project-local templates
templates/
└── old-workflow.yml

Parameters

Define workflow parameters with type checking:

parameters:
  environment:
    type: string
    required: true
    description: "Deployment environment"
    validation: "environment in ['dev', 'staging', 'prod']"

  replicas:
    type: number
    default: 3
    description: "Number of replicas"

  features:
    type: array
    default: []
    description: "Feature flags to enable"

  config:
    type: object
    required: false
    description: "Additional configuration"

Parameter Types

• string: Text value
• number: Numeric value (integer or float)
• boolean: true/false
• array: List of values
• object: Nested key-value pairs

Parameter Validation

Validation Expressions

Parameter validation uses boolean expressions. If validation fails, the workflow will not execute. Always test validation rules with expected inputs.

parameters:
  port:
    type: number
    validation: "port >= 1024 && port <= 65535"  # (1)!

  environment:
    type: string
    validation: "environment in ['dev', 'staging', 'prod']"  # (2)!

1. Numeric range validation for port numbers
2. String enum validation for allowed environments

Using Parameters

Parameters can be used in commands, environment variables, and merge workflows:

# Source: src/cook/workflow/composition/composer.rs:438-453
parameters:
  environment:
    type: string
    required: true
  api_key:
    type: string
    required: true

# In commands
- shell: "deploy.sh --env ${environment}"
- shell: "kubectl apply -f k8s/${environment}/"

# In environment variables
env:
  DEPLOY_ENV: "${environment}"
  API_KEY: "${api_key}"

# In merge workflows
merge:
  commands:
    - shell: "git merge origin/${environment}-branch"
    - claude: "/validate --env ${environment}"

Sub-Workflows

Execute workflows within workflows:

Modular Execution

Sub-workflows provide modular execution with independent contexts. Use them to break complex workflows into manageable, reusable pieces.

name: main-workflow

sub_workflows:
  build:
    name: build-subworkflow
    - shell: "cargo build --release"
    - shell: "cargo test --release"

  deploy:
    name: deploy-subworkflow
    parameters:
      environment: string               # (1)!
    - shell: "kubectl apply -f k8s/${environment}/"

# Execute sub-workflows
- workflow: build                       # (2)!

- workflow: deploy                      # (3)!
  parameters:
    environment: "staging"

# Access sub-workflow results
- shell: "echo Build completed: ${build.success}"  # (4)!

1. Sub-workflow parameters define expected inputs
2. Execute sub-workflow without parameters
3. Pass parameters to parameterized sub-workflow
4. Access sub-workflow results in parent workflow

Sub-Workflow Features

• Independent execution contexts
• Parameter passing between workflows
• Result capture and access
• Conditional execution

Implementation Status

Sub-workflow execution is currently in development. The data structures and configuration parsing are complete, but the execution integration is still being finalized.

Source: src/cook/workflow/composition/sub_workflow.rs

Template Registry

Manage reusable templates centrally:

# Source: src/cli/args.rs:831-851, src/cli/router.rs:210-233

# Register template
prodigy template register ./templates/rust-ci.yml --name rust-ci

# List available templates
prodigy template list

Template Usage

Templates are used by referencing them in workflow YAML files with the template: field, not via CLI flags. CLI-based parameter passing to templates is planned for a future release.

Registry Structure

~/.prodigy/template-registry/
├── rust-ci/
│   ├── template.yml
│   └── metadata.json
├── python-ci/
│   └── template.yml
└── deploy/
    └── template.yml

Template Metadata

Templates can include metadata:

name: rust-ci-template
version: "1.0.0"
description: "Standard Rust CI workflow with testing and linting"
author: "team@example.com"
tags: ["rust", "ci", "testing"]

parameters:
  # ... parameter definitions

Composition in Action

Here's how the composition features work together in real workflows:

sequenceDiagram
    participant User
    participant Composer
    participant Registry
    participant Executor

    User->>Composer: Load workflow.yml
    Composer->>Registry: Resolve imports
    Registry-->>Composer: Imported workflows
    Composer->>Registry: Resolve base (extends)
    Registry-->>Composer: Base workflow
    Composer->>Composer: Merge base + child
    Composer->>Composer: Interpolate parameters
    Composer->>Composer: Compose sub-workflows
    Composer-->>Executor: Composed workflow
    Executor->>Executor: Execute steps
    Executor->>Executor: Execute sub-workflows
    Executor-->>User: Results

    Note over Composer,Registry: Template caching<br/>optimization
    Note over Executor: Sub-workflow<br/>isolation

Figure: Workflow composition execution sequence showing resolution, merging, and execution phases.

Examples

Modular CI Pipeline

# .prodigy/templates/common-ci.yml
name: common-ci-steps

- shell: "git fetch origin"
- shell: "git diff --name-only origin/main"
  capture_output: changed_files

---
# rust-ci.yml
name: rust-project-ci
extends: ".prodigy/templates/common-ci.yml"

imports:
  - path: ".prodigy/templates/rust-lint.yml"
    alias: lint

- shell: "cargo build"
- workflow: lint.clippy
- shell: "cargo test"

Parameterized Deployment

name: deploy-workflow

parameters:
  environment:
    type: string
    required: true
    validation: "environment in ['dev', 'staging', 'prod']"

  image_tag:
    type: string
    required: true
    description: "Docker image tag to deploy"

  replicas:
    type: number
    default: 3
    validation: "replicas >= 1 && replicas <= 20"

env:
  DEPLOY_ENV: "${environment}"
  IMAGE_TAG: "${image_tag}"
  REPLICAS: "${replicas}"

- shell: "kubectl set image deployment/app app=${IMAGE_TAG}"
- shell: "kubectl scale deployment/app --replicas=${REPLICAS}"
- shell: "kubectl rollout status deployment/app"

Template with Sub-Workflows

name: comprehensive-ci
description: "Full CI/CD pipeline with build, test, and deploy"

parameters:
  deploy_enabled:
    type: boolean
    default: false

sub_workflows:
  build:
    - shell: "cargo build --release"
    - shell: "docker build -t app:latest ."

  test:
    - shell: "cargo test"
    - shell: "cargo clippy"

  deploy:
    parameters:
      environment: string
    - shell: "kubectl apply -f k8s/${environment}/"

# Main workflow
- workflow: build
- workflow: test

- workflow: deploy
  when: "${deploy_enabled} == true"
  parameters:
    environment: "staging"