Composition Metadata

Composition Metadata¶

Prodigy tracks metadata about workflow composition for debugging and dependency analysis. This metadata provides visibility into how workflows are composed, what dependencies exist, and when composition occurred.

Implementation Status: The composition metadata types and dependency tracking are fully implemented in the core composition system. These features are accessible programmatically via the WorkflowComposer API. CLI integration for viewing composition metadata in workflows is under development (Spec 131-133).

CompositionMetadata Structure¶

Every composed workflow includes metadata tracking all composition operations:

Source: src/cook/workflow/composition/mod.rs:153-170

/// Metadata about workflow composition
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CompositionMetadata {
    /// Source files involved in composition
    pub sources: Vec<PathBuf>,

    /// Templates used
    pub templates: Vec<String>,

    /// Parameters applied
    pub parameters: HashMap<String, Value>,

    /// Composition timestamp
    pub composed_at: chrono::DateTime<chrono::Utc>,

    /// Dependency graph
    pub dependencies: Vec<DependencyInfo>,
}

Field Details: - sources: File paths of all workflow files involved in composition (as PathBuf objects) - templates: Template names/sources used during composition - parameters: Final parameter values applied to the workflow (as serde_json::Value) - composed_at: ISO 8601 timestamp when composition occurred (UTC timezone) - dependencies: Complete dependency graph with all imports, extends, templates, and sub-workflows

Dependency Tracking¶

Each dependency includes detailed information:

Source: src/cook/workflow/composition/mod.rs:172-183

/// Information about workflow dependencies
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DependencyInfo {
    /// Source of the dependency
    pub source: PathBuf,

    /// Type of dependency
    pub dep_type: DependencyType,

    /// Resolved path or name
    pub resolved: String,
}

Field Details: - source: Source file path of the dependency (as PathBuf) - dep_type: Type of dependency (Import, Extends, Template, or SubWorkflow) - resolved: Resolved file path or template name (as String)

Dependency Types¶

Prodigy tracks four types of dependencies:

Source: src/cook/workflow/composition/mod.rs:185-193

/// Type of workflow dependency
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum DependencyType {
    Import,
    Extends,
    Template,
    SubWorkflow,
}

Import Dependencies:

imports:
  - path: "shared/utilities.yml"

# Creates DependencyInfo:
# dep_type: DependencyType::Import
# source: PathBuf::from("shared/utilities.yml")
# resolved: "/full/path/to/shared/utilities.yml"

Extends Dependencies:

extends: "base-config.yml"

# Creates DependencyInfo:
# dep_type: DependencyType::Extends
# source: PathBuf::from("base-config.yml")
# resolved: "/full/path/to/base-config.yml"

Template Dependencies:

template:
  source:
    registry: "ci-pipeline"

# Creates DependencyInfo:
# dep_type: DependencyType::Template
# source: PathBuf::from("registry:ci-pipeline")
# resolved: "~/.prodigy/templates/ci-pipeline.yml"

SubWorkflow Dependencies:

sub_workflows:
  - name: "tests"
    source: "workflows/test.yml"

# Creates DependencyInfo:
# dep_type: DependencyType::SubWorkflow
# source: PathBuf::from("workflows/test.yml")
# resolved: "/full/path/to/workflows/test.yml"

Viewing Composition Metadata¶

Note: CLI commands for viewing composition metadata in workflow execution are under development. Currently, metadata can be accessed programmatically via the WorkflowComposer API (see Programmatic Access below).

Future CLI Usage (planned):

# Show composition metadata (planned feature)
prodigy run workflow.yml --dry-run --show-composition

Expected Output:

Composition Metadata:
  Composed at: 2025-01-11T20:00:00Z

  Sources (3):
    - /path/to/workflow.yml
    - /path/to/base-config.yml
    - /path/to/shared/utilities.yml

  Templates (1):
    - registry:ci-pipeline

  Dependencies (3):
    [Import] shared/utilities.yml -> /path/to/shared/utilities.yml
    [Extends] base-config.yml -> /path/to/base-config.yml
    [Template] registry:ci-pipeline -> ~/.prodigy/templates/ci-pipeline.yml

  Parameters (2):
    environment: "production"
    timeout: 600

Programmatic Access¶

Access metadata in code using the WorkflowComposer API:

Source: src/cook/workflow/composition/composer.rs:21-37

use prodigy::cook::workflow::composition::{WorkflowComposer, TemplateRegistry};
use std::collections::HashMap;
use std::sync::Arc;
use std::path::Path;

// Create composer with template registry
let registry = Arc::new(TemplateRegistry::new());
let composer = WorkflowComposer::new(registry);

// Compose workflow with parameters
let params = HashMap::new();
let composed = composer.compose(Path::new("workflow.yml"), params).await?;

// Access metadata
let metadata = &composed.metadata;

// Inspect dependencies
for dep in &metadata.dependencies {
    println!("{:?}: {} -> {}",
        dep.dep_type,
        dep.source.display(),
        dep.resolved
    );
}

// Check composition timestamp
println!("Composed at: {}", metadata.composed_at);

// View final parameters
for (name, value) in &metadata.parameters {
    println!("Parameter {}: {:?}", name, value);
}

// List source files
println!("Sources:");
for source in &metadata.sources {
    println!("  - {}", source.display());
}

// List templates used
println!("Templates:");
for template in &metadata.templates {
    println!("  - {}", template);
}

Real-World Example (from src/cook/workflow/composition/composer.rs:45-51):

// Metadata is created during composition
let mut metadata = CompositionMetadata {
    sources: vec![source.to_path_buf()],
    templates: Vec::new(),
    parameters: params.clone(),
    composed_at: chrono::Utc::now(),
    dependencies: Vec::new(),
};

Dependency Graph Visualization¶

Metadata enables dependency visualization:

workflow.yml
├─ [Extends] base-config.yml
│  └─ [Import] shared/setup.yml
├─ [Import] shared/utilities.yml
└─ [Template] registry:ci-pipeline
   └─ [SubWorkflow] workflows/test.yml

Use Cases¶

Debugging Composition Issues: - Verify which files were loaded - Check parameter resolution order - Identify circular dependencies - Trace inheritance chains

Dependency Analysis: - Find all workflows using a template - Identify shared imports - Map workflow relationships - Audit composition complexity

Change Impact Assessment:

# Before changing base-config.yml, find all dependents
grep -r "extends.*base-config" workflows/

# View composition metadata programmatically
# (CLI integration for --show-composition is under development)

Compliance and Auditing: - Track template versions used - Record composition timestamps - Document parameter sources - Verify configuration origins

Metadata in Composed Workflows¶

Composed workflows carry metadata through the composition process:

Source: src/cook/workflow/composition/mod.rs:143-151

// ComposedWorkflow structure
pub struct ComposedWorkflow {
    /// The composed workflow
    pub workflow: ComposableWorkflow,

    /// Metadata about the composition
    pub metadata: CompositionMetadata,
}

// Access metadata from composed workflow
let composed = composer.compose(source, params).await?;
println!("This workflow was composed from {} sources",
    composed.metadata.sources.len());

Circular Dependency Detection¶

Metadata enables circular dependency detection:

# workflow-a.yml
extends: "workflow-b.yml"

# workflow-b.yml
extends: "workflow-a.yml"

Detection:

Error: Circular dependency detected
  workflow-a.yml -> workflow-b.yml -> workflow-a.yml

Dependency chain:
  1. workflow-a.yml (extends workflow-b.yml)
  2. workflow-b.yml (extends workflow-a.yml) <- Circular!

Parameter Tracking¶

Metadata tracks final parameter values applied during composition:

# workflow.yml
parameters:
  definitions:
    environment:
      type: string
      required: true
    timeout:
      type: integer
      default: 600

Metadata captures:

metadata.parameters = {
    "environment": "production",
    "timeout": 600,
}

The parameters field in CompositionMetadata stores the final resolved parameter values as a HashMap<String, Value>. This enables reproducibility and debugging of composed workflows.

Caching and Performance¶

Composition metadata enables future caching optimizations:

composed_at timestamp can be used for cache invalidation
sources list enables dependency-based cache busting
dependencies graph supports incremental composition
parameters hash can detect identical compositions

Note: Workflow caching is a planned feature. Currently, metadata is generated fresh on each composition.

Data Structure Properties¶

CompositionMetadata uses standard Rust types for broad compatibility:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CompositionMetadata {
    // All fields use standard types
    pub sources: Vec<PathBuf>,           // Cloneable
    pub templates: Vec<String>,          // Cloneable
    pub parameters: HashMap<String, Value>,  // Cloneable
    pub composed_at: chrono::DateTime<chrono::Utc>,  // Copy
    pub dependencies: Vec<DependencyInfo>,   // Cloneable
}

The struct derives Clone, making it easy to share metadata across components without requiring explicit synchronization primitives.

Template System - Template caching and loading
Workflow Extension - Inheritance tracking

Composition Metadata