Parallel Iteration with Foreach

Parallel Iteration with Foreach

Process multiple items in parallel using the foreach command.

Source: Configuration defined in src/config/command.rs:191-211

Basic Foreach

The foreach command uses a nested object structure where the foreach field specifies the input source:

- foreach:
    foreach: ["a", "b", "c"]
    do:
      - shell: "process ${item}"

Input Formats (src/config/command.rs:193-194): - List: Static array of items: ["item1", "item2"] - Command: Shell command that outputs items: "find . -name '*.rs'"

Source: Example from src/cook/execution/foreach_tests.rs:16-21

Dynamic Item Lists

Generate items dynamically from command output. Commands are executed via shell, and output is parsed line-by-line:

- foreach:
    foreach: "find . -name '*.rs'"
    do:
      - shell: "rustfmt ${item}"

Command Output Parsing (src/cook/execution/foreach.rs:201-206): - Output is split into lines - Empty lines are automatically filtered out - Each non-empty line becomes one item

Source: Example from src/cook/execution/foreach_tests.rs:59-64

Variables Available in Foreach

The following variables are automatically available in do block commands (src/cook/execution/foreach.rs:223-225):

• ${item} - Current item value
• ${index} - Zero-based index of current item (0, 1, 2, ...)
• ${total} - Total number of items being processed

Example with Variables:

- foreach:
    foreach: ["test1.rs", "test2.rs", "test3.rs"]
    do:
      - shell: "echo Processing ${item} (${index}/${total})"

This would output:

Processing test1.rs (0/3)
Processing test2.rs (1/3)
Processing test3.rs (2/3)

Parallel Execution

Control parallelism with the parallel field (src/config/command.rs:196-198). It accepts both boolean and numeric values:

Boolean - Default Parallelism:

- foreach:
    foreach: "ls *.txt"
    parallel: true  # Default: 10 concurrent workers
    do:
      - shell: "analyze ${item}"

Important: parallel: true uses a fixed default of 10 concurrent workers, not "all available cores" (src/cook/execution/foreach.rs:81).

Number - Explicit Concurrency Limit:

- foreach:
    foreach: "ls *.txt"
    parallel: 5  # Process 5 items concurrently
    do:
      - shell: "analyze ${item}"

Source: Example from src/cook/execution/foreach_tests.rs:102-114

Choosing Parallelism: - Use parallel: false (default) for sequential processing - Use parallel: true for moderate parallelism (10 workers) - Use parallel: N to specify exact concurrency level - Consider I/O limits, CPU cores, and rate limits when choosing N

Error Handling

Continue processing remaining items on failure:

- foreach:
    foreach: ["test1", "test2", "test3"]
    continue_on_error: true
    do:
      - shell: "run-test ${item}"

Behavior (src/config/command.rs:205-206): - continue_on_error: true - Process all items even if some fail - continue_on_error: false (default) - Stop on first failure - Failed items are tracked and reported in results

Limiting Items

Process only a subset of items:

- foreach:
    foreach: "find . -name '*.log'"
    max_items: 10  # Process first 10 items only
    do:
      - shell: "compress ${item}"

Source: Field definition in src/config/command.rs:209-210

Useful for: - Testing workflows on a small subset - Rate-limiting batch operations - Processing most recent items only

Nested Commands

Each item can execute multiple commands. Both shell and claude commands are supported in do blocks:

- foreach:
    foreach: "cargo metadata --format-version 1 | jq -r '.packages[].name'"
    do:
      - shell: "cargo build -p ${item}"
      - shell: "cargo test -p ${item}"
      - shell: "cargo doc -p ${item}"

Command Types Supported (src/cook/execution/foreach.rs:286-375): - shell - Execute shell commands - claude - Execute Claude commands

Each command in the do block has access to the same variables (${item}, ${index}, ${total}).

Practical Use Cases

Process Multiple Directories:

- foreach:
    foreach: ["frontend", "backend", "shared"]
    parallel: 3
    do:
      - shell: "cd ${item} && npm install"
      - shell: "cd ${item} && npm test"

Batch File Processing:

- foreach:
    foreach: "find src -name '*.rs'"
    parallel: 10
    continue_on_error: true
    do:
      - shell: "rustfmt ${item}"
      - shell: "cargo clippy --manifest-path=${item}"

When to Use Foreach vs MapReduce

Use Foreach when: - Simple iteration over items (< 100 items) - All items processed in the same worktree - No need for checkpoint/resume - Lightweight operations

Use MapReduce when: - Processing many items (100+) - Need checkpoint and resume capability - Need isolated worktrees per item - Complex failure handling and retry logic - Dead letter queue for failed items

See MapReduce for large-scale parallel processing.

Progress Tracking

During execution, a progress bar displays (src/cook/execution/foreach.rs:88): - Current item number - Total items - Processing rate - Estimated time remaining