Glob-based matching and sequential or parallel execution of tasks

[gfmio]

Description

Hi!

Thank you for the fantastic work on task. Ever since discovering it, I've been using it everywhere.

Context / Use Case

One thing I do quite frequently is to use task in monorepos or similar setups in a modular fashion, where I might e.g. have a Taskfile.yml and a Taskfile.root.yml in the project root, a Taskfile.yml inside packages (i.e. at packages/Taskfile.yml) and package-specific Taskfile.ymls in each package (i.e. at packages/*/Taskfile.yml).

I then include all packages/*/Taskfile.yml inside the packages/Taskfile.yml and I include the Taskfile.root.yml and the packages/Taskfile.yml inside the root-level Taskfile.yml.

This is nice and I can do task r:foo and task p:<packageName>:foo and so on.

Pain Point

However, while it's easy and convenient to run one task at a time, it isn't so convenient to run 20 or so. Since I tend to divide my projects into many small sub-projects, that tends to happen quite frequently.

My Proposal / What I've Built

Therefore, I built a small go tool that wraps task and allows me to match tasks with globs and globstars and subsequently execute all of them, either sequentially.

So if I have the following Taskfile.yml defined...

version: "3"

tasks:
  # Pattern matching tasks - must be defined before other tasks
  "p:*":
    desc: "Run tasks matching the pattern in parallel"
    vars:
      PATTERN: '{{index .MATCH 0}}'
      TASKFILE: '{{.TASKFILE}}'
    cmds:
      - ./scripts/task-runner "{{.PATTERN}}" parallel "{{.TASKFILE}}"
    silent: false

  "*":
    desc: "Run tasks matching the pattern"
    vars:
      PATTERN: '{{index .MATCH 0}}'
      TASKFILE: '{{.TASKFILE}}'
    cmds:
      - ./scripts/task-runner "{{.PATTERN}}" sequential "{{.TASKFILE}}"
    silent: false

  foo:
    cmd: echo "foo"

  bar:
    cmd: echo "bar"

  foo:bar:
    cmd: echo "foo bar"

  foo:foo:
    cmd: echo "foo foo"

  bar:foo:
    cmd: echo "bar foo"

  bar:bar:
    cmd: echo "bar bar"

  foo:bar:foo:
    cmd: echo "foo bar foo"

  foo:bar:bar:
    cmd: echo "foo bar bar"

  foo:foo:foo:
    cmd: echo "foo foo foo"

  foo:foo:bar:
    cmd: echo "foo foo bar"

  bar:foo:foo:
    cmd: echo "bar foo foo"

  bar:foo:bar:
    cmd: echo "bar foo bar"

  bar:bar:foo:
    cmd: echo "bar bar foo"

  bar:bar:bar:
    cmd: echo "bar bar bar"

...then the tool will e.g. do:

1. task "foo:*"

› task "foo:*"
task: [foo:*] task-runner "foo:*" sequential "/path/to/my/Taskfile.yml"
🔍 Finding tasks matching: foo:*
📋 Found 2 task(s):
  - foo:bar
  - foo:foo

▶️  Running tasks sequentially...

📌 Running: foo:bar
─────────────────────────────
task: [foo:bar] echo "foo bar"
foo bar
✅ Completed: foo:bar

📌 Running: foo:foo
─────────────────────────────
task: [foo:foo] echo "foo foo"
foo foo
✅ Completed: foo:foo

✅ All tasks completed successfully!

2. task "foo:**"

› task "foo:**"
task: [foo:**] task-runner "foo:**" sequential "/path/to/my/Taskfile.yml"
🔍 Finding tasks matching: foo:**
📋 Found 6 task(s):
  - foo:bar
  - foo:bar:bar
  - foo:bar:foo
  - foo:foo
  - foo:foo:bar
  - foo:foo:foo

▶️  Running tasks sequentially...

📌 Running: foo:bar
─────────────────────────────
task: [foo:bar] echo "foo bar"
foo bar
✅ Completed: foo:bar

📌 Running: foo:bar:bar
─────────────────────────────
task: [foo:bar:bar] echo "foo bar bar"
foo bar bar
✅ Completed: foo:bar:bar

📌 Running: foo:bar:foo
─────────────────────────────
task: [foo:bar:foo] echo "foo bar foo"
foo bar foo
✅ Completed: foo:bar:foo

📌 Running: foo:foo
─────────────────────────────
task: [foo:foo] echo "foo foo"
foo foo
✅ Completed: foo:foo

📌 Running: foo:foo:bar
─────────────────────────────
task: [foo:foo:bar] echo "foo foo bar"
foo foo bar
✅ Completed: foo:foo:bar

📌 Running: foo:foo:foo
─────────────────────────────
task: [foo:foo:foo] echo "foo foo foo"
foo foo foo
✅ Completed: foo:foo:foo

✅ All tasks completed successfully!

3. task p:**:bar:*

› task "p:**:bar:*"
task: [p:**:bar:*] task-runner "**:bar:*" parallel "/path/to/my/Taskfile.yml"
🔍 Finding tasks matching: **:bar:*
📋 Found 6 task(s):
  - bar:bar
  - bar:bar:bar
  - bar:bar:foo
  - bar:foo
  - foo:bar:bar
  - foo:bar:foo

🚀 Running tasks in parallel...
▶️  Starting: foo:bar:foo
▶️  Starting: bar:bar
▶️  Starting: bar:bar:foo
▶️  Starting: bar:bar:bar
▶️  Starting: bar:foo
▶️  Starting: foo:bar:bar
task: [bar:bar:foo] echo "bar bar foo"
task: [foo:bar:bar] echo "foo bar bar"
task: [bar:bar] echo "bar bar"
task: [bar:bar:bar] echo "bar bar bar"
task: [bar:foo] echo "bar foo"
task: [foo:bar:foo] echo "foo bar foo"
bar bar foo
foo bar bar
✅ Completed: foo:bar:bar
✅ Completed: bar:bar:foo
bar bar
bar bar bar
bar foo
foo bar foo
✅ Completed: bar:bar
✅ Completed: bar:bar:bar
✅ Completed: bar:foo
✅ Completed: foo:bar:foo

Note: The reason why the last example didn't match bar is because **:bar:* requires there to be a match after the bar segment. **:bar wouldn't require that and match bar.

I find it quite nifty, so I thought other people might find it useful too.

Interested?

I had a cursory look, but I didn't see that anything like it has been proposed or requested. Apologies if I've missed anything.

Would you be interested in adding this or something like it to task? If so, I'd be happy to create a PR.

I'm also all ears if you have other ideas and suggestions.

The Code

Anyway, here's the code.

Of course, this is just a small script I hacked together for my own purposes. The PR would of course be cleaner and try to match the code style of task.

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"os/exec"
	"regexp"
	"strings"
	"sync"
)

type TaskList struct {
	Tasks []struct {
		Name string `json:"name"`
	} `json:"tasks"`
}

func main() {
	if len(os.Args) < 2 {
		fmt.Println("❌ Error: Please provide a pattern")
		fmt.Println("Usage: task-runner <pattern> [parallel] [taskfile]")
		os.Exit(1)
	}

	pattern := os.Args[1]
	parallel := false
	taskfile := ""

	if len(os.Args) > 2 && os.Args[2] == "parallel" {
		parallel = true
	}
	if len(os.Args) > 3 {
		taskfile = os.Args[3]
	}

	fmt.Printf("🔍 Finding tasks matching: %s\n", pattern)

	// Get all tasks
	tasks, err := getTasks(taskfile)
	if err != nil {
		fmt.Printf("❌ Error getting tasks: %v\n", err)
		os.Exit(1)
	}

	// Filter tasks by pattern
	matched := filterTasks(tasks, pattern)

	if len(matched) == 0 {
		fmt.Printf("⚠️  No tasks found matching pattern: %s\n", pattern)
		os.Exit(0)
	}

	// Display matched tasks
	fmt.Printf("📋 Found %d task(s):\n", len(matched))
	for _, task := range matched {
		fmt.Printf("  - %s\n", task)
	}
	fmt.Println()

	// Run tasks
	if parallel {
		runParallel(matched, taskfile)
	} else {
		runSequential(matched, taskfile)
	}
}

func getTasks(taskfile string) ([]string, error) {
	args := []string{"--list-all", "--json"}
	if taskfile != "" {
		args = append([]string{"--taskfile", taskfile}, args...)
	}

	cmd := exec.Command("task", args...)
	output, err := cmd.Output()
	if err != nil {
		return nil, err
	}

	var taskList TaskList
	if err := json.Unmarshal(output, &taskList); err != nil {
		return nil, err
	}

	tasks := make([]string, 0, len(taskList.Tasks))
	for _, task := range taskList.Tasks {
		tasks = append(tasks, task.Name)
	}

	return tasks, nil
}

func filterTasks(tasks []string, pattern string) []string {
	// Convert glob pattern to regex
	regexPattern := globToRegex(pattern)
	re, err := regexp.Compile(regexPattern)
	if err != nil {
		fmt.Printf("❌ Invalid pattern: %v\n", err)
		os.Exit(1)
	}

	var matched []string
	for _, task := range tasks {
		// Skip our own pattern matching tasks to avoid recursion
		if task == "*" || task == "p:*" {
			continue
		}
		if re.MatchString(task) {
			matched = append(matched, task)
		}
	}

	return matched
}

func globToRegex(pattern string) string {
	// Handle brace expansion {a,b,c} -> (a|b|c)
	pattern = strings.ReplaceAll(pattern, "{", "(")
	pattern = strings.ReplaceAll(pattern, "}", ")")
	pattern = strings.ReplaceAll(pattern, ",", "|")

	// Escape special regex characters (except those we use)
	var result strings.Builder
	result.WriteString("^")

	i := 0
	for i < len(pattern) {
		switch {
		case i < len(pattern)-1 && pattern[i:i+2] == "**":
			// ** matches zero or more path segments
			if i == 0 {
				// **:foo matches "foo", "bar:foo", "bar:baz:foo"
				if i+2 < len(pattern) && pattern[i+2] == ':' {
					result.WriteString("(.*:)?")
					i += 3 // skip "**:"
				} else {
					result.WriteString(".*")
					i += 2 // skip "**"
				}
			} else if i > 0 && pattern[i-1] == ':' {
				// foo:** matches "foo", "foo:bar", "foo:bar:baz"
				if i+2 == len(pattern) {
					result.WriteString(".*")
					i += 2
				} else {
					result.WriteString(".*")
					i += 2
				}
			} else {
				result.WriteString(".*")
				i += 2
			}
		case pattern[i] == '*':
			// * matches any characters except :
			result.WriteString("[^:]*")
			i++
		case pattern[i] == '?':
			// ? matches single character except :
			result.WriteString("[^:]")
			i++
		case pattern[i] == '.':
			// Escape dots
			result.WriteString("\\.")
			i++
		case pattern[i] == '(' || pattern[i] == ')' || pattern[i] == '|':
			// These are already handled by brace expansion
			result.WriteByte(pattern[i])
			i++
		default:
			result.WriteByte(pattern[i])
			i++
		}
	}

	// Special handling for patterns ending with :**
	resultStr := result.String()
	if strings.HasSuffix(pattern, ":**") && !strings.HasSuffix(resultStr, ".*") {
		// This was already handled above
	}

	result.WriteString("$")
	return result.String()
}

func runSequential(tasks []string, taskfile string) {
	fmt.Println("▶️  Running tasks sequentially...")
	var failedTasks []string

	for _, task := range tasks {
		fmt.Println()
		fmt.Printf("📌 Running: %s\n", task)
		fmt.Println("─────────────────────────────")

		if err := runTask(task, taskfile); err != nil {
			fmt.Printf("❌ Failed: %s\n", task)
			failedTasks = append(failedTasks, task)
		} else {
			fmt.Printf("✅ Completed: %s\n", task)
		}
	}

	if len(failedTasks) > 0 {
		fmt.Println()
		fmt.Println("⚠️  Some tasks failed:")
		for _, task := range failedTasks {
			fmt.Printf("  - %s\n", task)
		}
		os.Exit(1)
	} else {
		fmt.Println()
		fmt.Println("✅ All tasks completed successfully!")
	}
}

func runParallel(tasks []string, taskfile string) {
	fmt.Println("🚀 Running tasks in parallel...")
	
	var wg sync.WaitGroup
	results := make(chan string, len(tasks))
	errors := make(chan string, len(tasks))

	for _, task := range tasks {
		wg.Add(1)
		go func(t string) {
			defer wg.Done()
			fmt.Printf("▶️  Starting: %s\n", t)
			if err := runTask(t, taskfile); err != nil {
				errors <- t
				fmt.Printf("❌ Failed: %s\n", t)
			} else {
				results <- t
				fmt.Printf("✅ Completed: %s\n", t)
			}
		}(task)
	}

	wg.Wait()
	close(results)
	close(errors)

	var failedTasks []string
	for task := range errors {
		failedTasks = append(failedTasks, task)
	}

	if len(failedTasks) > 0 {
		fmt.Println()
		fmt.Println("⚠️  Some tasks failed:")
		for _, task := range failedTasks {
			fmt.Printf("  - %s\n", task)
		}
		os.Exit(1)
	}
}

func runTask(task, taskfile string) error {
	args := []string{task}
	if taskfile != "" {
		args = append([]string{"--taskfile", taskfile}, args...)
	}

	cmd := exec.Command("task", args...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	return cmd.Run()
}

[trulede]

It could be an idea, for repos have the "monorepo" pattern, to have task behave like the go command:

$task build ./...

Which then dynamically imports taskfiles, starting from ./ and navigating to all folders under that path ...; and building a considated execution plan ... based on sources/generates of the build task in each found taskfile ... and calling build to realise that (so long that a circular dependency graph does not exist).

This would be somewhat similar to Bazel, perhaps, and a nice simple to use alterntaive (to Bazel) might be an idea.