Bug: `sources` are checked before running `deps`

[Dom324]

Hello,
firstly thanks for making such a great tool!
I believe I found a bug where sources are checked before deps complete:

  compile_sources:
    deps: [preprocess_sources]
    sources:
      - "./build/**/*.cc"
    generates:
      - "./build/**/*.o"
    cmds:
      - for: sources
        cmd: '{{.CXX}} {{.CPPFLAGS}} -c -o $(basename {{.ITEM}} .cc).o {{.ITEM}}'

  preprocess_sources:
    sources:
      - "./src/**/*.cc"
    generates:
      - "./build/**/*.cc"
    cmds:
      # Preprocess src/**/*.cc into build/**/*.cc
      - cp src/main.cc build/main.cc     # dummy command

The preprocess_sources task generates sources for compile_sources. But this is the output that I get:

❯ ls build                                                 # empty dir
❯ task compile_sources
task: [preprocess_sources] cp src/main.cc build/main.cc    # compile_sources does not run at all
❯ ls build
main.cc                                                    # no main.o
❯ task -f compile_sources                                  # force run
task: [preprocess_sources] cp src/main.cc build/main.cc
task: [compile_sources] clang++ .....                      # compile_sources is run
❯ ls build
main.cc  main.o                                            # main.o exists

What I believe is happening is:

1. compile_sources is called
2. compile_sources checks it's sources, generates checksum
3. Call dependencies
   3.1 preprocess_sources generates .cc files in build/
4. compile_sources does not generate .o files in build/, because build/ was empty when sources were checked

Expected behavior:

1. compile_sources is called
2. Call dependencies
   2.1 preprocess_sources generates .cc files in build/
3. compile_sources checks it's sources, generates checksum
4. compile_sources generates .o files from sources in build/

• Task version: 3.34.1
• Operating system: Ubuntu
• Experiments enabled: No

[living42]

I have the exactly same problem, seens whether a task is up to date is determined before any tasks executes, if its determined after dependency tasks finishs, the problem would be solved.

[trulede]

The purpose of sources is to prevent unnecessary work, a strategy which would be impossible is deps were always run. Could you rearrange your taskfile this way?

build:
  cmds:
    - task: preprocess_sources
    - task: compile_sources

compile_sources:
    sources:
      - "./build/**/*.cc"
    generates:
      - "./build/**/*.o"
    cmds:
      - for: sources
        cmd: '{{.CXX}} {{.CPPFLAGS}} -c -o $(basename {{.ITEM}} .cc).o {{.ITEM}}'

  preprocess_sources:
    sources:
      - "./src/**/*.cc"
    generates:
      - "./build/**/*.cc"
    cmds:
      # Preprocess src/**/*.cc into build/**/*.cc
      - cp src/main.cc build/main.cc     # dummy command

[Dom324]

Deps would not run always - they have their own sources (different than the main task).
Main task is invoked, calls deps. Depending tasks check their own sources and run if needed. Main task checks it's sources and runs only if needed.

I can rearrange my taskfile as you showed (actually I already did something similar), but I created the ticket because I believe the current behavior is very unintuitive - I spent quite some time debugging it and was very surprised when I finally realized what is going on.

[trulede]

You are right. From the code it seems that the deps are run first, then the sources are fingerprinted (glob expanded and then checksum'ed), after which the task runs.

If you run
task compile_sources
aa second time, does it skip the dependency preprocess_sources?

[trulede]

OK, the problem is actually caused by the for which is evaluated when the task is complied, which is before the deps are run. Because there are no sources at that point, there are no commands created by the for.

The rest of the mechanism is actually working as you expect.

[Dom324]

Ah, interesting, I did not think about the for! So it is indeed a bug?
Thanks for the work :)

[trulede]

It could be fixed by compiling the task a second time, if the sources had changed (which is the case here).

The maintainers might have a better idea though ...

[trulede]

This task file has a corrected For/Cmd for testing.

version: '3'

tasks:
  compile_sources:
    deps: [preprocess_sources]
    sources:
      - "./build/*.cc"
    generates:
      - "./build/*.o"
    cmds:
      - for: sources
        cmd: 'touch $(dirname {{.ITEM}})/$(basename {{.ITEM}} .cc).o; echo {{.ITEM}}'

  preprocess_sources:
    sources:
      - "./src/*.cc"
    generates:
      - "./build/*.cc"
    cmds:
      - cp src/main.cc build/main.cc

[lesinigo]

[pre s.] just found today about the existence of this tool; thank you to the author[s]!

I immediately stumbled on the same problem and I'm glad that there's already a commit with a proposed fix.

FWIW, my use case is that I have a task producing a bunch of zip files from different sources and another task that must find them all and use them to deploy updated AWS Lambdas.

Also, here is a simplified Taskfile for demonstrating this issue:

version: '3'
tasks:
  create_deps:
    generates:  # this is not actually needed but I wanted to check if it would make any difference
      - a.dep
      - b.dep
      - c.dep
    cmds:
      - touch a.dep b.deb c.dep
  use_deps:
    deps:
      - create_deps
    sources:
      - "*.dep"
    cmds:
      - for: sources
        cmd: echo {{ .ITEM }}

Use it with a command like: touch previous.dep; task -f use_deps and see that it is only getting the previously existing file previous.dep.

[cwrau]

I'm running into the same problem, is there a fix planned?