Loosing variables in non deterministic way

[joeblew999]

I have seen this now for a few days, where when you run task it looses variables.

HTML_DEEP_WHICH is changing under my feet.

task --version
3.43.3

go version
go version go1.24.3 darwin/arm64

I get same issue on windows too btw.

HTML_DEEP_NAME=x.html  task

HTML_DEEP_PATH              /Users/apple/workspace/go/src/github.com/joeblew999/pb-stack/mod/gui-web-native/test
HTML_DEEP_NAME              x.html
HTML_DEEP_WHICH             /x.html

then run it again without changing anything:

HTML_DEEP_PATH              /Users/apple/workspace/go/src/github.com/joeblew999/pb-stack/mod/gui-web-native/test
HTML_DEEP_NAME              x.html
HTML_DEEP_WHICH             /Users/apple/workspace/go/src/github.com/joeblew999/pb-stack/mod/gui-web-native/test/x.html

Either I am doing something wrong ?? non best practice ?

Its simple to show the error:

run.env:

HTML_DEEP_PATH={{.PWD}}/test
HTML_DEEP_NAME=index.html
HTML_DEEP_WHICH={{.HTML_DEEP_PATH}}/{{.HTML_DEEP_NAME}}

taskfile.yml:

# yaml-language-server: $schema=https://taskfile.dev/schema.json

version: '3'

# MUST have this ONLY in root for env variables to work.
dotenv: ['run.env', '{{.ENV}}/.env', '{{.HOME}}/.env']


env:
  # this allow overriding from CLI.
  HTML_DEEP_NAME: '{{.HTML_DEEP_NAME | default "test.html" }}'

var:
  HTML_DEEP_PATH: '{{.HTML_DEEP_PATH | default "{{.PWD}}" }}'
  HTML_DEEP_WHICH: '{{.HTML_DEEP_PATH}}/{{.HTML_DEEP_NAME}}'
  

tasks:
  default:
    cmds:
      - echo ''
      - echo 'HTML_DEEP_PATH              {{shellQuote .HTML_DEEP_PATH}}'
      - echo 'HTML_DEEP_NAME              {{.HTML_DEEP_NAME}}'
      - echo 'HTML_DEEP_WHICH             {{shellQuote .HTML_DEEP_WHICH}}'
      - echo ''
    silent: true

  

[joeblew999]

work around I found to not calculate the HTML_DEEP_WHICH as a variable.

Seems variable expansion is screwed up ?

when you run this its ALWAYS the same result....

task

HTML_DEEP_PATH              /Users/apple/workspace/go/src/github.com/joeblew999/pb-stack/mod/gui-web-native/test
HTML_DEEP_NAME              d.html
HTML_DEEP_WHICH             /Users/apple/workspace/go/src/github.com/joeblew999/pb-stack/mod/gui-web-native/test/d.html

taskfile:

# yaml-language-server: $schema=https://taskfile.dev/schema.json

version: '3'

# MUST have this ONLY in root for env variables to work.
dotenv: ['run.env', '{{.ENV}}/.env', '{{.HOME}}/.env']


env:
  # this allow overriding from CLI.
  HTML_DEEP_NAME: '{{.HTML_DEEP_NAME | default "test.html" }}'

var:
  HTML_DEEP_PATH: '{{.HTML_DEEP_PATH | default "{{.PWD}}" }}'

tasks:
  default:
    cmds:
      - echo ''
      - echo 'HTML_DEEP_PATH              {{shellQuote .HTML_DEEP_PATH}}'
      - echo 'HTML_DEEP_NAME              {{.HTML_DEEP_NAME}}'
      - echo 'HTML_DEEP_WHICH             {{.HTML_DEEP_PATH}}/{{.HTML_DEEP_NAME}}'
      - echo ''
    silent: true

  

[timrulebosch]

Seems like var HTML_DEEP_WHICH is shadowing envar HTML_DEEP_WHICH, and probably that is somehow a factor. But not sure how (is task caching something?). You might try with the experiment "Env Precedence" enabled.

Certainly odd. This would be a good test case.

[trulede]

@joeblew999 var should be vars - once changed ... it worked for me.

That is quite remarkable. The value can change from run to run, or stay stable a while and then change. IIRC tasks are resolved in part with go routines (parallel, for speed up) so could it be possible there is a race condition. Values in the environment of the ´sh´ were always correct. The templated sh commands however, could be one or the other.

[joeblew999]

Face plant on the vars spelling :)

Yep there seems to be a race condition …

When you say “ templated sh “ what do you mean ?

I have large task compositions and so am curious if that’s where I can work around the bug .

Would be fantastic if this was looked into deeper by Andrew .

[trulede]

What you are doing might become confusing, because, the env and var values are not the same - they have the same value only because you assigned them the same value. In use, can be confusing, because the value used in the templating and the env are not the same. My suggestion, avoid ghosting between env and var names, and be explicit with the templating env function if necessary.

The env values were processed correctly and available to the sh command (i.e. env | sort will show the correct values).

The expanded template values in the sh commands '{{...}}' which were calculated from the env variables, since you had the spelling wrong, were not reliable. That is this bug. It's something to do with how env are resolved since no vars are involved. And as soon as you move the resolution to any var or the template string itself ... the problem is resolved.

The problem might be that in the run.env file this line
HTML_DEEP_PATH={{.PWD}}/test
is the failing one, and that .PWD simply has not been calculated yet, sometimes.

I think you are doing fine, it's just a bug.

[joeblew999]

Thanks 🤩

That’s a huge help . It’s very kind of you to I’ve me your insight .

I got it all working in a simple test on Darwin and windows today .

However I reached the same understanding in that I need to name env and vars differently.

I will post a re worked example tomorrow . It may be useful to others too.

Then I will refactor my monster task files .

[trulede]

@joeblew999 I think I know what is happening. Envs are loaded into an ordered map (vars), so they are resolved by the task compiler in the order they are defined (in a Taskfile). However, the envars in the .dotenv file are loaded into a normal map (not ordered), and then added to the ordered map (vars) which means the order of the dotenv vars in the vars map might not match the order the envars were defined in the dotenv file.

If vars reference each other, then they might not resolve correctly. Exactly what we observed.

Also, usage of default here will not work, the default string in interpreted as a literal string, and not resolved. You would need an if/else construct.
HTML_DEEP_PATH: '{{.HTML_DEEP_PATH | default "{{.PWD}}" }}'

@andreynering if you wanted to fix this it's probably easiest to duplicate the necessary code, with modifications, from the module, AFAICS only Read() is used: https://github.com/joho/godotenv/blob/3a7a19020151b45a29896c9142723efe5b11a061/godotenv.go#L88

[joeblew999]

Thank you @trulede for the deep dive .

i got it working in a fashion by testing on windows and mac . Just trial and error for working out how it wants things to be setup .