dotenv: should override Taskfile env: just like the OS's environment does.

[d3dc]

.env files exist to supply a non-default value in a given deployment. If the default value just swallows them, what's the point?

(edit: I really like the tool that's been built and it's almost perfect, but it's driving me crazy when switching context 😅)

[d3dc]

I commented in #482

To me, it seems like the more correct way might be to opt-in to the inheritance.

env:
  PATH: $PATH

I think the closest parallel would be a Dockerfile, where the stacking goes:

• ENV directive
• .env file
• -e flag

[andreynering]

Hi @d3dc,

I'm not sure what is the most expected behavior here. I think hearing opinions from more users would be nice before changing.

In the meantime, you can use the default function (just like recommended by variables) to have the behavior you want:

version: '3'

tasks:
  echo:
    env:
      TEST: '{{default "foobar" .TEST}}'
    cmds:
      - echo $TEST

[d3dc]

@andreynering I want to set environment for all of my tasks. I’m not sure what you’re suggesting addresses my concern.

I want to use the Shell’s environment. I also want to use a dotenv. Why should I specify another call var?

[is-jonreeves]

Personally my "general" expectation with dotenv files is that they provide values that can be overriden by explicitly setting ENVs. This is commonly how it works in Webpack workflows that use dotenv or dotenv-flow. However, this is likely expected/acceptable because the inclusion of the dotenv file is passive (i.e.. the process reads it in by default, rather than the user explicitly applying a file to be used).

If we look at the docker run .. command. When the user actively includes the --env-file option they are explictly requesting it to override the base ENVs inside the container. Additionally though, the inclusion of --env key=value options will again explicitly override anything provided by the dotenv file.

The question worth asking is: Is the dotenv: directive designed to import ENVs into the Taskfile.yaml or into the Task. I think that declared in the global space it probably makes sense that you are making those ENVs available to the Taskfile.yml but not the Tasks themselves. If there is a means to specify the dotenv: on the Task level then maybe those should be automatically included into the Task's shell, that would seem expected to me...

version: '3'

dotenv: ['.env']    # ---> contains FOO=1 & BAR=2

tasks:
  example-optin:
    env:
      FOO: "{{.FOO}}"
    cmds:
      - echo $FOO    # ---> "1"
      - echo $BAR    # ---> ""
  
  example-include:
    dotenv: ['.env']
    cmds:
      - echo $FOO    # ---> "1"
      - echo $BAR    # ---> "2"
  
  example-include-override:
    dotenv: ['.env']
    env:
      BAR: "3"
    cmds:
      - echo $FOO    # ---> "1"
      - echo $BAR    # ---> "3"

For now though, I would prefer to explicitly opt-in if the dotenv: usage can only be in the global space. This is mainly because I have different Tasks in the same file that might be affected by ENVs of others, I basically wouldn't want cross-contamination.

I'm yet to use the dotenv: directive, but reading the documentation here, it looks like this isn't the way its currently implemented. It sounds like all Tasks get the .env file's ENVs by default. But I'm not sure.

Edit.. noticed this related feature request.

[antdking]

Hi @d3dc,

I'm not sure what is the most expected behavior here. I think hearing opinions from more users would be nice before changing.

In the meantime, you can use the default function (just like recommended by variables) to have the behavior you want:

version: '3'

tasks:
  echo:
    env:
      TEST: '{{default "foobar" .TEST}}'
    cmds:
      - echo $TEST

How do we do this for global scoped envs?

Dotenv doesn't seem to be available within Vars, and is disregarded when evaluating the global env.

I know v4 is in the works, but could we see a subset introduced into v3.9 to allow better access to scopes in gotemplates, or a way to have dotenv explicitly override the global scope?

dotenv:
  - file: .env
    override: true

[ghostsquad]

What needs to happen is that vars of all sorts need to define their own expectations, similar to make.

This variable will always have the value bar regardless of if it's set in the environment.

FOO := bar

This variable will have the value of bar only if an existing FOO does not exist (or is empty).

FOO ?= bar

The same thing needs to apply broadly. The typical CLI behavior for config is usually this priority (highest takes precedence):

1. Hard-coded values (intended by the developer to not be overridable)
2. command line args & flags
3. ENV variables
4. Config file
5. Discovery mechanisms (e.g. AWS metadata service for getting EC2 credentials)

I think Task needs to follow something similar.

Vars need to have the option of being considered constant OR overridable by other mechanisms.

I would consider an .env file as being config, which would be lower than explicit ENV vars.

Though, for additional flexibility, Task could also just give access to the various locations a variable could come from, and let the user make the decision in code as well, similar to github actions contexts: https://docs.github.com/en/actions/learn-github-actions/contexts

[nick4fake]

Hi @d3dc,

I'm not sure what is the most expected behavior here. I think hearing opinions from more users would be nice before changing.

In the meantime, you can use the default function (just like recommended by variables) to have the behavior you want:

version: '3'

tasks:
  echo:
    env:
      TEST: '{{default "foobar" .TEST}}'
    cmds:
      - echo $TEST

I am not quite sure how does it relate to dotfiles, default only works with variables.

Is there any workaround? Looks like currently dotenv files can't be used, as there is no way to define defaults (except on some bash level).

Edit: Looks like setting default env on taskfile level, and dotenv on task level works, but it is very verbose

[mjftw]

Hi @d3dc,
I'm not sure what is the most expected behavior here. I think hearing opinions from more users would be nice before changing.
In the meantime, you can use the default function (just like recommended by variables) to have the behavior you want:

version: '3'

tasks:
  echo:
    env:
      TEST: '{{default "foobar" .TEST}}'
    cmds:
      - echo $TEST

How do we do this for global scoped envs?

Dotenv doesn't seem to be available within Vars, and is disregarded when evaluating the global env.

I know v4 is in the works, but could we see a subset introduced into v3.9 to allow better access to scopes in gotemplates, or a way to have dotenv explicitly override the global scope?

dotenv:
  - file: .env
    override: true

Just as a bump, something like this would be really helpful for a problem I'm currently facing too.
In my project we're using direnv to automatically load the environment variables from .env.
This gives us an issue though, since this sets all the variables from .env as shell environment variables before task is run.
Our Taskfile has some tasks that need to use the variables from .env.test (snippet below), but because the .env file was already loaded, it means the variables loaded with dotenv: [".env.test"] are never used (since the .env versions take precedence).

An optional override: true flag here would completely solve the issue.
This would work much like the --override option the dotenv tool has available.

  test:ui:
    desc: Run the Vitest test suite with browser UI
    dotenv: [".env.test"]
    cmds:
      - ...

[skycaptain]

I'm also confused by the current behaviour. Taskfile is primarily a tool our developers use locally and occasionally in CI. I'd expect it to prioritise convenience. In terms of overriding values, I'd anticipate it working from the outside in, as this setup aligns with most users' experiences with other tools. The order, from least to most precedence, should be:

• Default in Taskfile.yml
• dotenv file(s) in reverse order of definition (last file takes precedence, i.e. overrides values from previous files)
• Environment variable (for example, MY_VAR="world" task hello or a variable in the job definition that overrides the .env file in the CI pipeline)

At the moment, we're using dotenv-cli as a workaround, which operates in exactly this manner.

[barrykaplan]

I am finding dotenv essentially useless. If an env was defined when executing task, there seems to be no way to use dotenv to override the value. LIke @mjftw, I use direnv and tend to run task from nested directories that have .envrc and .env, but want to run task from the parent directory and then down into several environemnts each with their own .env context.

Well, I can ensure that the task runs from the parent, but the envs are already polluted by the nested dir .env.

[barrykaplan]

Even trying to fork a new task carries with it all the env

  build:
    desc: Pre argo/kustomize build
    dir: "{{.ROOT_DIR}}"
    cmd: task _build
    
  _build:
     # same as what build used to do

[barrykaplan]

And I can't even raise an error if run from the wrong directory

  build:
    desc: Pre argo/kustomize build
    preconditions:
      - msg: "Must run build from root dir {{.ROOT_DIR}}"
        sh: '[ "$(pwd)" = "{{.ROOT_DIR}}" ]'
    cmds:
      - echo "pwd = $(pwd)"

❯ pwd
/home/bkaplan/dev.hbk/reliasoft/reli.gitops/reli/staging
❯ task build
task: [build] echo "pwd = $(pwd)"
[build] pwd = /home/bkaplan/dev.hbk/reliasoft/reli.gitops/reli

because task will run the taskfile in the parent directory, but all subdirectory .envs are already defined.

[barrykaplan]

strike that, this does it

  build:
    desc: Pre argo/kustomize build
    preconditions:
      - msg: "Must run build from root dir {{.ROOT_DIR}}"
        sh: '[ "{{.USER_WORKING_DIR}}" = "{{.ROOT_DIR}}" ]'
    cmds:
      - echo "pwd = $(pwd)"