Proposal: Merge `vars` and `env`

[pd93]

One of the things I see a lot, is confusion about the differences between variables and environment variables. They essentially serve the same purpose (holding data), but they have a few differences:

• Both can be parsed into Task via the CLI as strings (though they have different syntaxes)
  • TASK_VAR=foo task ... (environment variable)
  • task ... VAR=foo (variable)
• Both can be defined in the Taskfile (using different keywords) at a global or task level.
  • vars can also be set when calling another task, but env cannot.
• vars can be of any type. env only supports strings (though sort of works with other scalar values).
• vars are output using templating syntax and an env is output using shell variable syntax.
  • $FOO (environment variable)
  • {{.FOO}} (variable)
• They have slightly different inheritance/scoping rules.
  • At this point, I'm not even sure what the differences are without investigating properly.
• Environment variables defined on the CLI/shell are available as variables, but ones defined using the env keyword are not.

I'm not convinced that any of these distinctions are necessary and I feel like they add an additional layer of confusion for both new users and people wanting to contribute to the project.

Therefore, I propose removing the env key entirely. vars would then inherit from the system environment and be overridable in Task. If a variable is not a string, it would be output using the stringer version of its value (%v). All of the features below would then apply to vars:

• Variables are only passed into Task via the shell environment (whether that be in an .rc file of a shell command).
  • No more task ... VAR=foo syntax.
• Variables can be defined at a global or task level and passed when calling dependant tasks.
• Variables can be of any type, though they are still limited to strings when passed via the environment variables.
• Variables are output using templating syntax OR shell variable syntax depending on the user's needs.
• Variables have a consistent inheritance/scoping model as defined in Proposal: Variable scope and inheritance #2035.

Before:

version: '3'

env:
  FOO: 'foo'
vars:
  BAR: 'bar'

tasks:
  default:
    cmds:
      - echo '$FOO'
      - echo '{{.FOO}}'
      - echo '$BAR'
      - echo '{{.BAR}}'

foo


bar

After:

version: '3'

vars:
  FOO: 'foo'
  BAR: 'bar'

tasks:
  default:
    cmds:
      - echo '$FOO'
      - echo '{{.FOO}}'
      - echo '$BAR'
      - echo '{{.BAR}}'

foo
foo
bar
bar

One caveat to this proposal is that sometimes temporary variables are needed to perform operations and we don't always want to pollute the environment with these variables. To solve this, we can allow variables to be marked as internal, just like we would with tasks. This makes variables consistent with tasks and fulfills the same purpose that variables previously did.

Since most Task commands execute in a fairly specific context, this shouldn't be needed the majority of the time.

version: '3'

vars:
  FOO: 'foo'
  BAR:
    value: 'bar'
    internal: true
  BAZ:
    sh: 'echo "baz"'
    internal: true
  QUX:
    ref: 'lower .BAZ'
    internal: true

tasks:
  default:
    cmds:
      - 'echo "foo: {{.FOO}}"'
      - 'echo "bar: {{.BAR}}"'
      - 'echo "baz: {{.BAZ}}"'
      - 'echo "qux: {{.QUX}}"'

foo: foo
bar:
baz:
qux:

[andreynering]

Hey @pd93,

Two years ago, I opened an issue proposing something different, but that has the same objetive of reducing the confusion between vars and envs:

• Idea: Consider having functions to access variables and envs #1065

In that proposal, instead of merging vars and envs, we'd be actually making them completely separate. The idea is that env would be exclusively used to make environment variables available to the actual commands. For interpolation and passing data between tasks, users should use vars.

Environment variables defined on the CLI/shell are available as variables, but ones defined using the env keyword are not.

With my proposal, no env would be automatically be recognized as a variable, in any circustance.

Both proposals are valid, and I'm open to discuss both. /cc @vmaerten Any thoughts?

[trulede]

One caveat to this proposal is that sometimes temporary variables are needed to perform operations and we don't always want to pollute the environment with these variables.

This is the crux of the problem. Are vars and envs really the same thing? Or are they actually different.

I've developed a "mental model" that vars are something internal, between tasks, and envs are something that comes from the calling process, and might be forwarded to a called process - typically credentials (or similar). Having envs distinguishable from vars ($foo vs {{.foo}}) would be something too - since differentiating between them is currently not possible.

Also, deciding what to "pass along" becomes important, since it may be necessary to restrict what a called task can discover from the callee task (credential leaks come to mind).

[vmaerten]

Hey @pd93 & @andreynering,

I'll start by quoting you:

Both proposals are valid, and I'm open to discussing both.

Merging envs and vars

I like this approach because it provides a single, unified concept, reducing confusion.
However, I have a question: if we completely remove env, how would we set an environment variable for a specific task? For example, something like this:

version: '3'

tasks:
  deploy:
    env:
      AWS_PROFILE: 'dev'
    cmds:
      - aws s3 ....
      - aws lambda ....
      - aws cloudfront invalid ...

Additionally, it would become harder to distinguish between what comes from vars and what comes from envs, especially since environment variables can also be inherited from .zshrc, .bashrc, etc.

Completely separate envs and vars

I really like this idea. The only "downside" is that we wouldn’t be able to write something like LOG_LEVEL=info task ..., but I think that's just a habit to unlearn.

As @andreynering said, both approaches are valid. My initial preference would be to separate them.

[pd93]

if we completely remove env, how would we set an environment variable for a specific task?

version: '3'

tasks:
  deploy:
    vars:
      AWS_PROFILE: 'dev'
    cmds:
      - aws s3 ....
      - aws lambda ....
      - aws cloudfront invalid ...

@vmaerten It would be exactly the same as now, just using vars instead. You would just need to think of all variables in Task as environment variables, with the benefit of having additional sugar in the templating system (like types).

I don't personally see the need for two separate concepts. They provide mostly the same functionality, but with minor differences. This is where the confusion comes from. I have had multiple people ask me what the differences actually are.

I'm not absolutely against separating them, but I'm not a fan of having to use functions to access variables and env vars as in #1065. It doesn't seem as intuitive as directly referencing one and goes against most templating systems I've used.

---

For reference, GitHub actions does have variables and environment variables, but variables are only inputs. You cannot set them in the action config files. In the configs, you are only able to set variables using the env keyword and you access them in templating by calling {{ env.FOO }}. I would actually be in favour of following this and updating this proposal to merge the two concepts into the env keyword instead of into vars.

[trulede]

if we completely remove env, how would we set an environment variable for a specific task?
@vmaerten It would be exactly the same as now, just using vars instead. You would just need to think of all variables in Task as environment variables

Ugh, does that make things better?

The difficulty here is that you might get a lot of vars being set in the called commands. Normal vars plus whatever the Taskfile developer invents. So you are implicitly promoting internal task variables to environment variables.

I think this creates a different class of problem, one that can't be solved as easily as reading the manual.
Still, that can also be solved too. Its a design question.

{{ env.FOO }}
This is OK. Github also has a secrets mechanism (see workflow reuse) that helps prevent unexpected credential exposure.

I would suggest, perhaps again, that the primary confusion exists because envars and vars are already combined (as so far as variable resolution is concerned) and because of that, it's not possible to tell explicitly if you are referencing one or the other.

[13k]

👍 for the alternative complete separation proposal.

the primary confusion exists because envars and vars are already combined

I agree. And I'd expand on that:

If you consider task runners, like go-task, as middleware in the flow data/configuration -> middleware -> execution, then it's clear go-task needs to produce output that is compatible with consumers, which happens to be processes, which in turn use environment variables and commandline.

So when go-task merges env+vars on input, it loses meta-information on the data (provenance), and makes it harder/confusing to produce compatible output.

There's the temptation to make go-task behave like libraries that take input from several sources (env + file + commandline + HTTP + ...) and merge it all, using priorities, into a bag of key/values. They can do this because final apps don't care about provenance, it just wants to use the values.

I'd argue task runners can't: they need to pass data down to consumers (processes) in the format they accept (env + commandline) and if you're losing provenance, it's gonna be confusing.

[trulede]

@pd93 I think this would help greatly. It mirrors the behavior of bash scripts and simply removes the confusion.

Task starts, all envars are incorporated to the variables (according to the existing mechanisms).

Additional envars are loaded from .dotenv files (according to the existing mechanisms).

Additional vars can be set according to the existing schema.

Tasks and sh commands get the set of vars, either as vars or injected into the environment.

Original env values are available by the existing template function, or a similar notation.

Ideally, the env schema would be preserved, but depreciated to be backwards compatible ... or the behavior controlled by an option extension to the task schema ... rather than an experiment and additional schema, both of which complicate operation.

[twelvelabs]

• Variables are only passed into Task via the shell environment.
  • No more task ... VAR=foo syntax.

I agree the current implementation is confusing, but disagree that merging the two concepts improves matters. I'm much more in favor of strongly separating them per #1065.

Some concerns:

1. I commonly use vars to provide overridable defaults for scripts. As echoed by some of the other commenters, I don't like the idea of every one of those becoming an environment variable... or worse, unintentionally overriding an existing env var. For example, I can totally see someone writing task do:something PATH=some/dir and getting a nasty surprise.

2. I also really dislike losing that task ... key=val syntax, which is much more natural to write then putting the k/v pairs at the beginning of the command. Prefixing commands w/ env vars is ok for infrequent actions, but quickly becomes clunky for things you type everyday. For example:

# this...
$ task test file=some/test/file
# vs. this...
$ file=some/test/file task test