tasks outputs that enrich context of dependent downstream tasks

[osher]

Description

Usecase

Consider a setup where many tasks needs same credentials for different operations.

Therefore, the logical thing is to have a task that reads creds that MUST NOT be persisted on disk from some secret manager, and passes them in-memory to all its dependents.

Current state

The only way I could find how to do it now - is have the "reader" task print something that the downstream stasks have to pass to eval - e.g. a step like - eval $(task creds).

version: '3'

tasks:
  creds:
    cmds:
    - echo "export USR=me; export PWD=top-secret!!;"
   
  operation1:
    cmds:
    - >
       eval $(task creds);
       some-cli-tool -u $USR -p $PWD some operation1

  operation2:
    deps: [creds]
    cmds:
    - >
       eval $(task creds);
       some-cli-tool -u $USR -p $PWD some operation2

That is less than ideal.
Firstly because it is platform-dependent and shell dependent.
Second - if I want to debug something - I should print it to >&2 - and that's also a shell specific thing.
Third - the eval must happen on every command - it is not persisted even on the task level... If I want to pass it to env - it gets very clunky:

tasks:
  creds:
    cmds:
    - vault read ...
   
  operation1:
    vars:
       USR:  { sh: task creds | jq.USR }
       PWD:  { sh: task creds | jq.PWD}
    cmds:
    - some-cli-tool-expects-ENV-vars operation3

this negates all the reuse beause the operation<#>.vars is coppied between all the operations that depend on creds :(

Proposal

I would propose tasks to have "outputs", similar to how outputs work in GitHub actions, just less cumbersome...

e.g:

version: '3'

tasks:
  creds:
    vars:
      JSON:
        sh: aws sts get-parameter .... | jq -r .Parameter.Value
      USR: 
        sh: echo {{.JSON}} | jq .USR
        output: true  # makes $USR available on context of dependent tasks
      PWD: 
        sh: echo {{.JSON}} | jq .PWD
        output: { as: SVC_API_KEY } # makes it available on context of dependent tasks as `SVC_API_KEY`

  operation1:
    deps: [creds]
    cmds:
    - some-cli-tool -u $USR some operation1 # assumes SVC_API_KEY

Mind that on operation1 - it explicitly uses $USR and not {{.USR}} so that the value is not spat to screen.

It would be nice to support secret mode if the scope allows it:

  creds:
    vars:
      JSON:
        sh: aws sts get-parameter .... | jq -r .Parameter.Value
      USR: 
        sh: echo {{.JSON}} | jq .USR
        output: true  # make it available on contexts of dependent tasks
      PWD:
         sh: echo {{.JSON}} | jq .PWD
         output: true
         secret: true  # truncates the secret from being print, replacing it with `<secret>`

         # more suggested forms that worth considering to be supported:
         # (but if it makes the scope too big - it can be dealt in a separate feature PR)

         # secret: { replace: '..shh!!' }  - when print to logs - replaces each secret with `...shh!!` (you control the string)
         # secret: { mask: '*' }  - replaces every character with a `*`, so the length of the secret is visible
         # secret: { peek: 4, replace: true }  - keeps the first `peek` character visible 
         #         and replaces all the rest with `replace` so the output confirms you got the right secret, 
         #         but it is never spat raw.
         # secret: { peek: 4, replace: '<truncated>' }  - same like previous, but you control the replacement string
         # secret: { peek: 4, mask: '*' } - keeps the first `peek` character visible and replaces the rest of characters with `mask`
         #         so the secret can be identified also by the length as an additional confirmation
         #         but it is never spat raw.

[MalteMagnussen]

I have a use-case for this.

I often have to do the following at work, when someone requests a new S3 Bucket from my team:

• Run a tool to create the bucket
• Put the info and credentials into a "Burn on reading" tool for sharing credentials
• Send the link to the "Burn on reading" to the person that requested the bucket

I would like to be able to grab the output from step 1 to post to the pastebin.

And I'd like to get the pastebin link I get from step 2, so I can post to the person on Slack.

[madpipeline]

Here's another scenario that I'm currently stuck on:

• one task to generate/identify a key/tag for docker or git to use
• Multiple other tasks to be able to use that information in their process (tag and push a docker image)

Basically using a task as a function, to better organize the code, and get it's output as an input to other tasks.

[MalteMagnussen]

@madpipeline - Yeah, I basically want to use Task to create functions as well.

That is also why I am interested in #448 as well.

I want to make reproducible, portable "functions" that can be composed to make a workflow.

Things like GitHub actions and GitLab CI exists, but you can't run it locally.

Docker buildx bake with the docker-bake.hcl format is pretty close to a generic task runner, but not quite there yet, since they're stuck with the Dockerfile format, which is intended for building, and thus quite limiting for everything else. Great talk about using Dockerfile for fx linting as well

But I have moved certain tasks into docker-bake + Dockerfile, so I can run the exact same thing locally, as is being ran in CI. And so it is portable, and my team members don't need to install everything locally. It just works with task and docker.

I'm looking for a "universal" task runner, that can create "functions" that are reproducible, portable, and isolated. I want to run things in docker containers, with controlled inputs and outputs. I want to be able to run it locally, or remotely.

I love the Task format, and it is so close to perfect, but not quite there yet.

[osher]

yea. I see this best practice emerging - where pipeline workflows should not include logic, but invoke functions that developers can run locally with relevant context (vars & secrets) - this creates a situation where everything that happens on cloud - can be developed locally as long as the relevant permissions & secrets are available.
However, alas - often the purpose of running a workflow on a runner is that the runner is trusted, where developer machine is not.

Another issue is that pipelines should not include logic because their steps cannot be linted properly.
However - (and that's what's relevant to go-task) - this is the same with logic expressed in Task steps.

IMHO - being able to declare outputs bring us closer to a lint tool for Taskfile.yml that is able to check further than yml structure, because it can understand when a variable is an output of a dependent task, and trust it, feature it in autocomplete, etc.

(that also corresponds with @MalteMagnussen proposition for choosing supported interpreters)

[dragoscirjan]

I'm using this differently, but I really support what @osher wrote.

Being able to dynamically load variables from different sources would be quite useful.

I would go even further and say I would need to be able to load/create variables based on previously defined variables.

version: '3'

tasks:
  creds:
    vars:
      NODE_ENV:
        sh: cat .node_env
      JSON:
        sh: aws sts get-parameter .... | jq -r .Parameter.Value
        depends:
          - NODE_ENV

  operation1:
    deps: [creds]
    cmds:
    - some-cli-tool -u $USR some operation1 # assumes SVC_API_KEY

Or smth alike.

[trulede]

@osher @dragoscirjan @MalteMagnussen @madpipeline

can you not do this?

Taskfile-creds.yml

version: "3"

tasks:
  default:
    cmds:
      - echo "foo"

Taskfile.yml

version: "3"

vars:
  USER: 
    sh: task -s -t Taskfile-creds.yml

tasks:
  default:
    cmds:
      - echo "Cred/User is {{.USER}}"

which yields the expected result, the credential is available to all tasks ...

task -s
Cred/User is foo

Seems like it does what you want, already.

[osher]

@trulede - thank you for your question. I hope we can be very clear about the motive and the benefit of the proposition, so people can connect to it and contribute to the cause.

There are few problems with what you propose.

Your snippet almost repeats the example I gave when I explained it in the ticket text above, but puts the vars in global to the taskfile.
(which means I failed to explain myself 😞)

Let me try better:

---

Please look at this:

tasks:
  creds:
    cmds:
    - vault read ...
   
  operation1:
    vars:
       USR:  { sh: task creds | jq.USR }
       PWD:  { sh: task creds | jq.PWD}
    cmds:
    - some-cli-tool-expects-ENV-vars operation3

problem 1: isolation

The least of it - the way you arranged it makes the USER available on all jobs on the taskfile.
What if different jobs expect different creds under same env var?
That can be solved by putting the vars on the job.

I can also arrange all the tasks that require the same creds in a separate Taskfile but that interferes with how I can organize the code.
i.e. - That's not code organization, but a hack around limitations the proposition aims to solve.

problem 2: efficiency

The second problem is that it resorts to the shell to call task. That is quiet... how to say gently ...unthoughtful. Why should I need a subprocess from task to call task?

A bigger issue is that it exposes a single variable. If I want more - e.g. 5 of them HOST|PORT|DB|USER|PASSWORD - I need to repeat it once per variable.

Not only do I have to enter a shell subprocess to call task - I have to pipe it through jq, or have a task per parameter value...
so ok, I can abstract it away with a variable that names the env - but mind there is only one thing you cannot solve with abstraction - over abstraction. And I need to save my "abstraction budget" to keep the code simple and clear.

problem 3: usability / maintainability

But the greatest problem is that if I need these 5 creds in different jobs, and ONLY these jobs - I have to copy the vars section between these jobs - i.e. repeat it once per variable per consuming job.

In real world corporate production setups - this permutation becomes unmanageable either because it is duplicated, or because it hacks around limitations all over the setup to the point of unclarity.

---

I hope you can see why the current state is anything but elegant.
LMK if I still need to explain it better :)