Github Actions workflows - parallelism, macosx, windows

[dankamongmen]

i wanted to make a single place to discuss this stuff. @igo95862 @michaelsbradleyjr , heads-up

Parallelism

I think our Github Actions workflow could take advantage of parallelism, if any is available to us. Right now, for instance, python wrappers run serially against rust wrappers. They're independent, and ought be able to run in parallel.

It looks like jobs run in parallel by default, but steps are serialized, and furthermore that there are no plans to support parallel steps. We'd have to use backgrounded tasks or GNU parallel or something similar. So maybe this isn't worth it.

What kind of resources are available to us in our free tier?

[igo95862]

There is parallelization built in the actions.

The jobs run in parallel by default but the only way to share data between them is using artifacts https://docs.github.com/en/actions/guides/storing-workflow-data-as-artifacts. I have not looked in to that yet.

So the build phase will be one job, rust and python wrappers will depend on it.

[dankamongmen]

but we don't need to share between jobs, right? just from certain steps to others?

(i think maybe we're using different terminology? i thought the hierarchy was jobs/steps, where we have ubuntu as a job. you seem to be saying it's workflows/jobs, where we have ubuntu as a workflow. is that accurate?)

[igo95862]

Jobs have steps. Steps are always in sequence. If we want to create parallelization we need to put certain actions in to separated jobs. For example, the rust wrappers and python wrappers can be run in parallel so we can to put them in separated jobs. However, they both require compiled library. This means we can compile library in a separated job and use artifact system to upload compiled library to rust and python jobs.