Speed up publish CI pipeline

[michael-kerscher]

Currently a run of the publish pipeline is taking around 30 minutes.

Most of the time is spent in the "build all translation" step.

comprehensive-rust/.github/workflows/publish.yml

Lines 55 to 60 in c1773a7

	- name: Build all translations
	run: |
	for po_lang in ${{ env.LANGUAGES }}; do
	.github/workflows/build.sh $po_lang book/$po_lang
	mv book/$po_lang/html book/html/$po_lang
	done

This is the main loop for building the translations. This is currently writing files into the book/<language>/html folder, then moved into book/html/<language>. The build.sh script is heavily modifying the current state of the repository so this is not trivially parallelized in the same step.

Options:

• separate steps by using Githubs job matrix to run each translation as a separate job. Note: this requires all other steps in that job to run multiple times which is wasting resources. Installing mdbook in that environment should be cached properly. The English translation should then also be handled like other languages as this is currently a special case.
  • This also needs to take into account that the generated artifacts need to be used in later steps
• Alternative: Split the translation step into two steps (likely more efficient)
  • (for each language) restore the entire book source as currently and copy all necessary files into a new source/<language>/ directory
  • With all sources in each source/<language> folder, this can be easily parallelized after modifying the build script.

comprehensive-rust/.github/workflows/build.sh

Lines 27 to 28 in c1773a7

	rm -r src/ third_party/
	git restore --source "$(git rev-list -n 1 --before "$pot_creation_date" @)" src/ third_party/ book.toml

[michael-kerscher]

Current effort in refactoring the build.sh script google/mdbook-i18n-helpers#268 and the issue google/mdbook-i18n-helpers#268.

From my perspective the new tool could be an opportunity to create such an independent copy of all sources.

[djmitche]

Do you have any idea what is bounding that step? CPU? Disk bandwidth? I don't know how many cores the runners have, but either of those might mean parallelization within the step will not reduce overall runtime very much.

[michael-kerscher]

@djmitche it should be mostly CPU bound (with some disk spikes when everything is written), but mdbook is only using one core so it can benefit from using more cores (4 in Github CI runners).

I gave my idea a first test run and it seems the critical 25 minutes went down to 11 min. I expected a bit more improvement, target around 7 minutes, so the pdf generation might be more parallel than I expected - this is not properly set up on my system.

[michael-kerscher]

We see that the publish pipeline would benefit from some parallelism and this is possible in an easy way. But I would also like to take this opportunity and align both the grown build and the publish workflow. The publish workflow is very simple and we can easily make this parallel.
The build workflow is currently using the job matrix approach to build each translation in parallel. Making the workflows more complicated by using several different strategies does not feel right.

An interesting bit here is that a huge factor seems to be not the translation effort itself but installing tools.
Maybe it is time to completely refactor the grown pipelines to satisfy the current requirements while optimizing the process.

[mgeisler]

Hey folks, notice that we already run our tests in parallel (with a matrix config as you suggest). So we could adapt that to the publish pipeline as well.

[mgeisler]

The build workflow is currently using the job matrix approach to build each translation in parallel. Making the workflows more complicated by using several different strategies does not feel right.

Ah, I see you found that as well! 😄

The only complication I see is that publishing to GitHub pages requires a single final step: it would download the generated artifacts from each translation, unzip everything into a new directory and then do the publishing.

This is necessary (as far as I know) since publishing is an all or nothing operation. I don't think it's possible to write to just, say, da/ and leave the other files as they were at that point in time.

There are many interesting opportunities for speeding things up: imagine that we create a GitHub release for each translation. When we publish to GitHub pages, we could download the latest release, check if anything has changed, and only regenerate things if so. We would effectively cache the xx/ directories in GitHub releases and reuse them from there.

[michael-kerscher]

Yes, I have seen that and this was my reason for why this might be more complicated

This also needs to take into account that the generated artifacts need to be used in later steps

I did not yet write this in my comment above but I'm thinking about doing this in a way to combine actions/upload-artifact@v4 that uploads each language with https://github.com/actions/download-artifact that is a final job that downloads all languages, merges them appropriately and publishes them together.

note: using the merge-multiple: true parameter

This involves downloading all languages and I'm not sure how to properly reuse the language variable or how to ensure the language list is always in sync

[michael-kerscher]

I just saw in this example that this is exactly what we need

[michael-kerscher]

I think this might work. But somehow installing rust crates like mdbook and others is ultra-slow. While we have some caching it feels like this also needs some thorough look

[michael-kerscher]

The current state of #2772 looks promising but needs a bit of refactoring.