More internal docs

.gitignore

@@ -43,3 +43,5 @@
 /src/library/zip/src/tools/cmdzip
 /tools/build/linux/pak_*.tar.gz
 /tools/build/linux/*.done
+/vignettes/internals.html
+/vignettes/internals.md

vignettes/internals.Rmd → vignettes/internals.qmd

@@ -1,10 +1,9 @@
 ---
 title: "pak internals"
-output: github_document
-editor_options:
-  markdown:
-    wrap: sentence
-    canonical: true
+format:
+  gfm:
+    toc: true
+    toc-depth: 2
 ---
 
 # The embedded library
@@ -40,7 +39,7 @@ order, into `/library` in the package we are building.
 * The source package is bigger. Still not very big.
 * pak always needs a compiler now when it is being installed from source.
 
-## Undating embedded packages
+## Updating embedded packages
 
 The `embed.R` file has some tools for
 * listing the versions of the embedded packages,
@@ -65,6 +64,14 @@ again, with a special argument, to install dependencies into the user
 cache library. If the same version of a dependency is already installed,
 then that dependency is skipped, to make `load_all()` fairly quick.
 
+## Embedding dev packages from GitHub
+
+There is no support for packages from GitHub currently, that needs to
+be done manually:
+- package has to be added manually,
+- it has to be updated manually.
+I might add some support for them later, when I start to use them.
+
 ## Cross compilation
 
 pak supports cross-compilation now. You need to pass `--host=<platform>`
@@ -160,6 +167,12 @@ Othewise it runs daily:
 
 ## GitHub Packages
 
+All builds (via `Makefile`) upload the built packages to GitHub Packages:
+<https://github.com/r-lib/pak/pkgs/container/pak>.
+They also update `manifest.json` at
+<https://github.com/r-lib/pak/tree/packages>.
+`manifest.json` is the ground truth for the repository.
+
 ## GitHub Pages
 
 The challenge is that the site is very close to the 1GB limit, sometimes
@@ -176,6 +189,31 @@ https://raw.githubusercontent.com/r-lib/pak/repo/bin/macosx/...
 ```
 etc.
 
-### Challenges
+## Challenges
+
+### Concurrency
+
+Dealing with concurrency on GHA is a challenge on GHA.
+GitHub Packages requires that we push all architectures (=pak builds) at
+the same time. (The package files do not need to be present of the
+client if they stay the same.) So we treat `manifest.json` as the ground
+truth, and we update it for the pak build(s) that we are about to push.
+
+After pushing to GitHub Packages, we update `manifest.json` and push
+that to GitHub. If the push is not clean, then we need to do the whole
+update process again. With many concurrent builds finishing around the
+same time, this happens a lot, so in the end we do a lot of pushes to
+GitHub Packages.
+
+An alternative would be to store each pak build in its own package at
+GitHub Packages, instead of one multi-arch package. But it is also
+nice to have a single package, instead of about a hundred one. (One
+for each platform for each R version, for each stream.)
+
+### Cleanup
 
-TODO
+Old packages are not cleaned up on GitHub Packages, so ideally we
+would need to clean them up regularly, to avoid using too much
+space. E.g. we could clean up packages that are older than (say) a
+week, and that are not referenced from the `manifest.json` file.
+But we don't do that currently.