firmware: don't bake GIT_REVISION into firmware

[flokli]

This certainly causes the firmware to be a different one on every commit, and it'd become impossible to update the generated firmware alongside (as the commit ID is CA and would change).

There's also no commit ID to determine if we're just dealing with a raw archive of a specific revision, without a .git directory.

If we want to reintroduce some version ID into the firmware, it probably should be something that doesn't change on each commit, and is overridable externally, so build systems/packagers can pass the version string externally.

Asking pdm for the version string ¹ might check these boxes; we can define a less noisy version_format, it can be overridden with PDM_BUILD_SCM_VERSION, and a fallback can be defined, but maybe not baking one in at all for the sake of reproducibility is better?

Footnotes

1. https://backend.pdm-project.org/metadata/ ↩

[whitequark]

This certainly causes the firmware to be a different one on every commit

It doesn't. You should carefully study the code to understand what it does.

Here's the core part:

../glasgow/firmware $ git log -1 --abbrev=8 --pretty=%h HEAD .
b092d6a1
../glasgow/firmware $ git rev-parse HEAD
60cd63bbc38310e959816a996124d11a44e378b2

The git revision that gets baked into the firmware is the git revision of the firmware directory. This does mean that you have to land updates to the firmware in two separate steps, but it's the only way I found to have a reproducible build of the firmware that would still have a build identifier.

but maybe not baking one in at all for the sake of reproducibility is better?

Then how can you tell what firmware a device is running when you hit a bug?

[whitequark]

There's also no commit ID to determine if we're just dealing with a raw archive of a specific revision, without a .git directory.

Oh, and this isn't a supported way of building/using Glasgow software. Realistically, it isn't meant to be packaged at all, but I tolerate it with NixOS because it would be far too annoying otherwise.

[flokli]

The git revision that gets baked into the firmware is the git revision of the firmware directory.

Oh ok, I missed this indeed. If this stays, the comment could probably be a bit extended to mention this, it's easy to miss.

Then how can you tell what firmware a device is running when you hit a bug?

My proposal would be to use the checksum of the firmware. I think the CLI already does do something like this to determine whether it should flash something new? If we manage to get the firmware to be more reproducible, we don't really need version numbers, and there should be a way to map from a firmware hash to a git tree ID.

There's also no commit ID to determine if we're just dealing with a raw archive of a specific revision, without a .git directory.

Oh, and this isn't a supported way of building/using Glasgow software. Realistically, it isn't meant to be packaged at all, but I tolerate it with NixOS because it would be far too annoying otherwise.

NixOS isn't the only distribution building packages without a .git directory - that's also a reason why things like SETUPTOOLS_SCM_PRETEND_VERSION or PDM_BUILD_SCM_VERSION exist. If you want Glasgow to be widely packaged in other distro package managers at some point, I'd really recommend stopping on relying on being able to invoke git at build time.

[whitequark]

If we manage to get the firmware to be more reproducible, we don't really need version numbers, and there should be a way to map from a firmware hash to a git tree ID.

There is no such thing as "more reproducible": the build is either reproducible or it's not. The scheme I currently have should be, or at least, you're the first person to ever say that it's not, which would make it not reproducible due to a bug rather than due to the design.

My proposal would be to use the checksum of the firmware. I think the CLI already does do something like this to determine whether it should flash something new?

It does a byte-wise comparison of the occupied firmware bytes to save on flashing time. You can't do this to determine version since you don't know which bytes are occupied. Also, as the person who has to handle firmware bug reports, I reject the idea that allowing packagers to rebuild the firmware--something that is already unnecessary and unsupported--is worth my personal headache when triaging reports.

If you want Glasgow to be widely packaged in other distro package managers at some point, I'd really recommend stopping on relying on being able to invoke git at build time.

I do not. While I'm OK with distributions packaging my libraries, I actively do not want distributions packaging most of the end-user software I maintain. Distributions cultivate a brand of arrogance where people come to believe that just because they are a gatekeeper of what goes into the package index, they automatically understand the correct way to do something. This would in itself not be a problem if people who use that software did not expect support from upstream maintainers, usually without mentioning that they use a patched build that may or may not have broken functionality.

[flokli]

If we manage to get the firmware to be more reproducible, we don't really need version numbers, and there should be a way to map from a firmware hash to a git tree ID.

There is no such thing as "more reproducible": the build is either reproducible or it's not.

That's not really correct, you cannot know if a build is binary reproducible, or if you simply ended up being lucky / some of the variables affecting the build output didn't change.

The scheme I currently have should be, or at least, you're the first person to ever say that it's not, which would make it not reproducible due to a bug rather than due to the design.

The docker-based firmware build process currently simply uses debian:bookworm-slim (a tag which can point to another image tomorrow).
I doesn't pin sdcc or any of its underlying dependencies.

It might produce the same blob most of the time, but this can change whenever the online package repositories start shipping different packages.

My proposal would be to use the checksum of the firmware. I think the CLI already does do something like this to determine whether it should flash something new?

It does a byte-wise comparison of the occupied firmware bytes to save on flashing time. You can't do this to determine version since you don't know which bytes are occupied. Also, as the person who has to handle firmware bug reports, I reject the idea that allowing packagers to rebuild the firmware--something that is already unnecessary and unsupported--is worth my personal headache when triaging reports.

Ok, this won't work then. I think it might then still make sense to better document what commit id is baked into the firmware.

If you want Glasgow to be widely packaged in other distro package managers at some point, I'd really recommend stopping on relying on being able to invoke git at build time.

I do not. While I'm OK with distributions packaging my libraries, I actively do not want distributions packaging most of the end-user software I maintain. Distributions cultivate a brand of arrogance where people come to believe that just because they are a gatekeeper of what goes into the package index, they automatically understand the correct way to do something. This would in itself not be a problem if people who use that software did not expect support from upstream maintainers, usually without mentioning that they use a patched build that may or may not have broken functionality.

I'm not trying be arrogant or gatekeep things, I'd like to make it easy for NixOS users (including myself) to run a more recent version of glasgow, and am trying to find a balance between the two different requirements (I set neither of these).

I don't want to cause more work for you, wrt. having users report bugs and omitting the fact they're running the nix-built version. I could extend the PDM_BUILD_SCM_VERSION= env var to include a nixos suffix or something. If this gets logged in the output, it should become easy to spot in issue reports.

Regarding the firmware, one of my goals would be to binary reproduce the checked-in firmware with a from-source nix build (have our cake and eat it).