Add a GitHub Actions pipeline checking slendr against upcoming SLiM versions

[bodkan]

Continuing conversation with @bhaller from here:

@bhaller wrote:

It's actually super useful to have complicated SLiM clients like slendr do early testing prior to a release; you exercise code paths that nobody else exercises, just because you're doing such unusual things in slendr. :-> Perhaps we can try to co-ordinate on that in future? I'd very much appreciate it; I hate catching bugs like this right after a release is done. :-<

[...]

All sounds good. I'm glad one of your unit tests caught this bug, since it slipped through my unit tests completely! :-O It's really difficult to properly test software as complex as SLiM. I wish we had a full-time QC engineer.

I like your GHA idea. To keep it simple, how about a GHA action that just builds everything and runs slendr's standard test suite against a specified branch in SLiM, once a week say? Right now the branch would be master but it would've been multichrom before, and pretty soon it will hopefully be multitrait for the next stage of things. :-> If it's easy, it could just detect the branch where the most recent commit was made, and build/test against that branch. That would be right 99% of the time, and if it's wrong, no worries, it'll be right a week later. :-> Or we can just hard-code master for now, but then I have to remember to let you know when I switch to a side branch for a release. Anyhow, if you know how to do that sort of thing in GHA, that'd be great! GHA is a somewhat opaque black box to me; most of what is set up for SLiM was done by others.

Having thought about this more, I think a good tradeoff between "catching unintended SLiM problems early" and "false positives due to temporary errors due to WIP changes in slendr (and maybe SLiM)" could be this:

1. Create a macOS GHA instance as close to my own machine (I already have that as part of the normal GHA testing suite).
2. Install the latest stable slendr release from CRAN.
3. Clone and compile a specific release tag / branch of SLiM that's worth checking for unintended breakage of backwards compatibility.

Doing 1. makes it easy to immediately verify a problem locally.

Doing 2. avoids testing my potentially broken slendr master HEAD against SLiM (it could be broken for many reasons that have nothing to do with SLiM, because I sometimes make small changes without doing a dedicated branch/PR... I'm not a very disciplined git user).

Doing 3. saves me the trouble of checking potential GHA fails in situations which wouldn't matter that much, which might happen if I'm testing against the branch with the most recent commit (for instance, WIP changes on SLiM's part which are known and would be cleaned up before release). Similarly, together with 2. it would save both of us the time investigating false positives.

However, 3. would depend on you creating, at some point, a tag or branch for a release candidate... but given what you wrote above, it seems that you might be already doing this?

Basically, to sum up: investigating failing GHA logs is easily one of the parts of my job I dislike the most. And I'd like to minimize the time spent looking at GHA logs (and, by extension, minimize the number of GHA runs for this) as possible. Ideally also eliminate the need for manual intervention: for the slendr part, this would be install.packages("slendr"), for SLiM part, this could be git clone https://github.com/messerlab/slim; cd slim; git checkout rc (or some other appropriate name for the release candidate branch/tag that would ideally be always the same, eliminate need for coordination and communication of when should things be tested).

Just some immediate thoughts.

And don't worry about slendr being a hack! Real-world software almost always feel that way to the developers themselves, I think; it is a rare project indeed that has truly clean code. SLiM has a fair bit of messiness internally, even though I do try to keep it clean; and it has a handful of hacks of breathtaking ugliness. Such is life. :-> Don't hesitate to ping me if you think you've found a bug – like this one! The feedback is invaluable. :->

Thanks for the words of encouragement!

[bhaller]

OK, sounds good. Interesting idea having the tag name stay the same and just get moved around by me; that hadn't occurred to me, but is a good idea. I'll set that up soon and then ping you here. For now, gotta run. :->

[bhaller]

Pondering this further. Is there such a thing as a tag that stays at the head of a given branch? I'd like the "rc" tag to point to the head of the master branch now, and move forward with new commits; and then once I make a multitrait branch, I'll want it to point to the head of that branch, and move forward with new commits. Is that a thing? If not, we might need to hard-code the name of the target branch in your GHA script, and change it when we want to change it...?

[bodkan]

Hmm, I guess HEAD is the "top" of whatever branch is currently being checked out (so, on GHA, that would involve checking out a given branch... and running the unit tests against that). But yes, that would involve some minor amount of coordination... with me remembering to make the change to a branch name, and then checking the GHA results after I make the change. I'm bad with remembering anything. But maybe I could script it using some git voodoo I don't know yet (but could investigate).

I suppose automatically testing against the HEAD of your master (using the latest stable slendr release from CRAN, not its master which is often broken for in-development reasons) would work. But it would only work if SLiM's master is always in a consistent stable state. Which I'm not sure is true for SLiM. I think good git practice in general encourages the master to be always working... but for me personally, usually I cannot be bothered with that. As the only developer of all of my software, I can afford to be a bit messy... and I am, occasionally. :)

Maybe the following would be useful: getting the git branch carrying the most recent commit: https://betterstack.com/community/questions/how-can-i-get-ordered-list-of-git-branches/?

This should accomplish:

1. Only testing the branch which is worth testing (i.e., assumed-stable SLiM codebase against a stable slendr codebase) at times which they should be tested (SLiM close to a new release) -- avoid scanning long GHA logs for issues when it's not needed.
2. Do this in a fully automated way -- avoid the problem that I can never remember anything. :)

But this of course makes some assumptions about your development / release practice, namely that the most recently updated branch is the to-be-released-next SLiM version.

A yet another compromise could be if you'd add some kind of indicator to your to-be-released branch that would mark it as that release candidate, such as multitrait-rc. Then I could script this in a way that:

1. Fetches names of all SLiM branches.
2. Finds those with that particular marker (like rc in its name).
3. Compiles the one that is most recent.

But at this point I'm not sure if I'm not overcomplicating this too much. I'm just looking for a solution that is 100% automated. Maybe I should first think on my own instead of thinking out loud here. :D

[bhaller]

Hmm, I guess HEAD is the "top" of whatever branch is currently being checked out (so, on GHA, that would involve checking out a given branch... and running the unit tests against that). But yes, that would involve some minor amount of coordination... with me remembering to make the change to a branch name, and then checking the GHA results after I make the change. I'm bad with remembering anything. But maybe I could script it using some git voodoo I don't know yet (but could investigate).

I suppose automatically testing against the HEAD of your master (using the latest stable slendr release from CRAN, not its master which is often broken for in-development reasons) would work. But it would only work if SLiM's master is always in a consistent stable state. Which I'm not sure is true for SLiM. I think good git practice in general encourages the master to be always working... but for me personally, usually I cannot be bothered with that. As the only developer of all of my software, I can afford to be a bit messy... and I am, occasionally. :)

Yeah, I'm fairly messy with the master branch, but when it breaks, I do try to fix it quickly – I try not to leave it in a broken state for long. :-> But more to the point, big development can happen on a side branch for months at a time, and it'd be nice to have this automated testing giving us a sign of how things are going. I didn't merge the multichrom branch until just, like, a week or two before 5.0 shipped, because I wanted to be certain that it wasn't broken, for those who do live on the master branch HEAD; there aren't a lot of people who do that, but there are some.

Maybe the following would be useful: getting the git branch carrying the most recent commit: https://betterstack.com/community/questions/how-can-i-get-ordered-list-of-git-branches/?

This should accomplish:

1. Only testing the branch which is worth testing (i.e., assumed-stable SLiM codebase against a stable slendr codebase) at times which they should be tested (SLiM close to a new release) -- avoid scanning long GHA logs for issues when it's not needed.
2. Do this in a fully automated way -- avoid the problem that I can never remember anything. :)

But this of course makes some assumptions about your development / release practice, namely that the most recently updated branch is the to-be-released-next SLiM version.

Yeah, this is what I was suggesting before:

If it's easy, it could just detect the branch where the most recent commit was made, and build/test against that branch. That would be right 99% of the time, and if it's wrong, no worries, it'll be right a week later. :->

If you can figure out how to do it, I think this is the right option. Every now and then it might build against the wrong branch, but that'll be rare, and if it does, who cares? :->

A yet another compromise could be if you'd add some kind of indicator to your to-be-released branch that would mark it as that release candidate, such as multitrait-rc. Then I could script this in a way that:

1. Fetches names of all SLiM branches.
2. Finds those with that particular marker (like rc in its name).
3. Compiles the one that is most recent.

But at this point I'm not sure if I'm not overcomplicating this too much. I'm just looking for a solution that is 100% automated. Maybe I should first think on my own instead of thinking out loud here. :D

I think this is overcomplicating this too much. If it occasionally builds against the wrong branch, it doesn't matter.

[bhaller]

You and I are quite different in this respect, I guess. I automate almost nothing, because I find that the work to do the automation is more than the work to just do it myself. (Similarly, I have no interest in using AI to assist my coding, because the work to check the AI's code for correctness would be greater than the work to just write the code myself. Code review is harder than code authoring, especially if the code you're reviewing was written by someone you know can be rabidly incompetent in ways that are executed with total confidence! :-O)

[bodkan]

No, I think we're on a very similar page w.r.t. automation (and don't get me started on all the AI nonsense). It's just that investigating problems with GHA is awful, which is why I wanted to automate that without having to do much extra work. :) At least in my experience, my GHA workflows sometimes break for totally arbitrary reasons, even reasons which have nothing to do with slendr code. :(

If it's easy, it could just detect the branch where the most recent commit was made, and build/test against that branch. That would be right 99% of the time, and if it's wrong, no worries, it'll be right a week later. :->

Sounds good, let's agree on this compromise. I supposed this will be good enough for practical purposes, which is all that matters here.

To sum up, I will try to set up this pipeline so that it runs on a regularly scheduled basis but it runs independently of the other slendr GHA pipelines, using the latest stable slendr release from CRAN which is known to work with the latest SLiM release (by definition, as I don't submit new slendr release to CRAN unless it builds against the latest official SLiM release). After all, this is sort of how backwards compatibility-related issues have been discovered in the past, except by accident -- first finding out the master slendr breaks with new SLiM, then checking if it the stable slendr on CRAN works with the new SLiM (and realizing that it doesn't).

So, two parallel GHA workflows running in parallel to each other, each with a slightly different purpose, and the new one being of interest and worth investigating for problems more closely every time a new SLiM release is imminent.

Will ping you again once I put this together. Probably by copy pasting / editing obscure GHA YAML voodoo until it works (as is my GHA tradition). ¯_(ツ)_/¯

[bhaller]

Sounds good!