[CI] azure runners are flaky — remove them?

[orbeckst]

Even when all the GH actions runners succeed, many/all of the azure runners fail in some way. I have seen failures related to RDKIT and to timeouts with multiprocessing.

Do we know why the Azure runners appear to fail most of the time somehow?

At the moment we seem to have decided to ignore them and purely rely on GH ones so unless we figure out why the Azure ones are failing and work on fixing these issues, we might as well disable them and not bother because right now they don't seem to fulfill the purpose of guiding decisions on PR review.

[IAlibay]

The recent Azure pipeline failures are very recent and generally Azure is less flaky than GH actions.

As far as I can tell the rdkit failures are an actual issue that need to be addressed, it's a Windows (or pypi) thing not an azure thing.

At the moment we seem to have decided to ignore them and purely rely on GH ones so unless we figure out why the Azure ones are failing and work

I'm not aware of any such policy. This seems like a brand new thing? If Azure is failing so much that folks are doing this, why hasn't it been reported until now? It feels a bit much that we've jumped straight to "let's remove it", rather than "we'll fix it".

[IAlibay]

timeouts with multiprocessing

This isn't unique to azure pipelines, it happens with gh actions too

[orbeckst]

I'm not aware of any such policy. This seems like a brand new thing?

Every single PR that I have recently glanced over was ultimately merged while Azure was red. Basically, if it fails with something that is clearly not related to the content of the PR then we have been merging. The GH runners for Linux and macOS were green and that was "good enough" — "someone" probably should have raised issues for Windows things but at least I haven't so far as it wasn't clear to me that these were Windows specific issues.

On that note, would it make sense to dump Azure and just use GitHub for Windows, too, as this would simplify our CI?

timeouts with multiprocessing

This isn't unique to azure pipelines, it happens with gh actions too

This is true. It would be very nice if PR #4584 improved this situation – I think you wanted to offer some input there?

[orbeckst]

I opened #4687 for the RDKit test failure.

[orbeckst]

I don't think I fully understand what the rationale or strategy is for all our CI, i.e., at the big picture level what we decided was the priority to cover and how we decided to implement this strategy.

I started https://github.com/MDAnalysis/mdanalysis/wiki/CI-strategy with some notes but I am sure that I am missing important details — everyone is welcome to edit.

[IAlibay]

This is true. It would be very nice if PR #4584 improved this situation – I think you wanted to offer some input there?

This is indeed on my to-do list, but it's down prioritised currently, I'm happy to revise this as necessary.

[IAlibay]

I don't think I fully understand what the rationale or strategy is for all our CI, i.e., at the big picture level what we decided was the priority to cover and how we decided to implement this strategy.

I started https://github.com/MDAnalysis/mdanalysis/wiki/CI-strategy with some notes but I am sure that I am missing important details — everyone is welcome to edit.

This is another to-do item that I just haven't had the time to deal with, happy to have a call at some point, but I won't have time to prioritise this any time soon unfortunately.

[IAlibay]

On that note, would it make sense to dump Azure and just use GitHub for Windows, too, as this would simplify our CI?

No, dumping Azure means adding work to move what those pipelines do (which is not the same as what our GH actions pipelines do, even if you ignore the OS differences) to GH action, whilst retaining the exact same issues. Long term that might be a strategy, but if we're looking to reduce our workload then this is not the answer.

The provider isn't the issue, it's what we are covering.

[IAlibay]

"someone" probably should have raised issues for Windows things but at least I haven't so far as it wasn't clear to me that these were Windows specific issues.

This seems to be the crux of the issue here, if folks don't want to communicate issues then we can't prioritise and/or deal with them.