CI/CD Overhaul

[karosc]

There are three components to this PR. This may be quite a substantial set of changes to stomach, so consider this branch experimental until we get thorough testing of performance and stability from the maintainers. 🧪🧪

scikit-build-core migration

This is to move away from the deprecated build system scikit-build and into the more modern scikit-build-core backend that leverages pyproject.toml metadata. This largely amounted to moving most of our setup.py into a pyproject.toml file. However, there was one other change made to better accommodate the newer build-system:

1. It was difficult to dynamically move files from lib or bin into swmm_toolkit directory based on platform using the scikit-build-core backend. I elected to instead handle it in cmake in the install command. The linked libraries libswmm and smm-output are now included in the main package directory and the RPath was changed to ORIGIN to the python wrapper looks for the library in its own directory.
   • Consequently, the extra path kungfu in the __init__.py for the windows platform is no longer necessary

Stable ABI Overhaul

This explores restricting our build process to the python stable ABI. The main benefit of restricting to the stable ABI is we only need one wheel per system and architecture, it does not matter what python version you have. This means that future releases of python will be compatible with past releases of swmm-toolkit! 🥳🥳

Theoretically, all one needs to do to enable the stable api is set the compile time variable Py_LIMITED_API=0x03090000 to constrain our python api calls to those that are available in python>=3.9. However, in reality there were several other changes that were required:

1. The python wrapper extension had to be modified to use abi3 instead of Python3_SOABI....with the exception of windows, which does not seem to support the abi3 extension and consequently has none. Using _solver.abi3.pyd yielded the following error:
   ImportError: cannot import name '_solver' from 'swmm.toolkit' (C:\Users\stanley\Downloads\swmm\.venv\Lib\site-packages\swmm\toolkit\__init__.py) Removing the .abi3 component of the file name resolved the issue.

2. Remove explicit linking of python when using MSVC. When $<$<BOOL:$<C_COMPILER_ID:MSVC>>:Python3::Module>, cmake does not link to the stable ABI lib python3.lib and instead forces a link to the version-specific lib

Updates to the build_wheel.yml workflow

Since I last touched this file, github has released native arm64 runners for both macos and linux. I modified this workflow to use the native runners rather than cross compile. Additionally, since this branch uses the python stable ABI, it only builds once for each platform, but runs the unit tests for each version of python 3.9 through 3.14 to check compatibility.

Other notes

1. openmp libraries are now explicitly included in the distribution since I don't expect all our users to have the dlls. Consequently, I also am explicitly including the openmp-llvm license in our wheel to be safe.

2. If you don't want to compile a stable ABI binary and instead want to target your own python version (perhaps for optimizations?) just build the project with the environment variable NO_STABLE_ABI=0

[michaeltryby]

@karosc Nice work!

[karosc]

So I ran the following benchmark simulations on linux (debian trixie) using a medium size (1167/1151 links/nodes) RTK SWMM model with a month long simuation:

1. without stable ABI and without any swmm-toolkit getters/setters (STD NO CTL)
2. with stable ABI and without any swmm-toolkit getters/setters (ABI NO CTL)
3. without stable ABI and with swmm-toolkit getters/setters (STD CTL)
4. with stable ABI and with swmm-toolkit getters/setters (ABI CTL)

I attached the two scripts that run the model with and without swmm-toolkit getter/setter calls to show the extent of the getter and setter usage (lots of calls). Below is a table showing the runtime in seconds of each simulation in triplicate:

Iteration	STD NO CTL	ABI NO CTL	STD CTL	ABI CTL
Run 1	218	218	1200	1283
Run 2	216	217	1218	1236
Run 3	219	218	1203	1238
-	-	-	-	-
Average	218	218	1207	1252
Std Dev	2	1	10	27

It seems that adding the ABI does add some amount of overhead, slowing the simulation down by 3%. That being said, the standard deviation in ABI CTL runtime (27s) is about half the average slowdown (45s). That, and I think we`re doing heck of a lot of getting and setting in these sims, more than the typical workload......so we may still call this a win.

Right now this compilation of the stable ABI targets python 3.9 and these simulations were all run with 3.11. I wanted to check if targeting newer versions of the stable ABI would increase performance. In other words, if I increase the Py_LIMITED_API compile definition, does the stable ABI increase in performance? Below is a table with those added simulations.

TLDR, it does not affect things.

Iteration	STD CTL	ABI CTL	ABI 3.10 CTL	ABI 3.11 CTL
Run 1	1200	1283	1276	1265
Run 2	1218	1236	1247	1227
Run 3	1203	1238	1223	1259
-	-	-	-	-
Average	1207	1252	1249	1250
Std Dev	10	27	27	20

run_swmm_toolkit.py
run_no_controls.py

[karosc]

Build wheels workflow: https://github.com/karosc/swmm-python/actions/runs/17532877034

[michaeltryby]

Hey @karosc! I've said it before and I'll say it again, moving this build to Python's stable ABI is a big win for simplicity! I thought your performance analysis was very thoughtful. The degradation observed is expected and acceptable.

The only question I have is, what about the source distribution? We probably ought to post one when we publish to pypi.

Thanks for your work on this! It's greatly appreciated