Limiting AppVeyor tests

[MeanSquaredError]

This is definitely non-urgent, but probably needs to be resolved at some point in the future.

The number and complexity of tests have reached a point at which AppVeyor starts timing out (it seems that it has a limit of 1 hour or so per build).

One reasonable solution is to only build some limited subset of the tests when building for AppVeyor.

To do that we could add a CMake boolean setting (a.k.a. "option") called `BUILD_FOR_APPVEYOR``. E.g.

option(BUILD_FOR_APPVEYOR "Limit tests to the AppVeyor subset" OFF)

This boolean setting would only be used if BUILD_TESTING is enabled and would limit the built tests to only the essential subset used by AppVeyor.

Alternatively BUILD_TESTING could be changed from a boolean to a string with three values, e.g. NONE, ALL and APPVEYOR. Then in CMakeLists.txt it could be set with something like this:

set(BUILD_TESTING NONE CACHE STRING "Which tests to build (NONE, ALL, APPVEYOR)")

Then the CMakeLists.txt files in the tests/ subdirectories, could use BUILD_TESTING (+ optionally with BUILD_FOR_APPVEYOR) in order to choose which (if any) tests to build.

To me, the "recipe" tests seem like candidates for the non-essential tests. And I also want to turn examples/connection_pool/ into yet another "recipe" test, but have been postponing this because if the AppVeyor limits. And there are probably quite a few other tests that are "non-essential" and could be skipped when building for AppVeyor.

So what do you think about this approach to solving the AppVeyor issue?

BTW, on a side but related matter, most settings, like BUILD_MYSQL_CONNECTOR are being set via option(...) at the beginning of CMakeLists.txt:

sqlpp23/CMakeLists.txt

Line 37 in 54ca880

option(BUILD_MYSQL_CONNECTOR "Build MySQL Connector" OFF)

but BUILD_TESTING is not being set via option(BUILD_TESTING ...)

If I understand correctly, the only real difference that this makes, is that this way BUILD_TESTING is not cached, so it is not persisted across CMake runs (I might be wrong about this, though). I wonder if this (BUILD_TESTING not being set via option()) is intentional or just an omission?

[rbock]

Thanks for raising this as an issue.

Somehow the compile time changed pretty dramatically at the time we introduced modules (from 30-40 to 50-60 minutes). And we did not add that many tests in that time, I believe?

If that's correct then the compilation time per tests increased.

Picking a random compilation step from https://ci.appveyor.com/project/rbock/sqlpp23/builds/52749762 (2025-09-16)

[00:12:12]   Building Custom Rule C:/projects/sqlpp23/tests/core/serialize/data_types/CMakeLists.txt
[00:12:12]   Scanning sources for module dependencies...
[00:12:12]   blob.cpp
[00:12:13]   Compiling...
[00:12:13]   blob.cpp
[00:12:18]   sqlpp23_core_serialize_data_type_blob.vcxproj -> C:\projects\sqlpp23\build\tests\core\serialize\data_types\Release\sqlpp23_core_serialize_data_type_blob.exe
[00:12:19]   Building Custom Rule C:/projects/sqlpp23/tests/core/serialize/data_types/CMakeLists.txt
[00:12:19]   Scanning sources for module dependencies...
[00:12:19]   date.cpp
[00:12:19]   Compiling...
[00:12:19]   date.cpp
[00:12:28]   sqlpp23_core_serialize_data_type_date.vcxproj -> C:\projects\sqlpp23\build\tests\core\serialize\data_types\Release\sqlpp23_core_serialize_data_type_date.exe

And the equivalent from https://ci.appveyor.com/project/rbock/sqlpp23/builds/52314567 (2025-06-28)

[00:09:18]   Building Custom Rule C:/projects/sqlpp23/tests/core/serialize/data_types/CMakeLists.txt
[00:09:18]   blob.cpp
[00:09:22]   sqlpp23_core_serialize_data_type_blob.vcxproj -> C:\projects\sqlpp23\build\tests\core\serialize\data_types\Release\sqlpp23_core_serialize_data_type_blob.exe
[00:09:23]   Building Custom Rule C:/projects/sqlpp23/tests/core/serialize/data_types/CMakeLists.txt
[00:09:23]   date.cpp
[00:09:30]   sqlpp23_core_serialize_data_type_date.vcxproj -> C:\projects\sqlpp23\build\tests\core\serialize\data_types\Release\sqlpp23_core_serialize_data_type_date.exe

Looks like there is both more overhead and more time for compilation involved?

[MeanSquaredError]

@rbock
Thank you for the analysis. I have to admit that I have a difficult time finding the build settings of sqlpp23 on the appveyor website.

• Does the slowdown appear with msvc only, or it also appears with gcc and/or clang? Does appveyor actually test sqlpp23 with gcc/clang or just with msvc? I would expect that gcc/clang are being used too, but somehow I am unable to find that information.

• Does appveyor/msvc actually build the sqlpp23 tests with modules support? On the one hand I see that building each .cpp file is preceded by a line that says "Scanning sources for module dependencies...", and it seems that the message is emitted by msvc, so modules are being used. On the other hand I don't see the appveyor build using BUILD_WITH_MODULES anywhere. I see that it uses the following comand-line

cmake .. -DCMAKE_CXX_FLAGS="/EHsc /std:c++latest" %CMAKE_TOOLCHAIN_FILE% -DBUILD_MYSQL_CONNECTOR=ON -DBUILD_POSTGRESQL_CONNECTOR=ON -DBUILD_SQLITE3_CONNECTOR=ON

so it seems that modules support is not required when sqlpp23 tests are built. According to this article it seems that msvc enables modules support automatically with /std:c++20 or later, or /std:c++latest.

Maybe we could try disabling modules support as suggested by that article, using /experimental:module- and see if that will speed up the build process?

[rbock]

Only MSVC tests are run on appveyor (I used to use Travis.CI for gcc and clang but their free tier for open source projects became too difficult to use a few years ago and I haven't looked into alternatives for a while, github actions, maybe?).

I can't remember turning on modules for MSVC explicitly.

However, I re-ran https://ci.appveyor.com/project/rbock/sqlpp23/builds/52314567 (2025-06-28, 39:20)

• https://ci.appveyor.com/project/rbock/sqlpp23/builds/52749831 (47:56)
• https://ci.appveyor.com/project/rbock/sqlpp23/builds/52756537 (46:22)

It showed two things:

• No mention of modules despite using /std:c++latest, so apparently, some other option changed
• Some of the slowdown seems to be systemic.

[MeanSquaredError]

OK, thank you for the clarification. I have a machine with Windows 10 and MSVC 19.44.35216, so I will try to find some time this weekend to compile different versions of sqlpp23 (before and after modules were introduced) and measure the time. I will also try different compiler options to see if there is a way to lower the compile time.

[MeanSquaredError]

I did some very basic time measurements under Windows 10 + MSVC of two commits, an old one (before modules) and a new one (after modules), both of them configured with

cmake -B build -G Ninja  -DBUILD_TESTING=ON -DDEPENDENCY_CHECK=ON

(Essentially just the core tests were built, because I had problems with CMake finding the PostgreSQL/MySQL/SQLite3 libraries used by the connectors)

Then I built the two commits with

cmake --build build

Commit 829902e took 319 seconds
Commit 60185de took 379 seconds

So indeed, introducing modules, seems to add some noticeable increase in build time. It is not as bad as 40%, but then again, it is just the core without the actual connectors.

So before making final conclusions, I want to do more testing with some connectors enabled too.

[rbock]

Interesting. I tried with gcc and clang, too.

gcc-14, ninja
Commit 829902e took 5m:35s, 607 targets
Commit 3176bef took 5m:48s, 1205 targets

clang-20, ninja
Commit 829902e took 5m:34, 607 targets
Commit 3176bef took 6m:04s, 1205 targets

While there is little difference in overall time, there's almost a duplication in the number of targets. Any idea where that's coming from?

[MeanSquaredError]

I am still experimenting with core only because installing the PostgreSQL/MySQL/SQLite3 libraries on Windows and then getting them to work is a real pain in the neck.

However, one thing that I found is that the number of build targets is doubled (in my case from 360 to 720) by the change in main CMakeLists.txt from

cmake_minimum_required(VERSION 3.20)

to

cmake_minimum_required(VERSION 3.28)

A few more experiments showed that this is caused by CXX_SCAN_FOR_MODULES, which was added in CMake 3.28. I tried turning it off immediately after cmake_minimum_required, like this

cmake_minimum_required(VERSION 3.28)
set(CMAKE_CXX_SCAN_FOR_MODULES OFF)

and the number of build targets went down from 720 to 360 and the build time likewise went down from 379 seconds to 355 seconds. Not quite the 319 seconds that was with the old library but still a step in the right direction.

So one thing that we should probably do when building tests is turning off CXX_SCAN_FOR_MODULES for the tests if BUILD_WITH_MODULES is turned off.

I have a few more ideas on what to try next in order to further bring down the build time with MSVC, just need to find the time in the next few days to make a few more experiments.

[MeanSquaredError]

I tried adding the command-line option /experimental:module- to MSVC when building tests and while MSVC accepted the option without and warnings/errors, this did not affect build time (it went down by 1 second, which most likely is a fluctuation). The Microsoft documentation says that

This option is obsolete in Visual Studio 2019 version 16.11 and later.

so maybe that's the reason why this option did not make a big difference.

At this point I think that our best bet is turning off CXX_SCAN_FOR_MODULES when (BUILD_TESTING AND NOT BUILD_WITH_MODULES) and checking if AppVeyor build times will go down.

What do you think?

[rbock]

Thanks for the analysis! Giving it a try.

[rbock]

48 minutes is an improvement. Thanks for the suggestion!

[MeanSquaredError]

I think a further speed-up can be achieved by using the various all.h files as pre-compiled headers when BUILD_WITH_MODULES is OFF. I am going to do some experiments with that next weekend.

[rbock]

While I am sure precompiled headers would shave off some time, I am not too fond of the idea of using precompiled headers as they are not part of the standard.

[MeanSquaredError]

Well, I think that precompiled headers (PCH) are a compiler-specific detail that doesn't have to be a part of the C++ standard. IMHO it is OK to use PCH as long as they don't change the validity and the semantics (that is the behavior) of the program, which is mostly the case. I used "mostly" because PCH do require the header that is being precompiled to be the first included one, but that is a relatively small change in the tests' code.

I didn't have the time for any compile-time profiling, so I cannot say for sure which part of the tests takes most of the time to compile, but I strongly suspect it is the various all.h files, simply because most of the compiled code is contained in these headers and the headers that are included recursively from all.h. That's why I suspected that barring the scanning introduced by CMake 3.28, it is the addition of the all.h files that added for most of the slowdown. And then using PCH seemed like a natural next step to improve compile times.

Overall I am OK with NOT using PCH, but they pretty much concluded all of my ideas for further compile-time improvements, i.e. I am out of other ideas :-) But then again, I am OK with the current state, as long as AppVeyor doesn't time out on my PRs.

[rbock]

I think I found a couple of angles. I'll do some experiments in the next few days.