[viosock] Unable to build using defaults or for amd64 #1184
Comments
|
I see this is a a duplicate of #733, but perhaps now worth fixing...? I also note there were many, many CodeQL warnings... |
I should make it clear that this was for both Win10 and Win11. |
|
Hi @benyamin-codez, This is known issue. Unfortunately, you cannot define dependency from another platform, just another project as a dependency. |
This is Microsoft changes. You cannot use EWDK 24H2 / 2025 to build an x86 version of KMD/UMD. This is one of the reasons we use two EWDK in parallel. |
|
BTW: not Win11 32bit. |
|
Thanks for the feedback. Is it fair to say that this remains an issue that you cannot simply call Most, if not all of this, should be fairly easy to fix with just batch. Yan, just to be clear, I couldn't build Win11 at all with the EWDK versions listed. Also, I couldn't run the build again without cleaning up first. |
|
@benyamin-codez Sorry I might be missing something: As @kostyanf14 said - if you want to build everything by running buildall.bat you need two eDWKs, because 32bit binaries cannot be built with Win11 eWDK. Moreover - Win11 eWDK doesn't support first versions of Win10. So it is not a good idea to use it to build Windows 10 binaries in general, as you don't know the lowest Win10 version the user might use.
There is no such thing as Windows 11 x86. Only 64bit version of Win11 exists. @kostyanf14 I see that "buildall.bat Win11" doesn't work as intended. Also if you play with changing compilers\kits - run clean.bat between experiments. |
|
Some observations: Cleaning (using Building on Cobalt EWDK with Building on Germanium EWDK with
Are you saying we should still be using Titanium EWDK (19H1) for Win10 builds, or is Cobalt EWDK (21H2) ok? |
|
Just to me clear, building for So I think the simplest solution to this I could do something similar for
|
|
Progress update: Ok, so I've managed to get Cobalt EWDK working for all targets and architectures from the tree root and also in When done, I'll post all of that in a PR with some other SDV/CodeQL changes, basically updates to
I couldn't determine any reason as to why this would be the case. I've also done some I'll drop PR numbers here once submitted... |
What do you want to do? Currently, we did not see any problem with build in upstream and in RedHat downstream builds. Current build and artifacts matrix
|
|
Thanks, Kostiantyn..! That's a very helpful matrix..! Is it published somewhere? Maybe I can add it to my wiki PR?
Basically, when I touch a part of the tree, I want to be able to build that branch for all platforms and at the very least make sure it will compile without error on as many platforms as possible before raising relevant PRs. Therefore, if I touch something in the root, or in build, I want to be able to build the whole tree. It's worth pointing out that I raised this issue because I couldn't do any I imagine your in-house tooling does something similar. If I would hazard a guess: in your case - and in that of many of the other fine folk wearing red fedoras - you build the tree in your HCK toolkit, customising builds for various platforms, as part of your normal CI/CD operations ... but for me, I just want to build it from the EWDK, and preferably straight from the optical drive rather than from the hard drive (as this aids in upgrading EWDKs on multiple dev and test boxes). Secondly, and again to save time, I'd also prefer to build without SDV but with CodeQL instead, especially given SDV is deprecated - at least in part. The easiest way I found to do that (at least without making drastic changes) is to still use the SDV tag but add a WHCP_LEVEL argument to specify the branch. I ended up using WHCP_LEGACY and WHCP_24H2 as the two required levels. When using CodeQL CLI v2.15.4+, these map to the I say WHCP branches here, but I only use the repo for the suites (for which I control and check for updates based on a Git commit hash), and instead rely on the CodeQL package cache. For a variety of reasons (mostly for sanity), this requires additional steps in-flight, i.e. during build, but less steps to set up. These were the changes I wanted to add to the wiki, but thought I'd better build out the in-flight part first, raise a PR, get the buy-in of you and your colleagues, and then do another PR for necessary wiki changes based on where the chips might fall. In my cut of Does MS definitely still require SDV-based DVLs for ALL Windows 10 / pre-24H2 targets? I noticed we produce some DVL files for Win10 which are just a copy of Win11, and (sometimes) a COMPAT version too.
I presume that is a problem if SDV is indeed still needed for the WHCP. My reading was that CodeQL created DVLs were sufficient. This is based on text such as, e.g. "However, if you are certifying for Windows Server 2022 and below; CA, SDV and CodeQL can be used. Use WDK builds with the matching OS release versions." That being said, if a particular version of the HLK is required for a particular build, and the HLK fails the DVL for lack of SDV having run, then I can see how that would be an issue. If that is the case, I can certainly add a variable along the lines of USE_CODEQL_EXCLUSIVE, or similar, and resurrect the SDV orphans I deprecated on a conditional basis. That way it would run normally, i.e. with SDV, unless the variable was set. So then we have the oddity in the matrix of Win11 21H2/22H2/23H2 using Germanium EWDK (26100+) where SDV is most definitely deprecated. I presume here that the DVL is preferably sourced from a Cobalt EWDK (22000) build for Win10 (rather than just a copy of a Win11 DVL). This is what I am working on now, as discussed above. I don't think anything I am working on is likely to affect your existing tooling, but will instead likely extend your build time capability, especially around CodeQL. The advantage is really for external contributors like myself, who want to more thoroughly check their code before submitting PRs. I'll probably raise the various PRs as drafts, and from there we can discuss any issues and what you think is worth keeping. Apologies for the long post. I hope that clears things up somewhat. Obviously the details would be in any PRs. Best regards, |
Only in PR that switched to EWDK 24H2.
YES!
You can see the table of OS and HLK versions for it there https://learn.microsoft.com/en-us/windows-hardware/test/hlk/.
I don't see any reason for this. CodeQL results based on the WHCP_24H2 branch are good for both Win10 and Win11. We experimented with different CodeQL versions and with different WHCP* branches and see that there is no difference for HLK. This option just created a more complicated build environment in my opinion. |
|
Also, you can see |
|
@benyamin-codez @YanVugenfirer
Need to see your PR but I don't prefer the idea of building x86 as a dependency of AMD64 automatically. |
|
Just FYI. Jenkins config of build job: Files to archive the artifacts: There are no any workarounds. We always run the build from the root batch file on CI. |
|
Thanks for making the time to reply..!
That's good to know. I wasn't sure if MS would accept this as they state that you must use the @1.0.13 pack for pre-WHCP_24H2 (what I labelled WHCP_LEGACY). It's very good that they do accept @1.1.0 instead which will run the extra tests. I'm not familiar enough with what is submitted to know whether they can tell which one ran... I did default to WHCP_24H2, so you would have to explicitly set it to WHCP_LEGACY to enable @1.0.13. if /I "%ARG%"=="WHCP_LEGACY" set WHCP_LEVEL=WHCP_LEGACY& goto :argloop
if /I "%ARG%"=="WHCP_21H2" set WHCP_LEVEL=WHCP_LEGACY& goto :argloop
if /I "%ARG%"=="WHCP_21H2" set WHCP_LEVEL=WHCP_LEGACY& goto :argloop
rem WHCP_23H2 is a non-existent repo branch. Included here for completeness.
if /I "%ARG%"=="WHCP_23H2" set WHCP_LEVEL=WHCP_LEGACY& goto :argloop
rem WHCP_24H2 is the default. Set below during SDV builds only.
if /I "%ARG%"=="WHCP_24H2" set WHCP_LEVEL=WHCP_24H2& goto :argloop
rem ....
:runql
if exist %~dp1codeql_db (
echo "Removing previously created rules database"
rmdir /s /q %~dp1codeql_db
)
echo call "%~dp0SetVsEnv.bat" %~1 > %~dp1codeql.build.bat
echo msbuild.exe -maxCpuCount %~dp1%BUILD_FILE% /t:rebuild /p:Configuration="%~1" /P:Platform=%2 >> %~dp1codeql.build.bat
call %CODEQL_BIN% database create -l=cpp -s=%~dp1 -c "%~dp1codeql.build.bat" %~dp1codeql_db -j 0
IF ERRORLEVEL 1 (
set CODEQL_FAILED=1
set BUILD_FAILED=1
)
IF "%CODEQL_FAILED%" NEQ "1" (
rem Make sure the codeql-workspace.yml file is deleted from the repo so that CodeQL ignores the repo and relies on the cache instead
if exist %CODEQL_DRIVER_SUITES%\..\codeql-workspace.yml (del %CODEQL_DRIVER_SUITES%\..\codeql-workspace.yml)
rem The use of --model-packs=<name@range> is ignored and the most recent pack in the cache is always used.
rem Therefore, we must prepare the cache depending on the WHCP_LEVEL variable.
rem From CodeQL CLI v2.15.4 we use :
rem [a] microsoft/[email protected] for everything below WHCP_24H2
rem [b] microsoft/[email protected] for WHCP_24H2 [and potentially newer versions once released]
rem Note: This should also work with more recent versions of CodeQL CLI [tested to v2.19.2]
if "%WHCP_LEVEL%"=="WHCP_LEGACY" (
echo Configuring CodeQL database analysis cache for legacy WHCP with the microsoft/[email protected] CodeQL pack
if exist "%USERPROFILE%\.codeql\packages\microsoft\windows-drivers\1.1.0" (rmdir /s /q "%USERPROFILE%\.codeql\packages\microsoft\windows-drivers\1.1.0")
if not exist "%USERPROFILE%\.codeql\packages\microsoft\windows-drivers\1.0.13" (call %CODEQL_BIN% pack download -v microsoft/windows-drivers@1.0.13)
)
if "%WHCP_LEVEL%"=="WHCP_24H2" (
echo Configuring CodeQL database analysis cache for WHCP_24H2 with the microsoft/[email protected] CodeQL pack
if not exist "%USERPROFILE%\.codeql\packages\microsoft\windows-drivers\1.1.0" (call %CODEQL_BIN% pack download -v microsoft/windows-drivers@1.1.0)
)
if "%WHCP_LEVEL%"=="WHCP_NEXT" (
rem USE THIS AS A TEMPLATE FOR FUTURE VERSIONS
rem NOTE: Remember to add the directory removal conditionals to all WHCP_LEVELS above [per WHCP_LEGACY example]...
echo Configuring CodeQL database analysis cache for WHCP_NEXT with the microsoft/windows-drivers@__NEXTRANGE__ CodeQL pack
if not exist "%USERPROFILE%\.codeql\packages\microsoft\windows-drivers\__NEXTRANGE__" (call %CODEQL_BIN% pack download -v microsoft/windows-drivers@__NEXTRANGE__)
)
rem Do the analysis...
call %CODEQL_BIN% database analyze %~dp1codeql_db %CODEQL_DRIVER_SUITES%\windows_driver_recommended.qls --format=sarifv2.1.0 --output=%~dp1%BUILD_NAME%.sarif -j 0
)
IF ERRORLEVEL 1 (
set BUILD_FAILED=1
)
goto :eof
I presume the DVL files are copied from Titanium EDWK (19H1 / 18362) per the wiki. This is what I have been doing.
I could add a conditional for external devs like me, but I think even if it ran, it's pretty fast. rem We have pre / post 24H2 buildtime conditions
set WHCP_LEVEL=
set VSCMD_WHCP_LEVEL=
set VIOSOCK_PREBUILD_X86_LIBS=
rem Split MSBuild version and check if pre or post 24H2...
for /f "tokens=1,2,3 usebackq delims=." %%i in (`echo %%VSCMD_VER%%`) do @set VSCMD_VER_MAJOR=%%i && set VSCMD_VER_MINOR=%%j && set VSCMD_VER_BUILD=%%k
if not %VSCMD_VER_MAJOR% LSS 17 (
if not %VSCMD_VER_MINOR% LSS 8 (
set VSCMD_WHCP_LEVEL=WHCP_24H2
) else (
set VSCMD_WHCP_LEVEL=WHCP_LEGACY
)
) else (
set VSCMD_WHCP_LEVEL=WHCP_LEGACY
)
rem ... Near top
rem ... Much in-between
rem ... In the midst of :build_arch
rem Check for x86 viosock libraries and build them if needed...
echo Building : %BUILD_FILE%
if "%BUILD_FILE%"=="virtio-win.sln" (set VIOSOCK_PREBUILD_X86_LIBS=1)
if "%BUILD_FILE%"=="viosock.sln" (set VIOSOCK_PREBUILD_X86_LIBS=1)
if not "%VIOSOCK_PREBUILD_X86_LIBS%"=="" (
if %BUILD_ARCH%==amd64 (
if not exist "%BUILD_DIR%viosock\lib\x86\%TARGET%%BUILD_FLAVOR%\viosocklib.dll" (
echo WARNING - Need to build x86 viosock libraries before building for amd64...
if not "%VSCMD_WHCP_LEVEL%"=="WHCP_LEGACY" (
echo MSBuild version is %VSCMD_VER% - too high to build the viosock x86 libraries. Checking EWDK_DIR version...
rem ... Work In Progress here ...
) else (
echo MSBuild version is %VSCMD_VER% - already at the right level to build viosock x86 libraries.
setlocal
set VIRTIO_WIN_NO_ARM=1
pushd
if "%BUILD_FILE%"=="virtio-win.sln" (cd /d "%BUILD_DIR%viosock\lib")
if "%BUILD_FILE%"=="viosock.sln" (cd /d "%BUILD_DIR%\lib")
call ..\..\build\build.bat viosocklib.vcxproj %TARGET% x86
if ERRORLEVEL 1 (
popd
endlocal
set BUILD_FAILED=1
goto :eof
)
popd
endlocal
)
)
)
)Probably best to see my final cut in the PR, but would welcome any further comments.
I see. The renames make more sense now..! Thanks again, Kostiantyn. Best regards, |
|
I didn't notice before, but this part: pushd
if "%BUILD_FILE%"=="virtio-win.sln" (cd /d "%BUILD_DIR%viosock\lib")
if "%BUILD_FILE%"=="viosock.sln" (cd /d "%BUILD_DIR%\lib")...should be: if "%BUILD_FILE%"=="virtio-win.sln" (pushd "%BUILD_DIR%viosock\lib")
if "%BUILD_FILE%"=="viosock.sln" (pushd "%BUILD_DIR%\lib")Late night mangling... 8^d |
|
In similar vein, the following is not required: set VIRTIO_WIN_NO_ARM=1 |
|
Hope to drop fully tested PR in the next 24 to 48 hours. |
Building for amd64/x64 - whether called by
buildAll.batorviosock\buildAll.bat- fails for both Win10 and Win11 targets because of missing x86 components.Same result whether using:
Relevant log snippet for Win10 using master:
Attempting to build for x86 or all architectures using Germanium EWDK results in
'Win32' is not a valid architecture for Kernel mode drivers or UMDF drivers.Attempting to build Win11 for x86 or all architectures using Cobalt EWDK results in
fatal error LNK1181: cannot open input file '<redacted source location>\viosock\wsk\objfre_win11_x86\i386\viosockwsk.lib'.Attempting to build Win10 for x86 or all architectures using Cobalt EWDK succeeds...! Is this the only supported configuration?
Makes it hard to test
buildAll.batascall build\build.bat virtio-win.sln "Win10 Win11" %*fails...Win10_SDV and Win11_SDV do complete but don't appear to produce much. I see there's no entry in
viosock\buildAll.batfor Win10_SDV and Win11_SDV is limited tosys\viosock.vcxproj,wsk\wsk.vcxprojandviosock-wsk-test\viosock-wsk-test.vcxproj.What am I missing...?
The text was updated successfully, but these errors were encountered: