feat: yet another attempt to add windows builds #231

baszalmstra · 2024-04-05T12:55:37Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

Fixes #32

This PR is another attempt to add Windows builds (see #134) .

For now I disabled all other builds to be able to test the windows part first. I made this PR draft so we don't accidentally merge it.

conda-forge-webservices · 2024-04-05T12:56:09Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'libtorch' output doesn't have any tests.

recipe/meta.yaml

conda-forge-webservices · 2024-04-05T13:17:04Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Old-style Python selectors (py27, py35, etc) are only available for Python 2.7, 3.4, 3.5, and 3.6. Please use explicit comparisons with the integer py, e.g. # [py==37] or # [py>=37]. See lines [54]

For recipe:

It looks like the 'libtorch' output doesn't have any tests.

conda-forge-webservices · 2024-04-05T13:21:57Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'libtorch' output doesn't have any tests.

baszalmstra · 2024-04-06T06:36:53Z

Both pipelines failed because they ran out of disk space:

FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/static/te_wrapper.cpp.obj 
C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\cl.exe  /nologo /TP -DAT_PER_OPERATOR_HEADERS -DCAFFE2_BUILD_MAIN_LIB -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNOMINMAX -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_C10D_GLOO -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_MIMALLOC -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_DEPRECATE=1 -D_UCRT_LEGACY_INFINITY -Dtorch_cpu_EXPORTS -I%SRC_DIR%\build\aten\src -I%SRC_DIR%\aten\src -I%SRC_DIR%\build -I%SRC_DIR% -I%SRC_DIR%\third_party\onnx -I%SRC_DIR%\build\third_party\onnx -I%SRC_DIR%\third_party\foxi -I%SRC_DIR%\build\third_party\foxi -I%SRC_DIR%\third_party\mimalloc\include -I%SRC_DIR%\torch\csrc\api -I%SRC_DIR%\torch\csrc\api\include -I%SRC_DIR%\caffe2\aten\src\TH -I%SRC_DIR%\build\caffe2\aten\src\TH -I%SRC_DIR%\build\caffe2\aten\src -I%SRC_DIR%\build\caffe2\..\aten\src -I%SRC_DIR%\torch\csrc -I%SRC_DIR%\third_party\miniz-2.1.0 -I%SRC_DIR%\third_party\kineto\libkineto\include -I%SRC_DIR%\third_party\kineto\libkineto\src -I%SRC_DIR%\aten\src\ATen\.. -I%SRC_DIR%\c10\.. -I%SRC_DIR%\third_party\pthreadpool\include -I%SRC_DIR%\third_party\cpuinfo\include -I%SRC_DIR%\third_party\fbgemm\include -I%SRC_DIR%\third_party\fbgemm -I%SRC_DIR%\third_party\fbgemm\third_party\asmjit\src -I%SRC_DIR%\third_party\ittapi\src\ittnotify -I%SRC_DIR%\third_party\FP16\include -I%SRC_DIR%\third_party\fmt\include -I%SRC_DIR%\build\third_party\ideep\mkl-dnn\include -I%SRC_DIR%\third_party\ideep\mkl-dnn\src\..\include -I%SRC_DIR%\third_party\flatbuffers\include -external:I%SRC_DIR%\build\third_party\gloo -external:I%SRC_DIR%\cmake\..\third_party\gloo -external:I%SRC_DIR%\third_party\protobuf\src -external:I%SRC_DIR%\third_party\XNNPACK\include -external:I%SRC_DIR%\third_party\ittapi\include -external:I%SRC_DIR%\cmake\..\third_party\eigen -external:I%SRC_DIR%\third_party\ideep\mkl-dnn\include\oneapi\dnnl -external:I%SRC_DIR%\third_party\ideep\include -external:I%SRC_DIR%\caffe2 -external:W0 /DWIN32 /D_WINDOWS /GR /EHsc /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /utf-8 /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273 -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION /O2 /Ob2 /DNDEBUG /bigobj -DNDEBUG -std:c++17 -MD -DCAFFE2_USE_GLOO -DTH_HAVE_THREAD /EHsc /bigobj -O2 -DONNX_BUILD_MAIN_LIB -openmp:experimental /showIncludes /Focaffe2\CMakeFiles\torch_cpu.dir\__\torch\csrc\jit\runtime\static\te_wrapper.cpp.obj /Fdcaffe2\CMakeFiles\torch_cpu.dir\ /FS -c %SRC_DIR%\torch\csrc\jit\runtime\static\te_wrapper.cpp
%SRC_DIR%\torch\csrc\jit\runtime\static\te_wrapper.cpp : fatal error C1085: Cannot write compiler generated file: '%SRC_DIR%\build\caffe2\CMakeFiles\torch_cpu.dir\__\torch\csrc\jit\runtime\static\te_wrapper.cpp.obj': No space left on device

What would be the most idiomatic way to solve this issue?

weiji14 · 2024-04-06T07:14:05Z

Try following https://conda-forge.org/docs/maintainer/conda_forge_yml/#azure to clear some disk space. Set this in conda-forge.yml

azure:
  free_disk_space: true

and then rerender the feedstock.

Tobias-Fischer · 2024-04-06T07:45:34Z

I think there’s little we can do - the Azure free disk space setting is already enabled. I’d try and see if these build locally. Perhaps there is a way to use the Quansight servers for Windows as well, the same way they are used for Linux builds? If not, I guess if there are some volunteers to build these locally then this would be an option - I did that for aarch64 for a while for qt. Conda-forge has a windows server too, but disk space has always been quite restricted there too so it might be a bit of a pain.

jakirkham · 2024-04-06T07:49:24Z

Perhaps cross-compiling Windows from Linux is worth trying? Here is a different feedstock PR that does this ( conda-forge/polars-feedstock#187 )

If we were to use Quansight resources for Windows, being able to run the build on Linux (so cross-compiling) would be very helpful

baszalmstra · 2024-04-06T08:05:34Z

Try following conda-forge.org/docs/maintainer/conda_forge_yml/#azure to clear some disk space. Set this in conda-forge.yml
azure:
  free_disk_space: true

Sadly thats already set:

pytorch-cpu-feedstock/conda-forge.yml

Line 2 in 9e99e03

free_disk_space: true

I think there’s little we can do - the Azure free disk space setting is already enabled. I’d try and see if these build locally. Perhaps there is a way to use the Quantstack servers for Windows as well, the same way they are used for Linux builds?

I assume you mean the runners provided through open-gpu-server by Quantsight and MetroStar? This PR only build the cpu-only version but if we also start building for Cuda I think this is the only possible way forward (let alone for other related repositories like tensorflow). However, the open-gpu-servers don't seem to provide any Windows images. Do you know who I should contact to get the ball rolling?

If not, I guess if there are some volunteers to build these locally then this would be an option

That would be an option but Id prefer to automate and open-source things as much as possible. Having something hooked up to this repository would be ideal.

Perhaps cross-compiling Windows from Linux is worth trying?

The native code of the example you linked is using Rust which makes this much easier. I doubt that this would be easy to achieve with pytorch.

baszalmstra · 2024-04-06T08:06:48Z

I also expect another error when actual linking starts. On my local machine that takes at least 16GB of memory. The cuda version will mostly require more.

jakirkham · 2024-04-06T08:15:09Z

Perhaps cross-compiling Windows from Linux is worth trying?

The native code of the example you linked is using Rust which makes this much easier. I doubt that this would be easy to achieve with pytorch.

If we don't try, we won't know

baszalmstra · 2024-04-06T09:38:07Z

If we don't try, we won't know

Although that is technically true, its already hard enough to build pytorch natively. Adding cross-compilation in the mix seems to me to complicate this even further. Id much rather first focus on getting native builds working. Even if we need to modify the infrastructure to do so. I think having the ability to do resource intensive windows builds would be a huge benefit for the conda-forge ecosystem in general.

However, if all else fails cross-compiling seems like a worthwhile avenue to explore.

bkpoon · 2024-04-06T20:24:50Z

One thing to try is to move the build from D:\ to a directory that you have write access to on C:\. I have done this on a personal feedstock where I needed much more disk space. You can modify your conda-forge.yml file with

azure:
  settings_win:
    variables:
      CONDA_BLD_PATH: C:\\Miniconda\\envs\\

You should have roughly 70 GB free on C:\.

baszalmstra · 2024-04-06T20:52:21Z

Thanks! I added that to the PR. I quickly searched github and it seems c:\bld\ is used more often so I tried that.

bkpoon · 2024-04-06T20:57:07Z

Just make sure that the directory exists and is writeable. Also, you need to rerender for the variable to be set. This comment should trigger the bot.

@conda-forge-admin, please rerender

hmaarrfk · 2024-04-06T21:08:02Z

This PR only build the cpu-only version but if we also start building for Cuda I think this is the only possible way forward (let alone for other related repositories like tensorflow). However, the open-gpu-servers don't seem to provide any Windows images. Do you know who I should contact to get the ball rolling?

A bit of history. Back when this feedstock was created 6 years ago, the pytorch officially suggested that people install two distinct packages pytorch-cpu or pytorch-gpu. Therefore it felt appropriate to create pytorch-cpu package because it would throw an error for those trying to install pytorch-gpu. These instructions have changed upstream.

I personally feel like for windows users, we would HURT their experience to not have a GPU package in 2024.

baszalmstra · 2024-04-07T05:31:07Z

I personally feel like for windows users, we would HURT their experience to not have a GPU package in 2024.

Couldnt agree more. I started with CPU only to be able to make incremental progression. My goal is definitely to be able to build the cuda version too!

hmaarrfk · 2024-04-09T23:20:22Z

well few things:

I might try to build locally.
After locally works for 1 python, I might try to enable the mega builds. When you build locally, it saves all the pytorch library compilation and makes compilation take "1.2x" time instead of "4x" time due to the repeated compilaiton of the library for each python version.
Try to enable cuda using the CI.

Typically we "stop" the compilation on the CIs when we reach your stage (seems like it is working OK enough...).

Tobias-Fischer · 2024-05-06T22:43:52Z

Hi @baszalmstra @hmaarrfk - do you have any updates on this? It would be amazing to see this happen :)!

baszalmstra · 2024-05-07T04:56:56Z

@Tobias-Fischer Im still working on the Cuda builds but its a slow process because it takes ages to build them locally so iteration times are suuuper slow.

In parallel we are also looking into getting large Windows runners into the conda-forge infrastructure.

baszalmstra · 2024-05-11T14:33:24Z

Small update:

I have something compiling locally. Still lots of issues (like Windows builds of pytorch 2.1.2 dont compile with python 3.12) but making steady progress. Currently getting megabuilds to work. Will push when I have something reliably working.

baszalmstra · 2024-05-12T08:48:27Z

I got to the testing stage and noticed this:

pytorch-cpu-feedstock/recipe/meta.yaml

Line 300 in f9fd731

    
           - OMP_NUM_THREADS=4 python ./test/run_test.py || true  # [not win and not (aarch64 and cuda_compiler_version != "None")]

However this seems to always fail with (this is from the logs of the latest release):

Ignoring disabled issues:  ['']
Unable to import boto3. Will not be emitting metrics.... Reason: No module named 'boto3'
Missing pip dependency: pytest-rerunfailures, please run `pip install -r .ci/docker/requirements-ci.txt`

Some dependencies are missing. Particularly:

pytest-rerunfailures
pytest-shard (not on conda-forge)
pytest-flakefinder (not on conda-forge)
pytest-xdist

(as can be seen here https://github.com/pytorch/pytorch/blob/6c8c5ad5eaf47a62fafbb4a2747198cbffbf1ff0/test/run_test.py#L1705)

Given that the test is allowed to fail (due to || true). Should we just remove it? Or put in the effort to fix these tests?

h-vetinari · 2024-05-12T09:02:10Z

Given that the test is allowed to fail (due to || true). Should we just remove it? Or put in the effort to fix these tests?

The more we fix, the better. If it's really a lot of failures, we might not fix it right away (though depending on the severity of the failures, we might want to think twice about releasing something in that state).

In any case, let's leave the testing in, add the required dependencies, and pick up as many fixes as we can.

…st in merge

Tobias-Fischer · 2025-01-09T11:04:36Z

No additional test failures with the mkldnn tests enabled, new summary: 10 failed, 7492 passed, 1446 skipped, 13 deselected, 31 xfailed, 75976 warnings

Do we think it's worth trying some other configurations (non-mkl, cuda, ..) after the current run (assuming it goes ok)?

I would suggest to let the dust settle for a while. Having a first PyTorch package on Windows is high-value, the rest is lower-value. CUDA would add more than non-mkl, in case one would like to try a next build config later. The upstream Windows CUDA packages were discussed as candidate for dropping multiple times, since it's a lot of work and not all that relevant to production needs, only for local development (and it's possible to use WSL for that too).

Ok - let me see how the CUDA mkl build is going.

danpetry · 2025-01-09T14:44:22Z

if anyone has opinions about the dozen of test failures

For whatever it's worth, it looks fine to me. As pointed out, it's comparable to the number of failures in their pip package. Pip's getting typeerrors while these are all maths accuracy errors, so less critical afaics.

Tobias-Fischer · 2025-01-10T01:59:25Z

CUDA+mkl build succeeded - hooray!

I've marked this PR as ready for review @conda-forge/pytorch-cpu - looking for any feedback before enabling the full build pipeline.

My plan would be to mark the blas_impl: generic variant as unix-only (https://github.com/baszalmstra/pytorch-cpu-feedstock/blob/44c603513eb1166b44de29f5858763ae519ac340/recipe/conda_build_config.yaml#L9), then remove skip in https://github.com/baszalmstra/pytorch-cpu-feedstock/blob/44c603513eb1166b44de29f5858763ae519ac340/recipe/meta.yaml#L64-L66 and rerender.

h-vetinari · 2025-01-10T08:51:51Z

recipe/meta.yaml

+# TODO Temporary pin, remove
+{% set mkl = "<2025" %}


This is noted as temporary, what's the plan/status here?

To be honest I’m not sure if the right (compatible with mkl) version of intel-openmp would be pulled in without it. I can test after getting some more feedback, I want to avoid running CI more than needed now.

h-vetinari · 2025-01-10T09:23:07Z

I'm attempting a merge of this PR and #305 in #316. All the commits here are maintained 1:1, and this PR will show up as merged if/once #316 is merged. 🤞 we get everything passing this time

The only question that remains: who writes a blog post about this epic journey? 😛

huge thanks to @baszalmstra @Tobias-Fischer for the work here, and of course to prefix.dev for sponsoring the server!!! 🙏 🥳

Tobias-Fischer · 2025-01-10T09:59:48Z

Happy to write a blog post - very happy to jointly write with others involved @baszalmstra et al. :)

baszalmstra · 2025-01-10T10:59:20Z

Amazing! Id be happy to contribute to a blog post!

h-vetinari · 2025-01-26T04:31:25Z

Any updates on that blogpost @Tobias-Fischer @baszalmstra? 🙃 Does someone want to start a draft in a google-doc somewhere (probably private initially)?

h-vetinari · 2025-01-26T04:35:34Z

On that note: I randomly noticed today that we merged windows support exactly on the 4th anniversary of #32 🥳

Tobias-Fischer · 2025-01-27T23:14:33Z

Any updates on that blogpost @Tobias-Fischer @baszalmstra? 🙃 Does someone want to start a draft in a google-doc somewhere (probably private initially)?

I suggested to @baszalmstra that it might be worth waiting until some more downstream packages (besides torchvision) have incorporated Windows support, plus things like #333 are resolved. I'd hate if people try and then revert back because of teething issues they face. What do you think?

h-vetinari · 2025-01-27T23:16:37Z

Sure, we can choose the timing of publication to suit the circumstances, but first we need to write the post in the first place, and that may take some time as well.

Summary: Now that the [PyTorch conda package is available on Windows](conda-forge/pytorch-cpu-feedstock#231), we can enable building PyMomentum on this platform. ## Checklist: - [x] Adheres to the [style guidelines](https://facebookincubator.github.io/momentum/docs/developer_guide/style_guide) - [x] Codebase formatted by running `pixi run lint` Test Plan: CI on Windows Differential Revision: D71858372 Pulled By: jeongseok-meta

Summary: Now that the [PyTorch conda package is available on Windows](conda-forge/pytorch-cpu-feedstock#231), we can enable building PyMomentum on this platform. - [x] Adheres to the [style guidelines](https://facebookincubator.github.io/momentum/docs/developer_guide/style_guide) - [x] Codebase formatted by running `pixi run lint` Test Plan: CI on Windows Differential Revision: D71858372 Pulled By: jeongseok-meta

Summary: Now that the [PyTorch conda package is available on Windows](conda-forge/pytorch-cpu-feedstock#231), we can enable building PyMomentum on this platform. ## Checklist: - [x] Adheres to the [style guidelines](https://facebookincubator.github.io/momentum/docs/developer_guide/style_guide) - [x] Codebase formatted by running `pixi run lint` Test Plan: CI on Windows Differential Revision: D71858372 Pulled By: jeongseok-meta

Summary: Now that the [PyTorch conda package is available on Windows](conda-forge/pytorch-cpu-feedstock#231), we can enable building PyMomentum on this platform. ## Checklist: - [x] Adheres to the [style guidelines](https://facebookincubator.github.io/momentum/docs/developer_guide/style_guide) - [x] Codebase formatted by running `pixi run lint` Pull Request resolved: #238 Test Plan: CI on Windows Reviewed By: juliencbmeta Differential Revision: D71858372 Pulled By: jeongseok-meta fbshipit-source-id: a05d8d88c7fcd81472ff787349ef9deef6f3deb9

baszalmstra requested review from Tobias-Fischer, beckermr, benjaminrwilson, hmaarrfk and sodre as code owners April 5, 2024 12:55

baszalmstra marked this pull request as draft April 5, 2024 13:00

hmaarrfk reviewed Apr 5, 2024

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

baszalmstra mentioned this pull request Apr 6, 2024

Windows VMs Quansight/open-gpu-server#31

Closed

Tobias-Fischer added 2 commits January 9, 2025 21:01

Rerender + enable generic blas build + re-add migrations that were lo…

93e950d

…st in merge

Do not test non-mkl builds

44c6035

Tobias-Fischer marked this pull request as ready for review January 10, 2025 01:55

Tobias-Fischer requested a review from jeongseok-meta as a code owner January 10, 2025 01:55

h-vetinari reviewed Jan 10, 2025

View reviewed changes

h-vetinari mentioned this pull request Jan 10, 2025

Joint merge of windows and kineto PRs #316

Merged

hmaarrfk merged commit 44c6035 into conda-forge:main Jan 14, 2025
4 checks passed

jeongseok-meta mentioned this pull request Jan 15, 2025

Enable Windows conda-forge/rethinkdb-python-feedstock#6

Merged

5 tasks

dashagurova mentioned this pull request Jan 16, 2025

Blog post to address Pytorch channel deprecation conda-forge/conda-forge.github.io#2381

Closed

3 tasks

h-vetinari mentioned this pull request Jan 28, 2025

OpenMP follow-ups on windows #336

Open

jeongseok-meta mentioned this pull request Mar 19, 2025

Build pymomentum on Windows facebookresearch/momentum#238

Closed

2 tasks

Uh oh!

feat: yet another attempt to add windows builds #231

feat: yet another attempt to add windows builds #231

Uh oh!

Conversation

baszalmstra commented Apr 5, 2024

Uh oh!

conda-forge-webservices bot commented Apr 5, 2024

Uh oh!

Uh oh!

conda-forge-webservices bot commented Apr 5, 2024

Uh oh!

conda-forge-webservices bot commented Apr 5, 2024

Uh oh!

baszalmstra commented Apr 6, 2024

Uh oh!

weiji14 commented Apr 6, 2024

Uh oh!

Tobias-Fischer commented Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Apr 6, 2024

Uh oh!

baszalmstra commented Apr 6, 2024

Uh oh!

baszalmstra commented Apr 6, 2024

Uh oh!

jakirkham commented Apr 6, 2024

Uh oh!

baszalmstra commented Apr 6, 2024

Uh oh!

bkpoon commented Apr 6, 2024

Uh oh!

baszalmstra commented Apr 6, 2024

Uh oh!

bkpoon commented Apr 6, 2024

Uh oh!

hmaarrfk commented Apr 6, 2024

Uh oh!

baszalmstra commented Apr 7, 2024

Uh oh!

hmaarrfk commented Apr 9, 2024

Uh oh!

Tobias-Fischer commented May 6, 2024

Uh oh!

baszalmstra commented May 7, 2024

Uh oh!

baszalmstra commented May 11, 2024

Uh oh!

baszalmstra commented May 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

h-vetinari commented May 12, 2024

Uh oh!

Tobias-Fischer commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danpetry commented Jan 9, 2025

Uh oh!

Tobias-Fischer commented Jan 10, 2025

Uh oh!

h-vetinari Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

Tobias-Fischer Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

h-vetinari commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tobias-Fischer commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

baszalmstra commented Jan 10, 2025

Uh oh!

Uh oh!

h-vetinari commented Jan 26, 2025

Uh oh!

h-vetinari commented Jan 26, 2025

Tobias-Fischer commented Apr 6, 2024 •

edited

Loading

baszalmstra commented May 12, 2024 •

edited

Loading

Tobias-Fischer commented Jan 9, 2025 •

edited

Loading

Tobias-Fischer Jan 10, 2025 •

edited

Loading

h-vetinari commented Jan 10, 2025 •

edited

Loading

Tobias-Fischer commented Jan 10, 2025 •

edited

Loading