Exactly same libraries both Linux and Windows and Windows is widly slower what could be reason? FLUX Fine Tuning

[FurkanGozukara]

I just tested FLUX Fine Tuning on Windows (RTX 5090) and Linux (RunPod RTX 5090 and Massed Compute RTX 6000 PRO)

The thing is on Linux the training speed is at least 25% faster than Windows

I am using Adafactor optimizer and Full bf16 training

How could it be?

No block swap used in all tests since it fits into 32 GB VRAM

I am using Torch 2.8 and CUDA 12.9 exactly same libraries on both platforms

There weren't this much difference before

Moreover I was getting like 8.5 second / it before on RTX 3090 TI on Windows and now exactly same config is around 11 second / it on Windows

How can we debug reason you think? What could be the culprit? @kohya-ss

[kohya-ss]

There are several possible reasons, such as problems with the source code or your environment.
Could you please check out one of the past commit versions and see if the speed goes back to normal? If the speed doesn't change, it may be an environmental issue.
If the speed goes back to normal, please let me know that commit version. I will perform a test for the version.

[FurkanGozukara]

@kohya-ss yes i just tested a very old commit and same

definitely related to libraries

which library might be culprit you think?

here my pip freeze

Microsoft Windows [Version 10.0.26200.6725]
(c) Microsoft Corporation. All rights reserved.

F:\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\Scripts>activate

(venv) F:\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\Scripts>pip freeze
absl-py==2.3.1
accelerate==1.6.0
aiofiles==23.2.1
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.11.0
bitsandbytes==0.48.0
Brotli==1.1.0
certifi==2025.8.3
charset-normalizer==3.4.3
click==8.3.0
colorama==0.4.6
dadaptation==3.2
diffusers==0.32.1
docker-pycreds==0.4.0
easygui==0.98.3
einops==0.7.0
exceptiongroup==1.3.0
fairscale==0.4.13
fastapi==0.118.0
ffmpy==0.6.1
filelock==3.19.1
flash-attn @ https://huggingface.co/MonsterMMORPG/Wan_GGUF/resolve/main/flash_attn-2.8.2-cp310-cp310-win_amd64.whl#sha256=66d81656f02a39abc2982561018e74dc4ee7a747f43cad173028d42282e37ffa
fsspec==2025.9.0
ftfy==6.3.1
gitdb==4.0.12
GitPython==3.1.45
gradio==5.47.2
gradio_client==1.13.3
groovy==0.1.2
grpcio==1.75.1
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.34.3
idna==3.10
imagesize==1.4.1
importlib_metadata==8.7.0
invisible-watermark==0.2.0
Jinja2==3.1.6
-e git+https://github.com/kohya-ss/sd-scripts@fcb2ff010cf2e42c50b3745a17317f2d4b4319d9#egg=library
lion-pytorch==0.2.3
lycoris_lora==3.2.0.post2
Markdown==3.9
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mdurl==0.1.2
mpmath==1.3.0
networkx==3.4.2
numpy==1.26.4
omegaconf==2.3.0
onnx==1.16.1
open-clip-torch==2.20.0
opencv-python==4.10.0.84
orjson==3.11.3
packaging==25.0
pandas==2.3.3
pillow==11.3.0
platformdirs==4.4.0
prodigy-plus-schedule-free==1.9.2
prodigyopt==1.1.2
protobuf==3.20.3
psutil==7.1.0
pydantic==2.11.9
pydantic_core==2.33.2
pydub==0.25.1
Pygments==2.19.2
python-dateutil==2.9.0.post0
python-multipart==0.0.20
pytorch_optimizer==3.7.0
pytz==2025.2
PyWavelets==1.8.0
PyYAML==6.0.3
regex==2025.9.18
requests==2.32.5
rich==14.1.0
ruff==0.13.2
safehttpx==0.1.6
safetensors==0.4.5
sageattention @ https://huggingface.co/MonsterMMORPG/Wan_GGUF/resolve/main/sageattention-2.2.0-cp39-abi3-win_amd64.whl#sha256=5b278e4de9cbffc27d2cf0fd054846219eb7969c7b882ad6728d342bd099c42d
schedulefree==1.4
scipy==1.11.4
semantic-version==2.10.0
sentencepiece==0.2.1
sentry-sdk==2.39.0
setproctitle==1.3.7
shellingham==1.5.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
starlette==0.48.0
sympy==1.14.0
tensorboard==2.20.0
tensorboard-data-server==0.7.2
timm==1.0.15
tk==0.1.0
tokenizers==0.21.4
toml==0.10.2
tomlkit==0.13.3
torch==2.8.0+cu129
torchaudio==2.8.0+cu129
torchvision==0.23.0+cu129
tqdm==4.67.1
transformers==4.54.1
triton-windows==3.4.0.post20
typer==0.19.2
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
urllib3==2.5.0
uvicorn==0.37.0
voluptuous==0.15.2
wandb==0.18.0
wcwidth==0.2.14
websockets==15.0.1
Werkzeug==3.1.3
xformers @ https://huggingface.co/MonsterMMORPG/Wan_GGUF/resolve/main/xformers-0.0.33+c159edc0.d20250906-cp39-abi3-win_amd64.whl#sha256=acfd8cc8624ebc987b1ebca806273d7f0aab5a1b86a5f60c2caf7a17d43d0176
zipp==3.23.0

(venv) F:\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\Scripts>

[kohya-ss]

I think the cause is likely to be PyTorch, CUDA, or xformers. If the GPU usage rate is significantly different, it may be due to libraries related to training data, such as accelerate or transformers (when training text encoder) etc.

[FurkanGozukara]

@kohya-ss i see. i use exactly same libraries on linux and windows

[kohya-ss]

Hmm so accelerate or transformers are unlikely to be the cause, and it might be PyTorch, CUDA, or the version of the graphics driver.

[FurkanGozukara]

@kohya-ss pytorch cuda python all same. except graphics driver but i doubt it is the culprit. it usually makes no difference