GeForce RTX 5060 Incompatibility

[nshukla5]

Dear Relion community,
We are trying to install relion-5.0 on a workstation harboring the newly released NVIDIA GeForce RTX 5060 Ti 16GB and are running into issues with PyTorch compatibility. The machine has ubuntu 24.04.02 and has a compatible nvidia driver and cuda 12.8. If we do not install PyTorch separately before Relion installation, it results in an error when running 3D classification with Blush regularization.

Environment:

• OS: Ubuntu 24.04.02 LTS
• MPI runtime: OpenRTE 4.1.6
• RELION version: RELION-5.0.0-commit-adfec8
• Memory: 280 GB
• GPU: NVIDIA GeForce RTX 5060 Ti

Dataset:

• Box size: 60 px
• Pixel size: 2.2Å
• Number of particles: 80,000
• Description: helix

Job options:

• Type of job: Class3D
• Number of MPI processes:
• Number of threads: 1
• Full command:

`which relion_refine_mpi` --o Class3D/job305/run --i Select/job065/particles.star --ref InitialModel/job067/run_it100_class001.mrc --firstiter_cc --trust_ref_size --ini_high 15 --dont_combine_weights_via_disc --preread_images  --pool 10 --pad 2  --ctf --iter 25 --tau2_fudge 4 --particle_diameter 110 --K 1 --flatten_solvent --zero_mask --strict_highres_exp 12 --blush  --oversampling 1 --healpix_order 2 --offset_range 2 --offset_step 2 --sym C1 --norm --scale  --helix --helical_outer_diameter 72 --helical_nr_asu 3 --helical_twist_initial 100 --helical_rise_initial 10.2 --helical_z_percentage 0.4 --helical_symmetry_search --helical_twist_min 50 --helical_twist_max 150 --helical_twist_inistep 5 --helical_rise_min 5 --helical_rise_max 20 --helical_rise_inistep 2 --helical_keep_tilt_prior_fixed --sigma_tilt 3.33333 --sigma_psi 3.33333 --sigma_rot 0 --helical_sigma_distance 0.666667 --j 3 --gpu "0,1"  --pipeline_control Class3D/job305/

Error message 1:

/home/user/.conda/envs/relion-5.0/lib/python3.10/site-packages/torch/cuda/__init__.py:287:
UserWarning:
NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not
compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60
sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch,
please check the instructions at
https://pytorch.org/get-started/locally/

  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/.conda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/command_line.py",
line 335, in main
    class3d(
  File "/home/user/.conda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/command_line.py",
line 189, in class3d
    denoised_nv, _ = apply_model(
  File "/home/user/.conda/envs/relion-5.0/lib/python3.10/site-packages/torch/utils/_contextlib.py",
line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.conda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/util.py",
line 177, in apply_model
    infer_grid = torch.zeros_like(volume).to(device)
RuntimeError: CUDA error: no kernel image is available for execution
on the device
CUDA kernel errors might be asynchronously reported at some other API
call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Something went wrong in the external Python call...
Command: relion_python_blush
Class3D/job305/run_it001_class001_external_reconstruct.star  --gpu
0,1,

---------------------------------- PYTHON ERROR
---------------------------------
   Has RELION been provided a Python interpreter with the correct environment?
   The interpreter can be passed to RELION either during Cmake configuration by
     using the Cmake flag -DPYTHON_EXE_PATH=<path/to/python/interpreter>.
   NOTE: For some modules TORCH_HOME needs to be set to find pretrained models
---------------------------------------------------------------------------------

  Using python executable: /home/user/.conda/envs/relion-5.0/bin/python


--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero
status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[43628,1],1]
  Exit code:    1
--------------------------------------------------------------------------

We also tried to first install nightly build of PyTorch 2.9.0.dev20250717+cu18, but received an additional error message that only PyTorch versions up to 2.7.1 were compatible for Relion installation.

Error Message 2:

ERROR: Could not find a version that satisfies the requirement
torch==2.9.0.dev20250717+cu128 (from versions: 1.11.0, 1.12.0, 1.12.1,
1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1,
2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0, 2.7.0, 2.7.1)
ERROR: No matching distribution found for torch==2.9.0.dev20250717+cu128

Upon manually installing PyTorch 2.7.1, running the same 3D classification job as before gives us the same Error Message 1 and appears to also be incompatible with the RTX 5060. We were wondering if there are any other workarounds to properly install Relion 5.0 on this GPU? Thank you for your help!

[rbs-sci]

No solution, but can confirm reproducibility. All Python (PyTorch) related parts of RELION 5.0 are broken when running with Blackwell GPUs. I took a particle stack from good runs on Ampere generation hardware and tried 3D classification with BLUSH, DynaMight and Model-Angelo.

BLUSH output:

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/cuda/__init__.py:173: UserWarning: 
NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/command_line.py", line 320, in main
    class3d(
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/command_line.py", line 189, in class3d
    denoised_nv, _ = apply_model(
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/relion_blush/util.py", line 177, in apply_model
    infer_grid = torch.zeros_like(volume).to(device)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Something went wrong in the external Python call...
Command: relion_python_blush Class3D/job002/run_it001_class001_external_reconstruct.star  --gpu 0, 

---------------------------------- PYTHON ERROR ---------------------------------
   Has RELION been provided a Python interpreter with the correct environment?
   The interpreter can be passed to RELION either during Cmake configuration by
     using the Cmake flag -DPYTHON_EXE_PATH=<path/to/python/interpreter>.
   NOTE: For some modules TORCH_HOME needs to be set to find pretrained models
---------------------------------------------------------------------------------

  Using python executable: /opt/anaconda/envs/relion-5.0/bin/python


--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[33796,1],1]
  Exit code:    1
--------------------------------------------------------------------------

DynaMight output:

/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/cuda/__init__.py:173: UserWarning: 
NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
/opt/relion50/bin/relion_python_dynamight: line 30: 445765 Floating point exception(core dumped) "$python_executable" -c "from dynamight import cli; exit(cli())" "$@"

Model-Angelo output:

/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/cuda/__init__.py:173: UserWarning: 
NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
2025-07-25 at 15:24:47 | ERROR | Error in ModelAngelo
Traceback (most recent call last):
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/model_angelo/apps/build_no_seq.py", line 174, in main
    ca_cif_path = c_alpha_infer(ca_infer_args)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/model_angelo/c_alpha/inference.py", line 245, in infer
    meta_net_output = wrapper(meta_batch_list)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/model_angelo/models/multi_gpu_wrapper.py", line 160, in forward
    self.model(**send_dict_to_device(data, device)).to("cpu").to(torch.float32)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/tem/torch_home/hub/checkpoints/model_angelo_v1.0/nucleotides_no_seq/c_alpha/model.py", line 246, in forward
    return self.net(x)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/tem/torch_home/hub/checkpoints/model_angelo_v1.0/nucleotides_no_seq/c_alpha/model.py", line 186, in forward
    features_dict = self.prep_features_dict(x, **kwargs)
  File "/opt/tem/torch_home/hub/checkpoints/model_angelo_v1.0/nucleotides_no_seq/c_alpha/model.py", line 183, in prep_features_dict
    return {"x": self.normalize(x), **kwargs}
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda/envs/relion-5.0/lib/python3.10/site-packages/model_angelo/models/common_modules.py", line 72, in first_channel_only_forward
    mu = torch.mean(x_fc, dim=dims, keepdim=True)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[biochem-fan]

I don't have any Blackwell GPUs at hand so I cannot test myself but can you try
https://discuss.pytorch.org/t/how-do-i-use-pytorch-with-rtx-5060-ti/220926/8
after creating the RELION conda environment afresh? This updates PyTorch (and its dependencies).

@dkimanius @jamaliki @scheres Do any of you have 50x0 GPUs?

[rbs-sci]

Nightly didn't work for me. Latest stable does: 2.7.1

Solution:

conda activate relion-5.0
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

BLUSH running OK. Will check MA and DynaMight when that's finished running.

edit: Do not know if it breaks any of the Napari/tomo bits, several other Python packages were removed and newer versions installed, but no warnings thrown during update.

edit2: Model-Angelo throws a warning I've not seen before:

/opt/tem/torch_home/hub/checkpoints/model_angelo_v1.0/nucleotides_no_seq/c_alpha/model.py:180: UserWarning: `nn.functional.upsample` is deprecated. Use `nn.functional.interpolate` instead.
  return F.upsample(x, size=(H, W, D), mode="trilinear") + y

But appears to work.

edit3: Model was good.

DynaMight crashed, but that's a memory issue on the system I was testing on. Unless newer PyTorch breaks older cards, seems an easy update.

Here's my resulting relion-5.0 environment.yaml after the update if it'll help:

name: relion-5.0
channels:
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h4bc722e_7
  - ca-certificates=2025.6.15=hbd8a1cb_0
  - ld_impl_linux-64=2.40=hf3520f5_7
  - libexpat=2.7.0=h5888daf_0
  - libffi=3.4.2=h7f98852_5
  - libgcc=14.1.0=h77fa898_1
  - libgcc-ng=14.1.0=h69a702a_1
  - libgomp=14.1.0=h77fa898_1
  - liblzma=5.8.1=hb9d3cd8_2
  - liblzma-devel=5.8.1=hb9d3cd8_2
  - libnsl=2.0.1=hd590300_0
  - libsqlite=3.50.1=h6cd9bfd_6
  - libuuid=2.38.1=h0b41bf4_0
  - libxcrypt=4.4.36=hd590300_1
  - libzlib=1.3.1=h4ab18f5_1
  - ncurses=6.5=he02047a_1
  - openssl=3.5.0=h7b32b05_1
  - pip=25.1.1=pyh8b19718_0
  - python=3.10.18=hd6af730_0_cpython
  - python_abi=3.10=5_cp310
  - readline=8.2=h8228510_1
  - setuptools=59.5.0=py310hff52083_0
  - tk=8.6.13=noxft_h4845f30_101
  - wheel=0.44.0=pyhd8ed1ab_0
  - xz=5.8.1=hbcc6ac9_2
  - xz-gpl-tools=5.8.1=hbcc6ac9_2
  - xz-tools=5.8.1=hb9d3cd8_2
  - pip:
      - absl-py==2.1.0
      - alabaster==0.7.16
      - app-model==0.2.8
      - appdirs==1.4.4
      - asciitree==0.3.3
      - asttokens==2.4.1
      - attrs==24.2.0
      - babel==2.16.0
      - biopython==1.81
      - build==1.2.2
      - cachey==0.2.1
      - certifi==2024.8.30
      - charset-normalizer==3.3.2
      - click==8.1.7
      - cloudpickle==3.0.0
      - cmake==3.30.3
      - comm==0.2.2
      - contourpy==1.3.0
      - cycler==0.12.1
      - dask==2024.8.2
      - debugpy==1.8.5
      - decorator==5.1.1
      - dill==0.3.7
      - docstring-parser==0.16
      - docutils==0.17.1
      - dynamight==0.1.dev280+gd7d75e8
      - einops==0.8.0
      - exceptiongroup==1.2.2
      - executing==2.1.0
      - fair-esm==1.0.3
      - fastcluster==1.2.6
      - fasteners==0.19
      - filelock==3.16.0
      - flexcache==0.3
      - flexparser==0.3.1
      - fonttools==4.53.1
      - freetype-py==2.5.1
      - fsspec==2024.9.0
      - future==1.0.0
      - grpcio==1.66.1
      - heapdict==1.0.1
      - hsluv==5.0.4
      - idna==3.8
      - imageio==2.35.1
      - imagesize==1.4.1
      - importlib-metadata==8.4.0
      - in-n-out==0.2.1
      - ipykernel==6.29.5
      - ipython==8.27.0
      - jedi==0.19.1
      - jinja2==3.1.4
      - joblib==1.4.2
      - jsonschema==4.23.0
      - jsonschema-specifications==2023.12.1
      - jupyter-client==8.6.2
      - jupyter-core==5.7.2
      - kiwisolver==1.4.7
      - lazy-loader==0.4
      - libigl==2.5.1
      - lil-aretomo==0.1.1
      - lit==18.1.8
      - llvmlite==0.43.0
      - locket==1.0.0
      - loguru==0.7.0
      - lru-dict==1.3.0
      - magicgui==0.9.1
      - makefun==1.15.4
      - markdown==3.7
      - markdown-it-py==3.0.0
      - markupsafe==2.1.5
      - matplotlib==3.7.2
      - matplotlib-inline==0.1.7
      - mdocfile==0.0.8
      - mdurl==0.1.2
      - model-angelo==1.0.13
      - morphosamplers==0.0.13
      - mpmath==1.3.0
      - mrcfile==1.4.3
      - napari==0.4.18
      - napari-console==0.0.9
      - napari-plugin-engine==0.2.0
      - napari-plugin-manager==0.1.1
      - napari-svg==0.2.0
      - napari-threedee==0.0.26
      - nest-asyncio==1.6.0
      - networkx==3.3
      - npe2==0.7.7
      - numba==0.60.0
      - numcodecs==0.13.0
      - numpy==1.26.1
      - numpydoc==1.5.0
      - nvidia-cublas-cu11==11.10.3.66
      - nvidia-cublas-cu12==12.8.3.14
      - nvidia-cuda-cupti-cu11==11.7.101
      - nvidia-cuda-cupti-cu12==12.8.57
      - nvidia-cuda-nvrtc-cu11==11.7.99
      - nvidia-cuda-nvrtc-cu12==12.8.61
      - nvidia-cuda-runtime-cu11==11.7.99
      - nvidia-cuda-runtime-cu12==12.8.57
      - nvidia-cudnn-cu11==8.5.0.96
      - nvidia-cudnn-cu12==9.7.1.26
      - nvidia-cufft-cu11==10.9.0.58
      - nvidia-cufft-cu12==11.3.3.41
      - nvidia-cufile-cu12==1.13.0.11
      - nvidia-curand-cu11==10.2.10.91
      - nvidia-curand-cu12==10.3.9.55
      - nvidia-cusolver-cu11==11.4.0.1
      - nvidia-cusolver-cu12==11.7.2.55
      - nvidia-cusparse-cu11==11.7.4.91
      - nvidia-cusparse-cu12==12.5.7.53
      - nvidia-cusparselt-cu12==0.6.3
      - nvidia-nccl-cu11==2.14.3
      - nvidia-nccl-cu12==2.26.2
      - nvidia-nvjitlink-cu12==12.8.61
      - nvidia-nvtx-cu11==11.7.91
      - nvidia-nvtx-cu12==12.8.55
      - packaging==24.1
      - pandas==2.2.2
      - parso==0.8.4
      - partd==1.4.2
      - pexpect==4.9.0
      - pillow==10.4.0
      - pint==0.24.3
      - platformdirs==4.3.2
      - pooch==1.8.2
      - prompt-toolkit==3.0.47
      - protobuf==5.28.0
      - psutil==5.9.8
      - psygnal==0.11.1
      - ptyprocess==0.7.0
      - pure-eval==0.2.3
      - pyarrow==17.0.0
      - pyconify==0.1.6
      - pydantic==1.10.18
      - pydantic-compat==0.1.2
      - pygments==2.18.0
      - pyhmmer==0.7.1
      - pynndescent==0.5.13
      - pyopengl==3.1.7
      - pyparsing==3.0.9
      - pyproject-hooks==1.1.0
      - pyqt5==5.15.9
      - pyqt5-qt5==5.15.14
      - pyqt5-sip==12.15.0
      - python-dateutil==2.9.0.post0
      - pytz==2024.1
      - pyyaml==6.0.2
      - pyzmq==26.2.0
      - qtconsole==5.6.0
      - qtpy==2.4.1
      - referencing==0.35.1
      - relion==5.0.1.dev37+g1fdfb954
      - relion-blush==0.0.1
      - relion-classranker==0.0.1
      - requests==2.32.3
      - rich==13.8.0
      - rpds-py==0.20.0
      - scikit-image==0.24.0
      - scikit-learn==1.3.0
      - scipy==1.11.2
      - seaborn==0.12.2
      - six==1.16.0
      - snowballstemmer==2.2.0
      - sphinx==4.5.0
      - sphinxcontrib-applehelp==2.0.0
      - sphinxcontrib-devhelp==2.0.0
      - sphinxcontrib-htmlhelp==2.1.0
      - sphinxcontrib-jsmath==1.0.1
      - sphinxcontrib-qthelp==2.0.0
      - sphinxcontrib-serializinghtml==2.0.0
      - stack-data==0.6.3
      - starfile==0.5.13
      - superqt==0.6.7
      - sympy==1.13.3
      - tensorboard==2.17.1
      - tensorboard-data-server==0.7.2
      - threadpoolctl==3.5.0
      - tifffile==2024.8.30
      - tomli==2.0.1
      - tomli-w==1.0.0
      - toolz==0.12.1
      - topaz-em==0.2.5a0
      - torch==2.7.1+cu128
      - torchaudio==2.7.1+cu128
      - torchvision==0.22.1+cu128
      - tornado==6.4.1
      - tqdm==4.65.0
      - traitlets==5.14.3
      - triton==3.3.1
      - tsnecuda==3.0.1
      - typer==0.9.0
      - typing-extensions==4.12.2
      - tzdata==2024.1
      - umap-learn==0.5.3
      - urllib3==2.2.2
      - vispy==0.12.2
      - wcwidth==0.2.13
      - werkzeug==3.0.4
      - wrapt==1.16.0
      - yet-another-imod-wrapper==0.1.1
      - zarr==2.18.3
      - zipp==3.20.1
prefix: /opt/anaconda/envs/relion-5.0

[biochem-fan]

@rbs-sci Thank you very much for testing.

I wonder if changing

relion/environment.yml

Lines 9 to 10 in adfec82

	- torch==2.0.1
	- torchvision==0.15.2

into

      - torch==2.7.1+cu128
      - torchvision==0.22.1+cu128

or

      - torch==2.7.1
      - torchvision==0.22.1

is enough.

I am hesitant to add +cu128 because it breaks compatibility with older CUDA versions.

If +cu128 is essential, we have to provide two versions of the file.

[rbs-sci]

I'll see if I can test this evening. No system in the lab idling right now that I can potentially break (except the Blackwell box, given that it was not working correctly anyway) even if for a short time.

[biochem-fan]

Thank you very much!

[rbs-sci]

I installed the cu128 version on a system with RTX 3090s, driver 560.35 and CUDA 12.6.

After install:

Successfully installed nvidia-cublas-cu12-12.8.3.14 nvidia-cuda-cupti-cu12-12.8.57 nvidia-cuda-nvrtc-cu12-12.8.61 nvidia-cuda-runtime-cu12-12.8.57 nvidia-cudnn-cu12-9.7.1.26 nvidia-cufft-cu12-11.3.3.41 nvidia-cufile-cu12-1.13.0.11 nvidia-curand-cu12-10.3.9.55 nvidia-cusolver-cu12-11.7.2.55 nvidia-cusparse-cu12-12.5.7.53 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.8.61 nvidia-nvtx-cu12-12.8.55 sympy-1.13.3 torch-2.7.1+cu128 torchaudio-2.7.1+cu128 torchvision-0.22.1+cu128 triton-3.3.1

BLUSH works.

However, I just had a thought that it might have the old PyTorch versions still hiding in in the environment, so I need to remove it completely and test with a completely fresh environment from a custom yaml.

Then I'll test Model-Angelo and DynaMight as well.

But tentatively, it looks like specifying the cu128 versions should be OK for Turing (RTX 20x0) and newer. I know pre-Turing doesn't work with the open driver, which Blackwell requires, so... yes, Pascal (GTX 10x0) and Maxwell will break. So two environment files will be needed to keep Pascal/Maxwell users functional.

[rbs-sci]

OK. Ubuntu 24.04, A6000 (Ampere) GPU.

Default environment.yml, BLUSH and Model-Angelo work.

Modified environment.yml (torch==2.7.1 and torchvision==0.22.1), clean environment install: BLUSH and Model-Angelo work.

Modified environment.yml (torch==2.7.1+cu128 and torchvision==0.22.1+cu128), clean environment install: BLUSH and Model-Angelo work.

I created a pull request (still not completely comfortable with git, hope I got it right) with a new environment.yml file as this breaks the Python environment for pre-Turing GPUs.

edit: Just to clarify, +cu128 is not necessary for Turing -> Ada (it still works, though), and is essential for Blackwell. However, it breaks pre-Turing GPUs, so two environment installs to cover all supported GPUs. I'm uncertain how many Maxwell GPUs would still be in active use for RELION, but Pascal is still solid for tinkering/learning boxes.

[biochem-fan]

Great. Thank you very much for testing and your pull request! I merged your contribution.

Just to clarify, +cu128 is not necessary for Turing -> Ada (it still works, though), and is essential for Blackwell. However, it breaks pre-Turing GPUs, so two environment installs to cover all supported GPUs. I'm uncertain how many Maxwell GPUs would still be in active use for RELION, but Pascal is still solid for tinkering/learning boxes.

At both LMB and my current institute, 1080 Ti and P100 are part of the main cluster. So we definitely have to keep its support.