openBLAS warning and segmentation fault issue

[rvitancol]

I am encountering warnings on openBLAS and eventually segmentation fault that leads to stopping of execution for Novae: Happens during fine-tuning step

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 64 or lower
Segmentation fault (core dumped)

I have tried debugging and different settings for the environment variables that controls openblas number of threads but still getting issues. Here is the thread pool information:

=== Thread Pool Information ===
Active Threading Libraries:
  openblas        | Threads:  57 | Prefix: libopenblas | Version: 0.3.23.dev
  openblas        | Threads:  57 | Prefix: libscipy_openblas | Version: 0.3.28
  openmp          | Threads:  48 | Prefix: libgomp | Version: None
  openmp          | Threads:  57 | Prefix: libgomp | Version: None

PyTorch Threading:
  PyTorch version: 2.8.0+cu128
  Number of threads: 48
  Inter-op threads: 57

  Parallel info:
ATen/Parallel:
	at::get_num_threads() : 48
	at::get_num_interop_threads() : 57
OpenMP 201511 (a.k.a. OpenMP 4.5)
	omp_get_max_threads() : 48
Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
	mkl_get_max_threads() : 48
Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
std::thread::hardware_concurrency() : 96
Environment variables:
	OMP_NUM_THREADS : 57
	MKL_NUM_THREADS : 57
ATen parallel backend: OpenMP

=== CPU Information (lscpu) ===
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Byte Order:                              Little Endian
Address sizes:                           46 bits physical, 48 bits virtual
CPU(s):                                  96
On-line CPU(s) list:                     0-95
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               1
NUMA node(s):                            1
Vendor ID:                               GenuineIntel
CPU family:                              6

Do you have suggestions on other things to try to make this work.

Thank you.
Rona

[quentinblampey]

Hi @rvitancol, thanks for reporting!

I have not experienced this before. Could you please show me which environment variables you tried for OPENBLAS? I guess something like that?

os.environ['OPENBLAS_NUM_THREADS'] = '4'
# maybe other variables?

A few follow-up questions:

• How many workers did you use when running fine_tune?
• Do you frequently have this kind of issues when using pytorch, or is it only when using novae?

[rvitancol]

I revise my statement, I observed the error also even without fine tuning. I am using CPU and num_workers = 0.

Tried these 3 variables: max_threads we tried were 1,8,16,32, and some computation based values 60% of usable threads.
os.environ["OPENBLAS_NUM_THREADS"] = max_threads
os.environ["OMP_NUM_THREADS"] = max_threads
os.environ["MKL_NUM_THREADS"] = max_threads

I rarely use pytorch so I just encountered it in Novae.

[quentinblampey]

Okay, good to know. In that case, is it possible to trace where this warning happens? Maybe during torch import?

Also, does Segmentation fault (core dumped) happen right after the warning? Maybe the warning is not the actual issue

[rvitancol]

Warning happens after the compute representations, followed by the error from OpenBLAS.
For this log it happens when it was executing fine-tuning at the step where it was computing representations since the default for reference is not yet None. I included here so you can see where it happens and also the BLAS error message.

Creating spatial neighbors...
[INFO] (novae.utils.build) Computing graph on 245,455 cells (coord_type=generic, delaunay=True, radius=[0.0, 0.08], n_neighs=None)
Auto-selected human model: MICS-Lab/novae-human-0
Using pretrained model. Loading pretrained model...
Novae model
   ├── Known genes: 60697
   ├── Parameters: 32.1M
   ├── Model name: MICS-Lab/novae-human-0
   ├── Trained: True
   └── Multimodal: False
Fine-tuning for 50 epochs...
[INFO] (novae.utils._validate) Preprocessed 1 adata object(s) with sc.pp.normalize_total and sc.pp.log1p (raw counts were saved in adata.layers['counts'])
Computing representations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [04:31<00:00,  1.39s/it]
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 64 or lower
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
Segmentation fault (core dumped)
[1] 139

When I dont call fine-tuning, warning also happens in the Computing representations step.

[quentinblampey]

Do you know if other people in your institute have this type of warning when working with Pytorch? I think it's likely hardware-specific, and probably only related to pytorch.

Something you can try also is to make a new fresh environment and install novae with conda instead of pip, e.g.:

conda install bioconda::novae

Conda may be better at installing pytorch on some specific hardware setup

[rvitancol]

Hi @quentinblampey Thanks for the reply. We are encountering the same issue for this environment set-up but not in our personal smaller CPU core set-up. Unfortunately we cannot use conda.
To mitigate the error, we are changing the instance type to a lower number of CPU cores. This is now in testing. I will update if this resolves the issue.