[BUG] nn_descent slow down when Dask Cluster is active

[Intron7]

Describe the bug
nn_descent based algorithms like nn_descent and all_neighbors with nn_descent slow down significantly when a dask cluster is active. The dask cluster is just idle not doing anything

Steps/Code to reproduce bug

from dask_cuda import LocalCUDACluster
from dask.distributed import Client
preprocessing_gpus="0,1"#,2,3,4,5,6,7"
cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES=preprocessing_gpus)
client = Client(cluster)
from cuvs.neighbors import nn_descent
import cupy as cp
X = cp.array(data)
idxparams = nn_descent.IndexParams(
    graph_degree=15,
    intermediate_graph_degree=None,
    metric="sqeuclidean",
)
idx = nn_descent.build(
    idxparams,
    dataset=data,
)
neighbors = cp.array(idx.graph).astype(cp.uint32)

Note
This also effects UMAP. all_neighbors with ivf-pq seems to be not affected.