parallel processing with silence_periods

[JeffreyBoucher]

Hello,

In order to guard against a few periods of time in which some recordings of mine were saturated, I've:

• written and applied a function to detect these periods
• applied silence_periods with the "noise" function to them and the surrounding milliseconds.
• noted these periods for later exclusion from analysis

This process works quite well. However: silence_periods is quite slow when I don't run it parallel. The outer layer of this function does not appear to take an n_jobs kwarg; when I go into silence_periods.py myself and add the kwarg to get_noise_levels, it works, but the parallel pool doesn't properly close afterward, which leads to bugs later when I try to parallel process something else...

So, this post is either a feature request or a question as to how to do this properly if I am not. Is there a way to insist on parallel processing for silence_periods? If not, could you please implement this for future releases? Additionally: is there a spikeinterface-internal way I should be closing the parallel pool after I start one like this?

Thanks!

Jeff Boucher

[zm711]

@samuelgarcia I think you're best to explain this !

[samuelgarcia]

Hi Jeffrey.
I am not sure to understand the question.
This silence_periods return a lazy recording that compute on the fly the masking with noise around a trigger list.
In short the get_trace will trigger the "computation" like any other recording.

The parralell will happen if you do a recording.save(..., n_jobs=10) or detect_peak on it or any other operation that scan and read by chunk the entire recording.

[JeffreyBoucher]

Hello,

My understanding of SI is that how you describe things is how it should work. However, it's not how it works in my code. When I run the line:
"
recording = si.silence_periods(recording,windowsToSilenceArray, mode="noise")
"
I get the following output from my terminal:

noise_level (no parallelization): 100%|██████████| 20/20 [00:13<00:00, 1.50it/s]

I can parallelize this in the way I described above.

In addition to this and perhaps more importantly (though I can't confirm because of a sudden IT problem on the relevant computer), I believe this process occurs when I run "si.concatenate recordings", another function which does not take an n_jobs argument. I could also speed this up the way I described, but then had the problem with the pool not closing properly...

Anyway... If you can reproduce this, maybe the issue is that the silence_periods function is not properly lazy?

Also, for the record, I am using SI version 102.3

Thanks for your help!

Jeff Boucher

[zm711]

It looks to me like we are running into this here:

spikeinterface/src/spikeinterface/preprocessing/silence_periods.py

Lines 72 to 78 in 68c8d00

	if mode in ["noise"]:
	if noise_levels is None:
	random_slices_kwargs = random_chunk_kwargs.copy()
	random_slices_kwargs["seed"] = seed
	noise_levels = get_noise_levels(
	recording, return_scaled=False, random_slices_kwargs=random_slices_kwargs
	)

and this function takes in job_kwargs

spikeinterface/src/spikeinterface/core/recording_tools.py

Lines 697 to 704 in 68c8d00

	def get_noise_levels(
	recording: "BaseRecording",
	return_scaled: bool | None = None,
	return_in_uV: bool = True,
	method: Literal["mad", "std"] = "mad",
	force_recompute: bool = False,
	random_slices_kwargs: dict = {},
	**kwargs,

I think one way we could test this @JeffreyBoucher is could you set global job_kwargs and see if silence_periods speeds up and the pool doesn't get broken. If that is the case then we should probably appropriately expose these kwargs in the case someone gets noise levels. To set global job kwargs do:

si.set_global_job_kwargs(n_jobs=xx)

[JeffreyBoucher]

Okay, I will give this a try and let you know (as soon as the IT problem is solved)

Thanks!

[JeffreyBoucher]

Hello!

The IT problem was solved so I was able to try setting the global. This seems to be working quite well, thanks! This solves my short-term problem, though I still suspect that this is non-lazy behavior you'll want to fix...

Also, I was able to (fail to) confirm that si.concatenate recordings was my other problem. It's not: it was the function next to it, set_probe, run on the concatenated recording object.

Thanks for your help!

Jeff Boucher

[JeffreyBoucher]

Spoke a bit too soon I guess, oddly enough when I run detect_peaks just after all this I can neither set an n_jobs manually nor use a global argument... The problem specifically is with noise_level again, internal to this. I don't really understand this, but this used to be the stage where I had problems with my parallel pool not closing... I'll let you know if I can finagle something.

[zm711]

I will say getting noise levels is not a lazy process as it requires running to the actual data. So this does break the idea of pure laziness. I don't think this is being applied eagerly to the recording itself at this stage (ie the recording object is still lazy), but getting the noise chunks is not.

Spoke a bit too soon I guess, oddly enough when I run detect_peaks just after all this I can neither set an n_jobs manually nor use a global argument... The problem specifically is with noise_level again, internal to this. I don't really understand this, but this used to be the stage where I had problems with my parallel pool not closing... I'll let you know if I can finagle something.

yeah definitely let us know. Also could you let us know your OS and whether you're doing cluster (files + cpus remote) vs accessing data on a network (ie just files are remote analysis local)?

[JeffreyBoucher]

Getting things like this sometimes too:

[JeffreyBoucher]

The operating system I am using is Windows 11, with the IDE pycharm.

I'm using a local computer, not a cluster. The files on are a NAS.

Thanks!

Jeff Boucher

[JeffreyBoucher]

Alright, getting back into the SI code internals a bit...

I don't know the circumstances in which si.set_global_job_kwargs does or does not apply, but in the peak_detection code (within check_params), eventually this line is reached here, involving get_noise_levels again:

The random_chunk_kwargs are empty for me. Earlier in the code job_kwargs and method_kwargs are split:

The method_kwargs do transfer down to check_params, but only the detect_threshold argument (for me) is kept.

I think I might be able to hack a solution here in a few different ways, each of which might break the pool for me again... It seems odd to me that si.set_global_job_kwargs wouldn't work here; do you know why that might be?

Thanks for your help

[JeffreyBoucher]

Dug a bit deeper... The first time that fix_job_kwargs is called within this function, it properly extracts the value I set (15). Every other time it extracts some different globals which includes n_jobs of 1. I'll try to figure out when exactly this shift occurs...

[JeffreyBoucher]

Final set of info before I call it for the night:

I got this warning which may or may not be relevant just before it occured:

"RuntimeWarning: 'spikeinterface.core.job_tools' found in sys.modules after import of package 'spikeinterface.core', but prior to execution of 'spikeinterface.core.job_tools'; this may result in unpredictable behaviour"

The occurrence, also, happened in this section of the "process.py" file of concurrent

I do think I can solve my short term problems by forcing n_jobs = 15 somewhere after it changes, but let me know if this information helps develop a less hacky solution

Thanks!

Jeff Boucher