Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FileNotFoundError: [Errno 2] No such file or directory: 'data/meta\\masks_overlayed_eroded_0_dilated_0\\train\\masks\\000000150992 #235

Open
angeloprop opened this issue Feb 1, 2021 · 3 comments

Comments

@angeloprop
Copy link

Someone can help me to fix this problem
Immagine 2021-02-01 100606

Here is reported my neptune.YAML file
Immagine 2021-02-01 101009

@zeciro
Copy link

zeciro commented Mar 8, 2021

Hi,
I am having the exact same issue.
I am running a smaller data set with 1000 images each for the training and validation sets just to try things out.
Preparing the masks and the metadata works fine.
When I start training, I get the above "No such file or directory" issue. However I checked, the file does exist in that location.

Comments are appreciated. I am gonna try to run the entire data set for comparison.

Cheers

@zeciro
Copy link

zeciro commented Mar 8, 2021

I think it has to do with the file paths and running things under windows instead of linux.

Here is a similar discussion:
https://stackoverflow.com/questions/60635464/confusing-problem-filenotfounderror-errno-2-no-such-file-or-directory

I attempted to fix this in neptune.yaml by changing data/meta to data\meta but that did not change the result.
In src/utils.py I also changed line 150 and 162 to use \ instead of /

However the outcome is the same even though the file path now seems to be correct.

Traceback (most recent call last):
File "main.py", line 68, in
main()
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "main.py", line 30, in train
pipeline_manager.train(pipeline_name, dev_mode)
File "D:\machine learning\trial\src\pipeline_manager.py", line 42, in train
train(pipeline_name, dev_mode, self.logger, self.params, self.seed)
File "D:\machine learning\trial\src\pipeline_manager.py", line 136, in train
pipeline.fit_transform(data)
File "D:\machine learning\trial\src\steps\base.py", line 106, in fit_transform
step_inputs[input_step.name] = input_step.fit_transform(data)
File "D:\machine learning\trial\src\steps\base.py", line 106, in fit_transform
step_inputs[input_step.name] = input_step.fit_transform(data)
File "D:\machine learning\trial\src\steps\base.py", line 106, in fit_transform
step_inputs[input_step.name] = input_step.fit_transform(data)
[Previous line repeated 4 more times]
File "D:\machine learning\trial\src\steps\base.py", line 112, in fit_transform
return self._cached_fit_transform(step_inputs)
File "D:\machine learning\trial\src\steps\base.py", line 123, in _cached_fit_transform
step_output_data = self.transformer.fit_transform(**step_inputs)
File "D:\machine learning\trial\src\steps\base.py", line 262, in fit_transform
self.fit(*args, **kwargs)
File "D:\machine learning\trial\src\models.py", line 76, in fit
for batch_id, data in enumerate(batch_gen):
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 435, in next
data = self._next_data()
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 1111, in _process_data
data.reraise()
File "C:\ProgramData\Anaconda3\lib\site-packages\torch_utils.py", line 428, in reraise
raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\machine learning\trial\src\loaders.py", line 150, in getitem
Di = self.load_joblib(distance_filepath)
File "D:\machine learning\trial\src\loaders.py", line 135, in load_joblib
return joblib.load(filepath)
File "C:\ProgramData\Anaconda3\lib\site-packages\joblib\numpy_pickle.py", line 577, in load
with open(filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data\meta\masks_overlayed_eroded_0_dilated_0\train\masks\000000042954'

Any hints?

@zeciro
Copy link

zeciro commented Mar 11, 2021

It seems that running things in Ubuntu instead of Windows fixes these issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants