shutil.Error

[wwwwws12]

I'm not sure why this error keeps occurring. I'm wondering if it might be because I'm running this project on a Windows system?

[zjysteven]

Maybe delete the data folder and try again?

[wwwwws12]

Maybe delete the data folder and try again?

I tried it, but it still didn’t work. Additionally, there’s a post above discussing the installation issue of faiss-gpu, which I am also experiencing. My Windows system runs Python 3.12, but my Conda virtual environment is configured with Python 3.9. However, just like the other person, I couldn’t find faiss-gpu using pip. Instead, I installed version 1.9.0 of faiss-gpu using Conda. I’m not sure if this will work.

[wwwwws12]

Maybe delete the data folder and try again?

After deleting the `data` directory, running the program results in a "path not found" error.

[zjysteven]

That is very weird. I don't have a windows machine so unfortunately cannot test it. As a last resort, you can manually download relevant files, unzip them, and put under the data folder (and data needs to be put under the root directory of this repo).

For faiss-gpu, it should be fine as long as you don't see any errors when running scripts. Also it's just used in a small number of methods. So it's quite likely that you won't actually use it.

[wwwwws12]

That is very weird. I don't have a windows machine so unfortunately cannot test it. As a last resort, you can manually download relevant files, unzip them, and put under the data folder (and data needs to be put under the root directory of this repo).

For faiss-gpu, it should be fine as long as you don't see any errors when running scripts. Also it's just used in a small number of methods. So it's quite likely that you won't actually use it.

Thank you very much for your explanation. I will continue to try.

[GrandmasterFred]

Can confirm, i am on a windows machine and the download script is not working for me. Running it from a fresh install creates:

(openOOD) PS D:\gitprojects\openOOD> & C:/Users/PC/.conda/envs/openOOD/python.exe d:/gitprojects/openOOD/scripts/download/download.py
Traceback (most recent call last):
  File "d:\gitprojects\openOOD\scripts\download\download.py", line 192, in <module>
    gdown.download(id=download_id_dict['benchmark_imglist'],
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   output=store_path)
                   ^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\.conda\envs\openOOD\Lib\site-packages\gdown\download.py", line 304, in download
    for file in os.listdir(osp.dirname(output) or "."):
                ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: './data'

Running with the .\data folder created produces:

Downloading...
From (original): https://drive.google.com/uc?id=1lI1j0_fDDvjIt9JlWAw09X8ks-yrR_H1
From (redirected): https://drive.google.com/uc?id=1lI1j0_fDDvjIt9JlWAw09X8ks-yrR_H1&confirm=t&uuid=064ec01e-3bbb-4036-8de8-0e5c383731ec
To: D:\gitprojects\openOOD\data
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28.0M/28.0M [00:02<00:00, 10.2MB/s] 
Traceback (most recent call last):
  File "d:\gitprojects\openOOD\scripts\download\download.py", line 192, in <module>
    gdown.download(id=download_id_dict['benchmark_imglist'],
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   output=store_path)
                   ^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\.conda\envs\openOOD\Lib\site-packages\gdown\download.py", line 381, in download
    shutil.move(tmp_file, output)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\.conda\envs\openOOD\Lib\shutil.py", line 854, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path './data/fsefx8fz.part' already exists```

In essence, I cant really even brute force it. 

[GrandmasterFred]

Okay, ive modified the download.py so that it is hopefully more stable on both windows and linux (i dont have a linux machine so i can only hope), but this works for my windows machine:

import argparse
import os
import zipfile

import gdown
import shutil

benchmarks_dict = {
    'bimcv': [
        'bimcv', 'ct', 'xraybone', 'actmed', 'mnist', 'cifar10', 'texture',
        'tin'
    ],
    'mnist': [
        'mnist', 'notmnist', 'fashionmnist', 'texture', 'cifar10', 'tin',
        'places365', 'cinic10'
    ],
    'cifar-10': [
        'cifar10', 'cifar100', 'tin', 'mnist', 'svhn', 'texture', 'places365',
        'tin597'
    ],
    'cifar-100':
    ['cifar100', 'cifar10', 'tin', 'svhn', 'texture', 'places365', 'tin597'],
    'imagenet-200': [
        'imagenet_1k', 'ssb_hard', 'ninco', 'inaturalist', 'texture',
        'openimage_o', 'imagenet_v2', 'imagenet_c', 'imagenet_r'
    ],
    'imagenet-1k': [
        'imagenet_1k', 'ssb_hard', 'ninco', 'inaturalist', 'texture',
        'openimage_o', 'imagenet_v2', 'imagenet_c', 'imagenet_r'
    ],
    'misc': [
        'cifar10c',
        'fractals_and_fvis',
        'usps',
        'imagenet10',
        'hannover',
        # 'imagenet200_cae', 'imagenet200_edsr', 'imagenet200_stylized'
    ],
}

dir_dict = {
    'images_classic/': [
        'cifar100', 'tin', 'tin597', 'svhn', 'cinic10', 'imagenet10', 'mnist',
        'fashionmnist', 'cifar10', 'cifar100c', 'places365', 'cifar10c',
        'fractals_and_fvis', 'usps', 'texture', 'notmnist'
    ],
    'images_largescale/': [
        'imagenet_1k',
        'species_sub',
        'ssb_hard',
        'ninco',
        'inaturalist',
        'places',
        'sun',
        'openimage_o',
        'imagenet_v2',
        'imagenet_c',
        'imagenet_r',
        # 'imagenet200_cae', 'imagenet200_edsr', 'imagenet200_stylized'
    ],
    'images_medical/': ['actmed', 'bimcv', 'ct', 'hannover', 'xraybone'],
}

download_id_dict = {
    'osr': '1L9MpK9QZq-o-JrFHrfo5lM4-FsFPk0e9',
    'mnist_lenet': '13mEvYF9rVIuch8u0RVDLf_JMOk3PAYCj',
    'cifar10_res18': '1rPEScK7TFjBn_W_frO-8RSPwIG6_x0fJ',
    'cifar100_res18': '1OOf88A48yXFw4fSU02XQT-3OQKf31Csy',
    'imagenet_res50': '1tgY_PsfkazLDyI1pniDMDEehntBhFyF3',
    'cifar10_res18_v1.5': '1byGeYxM_PlLjT72wZsMQvP6popJeWBgt',
    'cifar100_res18_v1.5': '1s-1oNrRtmA0pGefxXJOUVRYpaoAML0C-',
    'imagenet200_res18_v1.5': '1ddVmwc8zmzSjdLUO84EuV4Gz1c7vhIAs',
    'imagenet_res50_v1.5': '15PdDMNRfnJ7f2oxW6lI-Ge4QJJH3Z0Fy',
    'benchmark_imglist': '1lI1j0_fDDvjIt9JlWAw09X8ks-yrR_H1',
    'usps': '1KhbWhlFlpFjEIb4wpvW0s9jmXXsHonVl',
    'cifar100': '1PGKheHUsf29leJPPGuXqzLBMwl8qMF8_',
    'cifar10': '1Co32RiiWe16lTaiOU6JMMnyUYS41IlO1',
    'cifar10c': '170DU_ficWWmbh6O2wqELxK9jxRiGhlJH',
    'cinic10': '190gdcfbvSGbrRK6ZVlJgg5BqqED6H_nn',
    'svhn': '1DQfc11HOtB1nEwqS4pWUFp8vtQ3DczvI',
    'fashionmnist': '1nVObxjUBmVpZ6M0PPlcspsMMYHidUMfa',
    'cifar100c': '1MnETiQh9RTxJin2EHeSoIAJA28FRonHx',
    'mnist': '1CCHAGWqA1KJTFFswuF9cbhmB-j98Y1Sb',
    'fractals_and_fvis': '1EZP8RGOP-XbMsKex3r-BGI5F1WAP_PJ3',
    'tin': '1PZ-ixyx52U989IKsMA2OT-24fToTrelC',
    'tin597': '1R0d8zBcUxWNXz6CPXanobniiIfQbpKzn',
    'texture': '1OSz1m3hHfVWbRdmMwKbUzoU8Hg9UKcam',
    'imagenet10': '1qRKp-HCLkmfiWwR-PXthN7-2dxIQVKxP',
    'notmnist': '16ueghlyzunbksnc_ccPgEAloRW9pKO-K',
    'places365': '1Ec-LRSTf6u5vEctKX9vRp9OA6tqnJ0Ay',
    'places': '1fZ8TbPC4JGqUCm-VtvrmkYxqRNp2PoB3',
    'sun': '1ISK0STxWzWmg-_uUr4RQ8GSLFW7TZiKp',
    'species_sub': '1-JCxDx__iFMExkYRMylnGJYTPvyuX6aq',
    'imagenet_1k': '1i1ipLDFARR-JZ9argXd2-0a6DXwVhXEj',
    'ssb_hard': '1PzkA-WGG8Z18h0ooL_pDdz9cO-DCIouE',
    'ninco': '1Z82cmvIB0eghTehxOGP5VTdLt7OD3nk6',
    'imagenet_v2': '1akg2IiE22HcbvTBpwXQoD7tgfPCdkoho',
    'imagenet_r': '1EzjMN2gq-bVV7lg-MEAdeuBuz-7jbGYU',
    'imagenet_c': '1JeXL9YH4BO8gCJ631c5BHbaSsl-lekHt',
    'imagenet_o': '1S9cFV7fGvJCcka220-pIO9JPZL1p1V8w',
    'openimage_o': '1VUFXnB_z70uHfdgJG2E_pjYOcEgqM7tE',
    'inaturalist': '1zfLfMvoUD0CUlKNnkk7LgxZZBnTBipdj',
    'actmed': '1tibxL_wt6b3BjliPaQ2qjH54Wo4ZXWYb',
    'ct': '1k5OYN4inaGgivJBJ5L8pHlopQSVnhQ36',
    'hannover': '1NmqBDlcA1dZQKOvgcILG0U1Tm6RP0s2N',
    'xraybone': '1ZzO3y1-V_IeksJXEvEfBYKRoQLLvPYe9',
    'bimcv': '1nAA45V6e0s5FAq2BJsj9QH5omoihb7MZ',
}


def require_download(filename, path):
    if not os.path.exists(path):
        return True
    for item in os.listdir(path):
        if item.startswith(filename) or filename.startswith(item):
            return False
    return True

def safe_extract_zip(zip_path, extract_to):
    """Extracts and deletes the zip file safely."""
    with zipfile.ZipFile(zip_path, 'r') as zip_file:
        zip_file.extractall(extract_to)
    os.remove(zip_path)

def download_file_from_gdrive(gdrive_id, output_path):
    """Downloads file from Google Drive with collision-safe logic."""
    tmp_file = output_path + '.part'
    if os.path.exists(output_path):
        print(f"[✓] Skipping download, already exists: {output_path}")
        return
    if os.path.exists(tmp_file):
        os.remove(tmp_file)
    gdown.download(id=gdrive_id, output=tmp_file, quiet=False)
    shutil.move(tmp_file, output_path)
    print(f"[↓] Downloaded and saved to: {output_path}")


def download_dataset(dataset, args):
    for key in dir_dict:
        if dataset in dir_dict[key]:
            store_path = os.path.join(args.save_dir[0], key, dataset)
            os.makedirs(store_path, exist_ok=True)
            break
    else:
        print(f'[!] Invalid dataset: {dataset}')
        return

    if require_download(dataset, store_path):
        print(f"[→] Downloading dataset: {dataset}")
        zip_path = os.path.join(store_path, dataset + '.zip')
        download_file_from_gdrive(download_id_dict[dataset], zip_path)
        safe_extract_zip(zip_path, store_path)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Download datasets and checkpoints')
    parser.add_argument('--contents',
                        nargs='+',
                        default=['datasets', 'checkpoints'])
    parser.add_argument('--datasets', nargs='+', default=['default'])
    parser.add_argument('--checkpoints', nargs='+', default=['all'])
    parser.add_argument('--save_dir',
                        nargs='+',
                        default=['./data', './results'])
    parser.add_argument('--dataset_mode', default='default')
    args = parser.parse_args()

    if args.datasets[0] == 'default':
        args.datasets = ['mnist', 'cifar-10', 'cifar-100']
    elif args.datasets[0] == 'ood_v1.5':
        args.datasets = [
            'cifar-10', 'cifar-100', 'imagenet-200', 'imagenet-1k'
        ]
    elif args.datasets[0] == 'all':
        args.datasets = list(benchmarks_dict.keys())

    if args.checkpoints[0] == 'ood':
        args.checkpoints = [
            'mnist_lenet', 'cifar10_res18', 'cifar100_res18', 'imagenet_res50'
        ]
    elif args.checkpoints[0] == 'ood_v1.5':
        args.checkpoints = [
            'cifar10_res18_v1.5', 'cifar100_res18_v1.5',
            'imagenet200_res18_v1.5', 'imagenet_res50_v1.5'
        ]
    elif args.checkpoints[0] == 'all':
        args.checkpoints = [
            'mnist_lenet', 'cifar10_res18', 'cifar100_res18', 'imagenet_res50',
            'osr'
        ]

    for content in args.contents:
        if content == 'datasets':
            store_path = args.save_dir[0]
            os.makedirs(store_path, exist_ok=True)

            # Benchmark imglist download
            benchmark_imglist_dir = os.path.join(store_path, 'benchmark_imglist')
            benchmark_imglist_zip = os.path.join(store_path, 'benchmark_imglist.zip')
            if not os.path.exists(benchmark_imglist_dir):
                print("[→] Downloading benchmark_imglist...")
                download_file_from_gdrive(download_id_dict['benchmark_imglist'], benchmark_imglist_zip)
                safe_extract_zip(benchmark_imglist_zip, store_path)

            if args.dataset_mode in ['default', 'benchmark']:
                for benchmark in args.datasets:
                    for dataset in benchmarks_dict[benchmark]:
                        download_dataset(dataset, args)

            elif args.dataset_mode == 'dataset':
                for dataset in args.datasets:
                    download_dataset(dataset, args)

        elif content == 'checkpoints':
            if 'v1.5' in args.checkpoints[0]:
                store_path = args.save_dir[1]
            else:
                store_path = os.path.join(args.save_dir[1], 'checkpoints')
            os.makedirs(store_path, exist_ok=True)

            for checkpoint in args.checkpoints:
                checkpoint_zip = os.path.join(store_path, checkpoint + '.zip')
                checkpoint_dir = os.path.join(store_path, checkpoint)
                if not os.path.exists(checkpoint_dir):
                    print(f"[→] Downloading checkpoint: {checkpoint}")
                    download_file_from_gdrive(download_id_dict[checkpoint], checkpoint_zip)
                    safe_extract_zip(checkpoint_zip, store_path)
                else:
                    print(f"[✓] Checkpoint already exists: {checkpoint_dir}")