Skip to content

Wrong behaviour of fs.get in "Directory to existing directory" case #1851

Open
@nsundalov

Description

@nsundalov

Problem:
fs.get copies source dir inside target dir but expected to copy content of source dir into target dir.
case 1e of the doc

Script to reproduce error:

import os

from fsspec import filesystem
from fsspec.implementations.dirfs import DirFileSystem

DATA_PATH = "dir"
REMOTE_PATH = "data/remote"
LOCAL_PATH = "data/local"


def init_test():
    # remote
    os.makedirs(REMOTE_PATH, exist_ok=True)
    data_dir = os.path.join(REMOTE_PATH, DATA_PATH)
    os.makedirs(data_dir, exist_ok=True)
    with open(os.path.join(data_dir, "f.txt"), "w") as f:
        f.write("hello world")

    # local
    os.makedirs(LOCAL_PATH, exist_ok=True)


def test():
    fs = filesystem("file")
    fs = DirFileSystem(REMOTE_PATH, fs)

    # copy from data/remote/dir to data/local/dir
    # first leads to data/local/dir/dir/f.txt if called 2 times
    fs.get(DATA_PATH, os.path.join(LOCAL_PATH, DATA_PATH), recursive=True)


init_test()
test()

After calling script 2 times I have following dir structure:

data
├── local
│   └── dir
│       └── dir
│           └── f.txt
└── remote
    └── dir
        └── f.txt

Expected to have following:

data
├── local
│   └── dir
│       └── f.txt
└── remote
    └── dir
        └── f.txt

Extra info:
fsspec 2025.5.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions