rm followed by exists returns true

[sshleifer]

Why does fs.exists() sometimes return True even after deleting directories using fsspec/s3fs? Is there a reliable way to confirm directory deletion on Cloudflare R2 in python?

If I rclone lsf r2://bucket-name/test from command line, contents are as I expect (deletion happened).

import fsspec
import s3fs

# Initialize filesystem pointing to Cloudflare R2 (or another S3-compatible service)
fs = s3fs.S3FileSystem(
    key='YOUR_ACCESS_KEY',
    secret='YOUR_SECRET_KEY',
    endpoint_url='https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com',
)

# Path to test
test_dir = 'bucket-name/test/checkpoint_dir'

# Create a test file
fs.touch(f'{test_dir}/file.txt')

# Verify existence
print("Exists before deletion:", fs.exists(test_dir))  # True expected

# Delete the directory
fs.rm(test_dir, recursive=True)

# Verify deletion (issue occurs here)
exists_after_delete = fs.exists(test_dir)
print(f'Exists after deletion: {exists_after_delete}')  # Expected False, but always true