I'm afraid this will be not possible to implement a perfect glob API, by perfect I mean only those objects matching the glob will be transmitted between OpenDAL and underlying storage services.

Given such conclusion, since OpenDAL offers BlockingObjectLister and ObjectLister, I suggest listing with filter.

use futures::TryStreamExt;
use glob::Pattern;
use opendal::{services::Fs, Operator};

#[tokio::main]
async fn main() {
    let mut op_builder = Fs::default();
    op_builder.root("/tmp/opendal/");
    op_builder.atomic_write_dir("/tmp/opendal/");
    let op = Operator::create(op_builder).expect("must success").finish();

    for i in 0..100 {
        let path = format!("valid/dir/test-{}.txt", i);
        op.object(&path).create().await.unwrap();
        let junk = format!("invalid/dir/junk-{}.txt", i);
        op.object(&junk).create().await.unwrap();
    }

    let gm = Pattern::new("valid/dir/test-*.txt").expect("should valid");

    let mut lister = op.object("/").scan().await.unwrap();

    // cannot:
    // while let Some(obj) = lister.try_next().await.unwrap().filter()
    // this will result in early endint of streaming
    while let Some(obj) = lister.try_next().await.unwrap() {
        if gm.matches(obj.path()) {
            println!("{} is valid", obj.path());
        }
    }
}

But, really, it's a little too verbose...

feat(core): support for matching file paths against Unix shell style patterns (glob). #1251

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions