using the read_gdal() function

[PMassicotte]

I am trying to use the read_gdal() function from the geoarrow crate. I am relatively new to rust and am unsure how to debug this issue.

I have a function like this that uses GDAL virtual file systems to download some data:

pub fn download<T: Downloadable>(
    downloadable_type: T,
    scale: Scale,
) -> Result<RecordBatchReader, GeoArrowError> {
    let url = downloadable_type.get_url(&scale);

    let dataset = Dataset::open(url)?;

    // Get the layer (assuming there is only one layer)
    let mut layer = dataset.layer(0)?;

    let reader = read_gdal(&mut layer, None)?;

    Ok(reader.into())
}

This partially works as I can get something from the read_gdal() function and further view the schema.

In my main.rs file, I am calling this function like this:

fn main() {
    let mut reader = download(url).unwrap();

    // This is working
    let schema = reader.schema().unwrap();
    println!("{:?}", schema);

    // This is not working (see the error below)
    for batch_result in reader.take().unwrap() {
        match batch_result {
            Ok(batch) => {
                println!("Batch ----");
                let value = batch.column(4);
                println!("{:?}", value);
            }
            Err(e) => eprintln!("Error reading batch: {:?}", e),
        }
    }

}

When I try to iterate over the RecordBatchReader, I get the following error:

Error reading batch: CDataInterface("Calling get_next() on a freed OGRLayer is not supported")
Error reading batch: CDataInterface("Calling get_next() on a freed OGRLayer is not supported")

Any ideas on what am I doing wrong here?

Thank you for your time and the work you are doing on this project.

[kylebarron]

I'm not at my computer right now but I'm not exactly sure. There might be issues on either the gdal or the rust side of closing the layer object when you access the schema?

Does it work if you don't access the schema first?

[PMassicotte]

Not, it does not matter if I read or not the schema. Thanks for looking into it.

[kylebarron]

@PMassicotte I verified in #778 that the GDAL example in the repo still works.

You can run the local GDAL example with (Note: you'll have to pull the latest main first)

> cargo run --example gdal --features gdal

and that should print

> cargo run --example gdal --features gdal
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.36s
     Running `target/debug/examples/gdal`
[examples/gdal.rs:14:5] &table.schema() = Schema {
    fields: [
        Field {
            name: "OGC_FID",
            data_type: Int64,
            nullable: false,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "kind",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "sort_key",
            data_type: Float64,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "is_link",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "is_tunnel",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "is_bridge",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "railway",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "highway",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
        Field {
            name: "wkb_geometry",
            data_type: Binary,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {
                "ARROW:extension:name": "geoarrow.wkb",
                "ARROW:extension:metadata": "{\"crs\":{\n  \"$schema\": \"https://proj.org/schemas/v0.7/projjson.schema.json\",\n  \"type\": \"GeographicCRS\",\n  \"name\": \"WGS 84\",\n  \"datum\": {\n    \"type\": \"GeodeticReferenceFrame\",\n    \"name\": \"World Geodetic System 1984\",\n    \"ellipsoid\": {\n      \"name\": \"WGS 84\",\n      \"semi_major_axis\": 6378137,\n      \"inverse_flattening\": 298.257223563\n    }\n  },\n  \"coordinate_system\": {\n    \"subtype\": \"ellipsoidal\",\n    \"axis\": [\n      {\n        \"name\": \"Geodetic latitude\",\n        \"abbreviation\": \"Lat\",\n        \"direction\": \"north\",\n        \"unit\": \"degree\"\n      },\n      {\n        \"name\": \"Geodetic longitude\",\n        \"abbreviation\": \"Lon\",\n        \"direction\": \"east\",\n        \"unit\": \"degree\"\n      }\n    ]\n  },\n  \"id\": {\n    \"authority\": \"EPSG\",\n    \"code\": 4326\n  }\n}}",
            },
        },
    ],
    metadata: {},
}

I'm not familiar with how GDAL VFS works in the Rust API. Can you first verify that the local example works for you?

[kylebarron]

Otherwise, I haven't used the read_gdal function at all. I've been putting more effort into the pure-Rust readers and writers

[PMassicotte]

Yes, this works now! Thank you very much. Aside question, would you recommend another way to read a shape file from a URL rather than using GDAL vrt?

[kylebarron]

That works from a local file, but does it still not work from a URL?

I made decent progress on a native shapefile reader in #759. Within the next couple months I'm likely to finish that, in which case you'll be able to load a Shapefile to GeoArrow without GDAL. If you're interested you're welcome to help push it over the line.

[PMassicotte]

That works from a local file, but does it still not work from a URL?

I made decent progress on a native shapefile reader in #759. Within the next couple months I'm likely to finish that, in which case you'll be able to load a Shapefile to GeoArrow without GDAL. If you're interested you're welcome to help push it over the line.

That's great to hear! I'd be happy to help if I can.

Right now, I'm working on a crate for accessing data from naturalearthdata.

https://github.com/PMassicotte/naturalearthdata

It’s a straightforward crate, but I think it could be a good fit for using geoarrow to read and process that kind of data.

[kylebarron]

That looks like a nice crate!

If it works for you using the GDAL integration then that sounds great.

#759 is roughly halfway done. The geometry parsing is implemented but we still need to manage property conversions.

[PMassicotte]

That is pretty cool. Thanks again for your work!