-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
structs: Support casting nullable fields to non-nullable if there are no null values #133
Comments
This is the case, the spec just defines the in-memory format. Anything implementing compute and casting behavior is outside the scope of the spec itself as different compute engines are free to have different behaviors. For a spec attempting to create standardized behavior on compute functionality, there is https://substrait.io That said, I agree the ideal would be for the go implementation to follow the C++ implementation where possible for these things.
By the Arrow spec, "nullability" is a feature of "fields" as it is simply a piece of metadata to inform consumers of data. So you can only mark something as non-nullable by way of a schema (i.e. top-level columns in a record batch can be marked non-nullable via their fields in the schema) or by virtue of it being a nested field of some other type (i.e. struct, list, map, etc. type) which contains fields to describe its children. So you can have a non-nullable uint16, if the schema/parent struct/parent list type defines the column field as non-nullable. Since casting doesn't accept record batches, only Array or ChunkedArray, the only cases where you might run into this nullability issue are in the child fields of these nested types. Since we do provide factories for constructing lists with non-nullable element fields, we probably should add tests to verify we handle nullability/non-nullability for lists/maps/etc. |
I came across a relevant issue regarding field nullability. I am trying to build a Personally, I think this is a bit confusing and error-prone. It's better if error is returned somewhere (preferably when |
This solves apache/arrow#33592
This is a sister PR to apache/arrow#43782, I was told to open a PR here to make this change on the Go side.
I need a few pointers though if you want me to do the implementation. Could you give me a high-level summary of what needs to happen? I did an initial exploration and had a few notes/questions:
The text was updated successfully, but these errors were encountered: