[Parquet] Support Variant Encoding for Parquet

[sfc-gh-mbojanczyk]

Describe the enhancement requested

Variant encoding looks like it's becoming part of the Parquet spec, and it'd be good to have Variant support in the Go library.

It looks like implementation of Variants is already under way in the Java Parquet library, as well as in Iceberg, so the spec seems to be settled enough that there shouldn't be massive changes in the base encoding. Looks like shredding still hasn't been tackled yet by any library, so that may be more in flux.

Variants will be a pretty powerful addon here for landing stuff in Iceberg- they generally have a good performance upside vs. using JSON or something similar.

Component(s)

Parquet

[zeroshade]

While I can work on adding Variant support in the Parquet package here, to add support at the Arrow level would require either creating an extension type to represent the Variant or, more likely, a proposal to add variant directly to the Arrow spec. The latter being the better route in my opinion.

[sfc-gh-mbojanczyk]

Oh dear, I hadn't thought of the implications on the Arrow side- seems like the Java folk have sidestepped that bit too by solely implementing Variant encoding and not plugging it through to Arrow. I'd figure that spec work would take a bit, so maybe an extension first while specs are being hashed out?

FWIW, I've been hacking away at a Go version of Variant encoding/decoding using Go primitives (getting all the bit packing out of the way), and should have something to share here in a draft PR shortly. That said, I'm a bit of a n00b when it comes to the finer bits of this library (as witnessed by me totally ignoring the Arrow bits), especially when it's going to come into the integration into the LogicalTypes and compiled Thrift definitions, so hopefully this can be a first step in integrating Variants.

[sfc-gh-mbojanczyk]

Got sniped by some things around here, but finally dusted off the change and I think I've got an initial stab at the base of Variant encoding (haven't yet plumbed this into Parquet or Arrow). I figure the Variant encoding and decoding would be common between the two, so it's a good place to start.

Unfortunately, I can't upload a draft PR here as I'm getting stuck in the precommit hooks for directories that I haven't even touched (eg. golangci-lint-full-internal among others are failing). I didn't see references to this in the CONTRIBUTING doc- any pointers here?

[zeroshade]

Hmm, can you put the issues you're seeing in this issue? We run the golangci-lint on the entire repo as part of CI, so there shouldn't be any issues for directories you haven't touched locally as far as I know.

[sfc-gh-mbojanczyk]

It's quite a load of output, but for example:

golangci-lint-full-internal..............................................Failed
- hook id: golangci-lint-full
- exit code: 1

json/json.go:28:16: undefined: json (typecheck)
type Decoder = json.Decoder
               ^
json/json.go:29:16: undefined: json (typecheck)
type Encoder = json.Encoder
               ^
json/json.go:30:18: undefined: json (typecheck)
type Marshaler = json.Marshaler

Looking closer at this, seems like however golangci-lint is being run on my box isn't respecting imports. Lemme hack on that a bit and see if there's an easy solution

[sfc-gh-mbojanczyk]

Alright, I sidestepped the precommit hooks for now to get some feedback on my approach in this draft PR. This only has Variant encoding and decoding- doesn't plumb it through to Parquet or Arrow.

[kris-gaudel]

I'm interested in contributing to this project and to the Apache organization. Are there any pointers for someone new to contributing to this project?

[zeroshade]

@kris-gaudel There's a good number of issues in the backlog here which could potentially be good starting points, when I get a chance I'll try to go through and add the "good first issue" tag to many of them, but for now I'd say that it would make sense to skim through the issues here and see if any are something you think you could take a stab at.

Another thing that is definitely needed in here, and would be a good starting point, is to improve the Examples that show up in the documentation and provide more examples of how to utilize this project in the docs or via more Example_* functions so that users can have an easier time onboarding than having to read through the unit tests.

[kris-gaudel]

That sounds good, I think creating some updated examples would help me get my feet wet aha, I can make an issue and work on it