fix: support reading compressed metadata #1802
Conversation
colinmarc
force-pushed
the
metadata-compressed
branch
2 times, most recently
from
654de6b to
cd16381
Compare
| let metadata_content = input_file.read().await?; | ||
| let metadata = serde_json::from_slice::<TableMetadata>(&metadata_content)?; | ||
|
|
||
| let metadata = if metadata_location.as_ref().ends_with(".gz.metadata.json") { |
There was a problem hiding this comment.
Do we want to optionally support the Java Iceberg alternative?
The Java reference implementation can additionally read GZIP compressed files with the suffix metadata.json.gz.
There was a problem hiding this comment.
Seems better to have one convention, to me, but happy either way.
Even better would be peeking at the file and looking for the gzip magic number. If there's interest in that I can implement it. The wording of the spec ("some implementations require") seems to suggest it would be better to have no naming requirement at all.
There was a problem hiding this comment.
Even better would be peeking at the file and looking for the gzip magic number. If there's interest in that I can implement it.
That would be a really elegant solution, I think.
colinmarc
force-pushed
the
metadata-compressed
branch
from
cd16381 to
9892bae
Compare
The spec mentions that metadata files "may be compressed with GZIP",
here:
https://iceberg.apache.org/spec/#table-metadata-and-snapshots
colinmarc
force-pushed
the
metadata-compressed
branch
from
9892bae to
011512a
Compare
2 participants
The spec mentions this naming convention here:
https://iceberg.apache.org/spec/#naming-for-gzip-compressed-metadata-json-files
Which issue does this PR close?
What changes are included in this PR?
Support for reading compressed metadata.
Are these changes tested?
Yes.