Data: Add partition stats writer and reader #11216

ajantha-bhat · 2024-09-26T17:00:06Z

Introduce APIs to write the partition stats into files in table default format using Iceberg generic writers and readers.

PartitionStatisticsFile partitionStatisticsFile =
        PartitionStatsHandler.computeAndWriteStatsFile(testTable, "b1");

testTable.updatePartitionStatistics().setPartitionStatistics(partitionStatisticsFile).commit();

core/src/main/java/org/apache/iceberg/data/PartitionStatsRecord.java

ajantha-bhat · 2024-09-27T01:52:39Z

core/src/test/java/org/apache/iceberg/TestTables.java

+      Schema schema,
+      PartitionSpec spec,
+      int formatVersion,
+      Map<String, String> properties) {


There was no option to pass the table properties before.
Needed to pass different file format for paramterized test.

Why not adding the parameter to the old create method and call the new method from the old one?

Like:

public static TestTable create( File temp, String name, Schema schema, PartitionSpec spec, SortOrder sortOrder, int formatVersion) { return create(temp, name, schema, spec, SortOrder.unsorted(), formatVersion, ImmutableMap.of()); } public static TestTable create( File temp, String name, Schema schema, PartitionSpec spec, int formatVersion, Map<String, String> properties) {

Followed a similar style when they added MetricsReporter,

Why not adding the parameter to the old create

It is public. So, need to modify all the callers.
But we can refactor a private method that can be helper to all these public method. I can do it in a follow up to keep minimal changes for this PR.

ajantha-bhat · 2024-09-27T02:00:04Z

@aokolnychyi: This PR is ready. But as we discussed previously, this PR wraps the PartitionStats into a Record as the writers cannot work with Iceberg internal objects yet.

I will explore adding the internal writers for Parquet and Orc. Similar to #11108.
If we fail to have it ready by 1.7.0, I think it makes sense to merge this PR and introduce the optimized writer in the next version by deprecating this writer.

ajantha-bhat · 2024-10-23T16:18:50Z

@RussellSpitzer: It would be good to have this in 1.7.0.
I am waiting from a month for a review.

aokolnychyi · 2024-10-24T04:34:52Z

I think we should try to use "internal" writers. @rdblue added "internal" readers recently.

Any guidance on how to add a writer, @rdblue? We can start with Avro for now. We will also need such readers/writers for Parquet.

ajantha-bhat · 2024-10-24T08:39:46Z

@aokolnychyi, @rdblue:

I already tried POC for internal writers on another branch,
c209bc9

The problems:
a) I am using PartitionData instead of Record for partition value, but the PartitionData get() method wraps the byte array to the byte buffer, which is a problem for internal writers, they expect byte[]. So, I didn't felt like using a new class instead of PartitionData just for this.

b) Also, Using partitionData in StructLikeMap is not working fine. Some keys are missing in the map (looks like equals() logic), If I use Record, it is fine.

Maybe in the next version we can have optimized writer and reader (without converter using internal reader and writers).
For end user it doesn't make any difference as new readers can also read the old partition stats parquet file and old readers can read the new partition stats parquet file. So, can we merge this?

core/src/main/java/org/apache/iceberg/PartitionStats.java

core/src/test/java/org/apache/iceberg/TestTables.java

data/src/test/java/org/apache/iceberg/data/TestPartitionStatsHandler.java

RussellSpitzer · 2024-10-28T18:24:06Z

Moving out of 1.7.0 since we still have a bit of discussion here

ajantha-bhat · 2024-11-19T15:31:28Z

@RussellSpitzer: I have added the Assertion for Partition type as you suggested and replied to #11216 (comment), do you have anymore comments for this PR?

aokolnychyi · 2024-11-20T20:57:57Z

I had a conversation with @rdblue today about internal writers. Ryan should have a bit of time to help/guide.
I will check the current implementation today too.

core/src/main/java/org/apache/iceberg/PartitionStats.java

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

jbonofre · 2024-11-28T06:55:13Z

@RussellSpitzer @aokolnychyi I'm reviewing the stale PRs, and this one is open for month. Do we have a way to move forward ? I can do a new review, but at the end of the day, it won't help for the merge (as only committers can merge PR).

ajantha-bhat · 2025-01-15T14:48:07Z

core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java

+
+  @SuppressWarnings("checkstyle:CyclomaticComplexity")
+  public static boolean isEqual(
+      Comparator<StructLike> partitionComparator, PartitionStats stats1, PartitionStats stats2) {


Cannot have Equals and HashCode for PartitionStats class as StructLike need to have comparator for equals() which forces that class extends StructLike to hold some more things. Setting comparator while serializing and deserializing that class will be a mess.

Hence, added this util method. Currently used only by tests. But can be useful for developers when they integrate partition stats to engines, they can use it for their tests. So, kept as a util.

Do I understand correctly that this will never been used in production code, just in tests?
Do we publish a test artifact? If so, we can put this code to the test artifact and users can depend on it.

ok. Can move to test. This is not a big code. Users can replicate it in their environment if required. Always safe to have the scope to minimum.

ajantha-bhat · 2025-01-15T18:21:44Z

@aokolnychyi, @rdblue, @RussellSpitzer: I have reworked on the PR to use Internal writers and readers. PR is much simpler and no need to handle those conversions now. I can rebase it once the Parquet internal writer PR is merged.

@deniskuzZ : Feel free to test the latest state. It doesn't have conversion layer. So, should behave as expected now.

deniskuzZ · 2025-01-16T09:08:04Z

@deniskuzZ : Feel free to test the latest state. It doesn't have conversion layer. So, should behave as expected now.

hi @ajantha-bhat, i need to include #11919, anything else?

ajantha-bhat · 2025-01-25T02:12:03Z

@aokolnychyi, @rdblue, @RussellSpitzer: I have worked on Internal writers, readers for Avro, parquet and PRs got merged.
I have rebased this PR to use the internal writers and readers.

So, this PR is very simple now (no converter logic) and it just writes stats to a file.

I think if we get a good review support it can be merged for 1.8.0 itself. Please take a look.
It was already reviewed before internal writers. So, I don't think much effort is needed. Thanks in advance.

deniskuzZ · 2025-01-25T11:42:34Z

hi @ajantha-bhat, what is the purpose of PartitionStats.totalRecordCount? it's always 0 and there is no external setter either.
Also SnapshotSummary.TOTAL_FILE_SIZE_PROP tracks all files (data + delete, see https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L288), whereas PartitionStats only data files totalDataFileSizeInBytes.
Could we extends the PartitionStats with totalFileSizeInBytes metric? I can open a PR with the change if that's ok.

ajantha-bhat · 2025-01-25T17:15:52Z

@deniskuzZ: While designing the spec (https://iceberg.apache.org/spec/#partition-statistics-file), we have added totalRecordCount to represent the record count after applying the delete file. It is optional field and hence not computed at the moment as it requires scanning all the data files and it can be expensive operation.

Could we extends the PartitionStats with totalFileSizeInBytes metric?

Let us wait for the merge of this PR. After that we can open the discussion to add additional stats for partition stats spec. For example some folks want min max stats also #11083.

ajantha-bhat · 2025-01-30T09:19:18Z

Ping.

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

deniskuzZ · 2025-01-31T09:19:54Z

data/src/test/java/org/apache/iceberg/data/TestPartitionStatsHandler.java

+    PartitionStatisticsFile partitionStatisticsFile =
+        PartitionStatsHandler.computeAndWriteStatsFile(testTable, "b1");
+    // creates an empty stats file since the dummy snapshot exist
+    assertThat(partitionStatisticsFile.fileSizeInBytes()).isEqualTo(0L);


test would be broken if default format changes, for example with avro format non-zero file would be created

But it won't be flaky. So, we can update the test if the behavior changes. This is as per current behavior.

true, I've already added +1

Thanks for your reviews. I hope we ship this feature soon and glad to know Hive, Trino are waiting for this feature.

I think it is bad practice to check in tests for things which are not a requirement, just "coincidentally" happens.
Could we check that the file could be read and actually empty?

Thinking more on it, behavior should be same as empty table testcase above. So, will update to not throw exceptions in this case.

deniskuzZ

LGTM, could we please get this merged? @pvary, would you be able to help

pvary · 2025-02-03T09:00:26Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+ * Computes, writes and reads the {@link PartitionStatisticsFile}. Uses generic readers and writers
+ * to support writing and reading of the stats in table default format.
+ */
+public final class PartitionStatsHandler {


Any reason for having this final?

Utility classes (with private constructor) ideally preferred to be final. I can remove it if not a requirement in this project.

pvary · 2025-02-03T09:02:32Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+   * @return a schema that corresponds to the provided unified partition type.
+   */
+  public static Schema schema(StructType partitionType) {
+    Preconditions.checkState(!partitionType.fields().isEmpty(), "table must be partitioned");


nit: Start the error message with capital letter

pvary · 2025-02-03T09:08:26Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+    if (currentSnapshot == null) {
+      Preconditions.checkArgument(
+          branch == null, "Couldn't find the snapshot for the branch %s", branch);
+      return null;


Is this for handling an empty table?
How users of this method will use the returned null value?

Should this be an exception? When we query empty table, it returns zero rows. Similarly, it returns null. I will update the java doc.

pvary · 2025-02-03T09:10:59Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+    try (DataWriter<StructLike> writer = dataWriter(dataSchema, outputFile); ) {
+      records.forEachRemaining(writer::write);
+    } catch (IOException e) {
+      throw new UncheckedIOException(e);


Why did we decide to convert an IOException to an unchecked exception?

pvary · 2025-02-03T09:11:24Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+      Table table, long snapshotId, Schema dataSchema, Iterator<PartitionStats> records) {
+    OutputFile outputFile = newPartitionStatsFile(table, snapshotId);
+
+    try (DataWriter<StructLike> writer = dataWriter(dataSchema, outputFile); ) {


nit: remove ;

pvary · 2025-02-03T09:15:18Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+  private static FileFormat fileFormat(String fileLocation) {
+    return FileFormat.fromString(fileLocation.substring(fileLocation.lastIndexOf(".") + 1));
+  }


Are we sure in this?
We usually depend on metadata files to deduce the file format. Depending on the filename seems brittle to me.

pvary · 2025-02-03T09:18:10Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+    FileFormat fileFormat =
+        fileFormat(
+            table.properties().getOrDefault(DEFAULT_FILE_FORMAT, DEFAULT_FILE_FORMAT_DEFAULT));


The fileFormat method parameter is a fileLocation, here we provide the actual FileFormat string... this seems like an issue for me

For the reader code, it has to infer from the input file extension. To keep reader and writer signature similar. It has done like this.

pvary · 2025-02-03T09:19:06Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+    return table
+        .io()
+        .newOutputFile(
+            ((HasTableOperations) table)


Do we need to check that the table implements HasTableOperations?

pvary · 2025-02-03T09:20:21Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+      case ORC:
+        // Internal writers are not supported for ORC yet.


Do we plan to support ORC Internal writers?
Or do we plan to support partition statistics file for ORC?

If people are intersted to contribute.
Last I discussed it with Ryan, Community is expecting the ORC users to contribute here.

pvary · 2025-02-03T09:24:16Z

data/src/test/java/org/apache/iceberg/data/TestPartitionStatsHandler.java

+
+  @Test
+  public void testPartitionStatsOnEmptyTable() throws Exception {
+    Table testTable = TestTables.create(tempDir("empty_table"), "empty_table", SCHEMA, SPEC, 2);


Are these tables cleaned up after the test methods?
If not, they leave a state for the tests which is a bad practice

It is similar to other existing tests. @tempdir annotation should clean up the folders.

pvary · 2025-02-03T09:38:09Z

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java

+   * @param partitionType unified partition schema type.
+   * @return a schema that corresponds to the provided unified partition type.
+   */
+  public static Schema schema(StructType partitionType) {


Why not schema(Table)? In this case we would not need the "Note" and make sure that it is calculated correctly.

To avoid computing partition type again again in computeAndWriteStatsFile. Also, It is recommended to pass only what is required for the method instead of the whole table.

ajantha-bhat · 2025-02-04T05:56:00Z

thanks @pvary for the review.
I have addressed the comments except (#11216 (comment)), It is because I wanted to keep the signatures of reader and writer similar. Let me know if you have any ideas. Thanks.

ajantha-bhat marked this pull request as draft September 26, 2024 17:00

github-actions bot added core data labels Sep 26, 2024

ajantha-bhat mentioned this pull request Sep 26, 2024

Data: Add a util to read write partition stats #10176

Closed

ajantha-bhat force-pushed the stats_writer branch 2 times, most recently from 941505a to 05a80f6 Compare September 27, 2024 01:43

ajantha-bhat commented Sep 27, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/data/PartitionStatsRecord.java Outdated Show resolved Hide resolved

ajantha-bhat commented Sep 27, 2024

View reviewed changes

ajantha-bhat added this to the Iceberg 1.7.0 milestone Sep 27, 2024

ajantha-bhat marked this pull request as ready for review September 27, 2024 02:00

ajantha-bhat requested a review from aokolnychyi September 27, 2024 02:00

ajantha-bhat mentioned this pull request Oct 16, 2024

Partition stats task tracker #8450

Open

11 tasks

RussellSpitzer reviewed Oct 25, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/PartitionStats.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 25, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/PartitionStats.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 25, 2024

View reviewed changes

core/src/test/java/org/apache/iceberg/TestTables.java Show resolved Hide resolved

RussellSpitzer reviewed Oct 25, 2024

View reviewed changes

data/src/test/java/org/apache/iceberg/data/TestPartitionStatsHandler.java Outdated Show resolved Hide resolved

RussellSpitzer modified the milestones: Iceberg 1.7.0, Iceberg 2.0.0 Oct 28, 2024

ajantha-bhat force-pushed the stats_writer branch from 05a80f6 to ee3b273 Compare November 19, 2024 15:23

aokolnychyi reviewed Nov 21, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/PartitionStats.java Outdated Show resolved Hide resolved

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java Outdated Show resolved Hide resolved

ajantha-bhat modified the milestones: Iceberg 2.0.0, Iceberg 1.8.0 Nov 22, 2024

ajantha-bhat commented Jan 15, 2025

View reviewed changes

ajantha-bhat force-pushed the stats_writer branch from 289957f to e1a4e89 Compare January 15, 2025 18:19

ajantha-bhat force-pushed the stats_writer branch from e1a4e89 to 0668d6a Compare January 25, 2025 01:52

github-actions bot removed the API label Jan 25, 2025

ajantha-bhat force-pushed the stats_writer branch from 0668d6a to dd2c45a Compare January 25, 2025 02:11

deniskuzZ reviewed Jan 30, 2025

View reviewed changes

data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java Show resolved Hide resolved

Data: Add partition stats writer and reader

4945ec1

ajantha-bhat force-pushed the stats_writer branch from dd2c45a to 4945ec1 Compare January 30, 2025 16:20

deniskuzZ reviewed Jan 31, 2025

View reviewed changes

deniskuzZ approved these changes Jan 31, 2025

View reviewed changes

pvary reviewed Feb 3, 2025

View reviewed changes

Address comments

2b61d23

ajantha-bhat force-pushed the stats_writer branch from 9324227 to 2b61d23 Compare February 4, 2025 05:59

Data: Add partition stats writer and reader #11216

Are you sure you want to change the base?

Data: Add partition stats writer and reader #11216

Conversation

ajantha-bhat commented Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajantha-bhat commented Sep 27, 2024

ajantha-bhat commented Oct 23, 2024

aokolnychyi commented Oct 24, 2024

ajantha-bhat commented Oct 24, 2024 • edited Loading

RussellSpitzer commented Oct 28, 2024

ajantha-bhat commented Nov 19, 2024 • edited Loading

aokolnychyi commented Nov 20, 2024

jbonofre commented Nov 28, 2024

ajantha-bhat Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajantha-bhat commented Jan 15, 2025

deniskuzZ commented Jan 16, 2025 • edited Loading

ajantha-bhat commented Jan 25, 2025

deniskuzZ commented Jan 25, 2025 • edited Loading

ajantha-bhat commented Jan 25, 2025

ajantha-bhat commented Jan 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvary Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deniskuzZ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvary Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

ajantha-bhat Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajantha-bhat commented Feb 4, 2025

ajantha-bhat commented Sep 26, 2024 •

edited

Loading

ajantha-bhat commented Oct 24, 2024 •

edited

Loading

ajantha-bhat commented Nov 19, 2024 •

edited

Loading

ajantha-bhat Jan 15, 2025 •

edited

Loading

deniskuzZ commented Jan 16, 2025 •

edited

Loading

deniskuzZ commented Jan 25, 2025 •

edited

Loading

pvary Feb 3, 2025 •

edited

Loading

pvary Feb 3, 2025 •

edited

Loading

ajantha-bhat Feb 4, 2025 •

edited

Loading