GH-46677: [C++] Expose an BinaryViewBuilder interface for append a binary and multiple subslice #46730

IndifferentArea · 2025-06-06T15:49:08Z

Rationale for this change

see #46677

What changes are included in this PR?

see #46677

Are these changes tested?

Yes

Are there any user-facing changes?

No

GitHub Issue: [C++] Expose an BinaryViewBuilder interface for append a binary and multiple subslice #46677

IndifferentArea · 2025-06-07T02:25:50Z

@mapleFU is currently implemented interface expected?

mapleFU · 2025-06-07T04:23:19Z

cpp/src/arrow/array/builder_binary.h

+    return AppendBlock(value.data(), static_cast<int64_t>(value.size()));
+  }
+
+  Status AppendViewFromBuffer(int32_t buffer_id, int32_t buffer_offset, int32_t start,


naming: from buffer or from block?

Personally both is ok for me, I prefer Buffer since a variable is buffer_index

mapleFU · 2025-06-07T04:28:03Z

cpp/src/arrow/array/builder_binary.h

@@ -645,6 +657,28 @@ class ARROW_EXPORT BinaryViewBuilder : public ArrayBuilder {
    UnsafeAppend(value.data(), static_cast<int64_t>(value.size()));
  }

+  Result<std::pair<int32_t, int32_t>> AppendBlock(const uint8_t* value,


Can we use more specific name rather than pair<i32, i32>?

can i directly use BinaryViewType::c_type since it already contains these two info we need?

The syntax is a bit weird here? Append a BinaryView and then append the sub-slice of the view?

mapleFU · 2025-06-07T04:29:02Z

cpp/src/arrow/array/builder_binary.cc

@@ -100,6 +100,13 @@ void BinaryViewBuilder::Reset() {
  data_heap_builder_.Reset();
 }

+Result<std::pair<int32_t, int32_t>> BinaryViewBuilder::AppendBlock(const uint8_t* value,
+                                                                   const int64_t length) {
+  DCHECK_GT(length, TypeClass::kInlineSize);


If length <= kInlineSize, should this return false or ok? Why just DCHECK here?

mapleFU · 2025-06-07T04:33:53Z

cpp/src/arrow/array/builder_binary.h

+  c_type GetViewFromBlock(int32_t block_id, int32_t block_offset, int32_t offset,
+                          int32_t length) const {
+    const auto* value = blocks_.at(block_id)->data_as<uint8_t>() + block_offset + offset;
+    if (length <= BinaryViewType::kInlineSize) {


Uses ToBinaryView?

IndifferentArea · 2025-06-07T07:06:45Z

Should we rename AppendBuffer or redesign the interface's semantics? since currently we don't really append a buffer/block, we will directly append to the last one if remaining size is enough.. I think It may introduce confusion.

Maybe aligning with arrow-rs's impl is fine..

mapleFU · 2025-06-07T12:01:42Z

Some personal thoughts:

AppendBuffer ( and etc ) which returns a StringView is a bit weird, Block is not a StringView
Now aligned with arrow-rs also a good way

IndifferentArea · 2025-06-08T12:43:59Z

Not sure why these 2 ci always failed..

cpp/src/arrow/array/array_test.cc

IndifferentArea · 2025-06-14T14:05:31Z

To implement interface aligned with what arrow-rs did, i have to change some behavior of StringHeapBuild::FinishLastBlock() and mark it as public.

More specific, before FinishLastBlock() just resize the last block. Now FinishLastBlock() reset internal states, including current_offset_, current_out_buffer_ and current_remaining_bytes_. I believe it's more safe and reasonable.
The interface was private before so don't mind external usage, for current internal usage of FinishLastBlock(), they always reset or change all these status.

If this change is unacceptable, plz let me know, i'll try to find another way.

mapleFU

General LGTM. Also cc @pitrou for your advices for this interface. This interface would be used for read binary as stringView from bytearray type faster

mapleFU · 2025-06-14T14:30:27Z

cpp/src/arrow/array/builder_binary.h

+    return AppendBuffer(reinterpret_cast<const uint8_t*>(value), length);
+  }
+
+  Result<int32_t> AppendBuffer(const std::string& value) {


Can std::string_view being used rather than const std::string&?

Can we remove this one since std::string_view is added here?

cpp/src/arrow/array/builder_binary.h

mapleFU · 2025-06-20T15:36:53Z

cpp/src/arrow/array/builder_binary.h

+  void UnsafeAppendViewFromBuffer(const int32_t buffer_idx, const int32_t start,
+                                  const int32_t length) {
+    UnsafeAppendToBitmap(true);
+    const auto v = data_heap_builder_.GetViewFromBuffer<false>(buffer_idx, start, length);


Suggested change

const auto v = data_heap_builder_.GetViewFromBuffer<false>(buffer_idx, start, length);

const auto v = data_heap_builder_.GetViewFromBuffer</*Safe=*/false>(buffer_idx, start, length);

mapleFU · 2025-06-20T15:37:10Z

cpp/src/arrow/array/builder_binary.h

+                              const int32_t length) {
+    ARROW_RETURN_NOT_OK(Reserve(1));
+    UnsafeAppendToBitmap(true);
+    ARROW_ASSIGN_OR_RAISE(const auto v, data_heap_builder_.GetViewFromBuffer<true>(


Suggested change

ARROW_ASSIGN_OR_RAISE(const auto v, data_heap_builder_.GetViewFromBuffer<true>(

ARROW_ASSIGN_OR_RAISE(const auto v, data_heap_builder_.GetViewFromBuffer</*Safe=*/true>(

mapleFU · 2025-06-26T07:50:48Z

Gentle ping @pitrou

pitrou · 2025-06-26T13:24:55Z

Isn't this approach wasteful? If you have lots of strings <= 12 bytes, you will still store their contents in a data buffer, while they're inlined in the string views.

mapleFU · 2025-06-26T13:32:35Z

Isn't this approach wasteful? If you have lots of strings <= 12 bytes, you will still store their contents in a data buffer, while they're inlined in the string views.

@pitrou I suppose this is used to append a whole parquet page and add buffer for it

pitrou · 2025-06-26T13:40:14Z

Is it a win, though? If most Parquet strings are <= 12 bytes we would pointlessly waste space and CPU time.

mapleFU · 2025-06-26T13:57:03Z

Is it a win, though? If most Parquet strings are <= 12 bytes we would pointlessly waste space and CPU time.

A nice question. I agree when most Parquet strings are <= 12 bytes, it would be memory wasted because a huge memcpy is applied. But when read large binary it would benefit a lot from this. I think usally a large memcpy might much faster than little un-continogous memcpy

Maybe we can also try to pick this way when average len is huge enough?

pitrou · 2025-06-26T14:09:38Z

A nice question. I agree when most Parquet strings are <= 12 bytes, it would be memory wasted because a huge memcpy is applied. But when read large binary it would benefit a lot from this. I think usally a large memcpy might much faster than little un-continogous memcpy

That's true, but another cost is to create the views themselves. It would be nice if a prototype could tell us which speedup we can expect.

Maybe we can also try to pick this way when average len is huge enough?

Yes, that's definitely a possibility.

mapleFU · 2025-06-26T14:36:22Z

So can we start to review this? We can set a ratio when average length > 12 or > 20

andishgar · 2025-06-29T16:09:07Z

@mapleFU @pitrou
I believe this pull request is related to several other PRs I've submitted. Here's a summary:

1- API and Handling of the Last Buffer
In this pull request, I demonstrated that it’s possible to share buffers without copying or finalizing the last buffer. This avoids relocating the buffer to remove blank space, which can be a costly operation when the unused space exceeds 64 bytes.

2-

Is it a win, though? If most Parquet strings are <= 12 bytes we would pointlessly waste space and CPU time.

In this pull request, I proposed a method that could help avoid memory bloat when buffers are shared. Additionally, in this issue, I think this metadata could help determine when CompactArray should be called.

Overall, my suggestion is to either modify this pull request or create a new API to support buffer sharing. It is possible to decide whether a created array should be compacted based on some metadata, in order to avoid memory bloat.

init: a draft version

57e0bec

github-actions bot added Component: C++ awaiting review Awaiting review labels Jun 6, 2025

IndifferentArea added 4 commits June 7, 2025 00:00

fix ci

1e1903d

try fixing ci

3780e56

try fixing ci

575d8e5

try fixing ci

ae7115a

mapleFU reviewed Jun 7, 2025

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 7, 2025

fix

7d3c084

IndifferentArea added 3 commits June 7, 2025 21:26

fix & add more check

53c4cf4

try fixing MSVC ci

a52df7b

Merge branch 'apache:main' into apacheGH-46677

ef8c814

IndifferentArea marked this pull request as ready for review June 8, 2025 12:44

mapleFU reviewed Jun 8, 2025

View reviewed changes

cpp/src/arrow/array/array_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/array/array_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/array/array_test.cc Outdated Show resolved Hide resolved

add more test, change FinishLastBlock

22dc342

mapleFU reviewed Jun 14, 2025

View reviewed changes

fix

7ed94dc

IndifferentArea requested a review from mapleFU June 18, 2025 12:21

mapleFU approved these changes Jun 20, 2025

View reviewed changes

mapleFU requested a review from pitrou June 20, 2025 15:38

fix

f896a97

	const auto v = data_heap_builder_.GetViewFromBuffer<false>(buffer_idx, start, length);
	const auto v = data_heap_builder_.GetViewFromBuffer</Safe=/false>(buffer_idx, start, length);

	ARROW_ASSIGN_OR_RAISE(const auto v, data_heap_builder_.GetViewFromBuffer<true>(
	ARROW_ASSIGN_OR_RAISE(const auto v, data_heap_builder_.GetViewFromBuffer</Safe=/true>(

GH-46677: [C++] Expose an BinaryViewBuilder interface for append a binary and multiple subslice #46730

Are you sure you want to change the base?

GH-46677: [C++] Expose an BinaryViewBuilder interface for append a binary and multiple subslice #46730

Conversation

IndifferentArea commented Jun 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

IndifferentArea commented Jun 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IndifferentArea Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IndifferentArea commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mapleFU commented Jun 7, 2025

Uh oh!

IndifferentArea commented Jun 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IndifferentArea commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

mapleFU Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU commented Jun 26, 2025

Uh oh!

pitrou commented Jun 26, 2025

Uh oh!

mapleFU commented Jun 26, 2025

Uh oh!

pitrou commented Jun 26, 2025

Uh oh!

mapleFU commented Jun 26, 2025

Uh oh!

pitrou commented Jun 26, 2025

Uh oh!

mapleFU commented Jun 26, 2025

Uh oh!

andishgar commented Jun 29, 2025

Uh oh!

Uh oh!

IndifferentArea commented Jun 6, 2025 •

edited by github-actions bot

Loading

IndifferentArea Jun 7, 2025 •

edited

Loading

IndifferentArea commented Jun 7, 2025 •

edited

Loading

IndifferentArea commented Jun 14, 2025 •

edited

Loading

mapleFU Jun 14, 2025 •

edited

Loading