GH-46092: [C++] Add filesystem related options to Meson #46101

WillAyd · 2025-04-10T21:05:50Z

Rationale for this change

This continues adding support for the Meson build system by adding filesystem support

What changes are included in this PR?

This implements the filesystem directory and adds support for Azure. GCS and S3 are stubbed out but will raise an error when a user attempts to enable them, strictly because they look like larger integrations to get the dependency chain properly configured.

Are these changes tested?

Locally yes

Are there any user-facing changes?

No

GitHub Issue: [C++] Add filesystem directory to Meson #46092

WillAyd · 2025-04-10T21:13:34Z

@github-actions crossbow submit *meson

github-actions · 2025-04-10T21:15:52Z

Revision: a2bdd12

Submitted crossbow builds: ursacomputing/crossbow @ actions-df2349b5fe

Task	Status
test-conda-cpp-meson

kou · 2025-04-10T21:50:47Z

cpp/meson.build

 needs_hdfs = get_option('hdfs')
-needs_filesystem = false or needs_hdfs
+needs_s3 = get_option('s3')
+needs_filesystem = get_option('filesystem') or needs_azure or needs_gcs or needs_s3 or needs_hdfs


Suggested change

needs_filesystem = get_option('filesystem') or needs_azure or needs_gcs or needs_s3 or needs_hdfs

needs_filesystem = get_option('filesystem') or needs_azure or needs_gcs or needs_hdfs or needs_s3

kou · 2025-04-10T21:52:02Z

cpp/meson.options

+option(
+    'gcs',
+    type: 'boolean',
+    description: 'Build Arrow with GCS support (requires the GCloud SDK for C++)',


Based on its README: https://github.com/googleapis/google-cloud-cpp?tab=readme-ov-file#google-cloud-platform-c-client-libraries

Suggested change

description: 'Build Arrow with GCS support (requires the GCloud SDK for C++)',

description: 'Build Arrow with GCS support (requires the Google Cloud Platform C++ Client Libraries)',

No problem - do you want me to update DefineOptions.cmake with the new description as well?

Yes, please.

kou · 2025-04-10T21:54:11Z

cpp/src/arrow/filesystem/meson.build

+if needs_s3
+endif
+


Suggested change

if needs_s3

endif

kou · 2025-04-10T21:54:23Z

cpp/src/arrow/filesystem/meson.build

+if needs_s3
+endif
+
+if needs_hdfs


Can we remove this?

@WillAyd I think that we can merge this once this is resolved.

I think I am misunderstanding the request - I removed all the empty branches but this one has a test in it. I think we want to keep this around right?

Ah, sorry. You're right. I misunderstood.

I thought that we have disabler() dependency for needs_hdfs here but it doesn't exist. We need this if needs_hdfs.

kou · 2025-04-10T21:57:43Z

cpp/src/arrow/meson.build

+            # TODO: couldn't find any usage of this in Arrow - do we need?
+            # set(ENV{AZURE_SDK_DISABLE_AUTO_VCPKG} TRUE)


Azure SDK for C++ prepared vcpkg automatically without this: https://github.com/Azure/azure-sdk-for-cpp/blob/27c2e5d335c9306d08aa49bd5fe92d0cc1628fe0/cmake-modules/AzureVcpkg.cmake#L17-L41

We don't want to use vcpkg here.

Ok great - I will set to FALSE then. Should I update the CMake configuration as well?

Ah nevermind - ignore comment above. I was thinking this ENABLED something but I see it DISABLEs

kou · 2025-04-10T21:58:01Z

cpp/src/arrow/meson.build

+        arrow_filesystem_srcs += [
+            'filesystem/gcsfs.cc',
+            'filesystem/gcsfs_internal.cc',
+        ]
+        gcs_dep = dependency('google-cloud-cpp')
+        arrow_filesystem_deps += [gcs_dep]


Could you remove this for now?

No problem at all (for gcs and s3)

kou · 2025-04-10T21:58:09Z

cpp/src/arrow/meson.build

+
+    if needs_s3
+        error('s3 filesystem support is not yet implemented in Meson')
+        arrow_filesystem_srcs += ['filesystem/s3fs.cc']


Could you remove this for now?

WillAyd · 2025-04-11T12:46:30Z

@github-actions crossbow submit *meson

github-actions · 2025-04-11T12:48:41Z

Revision: 3355228

Submitted crossbow builds: ursacomputing/crossbow @ actions-f11410af11

Task	Status
test-conda-cpp-meson

kou · 2025-04-14T01:48:08Z

cpp/cmake_modules/DefineOptions.cmake

@@ -346,7 +346,7 @@ takes precedence over ccache if a storage backend is configured" ON)
                ARROW_WITH_UTF8PROC)

  define_option(ARROW_GCS
-                "Build Arrow with GCS support (requires the GCloud SDK for C++)"
+                "Build Arrow with GCS support (requires the Google Cloud Platform C++ Client Libraries)"


Could you fix this failure?

https://github.com/apache/arrow/actions/runs/14430422114/job/40464526899?pr=46101#step:7:423

CMake Error at cmake_modules/DefineOptions.cmake:27 (message): description for ARROW_GCS contained a line 86 characters long! (max is 80). Split it into more lines with semicolons Call Stack (most recent call first): cmake_modules/DefineOptions.cmake:47 (check_description_length) cmake_modules/DefineOptions.cmake:348 (define_option) CMakeLists.txt:224 (include)

Whoops - should be fixed now

kou

+1

github-actions · 2025-04-14T21:45:01Z

⚠️ GitHub issue #46092 has been automatically assigned in GitHub to PR creator.

conbench-apache-arrow · 2025-04-15T05:07:19Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6b64396.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.

…#46101) ### Rationale for this change This continues adding support for the Meson build system by adding filesystem support ### What changes are included in this PR? This implements the filesystem directory and adds support for Azure. GCS and S3 are stubbed out but will raise an error when a user attempts to enable them, strictly because they look like larger integrations to get the dependency chain properly configured. ### Are these changes tested? Locally yes ### Are there any user-facing changes? No * GitHub Issue: apache#46092 Authored-by: Will Ayd <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

WillAyd force-pushed the meson-add-filesystem branch from 2bb0e73 to a2bdd12 Compare April 10, 2025 21:06

github-actions bot added Component: C++ awaiting review Awaiting review labels Apr 10, 2025

kou reviewed Apr 10, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Apr 10, 2025

WillAyd added 2 commits April 11, 2025 08:45

apacheGH-46092: [C++] Add filesystem option to Meson

482279f

kou feedback

3355228

WillAyd force-pushed the meson-add-filesystem branch from a2bdd12 to 3355228 Compare April 11, 2025 12:45

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 11, 2025

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Apr 13, 2025

kou reviewed Apr 14, 2025

View reviewed changes

Updated DefineOptions

320167c

WillAyd force-pushed the meson-add-filesystem branch from 999cf12 to 320167c Compare April 14, 2025 13:02

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 14, 2025

kou approved these changes Apr 14, 2025

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Apr 14, 2025

kou changed the title ~~GH-46092: [C++] Add filesystem option to Meson~~ GH-46092: [C++] Add filesystem related options to Meson Apr 14, 2025

kou merged commit 6b64396 into apache:main Apr 14, 2025
37 of 39 checks passed

kou removed the awaiting merge Awaiting merge label Apr 14, 2025

kou mentioned this pull request Apr 14, 2025

[C++] Add filesystem directory to Meson #46092

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-46092: [C++] Add filesystem related options to Meson #46101

GH-46092: [C++] Add filesystem related options to Meson #46101

WillAyd commented Apr 10, 2025 •

edited by github-actions bot

Loading

WillAyd commented Apr 10, 2025

github-actions bot commented Apr 10, 2025

kou Apr 10, 2025

kou Apr 10, 2025

WillAyd Apr 11, 2025

kou Apr 13, 2025

kou Apr 10, 2025

kou Apr 10, 2025

kou Apr 13, 2025

WillAyd Apr 13, 2025

kou Apr 14, 2025

kou Apr 10, 2025

WillAyd Apr 11, 2025

WillAyd Apr 11, 2025

kou Apr 10, 2025

WillAyd Apr 11, 2025

kou Apr 10, 2025

WillAyd commented Apr 11, 2025

github-actions bot commented Apr 11, 2025

kou Apr 14, 2025

WillAyd Apr 14, 2025

kou left a comment

github-actions bot commented Apr 14, 2025

conbench-apache-arrow bot commented Apr 15, 2025

	needs_filesystem = get_option('filesystem') or needs_azure or needs_gcs or needs_s3 or needs_hdfs
	needs_filesystem = get_option('filesystem') or needs_azure or needs_gcs or needs_hdfs or needs_s3

	description: 'Build Arrow with GCS support (requires the GCloud SDK for C++)',
	description: 'Build Arrow with GCS support (requires the Google Cloud Platform C++ Client Libraries)',

		# TODO: couldn't find any usage of this in Arrow - do we need?
		# set(ENV{AZURE_SDK_DISABLE_AUTO_VCPKG} TRUE)

GH-46092: [C++] Add filesystem related options to Meson #46101

GH-46092: [C++] Add filesystem related options to Meson #46101

Conversation

WillAyd commented Apr 10, 2025 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

WillAyd commented Apr 10, 2025

github-actions bot commented Apr 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Apr 11, 2025

github-actions bot commented Apr 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 14, 2025

conbench-apache-arrow bot commented Apr 15, 2025

WillAyd commented Apr 10, 2025 •

edited by github-actions bot

Loading