Upgrade to protobuf 6.31.1 by gongsu832 · Pull Request #3275 · onnx/onnx-mlir

gongsu832 · 2025-09-08T16:55:15Z

Upgrade to protobuf 6.31.1, which is the 1st version that uses rules_rust 0.56.0 with s390x support we managed to upstream. Some caveats apply:

bazel related toolchain support on ppc64le is even worse than that on s390x so this PR will not work on ppc64le. As a result, we will have to retire our ppc64le Jenkins CI after this PR (we can resurrect it once the bazel toolchain on ppc64le catches up). ppc64le Jenkins CI is not required for merging
onnx cmake build file needs to be patched to add some missing dependencies on abseil, which should eventually be upstreamed

gongsu832 · 2025-09-10T01:34:02Z

OK MacOS CI build is fixed. The requirements.txt still had the old version of protobuf specified, which overrode the new version. The PR is ready to be merged.

rules_rust 0.56.0 with s390x support we managed to upstream. Some caveats apply: - bazel related toolchain support on ppc64le is even worse than that on s390x so this PR will not work on ppc64le. As a result, we will have to retire our ppc64le Jenkins CI after this PR (we can resurrect it once the bazel toolchain on ppc64le catches up). ppc64le Jenkins CI is not required for merging - onnx cmake build file needs to be patched to add some missing dependencies on abseil, which should eventually be upstreamed Signed-off-by: Gong Su <[email protected]>

Signed-off-by: Gong Su <[email protected]>

- subdivide jobs so mlir cache can be posted early Signed-off-by: Gong Su <[email protected]>

Signed-off-by: Gong Su <[email protected]>

onnx doesn't get installed then reinstalled Signed-off-by: Gong Su <[email protected]>

chentong319 · 2025-09-10T14:51:43Z

Do we need to update the document(such as BuildOnLinuxOSX.md) accordingly?

gongsu832 · 2025-09-10T17:17:56Z

Do we need to update the document(such as BuildOnLinuxOSX.md) accordingly?

There is really nothing changed in terms of building onnx-mlir itself. Most of the changes are for building protobuf with C/C++ backend on s390x. We can certainly describe what's need to be done in BuildOnLinuxOSX.md but unless you are a developer you probably don't care about that. This is also the reason we typically recommend people to use the prebuilt docker image instead of trying to do all these dirty hack themselves.

Sunny-Anand · 2025-09-12T18:43:33Z

@gongsu832 I am running some tests based off this pr especially on the abseil dependency part. I will update the results here with my experiment. Sorry for the delay caught up with a few urgent issues. Will try to post the results and findings by next week.

Sunny-Anand

I am not very well acquainted with Bazel and Protobuf. It would be wonderful if we could store the dependency matrix and also the patch for each broken dependency.

Otherwise looks good to me. Approving as based on my newer tests based off this pr, the builds are still running into broken dependencies when trying with bazel 7.2 and using bzlmod and rust.

Sunny-Anand · 2025-10-08T17:17:03Z

utils/install-protobuf.sh

+rm -rf ${ABSL_DIR} ${HOME}/.cache
+
+# Install protobuf
+PROTOBUF_VERSION=6.31.1


Do we want to keep the same version as 6.32 or do we want to update requirements.txt with 6.31.1

Sunny-Anand · 2025-10-08T17:29:45Z

utils/install-protobuf.sh

-cd python
-python3 setup.py install --cpp_implementation
+# Install absl for new version of protobuf
+ABSL_VERSION=20240722.1


Same as above for ABSEIL version.

Sunny-Anand · 2025-10-08T17:33:09Z

utils/install-protobuf.sh

+cd ${PROTOBUF_DIR}/build
+# Must specify -Dprotobuf_BUILD_TESTS=OFF otherwise find_package(absl)
+# in onnx will fail due to missing protobuf::gmock target
+cmake -DCMAKE_INSTALL_LIBDIR=lib \


Same as above for using cmake and bazel together.

Sunny-Anand · 2025-10-08T17:35:33Z

utils/install-onnx.sh

@@ -0,0 +1,9 @@
+git fetch --prune --unshallow --tags


Same comment as above. It seems this patching is needed for all CI platforms, is this correct? Then should we get someone from onnx take a look at this?

Sunny-Anand · 2025-10-08T17:53:26Z

docker/Dockerfile.onnx-mlir

 RUN ONNX_ROOT=${WORK_DIR}/onnx-mlir/third_party/onnx \
    && cd ${ONNX_ROOT} \
-    && CMAKE_ARGS="-DCMAKE_INSTALL_LIBDIR=lib" python3 -m pip install .
+    # Require patching until upstreamed


Why is this patching needed for onnx here for abseil? Doesn't onnx as third-party install its own protoc using cmake and abseil? Is this something we need to report to onnx as a breakage?

Sunny-Anand · 2025-10-08T17:55:28Z

docker/Dockerfile.llvm-project

+    && cargo install cargo-bazel --version ${CARGO_BAZEL_VERSION}
+
+# Install absl for new version of protobuf
+ARG ABSL_VERSION=20240722.1


Should this be updated to the latest version of ABSEIL for protobuf 32.0 https://registry.bazel.build/modules/protobuf/32.0

Sunny-Anand · 2025-10-08T17:56:28Z

docker/Dockerfile.llvm-project

-RUN git clone -b v${PROTOBUF_VERSION} --recursive https://github.com/protocolbuffers/protobuf.git \
-    && cd protobuf && ./autogen.sh \
-    && ./configure --enable-static=no \
+ARG PROTOBUF_VERSION=6.31.1


Do we want to keep the same version as 6.32 or do we want to update requirements.txt with 6.31.1

Sunny-Anand · 2025-10-08T17:58:11Z

docker/Dockerfile.llvm-project

+    # Must specify -Dprotobuf_BUILD_TESTS=OFF otherwise find_package(absl)
+    # in onnx will fail due to missing protobuf::gmock target
+    && CC=clang CXX=clang++ \
+       cmake -DCMAKE_INSTALL_LIBDIR=lib \


I did try to change the build from cmake to bazel for building and installing protobuf and building the protoc; however as usual, it runs into issues with missing upstreams for rust_rules for s390x. The only comment I have here is, if we had found a way to use Bazel for this step, it would avoid needing us to build abseil as it is a dependency of protobuf and if bazel is used for building protobuf then it would also be able to fetch and build abseil own its own. https://registry.bazel.build/modules/protobuf/32.0

Sunny-Anand · 2025-10-13T16:59:24Z

@AlexandreEichenberger @gongsu832 if you guys are ok with merging this pr, I can merge this now. Also do we want to update https://github.com/onnx/onnx-mlir/blob/main/README.md to drop ppc64le-Linux and also remove/stop the CI from running?

jenkins-droid · 2025-10-13T23:33:23Z

Jenkins Linux amd64 Build #17470 [push] Upgrade to protobuf 6.31... started at 18:33

jenkins-droid · 2025-10-13T23:33:23Z

Jenkins Linux s390x Build #17472 [push] Upgrade to protobuf 6.31... started at 19:33

jenkins-droid · 2025-10-14T02:04:43Z

Jenkins Linux s390x Build #17472 [push] Upgrade to protobuf 6.31... passed after 2 hr 31 min

jenkins-droid · 2025-10-14T02:46:49Z

Jenkins Linux amd64 Build #17470 [push] Upgrade to protobuf 6.31... passed after 3 hr 13 min

AlexandreEichenberger · 2025-10-14T14:35:21Z

FYI, on my Mac I had to install the package below

brew install abseil

gongsu832 force-pushed the upgrade_protobuf_v6.31.1 branch 2 times, most recently from 5b23648 to 22ac878 Compare September 10, 2025 01:33

gongsu832 added 14 commits September 9, 2025 23:53

Fix MacOS build

7edc15d

Signed-off-by: Gong Su <[email protected]>

More MacOS build fixes

a9912b7

Signed-off-by: Gong Su <[email protected]>

Remove bazelisk

11fad30

Signed-off-by: Gong Su <[email protected]>

Try python 3.12

729d2ed

Signed-off-by: Gong Su <[email protected]>

Try prebuilt protobuf

a76c477

Signed-off-by: Gong Su <[email protected]>

Make cache-mlir key depend on protobuf version

f631874

Signed-off-by: Gong Su <[email protected]>

MacOS sed does not take multiple -e options

fd4096e

Signed-off-by: Gong Su <[email protected]>

Trying to debug MacOS build

b80e99e

Signed-off-by: Gong Su <[email protected]>

- brew install protobuf@32

8b9f593

- subdivide jobs so mlir cache can be posted early Signed-off-by: Gong Su <[email protected]>

add missing runs-on for jobs

1ac82fd

Signed-off-by: Gong Su <[email protected]>

save MLIR cache immediately after successful build

491976d

Signed-off-by: Gong Su <[email protected]>

Update requirements.txt

3681945

Signed-off-by: Gong Su <[email protected]>

Move installing python requirements after third_party/onnx so that

046183c

onnx doesn't get installed then reinstalled Signed-off-by: Gong Su <[email protected]>

gongsu832 force-pushed the upgrade_protobuf_v6.31.1 branch from 22ac878 to 046183c Compare September 10, 2025 03:54

Sunny-Anand self-requested a review September 12, 2025 18:43

Sunny-Anand added 2 commits October 7, 2025 21:43

Merge branch 'main' into upgrade_protobuf_v6.31.1

5fdefa7

Merge branch 'main' into upgrade_protobuf_v6.31.1

0a8d1d8

Sunny-Anand approved these changes Oct 8, 2025

View reviewed changes

Merge branch 'main' into upgrade_protobuf_v6.31.1

f344031

Sunny-Anand mentioned this pull request Oct 13, 2025

protobuf upgrade #3255

Closed

Sunny-Anand merged commit f24291d into onnx:main Oct 13, 2025
6 checks passed

christopherlmunoz mentioned this pull request Oct 14, 2025

start of llvm/stablehlo bump #3302

Merged

Conversation

gongsu832 commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gongsu832 commented Sep 10, 2025

Uh oh!

chentong319 commented Sep 10, 2025

Uh oh!

gongsu832 commented Sep 10, 2025

Uh oh!

Sunny-Anand commented Sep 12, 2025

Uh oh!

Sunny-Anand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sunny-Anand commented Oct 13, 2025

Uh oh!

Uh oh!

jenkins-droid commented Oct 13, 2025

Uh oh!

jenkins-droid commented Oct 13, 2025

Uh oh!

jenkins-droid commented Oct 14, 2025

Uh oh!

jenkins-droid commented Oct 14, 2025

Uh oh!

AlexandreEichenberger commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gongsu832 commented Sep 8, 2025 •

edited

Loading