protobuf upgrade #3255

Sunny-Anand · 2025-08-07T03:01:23Z

Protobuf upgrade from 4.21.12 to 4.25.1 which is the new minimum required for onnx 1.18.0. With protobuf 4.22.0 autotools support is dropped, so we need to build and install protobuf from the source in our docker with the abseil lib without the python setup.py

Signed-off-by: Sunny Anand <[email protected]>

chentong319 · 2025-08-12T14:48:56Z

docker/Dockerfile.llvm-project

-    && make -j${NPROC} install && ldconfig \
-    && cd python && python3 setup.py install --cpp_implementation \
-    && cd ../.. && rm -rf protobuf
+ARG PROTOBUF_VERSION=25.1


The version is 4.25.1 in other places. What's the reason?

There is no difference between the 2 as you can see the commit is the same. The different naming style is due to the fact that 3. or 4. or 5. is a different release style adopted by protobuf for python verions.

So all the places where there is a python pip reference the 4.25.1 version is used while in onnx-mlir we rely on the c++ protobuf so the verion is 25.1

https://pypi.org/project/protobuf/4.25.1/

Signed-off-by: Sunny Anand <[email protected]>

Sunny-Anand · 2025-08-13T18:23:58Z

@chentong319 @gongsu832, this pr is ready for review.

gongsu832 · 2025-08-13T20:26:38Z

You removed the part that installs the protobuf python binding with cpp_implementation, which typically should fail the build. But the build didn't fail. That's because onnx apparently is installing its own version of protobuf anyway, which you can see from the build log:

Processing /workdir/onnx-mlir/third_party/onnx
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numpy>=1.20 in /usr/lib/python3/dist-packages (from onnx==1.17.0) (1.21.5)
Collecting protobuf>=3.20.2 (from onnx==1.17.0)
  Using cached protobuf-6.31.1-py3-none-any.whl.metadata (593 bytes)
Using cached protobuf-6.31.1-py3-none-any.whl (168 kB)
Building wheels for collected packages: onnx
  Building wheel for onnx (pyproject.toml): started
  Building wheel for onnx (pyproject.toml): finished with status 'done'
  Created wheel for onnx: filename=onnx-1.17.0-cp310-cp310-linux_s390x.whl size=17690044 sha256=049180da11bdf03ba6cc827a6c7aba6ab66df5e38950110f90f81ab450917078
  Stored in directory: /tmp/pip-ephem-wheel-cache-vi37mcv8/wheels/fc/22/72/c01d6d50556199cf4a3117cda7bbabfedb2a7da0145cfc7502
Successfully built onnx
Installing collected packages: protobuf, onnx

Successfully installed onnx-1.17.0 protobuf-6.31.1
...

The problem with this version of protobuf is that its internal implementation is pure python rather than C/C++ so its performance is going to be a lot worse.

# python3
Python 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.protobuf.internal import api_implementation
>>> implementation_type = api_implementation.Type()
>>> print(f"Current Protobuf implementation: {implementation_type}")
Current Protobuf implementation: python

For a C/C++ implementation, you should see upb instead. Although how much impact it will have on the ONNX-MLIR compiler is not clear.

Sunny-Anand · 2025-08-13T20:48:42Z

For a C/C++ implementation, you should see upb instead. Although how much impact it will have on the ONNX-MLIR compiler is not clear.

I did try with the initial approach of keeping the c++ implementation with setup.py but it either never built or completed the tests.

I will add the change again and see if the failure happens again with setup.py, one thing to remember is there is std-c++=14 code level with the setup.py with --cpp_implementation

# Exit immediately if a command exits with a non-zero status.
set -e
# Check out protobuf source code and build and install it
PROTOBUF_VERSION=25.1
INSTALL_PROTOBUF_PATH=~/work/protobuf_install # Changed to a dedicated install directory
git clone -b v${PROTOBUF_VERSION} --depth 1 --recursive https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
mkdir build_source && cd build_source
cmake -G Ninja ../ \
    -Dprotobuf_BUILD_SHARED_LIBS=OFF \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_PROTOBUF_PATH \
    -Dprotobuf_BUILD_TESTS=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    -DCMAKE_CXX_STANDARD=17 \
    -DABSL_PROPAGATE_CXX_STD=ON \
    ..
cmake --build . --target install

# Verify that protoc is installed correctly before proceeding
echo "Verifying protoc installation at $INSTALL_PROTOBUF_PATH/bin/protoc..."
if [ -f "$INSTALL_PROTOBUF_PATH/bin/protoc" ]; then
    echo "protoc found."
    "$INSTALL_PROTOBUF_PATH/bin/protoc" --version
else
    echo "Error: protoc not found at $INSTALL_PROTOBUF_PATH/bin/protoc. Installation might have failed."
    exit 1
fi

# Now navigate and run the python setup.py
# Use a subshell to temporarily modify PATH and LDFLAGS for this specific command,
# ensuring our installed protoc and libraries are found first.
# Pass library/include paths directly to setup.py
if [ -d "$HOME/work/protobuf/python/build" ]; then
  echo "Removing existing build directory..."
  rm -rf "$HOME/work/protobuf/python/build"
else
  echo "No build directory to remove."
fi

cd ~/work/protobuf/python
# Temporarily modify setup.py to remove the hardcoded -std=c++14 flag
# Use a backup extension like '.bak' for macOS sed.
sed -i '.bak' 's/extra_compile_args\.append('\''-std=c++14'\'')/#extra_compile_args.append('\''-std=c++14'\'') # Commented out by install script/g' setup.py

(   export PATH="$INSTALL_PROTOBUF_PATH/bin:$PATH" && \
    export CC="clang" && \
    export CXX="clang++" && \
    export CFLAGS="-std=c++17" && \
    export CXXFLAGS="-std=c++17" && \
    export LDFLAGS="-L$INSTALL_PROTOBUF_PATH/lib" &&\
    export CPPFLAGS="-I$INSTALL_PROTOBUF_PATH/include" &&\
    python3 setup.py install --cpp_implementation \
    build_ext --library-dirs="$INSTALL_PROTOBUF_PATH/lib" --include-dirs="$INSTALL_PROTOBUF_PATH/include")

mv setup.py.bak setup.py

# Update the main shell's PATH for subsequent commands like 'protoc --version'
export PATH="$INSTALL_PROTOBUF_PATH/bin:$INSTALL_PROTOBUF_PATH/include:$INSTALL_PROTOBUF_PATH/lib:$PATH"

protoc --version
echo "protobuf installed"

gongsu832 · 2025-08-13T20:55:35Z

It will fail. --cpp_implementation no longer exists in the new protobuf versions. The C/C++ implementation must be built with bazel.

…as it fails to build on s390x Signed-off-by: Sunny Anand <[email protected]>

Signed-off-by: Sunny Anand <[email protected]>

Sunny-Anand · 2025-08-26T21:18:51Z

Here is a complete breakdown of all the experiments I have run trying to find a solution for protobuf install such that the new protobuf upb c implementation can be used for the python bindings.

Before moving to the bazel approach, below is the approach I took to see if using the cmake approach I can get the python bindings to use the new c backend. This approach however proves that pip always now defaults to python implmenetation of the api and the only time pip install worked with upb installation was when a new protobuf was installed apart from the one installed by cmake.

RUN python3 -m pip install --no-binary :all: protobuf==6.31.1 # ----> this works as it compiles the c backend and upb is the default backend. However we have 2 separate protobuf instances.

Dockerfile1

FROM registry.access.redhat.com/ubi9/ubi

ARG PROTOBUF_VERSION=29.2
ARG INSTALL_PROTOBUF_PATH=/usr/local
ARG BUILD_TYPE=Release
ARG CORE_NUMBER=2

# Install dependencies
RUN yum install -y unzip wget git gcc-c++ make ninja-build python3 python3-pip cmake python3-devel

# Clone the Protobuf repo (with submodules)
RUN git clone --recurse-submodules -b v${PROTOBUF_VERSION} https://github.com/protocolbuffers/protobuf.git /protobuf-${PROTOBUF_VERSION}

RUN test -f /protobuf-${PROTOBUF_VERSION}/src/google/protobuf/descriptor.proto

# Build and install Protobuf C++ core
# RUN cd /protobuf-${PROTOBUF_VERSION} && \
#    mkdir build_source && cd build_source && \
#    cmake -G Ninja .. \
#        -DBUILD_SHARED_LIBS=OFF \
#        -DCMAKE_INSTALL_PREFIX=${INSTALL_PROTOBUF_PATH} \
#        -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
#        -Dprotobuf_BUILD_TESTS=OFF \
#        -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
#        -DCMAKE_CXX_STANDARD=17 \
#        -DABSL_PROPAGATE_CXX_STD=on && \
#    cmake --build . --target install --parallel ${CORE_NUMBER}

# Set environment variables for Protobuf
#ENV PATH=${INSTALL_PROTOBUF_PATH}/bin:$PATH
#ENV LD_LIBRARY_PATH=${INSTALL_PROTOBUF_PATH}/lib:$LD_LIBRARY_PATH
#ENV CPATH=${INSTALL_PROTOBUF_PATH}/include:$CPATH
#ENV LIBRARY_PATH=${INSTALL_PROTOBUF_PATH}/lib:$LIBRARY_PATH

# Set environment variables to enable UBP Python backend
#ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb
#ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=2

# Upgrade pip and install build backend
RUN python3 -m pip install --upgrade pip setuptools wheel build

# Build and install Protobuf Python bindings (UBP backend)
#Fails with error as it finds no upb backend --
#RUN cd /protobuf-${PROTOBUF_VERSION}/python && \
#    pip3 install .
#Approach 2 builds the python backend and not the cpp or upb backend
#RUN cd /protobuf-${PROTOBUF_VERSION}/python && \
#    python3 -m pip install --upgrade pip setuptools wheel build && \
#    python3 -m build --wheel && \
#    pip3 install dist/*.whl
# Approach 3 still fails to build upb backend and stills python backend
#RUN cd /protobuf-${PROTOBUF_VERSION}/python && \
#    python3 -m pip install --upgrade pip setuptools wheel && \
#    python3 setup.py bdist_wheel && \
#    pip3 install dist/*.whl

# Approach 4 tryong to see if Cmake may work but doesn't work as the python folder  has no CMAKelist in it.
# Build Python bindings with UPB backend using CMake
#RUN cd /protobuf-${PROTOBUF_VERSION} && \
#    mkdir build_python && cd build_python && \
#    cmake -G Ninja ../python \
#        -Dprotobuf_BUILD_PYTHON=True \
#        -DCMAKE_INSTALL_PREFIX=${INSTALL_PROTOBUF_PATH} \
#        -Dprotobuf_ENABLE_UPB=ON \
#        -Dprotobuf_PYTHON_INSTALL_PATH=$(python3 -c "import site; print(site.getsitepackages()[0])") && \
#    cmake --build . --target _pb_python && \
#    cmake --install .

# Upgrade pip and install tools
#RUN python3 -m pip install --upgrade pip setuptools wheel

# Set the runtime environment to use UPB backend
#ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb
#ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=2

# Final stage: install Python protobuf with UPB via official wheel
# The official wheel files don't have the cpython .so files hence falling back to the python imeplementation

#RUN python3 -m pip install --upgrade pip
#RUN python3 -m pip install protobuf==6.31.1

#Check if the latest pypi download have upb backend by default for it we need to install protoc compiler first for the latest s390
#ARG PROTOC_VERSION="31.1" # Adjust to the version you need
#ARG PLATFORM="linux-s390_64" # Adjust platform if needed for your environment

# Download and install protoc
#RUN PB_REL="https://github.com/protocolbuffers/protobuf/releases" && \
#    wget $PB_REL/download/v$PROTOC_VERSION/protoc-$PROTOC_VERSION-$PLATFORM.zip && \
#    unzip protoc-$PROTOC_VERSION-$PLATFORM.zip -d /usr/local && \
#    rm protoc-$PROTOC_VERSION-$PLATFORM.zip

# Add protoc to the PATH
#ENV PATH="$PATH:/usr/local/bin"

# Verify the protoc version
#RUN protoc --version

# Set environment variables *before* installing protobuf
ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb
ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=2
ENV PROTOCOL_BUFFERS_PYTHON_CAN_USE_C_IMPL=true

# Force pip to build the Python protobuf extensions from source
# This will attempt to compile the C extensions (including upb)
#RUN cd /protobuf-${PROTOBUF_VERSION}/python && \ # --> this doesn't work as the python dir has no setup and pyproject.toml
RUN python3 -m pip install --no-binary :all: protobuf==6.31.1 # ----> this works as it compiles the c backend and upb is the default backend. However we have 2 separate protobuf instances.

# Verify installation and UBP backend
#RUN protoc --version && \
RUN python3 -c "from google.protobuf.internal import api_implementation; \
    print('protobuf version:', api_implementation.Version()); \
    print('implementation type:', api_implementation.Type())" ;
    #print('implementation version:', api_implementation._IMPLEMENTATION_VERSION)" && \
    #ls -l /usr/local/lib/python3.9/site-packages/google/protobuf/pyext/
RUN python3 -c "import google.protobuf; import os; print(os.path.dirname(google.protobuf.__file__))"

RUN python3 -c "import google.protobuf; import os; print(os.path.dirname(google.protobuf.__file__))" && \
    ls -l $(python3 -c "import google.protobuf; import os; print(os.path.dirname(google.protobuf.__file__))")


RUN protoc --version && \
    ( python3 -c "import google.protobuf; \
    from google.protobuf.internal import api_implementation; \
    import os; \
    pkg_path = os.path.dirname(google.protobuf.__file__); \
    print('protobuf version:', api_implementation.Version()); \
    print('implementation type:', api_implementation.Type()); \
    print('Protobuf package path:', pkg_path); \
    print('Contents of pyext folder:')" && \
    (ls -l /usr/local/lib64/python3.9/site-packages/google/protobuf/pyext))# Use || true to prevent the build from failing if the directory is missing

https://github.com/protocolbuffers/protobuf/blob/main/src/README.md

After exhausting this approach I turned to building protobuf with bazel. I tried with bazel 6.5.0 and protobuf v25.1 and it ran into the below errors for both rhel and ubuntu on s390x where the s390x arch is not recognized.

Error from UBI image

[sunny@aqlinux2 bazel-build]$ docker build . -f Dockerfile-step-by-step-1 -t new-test
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/12: FROM registry.access.redhat.com/ubi9/ubi
STEP 2/12: ENV LANG=en_US.UTF-8     LC_ALL=en_US.UTF-8     PYTHONUNBUFFERED=1     BAZEL_VERSION=7.2.0     PROTOBUF_VERSION=v25.1     PROTOBUF_PREFIX=/usr/local/protobuf
--> Using cache 58c481a5a7a5a371846f3c793ffbc16a7ce2f1e2ec35e6c0c3d709693025e76c
--> 58c481a5a7a5
STEP 3/12: RUN dnf -y update &&     dnf install -y         git         gcc         gcc-c++         make         unzip         zip         wget         tar         python3         python3-pip         python3-devel         java-11-openjdk-devel         which         findutils         hostname         procps         java-11-openjdk-devel         file  &&     dnf clean all
--> Using cache 02292a1651f39cf7cee46e965563fdc8e78cc8d392a514ec191602137c48890e
--> 02292a1651f3
STEP 4/12: WORKDIR /opt
--> Using cache 9ed274811c4d94fdc8c907ec81e8651e44a90ef2bdde4983f46bedd5592fed7d
--> 9ed274811c4d
STEP 5/12: RUN  mkdir /usr/local/bazel && cd /usr/local/bazel &&   curl -LO https://github.com/bazelbuild/bazel/releases/download/6.5.0/bazel-6.5.0-dist.zip  &&   unzip bazel-6.5.0-dist.zip &&     chmod -R +w . &&     EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh && cp output/bazel /usr/local/bin/
--> Using cache db165bfcfea6d980905dcafae358b55fc13b056ef76690c887e22823e0e473e0
--> db165bfcfea6
STEP 6/12: WORKDIR /opt
--> Using cache e372c8b4ecef51441178ea8875bd35dc983472bd62a1e4018a8c70225e1f35c7
--> e372c8b4ecef
STEP 7/12: RUN git clone --depth 1 --branch ${PROTOBUF_VERSION} https://github.com/protocolbuffers/protobuf.git
--> Using cache 83f31f3a759912b2a50b4fd86da2e022ea68e78a0913b6480b1c3f5c90f9e379
--> 83f31f3a7599
STEP 8/12: WORKDIR /opt/protobuf
--> Using cache 3effbaa69874518ecc70c6a21fd92d7504b2b01682b4c7fbe81f4cb5ed20c352
--> 3effbaa69874
STEP 9/12: RUN /usr/local/bin/bazel build --enable_bzlmod :protoc :protobuf
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading:
Loading:
Loading:
Loading:
DEBUG: /root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_ruby/ruby/private/toolchains/ruby_runtime.bzl:297:14: WARNING: no system ruby available, builds against system ruby will fail
DEBUG: /root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/system_ruby/bundle.bzl:3:10: WARNING: no system ruby found for bundle
Loading:
INFO: Repository crate_index instantiated at:
  /opt/protobuf/WORKSPACE:188:18: in <toplevel>
Repository rule crates_repository defined at:
  /root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_rust/crate_universe/private/crates_repository.bzl:107:36: in <toplevel>
ERROR: An error occurred during the fetch of repository 'crate_index':
   Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_rust/crate_universe/private/crates_repository.bzl", line 25, column 34, in _crates_repository_impl
		host_triple = get_host_triple(repository_ctx)
	File "/root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_rust/rust/platform/triple.bzl", line 177, column 38, in get_host_triple
		cpu = _query_cpu_architecture(repository_ctx, supported_architectures["linux"])
	File "/root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_rust/rust/platform/triple.bzl", line 129, column 13, in _query_cpu_architecture
		fail("{} is not a expected cpu architecture {}\n{}".format(
Error in fail: s390x is not a expected cpu architecture ["aarch64", "x86_64"]
s390x
ERROR: /opt/protobuf/WORKSPACE:188:18: fetching crates_repository rule //external:crate_index: Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/59ee78e771eb6ceac2593b46fd0375fb/external/rules_rust/crate_universe/private/crates_repository.bzl", line 25, column 34, in _crates_repository_impl

Error from Ubuntu image:


[sunny@aqlinux2 bazel-build]$ docker build . -f Dockerfile-step-by-step-2 -t new-test
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/10: FROM ghcr.io/onnxmlir/ubuntu:jammy
STEP 2/10: RUN apt-get update &&     apt-get install -y build-essential openjdk-11-jdk python3-dev zip unzip gcc  wget g++ git vim
--> Using cache d90041e087d47844221d51b39f7b9372388eee3a3cb9ca44edeb351b719999f2
--> d90041e087d4
STEP 3/10: RUN mkdir /usr/local/bazel
--> Using cache 07a7a764b7dce609caf9f28dd19faa28e5e641eca3f36e46f2decd253c170d05
--> 07a7a764b7dc
STEP 4/10: RUN cd /usr/local/bazel &&  wget https://github.com/bazelbuild/bazel/releases/download/6.5.0/bazel-6.5.0-dist.zip                               --progress=dot:giga &&                               unzip bazel-6.5.0-dist.zip &&                               chmod -R +w . &&                               EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
--> Using cache 780a1229f528d767d391d6142b300aba2a0558eba878801d94f048752da96b91
--> 780a1229f528
STEP 5/10: RUN cp /usr/local/bazel/output/bazel /usr/local/bin/
--> Using cache 50af3107934fef3582ebda868a735faf3be8de839cf27a507c24f3b0b537c490
--> 50af3107934f
STEP 6/10: RUN bazel --version
--> Using cache ea0fd11dd95abc97fd55525505edc8bc4d0beedab0f11b827950be375dc040b3
--> ea0fd11dd95a
STEP 7/10: WORKDIR /workdir
--> Using cache 8ac3ff82c45e9753143a58853cd071186dedade1dfe33ffacb07289cc8ae68b3
--> 8ac3ff82c45e
STEP 8/10: RUN git clone --recursive https://github.com/protocolbuffers/protobuf.git
--> Using cache 0ce42e1032942b4c108a8c30e7d04badaca74ddb00adf0e0c0d6a220e8029f36
--> 0ce42e103294
STEP 9/10: RUN cd /workdir/protobuf && git checkout v25.1 && git submodule update --init --recursive
--> Using cache 2ab93b1aef277dc3345878d4c52e30bd790701f4ae74a653731cb7224868f4b0
--> 2ab93b1aef27
STEP 10/10: RUN cd /workdir/protobuf && bazel build :protoc :protobuf
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading:
Loading:
Loading:
Loading:
DEBUG: /root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_ruby/ruby/private/toolchains/ruby_runtime.bzl:297:14: WARNING: no system ruby available, builds against system ruby will fail
DEBUG: /root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/system_ruby/bundle.bzl:3:10: WARNING: no system ruby found for bundle
Loading:
INFO: Repository crate_index instantiated at:
  /workdir/protobuf/WORKSPACE:188:18: in <toplevel>
Repository rule crates_repository defined at:
  /root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/crate_universe/private/crates_repository.bzl:107:36: in <toplevel>
ERROR: An error occurred during the fetch of repository 'crate_index':
   Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/crate_universe/private/crates_repository.bzl", line 25, column 34, in _crates_repository_impl
		host_triple = get_host_triple(repository_ctx)
	File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/rust/platform/triple.bzl", line 177, column 38, in get_host_triple
		cpu = _query_cpu_architecture(repository_ctx, supported_architectures["linux"])
	File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/rust/platform/triple.bzl", line 129, column 13, in _query_cpu_architecture
		fail("{} is not a expected cpu architecture {}\n{}".format(
Error in fail: s390x is not a expected cpu architecture ["aarch64", "x86_64"]
s390x
ERROR: /workdir/protobuf/WORKSPACE:188:18: fetching crates_repository rule //external:crate_index: Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/crate_universe/private/crates_repository.bzl", line 25, column 34, in _crates_repository_impl
		host_triple = get_host_triple(repository_ctx)
	File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/rust/platform/triple.bzl", line 177, column 38, in get_host_triple
		cpu = _query_cpu_architecture(repository_ctx, supported_architectures["linux"])
	File "/root/.cache/bazel/_bazel_root/e2b3cd6608d3f99f8f812345194379e4/external/rules_rust/rust/platform/triple.bzl", line 129, column 13, in _query_cpu_architecture
		fail("{} is not a expected cpu architecture {}\n{}".format(
Error in fail: s390x is not a expected cpu architecture ["aarch64", "x86_64"]
s390x
ERROR: Error computing the main repository mapping: no such package '@crate_index//': s390x is not a expected cpu architecture ["aarch64", "x86_64"]
s390x
Error: building at STEP "RUN cd /workdir/protobuf && bazel build :protoc :protobuf": while running runtime: exit status 1

Now changing the Dockerfiles to point to the latest Bazel 8.3.0 and Protobuf 6.32.0 I run into different issue.

[sunny@aqlinux2 bazel-build]$ docker build . -f Dockerfile-step-by-step-2 -t new-test
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/10: FROM ghcr.io/onnxmlir/ubuntu:jammy
STEP 2/10: RUN apt-get update &&     apt-get install -y build-essential openjdk-21-jdk python3-dev zip unzip gcc  wget g++ git vim
--> Using cache 45e47f8e727fdf82fc0ca186ab01e5a18443e19bc6c49a7426fbf4d5d7998ca3
--> 45e47f8e727f
STEP 3/10: RUN mkdir /usr/local/bazel
--> Using cache 7ba1990ed1e9aa0c675b3b7a7b25ac1d0c8f7761c2df57c9468afe865953877c
--> 7ba1990ed1e9
STEP 4/10: RUN cd /usr/local/bazel &&  wget https://github.com/bazelbuild/bazel/releases/download/8.3.0/bazel-8.3.0-dist.zip                               --progress=dot:giga &&                               unzip bazel-8.3.0-dist.zip &&                               chmod -R +w . &&                               EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
--> Using cache e868b9bfce077b7cebdd25a020f3dfa39a31916bf62717cb2c403c354dcaf066
--> e868b9bfce07
STEP 5/10: RUN cp /usr/local/bazel/output/bazel /usr/local/bin/
--> Using cache 9cceedfbc113c88df369f3ea6f3fa007d5ecd67fbd5e3de946594245f75ea838
--> 9cceedfbc113
STEP 6/10: RUN bazel --version
--> Using cache 52d22b8cc3aafaebdad72b39ad36823fa74bc02678879baf92938f6a9af98d61
--> 52d22b8cc3aa
STEP 7/10: WORKDIR /workdir
--> Using cache 1cd23f23d9f85a3123f264f458513c29c1cd8e4a39c0519234596292ef9bccd9
--> 1cd23f23d9f8
STEP 8/10: RUN git clone --recursive https://github.com/protocolbuffers/protobuf.git
--> Using cache 56c22b6bebe6726acb63c3ae4fd37baf860f7343c7fdec85a82434d500c3c50e
--> 56c22b6bebe6
STEP 9/10: RUN cd /workdir/protobuf && git checkout v32.0 && git submodule update --init --recursive
--> Using cache 46fdeb884ca48645fc5e9239540cd01e40907df9be6dacbb2dec9da0fbfc98da
--> 46fdeb884ca4
STEP 10/10: RUN cd /workdir/protobuf && bazel build :protoc :protobuf
Extracting Bazel installation...
Starting local Bazel server (8.3.0- (@non-git)) and connecting to it...
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
WARNING: For repository 'rules_cc', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'proto_bazel_features', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_java', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_shell', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'googletest', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
Computing main repo mapping:
Loading:
Loading: 1 packages loaded
Loading: 1 packages loaded
    currently loading:
Analyzing: 2 targets (2 packages loaded, 0 targets configured)
Analyzing: 2 targets (2 packages loaded, 0 targets configured)

INFO: Repository rules_buf++buf+rules_buf_toolchains instantiated at:
  <builtin>: in <toplevel>
Repository rule buf_download_releases defined at:
  /root/.cache/bazel/_bazel_root/b7d198b8b7d10edd46c67287a46567ba/external/rules_buf+/buf/internal/toolchain.bzl:170:40: in <toplevel>
ERROR: /root/.cache/bazel/_bazel_root/b7d198b8b7d10edd46c67287a46567ba/external/rules_buf+/buf/internal/toolchain.bzl:119:13: An error occurred during the fetch of repository 'rules_buf++buf+rules_buf_toolchains':
   Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/b7d198b8b7d10edd46c67287a46567ba/external/rules_buf+/buf/internal/toolchain.bzl", line 119, column 13, in _buf_download_releases_impl
		fail("Unsupported operating system or cpu architecture ")
Error in fail: Unsupported operating system or cpu architecture
ERROR: Analysis of target '//:protoc' failed; build aborted: Unsupported operating system or cpu architecture
INFO: Elapsed time: 10.650s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED:
Error: building at STEP "RUN cd /workdir/protobuf && bazel build :protoc :protobuf": while running runtime: exit status 1

I have opened an issue for the above at protobuf: protocolbuffers/protobuf#23241

Dockerfiles used for above experiments

Dockerfile-step-by-step-2

ARG BASE_IMAGE="ghcr.io/onnxmlir/ubuntu:jammy"
FROM ${BASE_IMAGE}

RUN apt-get update && \
    apt-get install -y build-essential openjdk-21-jdk python3-dev zip unzip gcc  wget g++ git vim

RUN mkdir /usr/local/bazel
RUN cd /usr/local/bazel &&  wget https://github.com/bazelbuild/bazel/releases/download/8.3.0/bazel-8.3.0-dist.zip \
                              --progress=dot:giga && \
                              unzip bazel-8.3.0-dist.zip && \
                              chmod -R +w . && \
                              EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh

RUN cp /usr/local/bazel/output/bazel /usr/local/bin/
RUN bazel --version

WORKDIR /workdir
RUN git clone --recursive https://github.com/protocolbuffers/protobuf.git
RUN cd /workdir/protobuf && git checkout v32.0 && git submodule update --init --recursive
RUN cd /workdir/protobuf && bazel build :protoc :protobuf

Dockerfile-step-by-step-1

FROM registry.access.redhat.com/ubi9/ubi

# Environment setup
ENV LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8 \
    PYTHONUNBUFFERED=1 \
    BAZEL_VERSION=7.2.0 \
    PROTOBUF_VERSION=v25.1 \
    PROTOBUF_PREFIX=/usr/local/protobuf

# Install required tools and JDK
RUN dnf -y update && \
    dnf install -y \
        git \
        gcc \
        gcc-c++ \
        make \
        unzip \
        zip \
        wget \
        tar \
        python3 \
        python3-pip \
        python3-devel \
        java-11-openjdk-devel \
        which \
        findutils \
        hostname \
        procps \
        java-11-openjdk-devel \
        file  && \
    dnf clean all

# Install Bazel for s390x architecture
WORKDIR /opt

RUN  mkdir /usr/local/bazel && cd /usr/local/bazel &&   curl -LO https://github.com/bazelbuild/bazel/releases/download/6.5.0/bazel-6.5.0-dist.zip  &&   unzip bazel-6.5.0-dist.zip &&     chmod -R +w . &&     EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh && cp output/bazel /usr/local/bin/
# Clone protobuf repository
WORKDIR /opt
RUN git clone --depth 1 --branch ${PROTOBUF_VERSION} https://github.com/protocolbuffers/protobuf.git

WORKDIR /opt/protobuf

# Build Protobuf components
RUN /usr/local/bin/bazel build --enable_bzlmod :protoc :protobuf

# Install protoc and headers/libraries to the specified prefix
RUN mkdir -p ${PROTOBUF_PREFIX}/bin ${PROTOBUF_PREFIX}/lib ${PROTOBUF_PREFIX}/include && \
    cp -v bazel-bin/protoc ${PROTOBUF_PREFIX}/bin/protoc && \
    cp -v bazel-bin/libprotobuf.a ${PROTOBUF_PREFIX}/lib/ && \
    cp -rv src/google ${PROTOBUF_PREFIX}/include/

# Set environment variables so protoc and protobuf libraries are discoverable
ENV PATH="${PROTOBUF_PREFIX}/bin:${PATH}" \
    CPPFLAGS="-I${PROTOBUF_PREFIX}/include"

# Default command to run when the container starts
CMD ["/bin/bash"]

@gongsu832 @chentong319 let me know if you guys have any suggestions I can try here.
fyi @AlexandreEichenberger , this is going to become a blocker for future onnx upgrades as the min level for new onnx releases is 4.25.1

Sunny-Anand · 2025-09-05T15:58:35Z

When rules_buf is commented out in MODULE.bazel

#bazel_dep(name = "rules_buf", version = "0.3.0", dev_dependency = True)

the protobuf gets built at bazel-bin/protoc
However then trying the below the error gets generated for CARGO_BAZEL

root@77c414183d77:/usr/local/protobuf# bazel build --enable_workspace  --noenable_bzlmod //python/dist:binary_wheel
WARNING: WORKSPACE support will be removed in Bazel 9 (late 2025), please migrate to Bzlmod, see https://bazel.build/external/migration.
DEBUG: /root/.cache/bazel/_bazel_root/f1ce3c76545679fe06b35df989ced517/external/rules_java/java/repositories.bzl:367:10: DEPRECATED: use rules_java_dependencies() from rules_java_deps.bzl
INFO: Repository crate_index instantiated at:
  /usr/local/protobuf/WORKSPACE:267:18: in <toplevel>
Repository rule crates_repository defined at:
  /root/.cache/bazel/_bazel_root/f1ce3c76545679fe06b35df989ced517/external/rules_rust/crate_universe/private/crates_repository.bzl:124:36: in <toplevel>
ERROR: /root/.cache/bazel/_bazel_root/f1ce3c76545679fe06b35df989ced517/external/rules_rust/crate_universe/private/generate_utils.bzl:60:13: An error occurred during the fetch of repository 'crate_index':
   Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/f1ce3c76545679fe06b35df989ced517/external/rules_rust/crate_universe/private/crates_repository.bzl", line 28, column 48, in _crates_repository_impl
                generator, generator_sha256 = get_generator(repository_ctx, host_triple.str)
        File "/root/.cache/bazel/_bazel_root/f1ce3c76545679fe06b35df989ced517/external/rules_rust/crate_universe/private/generate_utils.bzl", line 60, column 13, in get_generator
                fail((
Error in fail: No generator URL was found either in the `CARGO_BAZEL_GENERATOR_URL` environment variable or for the `s390x-unknown-linux-gnu` triple in the `generator_urls` attribute
ERROR: Error computing the main repository mapping: no such package '@@crate_index//': No generator URL was found either in the `CARGO_BAZEL_GENERATOR_URL` environment variable or for the `s390x-unknown-linux-gnu` triple in the `generator_urls` attribute
Computing main repo mapping:

Sunny-Anand and others added 28 commits August 6, 2025 22:00

protobuf upgrade

6a7d306

Signed-off-by: Sunny Anand <[email protected]>

fix git error for macos

e745797

Signed-off-by: Sunny Anand <[email protected]>

updates

2bc50fa

Signed-off-by: Sunny Anand <[email protected]>

updates

c4a7ae1

Signed-off-by: Sunny Anand <[email protected]>

updates

f32f22a

Signed-off-by: Sunny Anand <[email protected]>

updates

bae4408

Signed-off-by: Sunny Anand <[email protected]>

updates osx

a1d9a3a

Signed-off-by: Sunny Anand <[email protected]>

updates osx

7efe63f

Signed-off-by: Sunny Anand <[email protected]>

updates osx

a3d4701

Signed-off-by: Sunny Anand <[email protected]>

updates osx

c69a66f

Signed-off-by: Sunny Anand <[email protected]>

updates osx

f0c3c9b

Signed-off-by: Sunny Anand <[email protected]>

updates osx

a269c63

Signed-off-by: Sunny Anand <[email protected]>

updates osx

f88f3bd

Signed-off-by: Sunny Anand <[email protected]>

updates osx

ab79c14

Signed-off-by: Sunny Anand <[email protected]>

updates osx

c1b6106

Signed-off-by: Sunny Anand <[email protected]>

updates osx

c543bae

Signed-off-by: Sunny Anand <[email protected]>

updates osx

00b384e

Signed-off-by: Sunny Anand <[email protected]>

updates osx

90e06b4

Signed-off-by: Sunny Anand <[email protected]>

updates osx

6b93539

Signed-off-by: Sunny Anand <[email protected]>

updates osx

b12664a

Signed-off-by: Sunny Anand <[email protected]>

updates osx

c1bc4ac

Signed-off-by: Sunny Anand <[email protected]>

updates osx

2594aeb

Signed-off-by: Sunny Anand <[email protected]>

updates osx

28d335f

Signed-off-by: Sunny Anand <[email protected]>

Merge branch 'main' into protobuf-update

0e55aa6

add cppcheck suppression

6750e53

Signed-off-by: Sunny Anand <[email protected]>

update s390x

7a2474b

Signed-off-by: Sunny Anand <[email protected]>

update s390x

f87d82d

Signed-off-by: Sunny Anand <[email protected]>

update s390x

92bdbce

Signed-off-by: Sunny Anand <[email protected]>

chentong319 reviewed Aug 12, 2025

View reviewed changes

updates

9beeb47

Signed-off-by: Sunny Anand <[email protected]>

updates

137e940

Signed-off-by: Sunny Anand <[email protected]>

Sunny-Anand requested a review from gongsu832 August 13, 2025 18:23

Merge branch 'main' into protobuf-update

0394448

Merge branch 'main' into protobuf-update

d298915

Sunny-Anand and others added 5 commits August 14, 2025 00:36

revert to using cpp implementation using whl files - can't use bazel …

c972e44

…as it fails to build on s390x Signed-off-by: Sunny Anand <[email protected]>

comment the install

e08b36c

Signed-off-by: Sunny Anand <[email protected]>

update the clone path

577f5bf

Signed-off-by: Sunny Anand <[email protected]>

update the clone path

59518c5

Signed-off-by: Sunny Anand <[email protected]>

Merge branch 'main' into protobuf-update

8774fe6

Merge branch 'main' into protobuf-update

e296fd6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

protobuf upgrade #3255

protobuf upgrade #3255

Uh oh!

Sunny-Anand commented Aug 7, 2025 •

edited

Loading

Uh oh!

chentong319 Aug 12, 2025

Uh oh!

Sunny-Anand Aug 13, 2025

Uh oh!

Sunny-Anand commented Aug 13, 2025

Uh oh!

gongsu832 commented Aug 13, 2025

Uh oh!

Sunny-Anand commented Aug 13, 2025 •

edited

Loading

Uh oh!

gongsu832 commented Aug 13, 2025 •

edited

Loading

Uh oh!

Sunny-Anand commented Aug 26, 2025 •

edited

Loading

Uh oh!

Sunny-Anand commented Sep 5, 2025

Uh oh!

Uh oh!

protobuf upgrade #3255

Are you sure you want to change the base?

protobuf upgrade #3255

Uh oh!

Conversation

Sunny-Anand commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chentong319 Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Sunny-Anand Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Sunny-Anand commented Aug 13, 2025

Uh oh!

gongsu832 commented Aug 13, 2025

Uh oh!

Sunny-Anand commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gongsu832 commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sunny-Anand commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sunny-Anand commented Sep 5, 2025

Uh oh!

Uh oh!

Sunny-Anand commented Aug 7, 2025 •

edited

Loading

Sunny-Anand commented Aug 13, 2025 •

edited

Loading

gongsu832 commented Aug 13, 2025 •

edited

Loading

Sunny-Anand commented Aug 26, 2025 •

edited

Loading