Skip to content

[CI] Fix Failing Build Since CI updates #1374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
3 tasks
nedvedba opened this issue Apr 14, 2025 · 2 comments · May be fixed by #1390
Open
3 tasks

[CI] Fix Failing Build Since CI updates #1374

nedvedba opened this issue Apr 14, 2025 · 2 comments · May be fixed by #1390
Assignees
@nedvedba
Labels
Component: CI Priority: High Highest priority
Milestone
OpenShift Compati...

Comments

@nedvedba
Copy link
Collaborator

nedvedba commented Apr 14, 2025 •
edited by JoshuaSBrown
Loading

Description

Currently the CI job is failing that is responsible for provisioning the DataFed Python Client VM. The issue with the CI is that the protobuf build does not support Python 3.12 and its deprecation of the pkg_resources package as of yet. I have tried bumping the versions of the protobuf and setuptools python libraries, all leading to the same issue. The current plan is to setup a Python 3.11 virtual environment that can be used by the install and build scripts, until a more permanent solution can be decided.

Seeing the problem with both latest version of protobuf>=6.30.2, and with currently running version 5.27.1.

Failing job shown here: https://code.ornl.gov/dlsw/datafed/datafed/-/jobs/3178328

Installing: /shared/install/lib/cmake/protobuf/protobuf-generate.cmake
Requirement already satisfied: numpy in /shared/install/python/datafed/lib/python3.12/site-packages (2.2.4)
Traceback (most recent call last):
  File "/home/gitlab-runner/builds/gYxDkX87B/1/dlsw/datafed/datafed/external/protobuf/python/setup.py", line 16, in <module>
    import pkg_resources
  File "/shared/install/python/datafed/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit status 1

Here is the relevant MR for the CI repository on installing a separate version of Python.
https://code.ornl.gov/dlsw/datafed/ci/-/merge_requests/124

Acceptance

CI runs without failing after upgrading to ubuntu 24.

  • Pin Python version to 3.11
  • Get client provisioning job to pass
  • Get dependency containers to build correctly with python 3.11
@nedvedba nedvedba self-assigned this Apr 14, 2025
@JoshuaSBrown
Copy link
Collaborator

Here is where it is failing.

DataFed/scripts/dependency_install_functions.sh

Line 153 in 403636d

LD_LIBRARY_PATH="$LD_LIBRARY_PATH" PATH="$PATH" python3 setup.py build

DataFed/scripts/dependency_install_functions.sh at 403636d · ORNL/DataFed
A Federated Scientific Data Management System. Contribute to ORNL/DataFed development by creating an account on GitHub.

It's because the protobuf requriements.txt file does not include the needed module. I would recommend just adding a line the dependency install script that captures installation of the needed python dependencies as part of the protobuf_install. Or however else you want to fix this. If you want to fix it from the ansible side you will need to add a new role that installs the dependency as part of the setup process for the DataFedPythonClient playbook.

@JoshuaSBrown JoshuaSBrown linked a pull request Apr 23, 2025 that will close this issue
Fix ci build #1390
Open
7 tasks
@JoshuaSBrown JoshuaSBrown added the Priority: High Highest priority label Apr 23, 2025
@nedvedba
Copy link
Collaborator Author

Some developments have been made, and the GitLab CI was passing however the GitHub CI was not, I believe that after further research protobuf only supports Python 3.9 as the highest version and not 3.11 which will still allow protobuf to work but will not allow the tests to pass. To get the GitHub CI to work I have updated the python version that is installed to 3.9, along with setting it up to pull from the scripts/dependency_versions.sh file. We will see if this works when the CI finishes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nedvedba nedvedba

Labels
Component: CI Priority: High Highest priority
Projects
None yet
Milestone
OpenShift Compatible containers
Development

Successfully merging a pull request may close this issue.

Fix ci build ORNL/DataFed
2 participants
@JoshuaSBrown @nedvedba