Install Python 3.10 on all nodes

[mauritsvanrees]

Zope is almost compatible with Python 3.10. See comment, although I don't know if all individual packages are tested yet. But this means we could start testing how well Plone does on 3.10.

There are at least two things that I can think of that first need to happen:

• Make Python 3.10 available on the nodes.
• Create a normal job and a PR job for Plone 6.0 with Python 3.10. For now, they should only be started manually. We don't want to test 3.10 for all PRs and commits yet.

Anything else?

I have no idea how much work it is to fix Plone for 3.10. I have not tried anything yet.
If it is easy, it would be interesting to officially support 3.10 when 6.0 final comes out. But currently it would be fine to support 3.9 as highest Python. So no pressure.

[gforcada]

I will try to work on that during the weekend, it should mostly take time as it compiles python to get it to run faster.

[gforcada]

I was worse than I expected during the weekend, I got my second vaccination, and now that I finally I'm human again, I'm already starting the process: python 3.10 is already added on https://github.com/gforcada/ansible-compile-python and jenkins nodes are starting to compile python :)

[gforcada]

I'm rather surprised that it went so fast, but 3.10 is already on the nodes 🎉 I will create a jenkins job and set it to run 🤞🏾

[gforcada]

😱 it fails not on any plone code but rather at trying to create a virtual environment for it to run on 😕 the blame should go on shiningpanda jenkins plugin... we used this plugin specially because it was handling the nice thing to have a separate virtual environment for every job running, so we could install dependencies etc etc without us having to manually maintain the virtual environment...

We might have to switch back to it, at least for 3.10 onwards 😕

[gforcada]

I created an upstream ticket: https://issues.jenkins.io/browse/JENKINS-67305

[mauritsvanrees]

Thanks for your effort so far, Gil!

[fredvd]

@gforcada When I searched on that last error line I found the following issue on pip's repo: pypa/pip#10253

Apparently that traceback was solved in pip 21.2.3. . Which version of pip is being used to create the virtualenv in the plugin is that configurable? Maybe we need to install that updated pip first in the global python 3.10, I'm not sure where virtualenv or 'python3.10 -mvenv` get their pip from that they install into the venv.

[gforcada]

@fredvd thanks for looking that up, I noticed though that this is coming from the bootstrap script from the shiningpanda plugin itself: https://github.com/jenkinsci/shiningpanda-plugin/blob/f4b5d326b6cec5ef09edc54e303b2b30aa79277b/src/main/resources/jenkins/plugins/shiningpanda/workspace/virtualenv.py#L1521

I see that it always downloads and install pip/setuptools/wheel with the following versions:
wheel: '0.31.1'
setuptools: '39.1.0'
pip: '10.0.1'

[fredvd]

Ouch, those are terribly old versions. I was looking at the python file to see any version numbers, but the 3 wheels are in the repo directory as raw archives with a 'zero' version. Off course in 2018 Python 2 was still much more in use.

I'm amazed that previous Python versions worked at all... But then we don't really use pip extensively when building buildout 2. (allthough I found out this month resolving sub-dependencies of zope.packages does insert wheels in my local pip cache, even with zc.buildout 2... different story)

I don't think shiningpandas is still very active as a company / saas service. Their last blog / updated item on the website is from 2018 :-O

There are quite a bit of forks on the jenkins plugin repo, but these also seem to be 3> years old. So we might have to squeeze and bend the plugin ourselves to get it working with 3.10

[gforcada]

If you look at the usage: https://plugins.jenkins.io/ui/search?query=python the python one, either is strong solid as it can get, because it did not had a release for the last 6 years, but at the same time it is used more than 3 times the shiningpanda one.

So I guess we will have to turn all our shiningpanda wrapped scripts into plain scripts and manage the virtualenvs ourselves 🤷🏾

At least, if we only do it for the python3 jobs, it is not that cumbersome to do anyway, but still more code to write and maintain 😑

[mauritsvanrees]

Latest run on Jenkins is green. Yeehah!

Is there anything left to do here? Python 3.10 seems to be installed everywhere, at least on the active nodes.

There is some followup that we could do, but that can be a new issue:

• Create a PR job for 3.10.
• Split the current 3.10 job into Python tests and Robot tests. (The current one only runs bin/test, so no robot tests.)
• Setup mr.roboto to start these jobs.

[gforcada]

Sounds like a plan, I don't have much time, but I will want to get out of my tasks every now and then, so I will try my best to set up all this, not sure when though, I'm fine reviewing changes from others though 👍🏾 😃

[fredvd]

@gforcada . @mauritsvanrees and me just tuned some settings on robotframework tests and percentage of failed tests to be counted as unstable and we added Python3.10 Plone 6 robotframework tests.

We get the same error as we had in december though as reported in this issue with shiningpandas and the pip 21.x issue .

But if I remember correctly you did change something later in the scripts to fix this, or? The Normal Python3.10 Plone 6.0 was running fine, until I updates the jobs from this repo. Was something changed maybe manually on jenkins in between?

[edit] Oh, apparently now only the Python 3.10 jobs are not working correctly. I first thought the Python 3.9 jobs for Plone 6.0 were also broken after I pushed the jobs from this repo, but that is not the case.

I'll split the Python 3.10 / Plone 6 jobs into a new job template and try to set that up using the normal Python jenkins plugins and script and not shiningpandas, as discussed above.

[gforcada]

Sorry to come back to this one this late 😅

I, manually, replaced the scripts on the 2 jenkins jobs for Plone 6.0 on python 3.10 and they are running as I type:

https://jenkins.plone.org/job/plone-6.0-python-3.10-robot-chrome/315/
https://jenkins.plone.org/job/plone-6.0-python-3.10/672/

So far they are installing dependencies, which is much more than what it used to do until then 😃

[petschki]

Seems that the jobs are green now. Should we enable py3.10 for PRs also? or too early...