[ci] Drop support for Python 3.7 and Python 3.8

[borchero]

Motivation

In our CI, we are still testing for old Python versions, namely Python 3.7 and Python 3.8. However:

• Python 3.7 and Python 3.8 have both reached end of life (see https://devguide.python.org/versions/)
• Almost no other package still supports Python 3.7 and 3.8 (XGBoost v3.0.0 released a couple weeks ago has even dropped support for Python 3.9)
• conda-forge does not build packages for Python <3.9 anymore

I'd argue that we can simplify our CI by dropping support for 3.7 and 3.8 and, instead, test on 3.9 the new "oldest supported Python version".

Changes

• Adjust relevant CI jobs to run for Python 3.9 instead of 3.8 (and 3.7)

[borchero]

@jameslamb the AppVeyor CI fails to solve the environment. However, I can't replicate this behavior locally (even for Windows) -- any idea how to debug this?

[jameslamb]

the AppVeyor CI fails to solve the environment.

Links and a bit of plaintext logs would be helpful, to narrow it down. "fails to solve the environment" could take many different forms.

I just looked in the history of Appveyor runs
and found this one: https://ci.appveyor.com/project/guolinke/lightgbm/builds/51955916

It looks like the solve setting up the environment succeeded as did all of the Python unit tests.

...
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate test-env
...
==== 878 passed, 33 skipped, 4 xfailed, 100 warnings in 89.26s (0:01:29) =====

I guess you probably mean this later solve, after the tests:

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... 

Since it's after the tests, I guess it must be this one:

LightGBM/.ci/test-windows.ps1

Line 146 in ec5492f

conda install -y -n $env:CONDA_ENV "h5py>=3.10" "ipywidgets>=8.1.2" "notebook>=7.1.2"

any idea how to debug this?

Do this initial conda env creation locally on windows:

LightGBM/.ci/test-windows.ps1

Lines 79 to 85 in ec5492f

	$condaParams = @(
	"-y",
	"-n", "$env:CONDA_ENV",
	"--file", "$env:CONDA_REQUIREMENT_FILE",
	"python=$env:PYTHON_VERSION[build=*_cp*]"
	)
	conda create @condaParams ; Assert-Output $?

Run conda env export --name test-env and compare the package versions to what you see in the Appveyor logs.

If there are differences in any package versions, then that means there's something different between your local environment and CI, and those differences are the result of the different behavior. The relevant difference will probably exist in the output of these commands:

conda config --get
conda info
conda --version

If all of the versions and conda info stuff are identical, then you could start investigating other, less likely, root causes like available memory / disk, network bandwidth, environment variables, etc.

[borchero]

Thanks for the hint @jameslamb, tornado caused some issues :)

[jameslamb]

Thanks for this! I left one small suggestion for your consideration.

Overall, I'm supporting of changing the Python support policy in this project to Python version support is dropped when a version of Python reaches end-of-life, according to https://devguide.python.org/versions/.

That'd be more aggressive than we've been in the past, but I think it's defensible given that many of lightgbm's dependencies are even more aggressive (SPEC 0) and so is conda-forge (https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-python.html#managing-python).

But let's not merge this until @StrikerRUS has a chance to comment as well, I'd like to hear his thoughts.

---

Also... @borchero , I know it's been a little while since your last PR here. As a reminder... every PR in this repo needs to have a label from the set of release notes sections:

LightGBM/.github/release-drafter.yml

Lines 3 to 15 in ec5492f

	categories:
	- title: '💡 New Features'
	label: 'feature'
	- title: '🔨 Breaking'
	label: 'breaking'
	- title: '🚀 Efficiency Improvement'
	label: 'efficiency'
	- title: '🐛 Bug Fixes'
	label: 'fix'
	- title: '📖 Documentation'
	label: 'doc'
	- title: '🧰 Maintenance'
	label: 'maintenance'

Please add them on all of the PRs you've opened. This one would be breaking, similar to when we dropped Python 3.6: #6459

[jameslamb]

I'd support updating the minimum for dask in the [dask] extra to >=2022.12.0 to match this.

Here:

LightGBM/python-package/pyproject.toml

Line 42 in ec5492f

"dask[array,dataframe,distributed]>=2.0.0",

That's 2+ years old at this point, which is a LOT of time given the speed of development in Dask.

[jameslamb]

I'll do this in a follow-up PR, let's not grow the size of this PR too much more.

[StrikerRUS]

I don't mind to change the Python support policy. I believe this change will be understandable from the community point of view given our small team size.

Also, nowadays there are a lot of [cool] environment managing tools besides conda that extremely simplify the process of creating new envs. So for newcomers it should be a problem. And for old users, as @jameslamb already mentioned, they are locked with old package versions including the deps.

P.S. Another example of harsh policy for version retirement: #6520.

[StrikerRUS]

LGTM!

[jameslamb]

It's been 3 months without a response from @borchero (who mentioned to me offline that he has not had much time for LightGBM recently), so I took this over.

I just pushed the following changes:

• merged in latest master
• removed all the unnecessary quoting changes ([ci] Drop support for Python 3.7 and Python 3.8 #6896 (comment))
• added a short section to the Python docs documenting the new Python support policy
• clarified a comment about dask / distributed versions

Since I pushed changes here, my review shouldn't count towards a merge.

@StrikerRUS if you don't mind, could you please re-review?

dismissing because now I've pushed changes here

[StrikerRUS]

@jameslamb Thanks a lot for picking this up!