[ci] [python-package] [R-package] support MUSL-based Linux distributions (like Alpine)

[jameslamb]

Summary

Opening this issue to track the idea of adding support for MUSL-based Linux distributions (particularly Alpine Linux) here.

Motivation

Alpine Linux is a really popular options for containerized environments that place high importance on small image sizes (e.g. for reduced disk usage and fast spin-up). Especially helpful for serving LightGBM models.

For example, the latest Alpine-based official Python 3.13 container image (python:3.13-alpine2.21) is about 16MB compressed, more than 2x smaller than the comparable Debian-based image (python:3.13-slim-bullseye).

For more on the benefits of MUSL-based distributions, see https://en.wikipedia.org/wiki/Musl.

Today, if you try to install lightgbm's Python package in an official Python Alpine image, PyPI will serve back the sdist and try to compile it from source.

docker run \
  --rm \
  -it python:3.12-alpine3.21 \
  sh

pip install --no-deps 'lightgbm==4.6.0'

That fails like this:

CMake Error at /tmp/pip-build-env-6drbwikx/normal/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CMakeDetermineCCompiler.cmake:49 (message):
Could not find compiler set in environment variable CC:
  gcc.
Call Stack (most recent call first):
CMakeLists.txt:36 (project)

CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

logs (click me)

Collecting lightgbm==4.6.0
  Using cached lightgbm-4.6.0.tar.gz (1.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: lightgbm
  Building wheel for lightgbm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for lightgbm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      2025-02-15 07:29:28,311 - scikit_build_core - INFO - RUN: /tmp/pip-build-env-6drbwikx/normal/lib/python3.12/site-packages/cmake/data/bin/cmake -E capabilities
      2025-02-15 07:29:28,313 - scikit_build_core - INFO - CMake version: 3.31.4
      *** scikit-build-core 0.10.7 using CMake 3.31.4 (wheel)
      2025-02-15 07:29:28,315 - scikit_build_core - INFO - Build directory: /tmp/tmp5_9pg2uz/build
      *** Configuring CMake...
      2025-02-15 07:29:28,319 - scikit_build_core - INFO - RUN: ninja --version
      2025-02-15 07:29:28,319 - scikit_build_core - INFO - Ninja version: 1.11.1
      2025-02-15 07:29:28,320 - scikit_build_core - INFO - RUN: /tmp/pip-build-env-6drbwikx/normal/lib/python3.12/site-packages/cmake/data/bin/cmake -S. -B/tmp/tmp5_9pg2uz/build -DCMAKE_BUILD_TYPE:STRING=Release -C/tmp/tmp5_9pg2uz/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/tmp/tmp5_9pg2uz/wheel/platlib -DCMAKE_MAKE_PROGRAM=ninja -D__BUILD_FOR_PYTHON:BOOL=ON
      loading initial cache file /tmp/tmp5_9pg2uz/build/CMakeInit.txt
      CMake Error at /tmp/pip-build-env-6drbwikx/normal/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CMakeDetermineCCompiler.cmake:49 (message):
        Could not find compiler set in environment variable CC:

        gcc.
      Call Stack (most recent call first):
        CMakeLists.txt:36 (project)


      CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
      CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
      -- Configuring incomplete, errors occurred!

      *** CMake configuration failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lightgbm
Failed to build lightgbm

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (lightgbm)

Providing Python wheels for that platform would allow users to install lightgbm on distributions like that without needing a build toolchain.

Description

At a minimum, closing this issue should involve the following:

• add 1 CI job building lightgbm Python wheels (x86_64) on Alpine Linux
• add 1 CI job building {lightgbm} R package on Alpine Linux
• distribute *_musllinux_* wheel(s) on PyPI

Nice to have:

• also building aarch64 Python wheels

challenge: support for dependencies

Not all dependencies support MUSL-based distributions... for example, scipy has only x86_64 musllinux wheels (https://pypi.org/project/scipy/1.15.1/#files) and pyarrow does not have any (https://pypi.org/project/pyarrow/19.0.0/#files).

Similarly, I expect that not all R packages {lightgbm} depends on will build successfully on such distributions (for reasons like @bastistician mentioned in #6743 (comment)).

So testing might be more limited, for example "compiles successfully and a small smoke test runs without error".

References

• Installation of Python lib on Alpine Linux #2614
• [Python] Crashes and/or segfaults in Alpine Docker containers #1227
• [R] compilation failure with missing libexecinfo on Alpine Linux 3.17 dmlc/xgboost#8595
• Build failure on musl (CMAKE) dmlc/xgboost#5510
• Catboost installation on python 3.12 Alpine Docker image issue  catboost/catboost#2702
• Alpine linux install instructions catboost/catboost#1991

[bastistician]

Similarly, I expect that not all R packages {lightgbm} depends on will build successfully on such distributions (for reasons like @bastistician mentioned in #6743 (comment)).

The newly released lightgbm 4.6.0 from CRAN installs and checks fine on my Alpine Linux server. (Last night's run tells me we are down to only 31 failing compilations from a total of 22021 CRAN packages.)

To test interactively:
In docker run --rm -it alpine:latest:

apk add R R-dev R-doc g++
R -s -e 'install.packages("lightgbm", repos = "https://cloud.R-project.org/")'

Or using docker run --rm -it rhub/r-minimal /bin/bash (from https://github.com/r-hub/r-minimal):

installr -d -t gfortran lightgbm

[jameslamb]

Last night's run tells me we are down to only 31 failing compilations from a total of 22021 CRAN packages.

Oh wow, amazing,thank you! As part of this issue, we will try to keep that support working in lightgbm.