pip install clickhouse-connect causes flow to fail in io.kestra.plugin.scripts.python.Script

[vndv]

Describe the issue

When attempting to install the clickhouse-connect package using pip install clickhouse-connect within a io.kestra.plugin.scripts.python.Script, the flow fails with an error. The issue occurs during the execution of the script, and the error message is not clear enough to diagnose the root cause.

namespace: company.team

tasks:
  - id: working_dir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: python_script
        type: io.kestra.plugin.scripts.python.Script
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        warningOnStdErr: false
        beforeCommands:
          - python -m venv venv
          - . venv/bin/activate
          - pip install clickhouse-connect
        script: |
          print('Hello')
    cache:
      patterns:
        - venv/**
      ttl: PT24H

Expected Behavior:

The clickhouse-connect package should be installed successfully, and the flow should continue execution without errors.

Actual Behavior:

The flow fails with an error during the installation of the clickhouse-connect package.

Error Message:

2025-02-06T12:28:30.910Z DEBUG Using task runner 'io.kestra.plugin.core.runner.Process'
2025-02-06T12:28:30.910Z TRACE Provided 0 input(s).
2025-02-06T12:28:30.912Z TRACE Provided 1 input(s).
2025-02-06T12:28:30.913Z DEBUG Starting command with pid 1524 [/bin/sh -c set -e
python -m venv venv
. venv/bin/activate
pip install clickhouse-connect
python /tmp/kestra-wd/tmp/3HKhPWqtuePnIyhuXoSLuk/11688218610340038108.py]
2025-02-06T12:28:32.291Z INFO   Using cached clickhouse_connect-0.8.15-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.4 kB)
2025-02-06T12:28:32.291Z INFO Collecting clickhouse-connect
2025-02-06T12:28:32.293Z INFO Requirement already satisfied: urllib3>=1.26 in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2.3.0)
2025-02-06T12:28:32.293Z INFO Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2025.1.31)
2025-02-06T12:28:32.294Z INFO Requirement already satisfied: pytz in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2025.1)
2025-02-06T12:28:32.367Z INFO Collecting zstandard (from clickhouse-connect)
2025-02-06T12:28:32.368Z INFO   Using cached zstandard-0.23.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.0 kB)
2025-02-06T12:28:32.449Z INFO Collecting lz4 (from clickhouse-connect)
2025-02-06T12:28:32.450Z INFO   Using cached lz4-4.4.3.tar.gz (171 kB)
2025-02-06T12:28:32.465Z INFO   Installing build dependencies: started
2025-02-06T12:28:33.724Z INFO   Installing build dependencies: finished with status 'done'
2025-02-06T12:28:33.725Z INFO   Getting requirements to build wheel: started
2025-02-06T12:28:33.794Z INFO   Getting requirements to build wheel: finished with status 'done'
2025-02-06T12:28:33.795Z INFO   Preparing metadata (pyproject.toml): started
2025-02-06T12:28:33.880Z INFO   Preparing metadata (pyproject.toml): finished with status 'done'
2025-02-06T12:28:33.884Z INFO Using cached clickhouse_connect-0.8.15-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (963 kB)
2025-02-06T12:28:33.886Z INFO Using cached zstandard-0.23.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.9 MB)
2025-02-06T12:28:33.896Z INFO   Building wheel for lz4 (pyproject.toml): started
2025-02-06T12:28:33.896Z INFO Building wheels for collected packages: lz4
2025-02-06T12:28:33.978Z INFO   Building wheel for lz4 (pyproject.toml): finished with status 'error'
2025-02-06T12:28:33.980Z ERROR   │ exit code: 1
2025-02-06T12:28:33.980Z ERROR   error: subprocess-exited-with-error
2025-02-06T12:28:33.980Z ERROR   × Building wheel for lz4 (pyproject.toml) did not run successfully.
2025-02-06T12:28:33.980Z ERROR   ╰─> [18 lines of output]
2025-02-06T12:28:33.980Z ERROR   
2025-02-06T12:28:33.980Z ERROR       /tmp/pip-build-env-7xs4xhjb/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'tests_require'
2025-02-06T12:28:33.981Z ERROR       running build_py
2025-02-06T12:28:33.981Z ERROR         warnings.warn(msg)
2025-02-06T12:28:33.981Z ERROR       running bdist_wheel
2025-02-06T12:28:33.981Z ERROR       creating build/lib.linux-aarch64-cpython-310/lz4
2025-02-06T12:28:33.981Z ERROR       copying lz4/version.py -> build/lib.linux-aarch64-cpython-310/lz4
2025-02-06T12:28:33.981Z ERROR       building 'lz4._version' extension
2025-02-06T12:28:33.981Z ERROR       running build
2025-02-06T12:28:33.981Z ERROR       creating build/lib.linux-aarch64-cpython-310/lz4/block
2025-02-06T12:28:33.981Z ERROR       creating build/lib.linux-aarch64-cpython-310/lz4/frame
2025-02-06T12:28:33.981Z ERROR       creating build/temp.linux-aarch64-cpython-310/lz4
2025-02-06T12:28:33.981Z ERROR       [end of output]
2025-02-06T12:28:33.981Z ERROR       error: command 'aarch64-linux-gnu-gcc' failed: No such file or directory
2025-02-06T12:28:33.981Z ERROR   
2025-02-06T12:28:33.981Z ERROR   ERROR: Failed building wheel for lz4
2025-02-06T12:28:33.981Z ERROR ERROR: Failed to build installable wheels for some pyproject.toml based projects (lz4)
2025-02-06T12:28:33.981Z ERROR   note: This error originates from a subprocess, and is likely not a problem with pip.
2025-02-06T12:28:33.981Z ERROR       creating build/temp.linux-aarch64-cpython-310/lz4libs
2025-02-06T12:28:33.981Z ERROR       copying lz4/frame/__init__.py -> build/lib.linux-aarch64-cpython-310/lz4/frame
2025-02-06T12:28:33.981Z INFO Failed to build lz4
2025-02-06T12:28:33.981Z ERROR       copying lz4/__init__.py -> build/lib.linux-aarch64-cpython-310/lz4
2025-02-06T12:28:33.981Z ERROR       copying lz4/block/__init__.py -> build/lib.linux-aarch64-cpython-310/lz4/block
2025-02-06T12:28:33.981Z ERROR       running build_ext
2025-02-06T12:28:33.981Z ERROR       aarch64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ilz4libs -I/usr/include/python3.10 -c lz4/_version.c -o build/temp.linux-aarch64-cpython-310/lz4/_version.o -O3 -Wall -Wundef
2025-02-06T12:28:34.022Z TRACE Captured 0 output(s).
2025-02-06T12:28:34.022Z ERROR io.kestra.core.models.tasks.runners.TaskException: Command failed with exit code 1
2025-02-06T12:28:34.022Z ERROR Command failed with exit code 1
2025-02-06T12:28:34.022Z TRACE io.kestra.core.models.tasks.RunnableTaskException: io.kestra.core.models.tasks.runners.TaskException: Command failed with exit code 1
	at io.kestra.plugin.scripts.exec.scripts.runners.CommandsWrapper.run(CommandsWrapper.java:184)
	at io.kestra.plugin.scripts.python.Script.run(Script.java:216)
	at io.kestra.plugin.scripts.python.Script.run(Script.java:26)
	at io.kestra.core.runners.WorkerTaskCallable.doCall(WorkerTaskCallable.java:78)
	at io.kestra.core.runners.AbstractWorkerCallable.call(AbstractWorkerCallable.java:62)
	at io.kestra.core.runners.WorkerSecurityService.callInSecurityContext(WorkerSecurityService.java:10)
	at io.kestra.core.runners.Worker.lambda$callJob$19(Worker.java:837)
	at io.kestra.core.trace.NoopTracer.inCurrentContext(NoopTracer.java:15)
	at io.kestra.core.runners.Worker.callJob(Worker.java:833)
	at io.kestra.core.runners.Worker.runAttempt(Worker.java:792)
	at io.kestra.core.runners.Worker.run(Worker.java:640)
	at io.kestra.core.runners.Worker.handleTask(Worker.java:372)
	at io.kestra.core.runners.Worker.lambda$run$8(Worker.java:261)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.kestra.core.models.tasks.runners.TaskException: Command failed with exit code 1
	at io.kestra.plugin.core.runner.Process.run(Process.java:155)
	at io.kestra.plugin.scripts.exec.scripts.runners.CommandsWrapper.run(CommandsWrapper.java:164)
	... 16 more

Environment

• Kestra Version: develop

[anna-geller]

hey, I will close as this is not really a kestra issue but the Python package distribution issue :) try using one of our Docker images that have sqlalchemy built it like the polars one https://github.com/kestra-io/examples/blob/main/dockerfiles/polars.Dockerfile. This flow will work:

id: goldfish_343874
namespace: company.team

tasks:
  - id: python_script
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/polars:latest
    warningOnStdErr: false
    beforeCommands:
      - pip install clickhouse-connect
    script: |
      print('Hello')