Unable to pass pypi credentials to poetry running in Gitlab CI

[dromer]

We have several projects that use poetry for project management and tox for CI and that require access to our private pypi repository in Gitlab for packages.

Our pyproject.toml has:

[[tool.poetry.source]]
name = "mygitlab"
url = "https://mygitlab.com/api/v4/projects/411/packages/pypi/simple"
priority = "supplemental"

I've tried setting in tox.ini:

commands_pre =
    poetry config http-basic.mygitlab __token__ {env:CI_JOB_TOKEN}

Or in .gitlab-ci.yml:

variables:
  POETRY_HTTP_BASIC_MYGITLAB_PASSWORD: ${CI_JOB_TOKEN}
  POETRY_HTTP_BASIC_MYGITLAB_USERNAME: __token__

With either in tox.ini:

pass_env = 
    POETRY_HTTP_BASIC_MYGITLAB_PASSWORD
    POETRY_HTTP_BASIC_MYGITLAB_USERNAME

or

set_env =
    POETRY_HTTP_BASIC_MYGITLAB_PASSWORD="{env:POETRY_HTTP_BASIC_MYGITLAB_PASSWORD}"
    POETRY_HTTP_BASIC_MYGITLAB_USERNAME="{env:POETRY_HTTP_BASIC_MYGITLAB_USERNAME}"

These strategies seem to work locally, but not when running in Gitlab.

Does anyone have any suggestions how to properly run poetry inside of tox, on Gitlab, with access to a private repo?

[webknjaz]

Sounds like you're not setting said env vars on the GL side. My guess is that you're trying to access secrets as env vars. But I haven't used GL in a while so not sure. Try verifying with a dummy arg (and a non-secret value printed in the log, that it's actually getting into the tox env (and also log outside tox invocation).

It sounds like this might be a poetry question rather than tox. FWIW, I'd just use standard tooling (Twine) wrapped w/ tox to avoid potential weirdness with that.

[webknjaz]

Also, you can invoke your POETRY_HTTP_BASIC_MYGITLAB_PASSWORD=dummy-sentinel tox run -e upload-to-pypi and check that dummy-sentinel appears in your poetry config. Although, it does seem like you're using multiple indentifiers — why are you setting POETRY_HTTP_BASIC_MYGITLAB_PASSWORD but trying to access CI_JOB_TOKEN?

[dromer]

As I showed in the examples I most definitely did set those envvarrs.
I can also see them redacted in the CI logs. Somehow poetry is not receiving them.

The password is never shown in the poetry config unfortunately.

And yeah maybe I should move this to poetry discussion, it's hard to pinpoint where the failure is here ..

The strange thing is that when I run things locally (and setting the vars) they work correctly, only during CI do things break ..

ps: I'm not uploading to pypi, I just need to install custom libs from our pypi.
also: I tried both these variants of passing the token, as I explained in the OG post.

[webknjaz]

It's unclear which of those snippets you tried together so it seems you might've been combining them in weird ways.

[dromer]

Sorry, I described 3 tested scenarios:

• only setting poetry config in tox.ini
• gitlab POETRY_HTTP_BASIC variables - with pass_env
• gitlab POETRY_HTTP_BASIC variables - with set_env

All of these I've been able to run with tox locally, but not in the CI.

I won't be able to until Monday, but I will try and setup an example repo that shows the different cases I tried.