SSH remotes, docker extra hosts, unique pipe_ids

[macmacal]

Hi there!

I came across the JayporeCI (JCI) during extensive research of possible CI/CD plugins for self-hosted Gitea instance. The idea of running pipelines on local (or remote) machine's docker engine with minimal overhead of configuration bought me.

However, in my case, JCI didn't work well right outside the box:

1. The Gitea server is hosted on a workstation inside academia network - a custom DNS server is currently out of my reach, so all development machines have custom aliases in their /etc/hosts files. To apply these custom aliases to docker containers, a new parameter (docker run --add-host / extra_hosts: {}) must be passed.
2. What's more, (of course) this setup required self-signed SSL certificate to be passed for all pyrequest calls...,
3. All git remotes to this Gitea server are setup as ssh ones. I was in quite shock, when i read the source code ( here ) that JCI supports only https remotes. AFAIK it is just a matter of rewriting the URL from user@server:/repo/path.git to https://server/repo/path.git.
4. Running JCI on Ubuntu 22.04 gave me job containers' names without any unique name (i.e. in my case, the pipe_id variable based on "name=" parameter in /proc/self/cgroup was always empty). Probably a fallback to an other method for obtaining unique tag (e.g. unix timestamps) would fix this inconvenience.
   
   
5. And finally, the setup section of JCI's docs doesn't have any info about "plaintext" and "secrets" concepts. Not fun at all!

All these problems motivated me to do something with them. After a long ~weekend (with an unexpected problem with official docker image arjoonn/jci (the docker wasn't installed)) I proudly present my fresh setup of JCI + Gitea:

• First, in my repo @ gitea, I mitigated the problems with "pipe" containers by adding --add-host and -e REQUESTS_CA_BUNDLE to the hook() function:

echo "> Running pipeline"
docker run \
    -d \
    --name $PIPE_NAME \
    -e JAYPORE_CODE_DIR=$JAYPORE_CODE_DIR \
    -e SHA=$SHA \
    -e REQUESTS_CA_BUNDLE=/jaypore_ci/repo/secrets/remote.pem \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $REPO_ROOT:/jaypore_ci/repo:ro \
    -v /tmp/jayporeci__src__$SHA:/jaypore_ci/run \
    --workdir /jaypore_ci/run \
    --add-host $REMOTE_ALIAS:$REMOTE_IP \
    $CI_IMAGE_TAG \
    bash -c "ENV=$ENV bash /jaypore_ci/repo/$JAYPORE_CODE_DIR/pre-push.sh run"

• In my fork of JCI, I have added a feature to pass extra hosts to job containers through pipeline definition:
  
• Of course, I had to add a ssh-to-https parser to the Repo definition with fake url injection for unit testing. I didn't know how to properly set the development of JCI with github (maybe a small "How To" in docs would be more helpful), but manual launch of all tests gave me all greens:
  
• Finally, to address problems with "empty" pipe_id for job containers in Ubuntu 22.04 (and in Arch too) I have added a small fallback to unix timestamp, when the length of the obtained name is equal to 0. IMHO it looks good enough.
  

Now, I'm a happy user of JCI and I'm ready to start doing some refactor to my current code base:

That was an adventure! I hope someone will find these things useful 😉 .

Also, as you can see, these 3 main modifications are already separated into three branches - if there is will and desire I can start creating PRs for them.

[theSage21]

• First of all thank you so much for trying out JayporeCI 😄 .
• I'll try to address each of your points. Jci started as a way to solve the problems I was facing in my workflows so I'm really happy to see how other people use it.

1. Regarding extra_hosts: I think we can simply add something like docker_kwargs to a job and have that entire kwargs dictionary passed to the docker run function call. If we do that then any time a new requirement for some docker feature comes we don't have to change the codebase.
2. For the SSL certificates I was looking at your solution and it seems like a good enough way to proceed. One consideration might be to have the volume mounts also passed on to the child jobs. This would make it easier to mount a cache / secrets / data volume and each job can have access to it.
3. During the early stages of jci we were using https clones with stored API keys as passwords. That is where Jci was taking the gitea api token from and so we only supported https clones at the time. Currently it is expected that the token will be provided as a secret so your suggestion of simply adding support for ssh is great 👍 . All we need to do is remove the https restriction since nothing actually depends on it.
4. I think I'll change the cgroup logic to simply use --cidfile. The unique name is there to relate the CI run each job that is part of the CI run. The intention is not to have a unique name, rather to be able to tell which CI run each job was a part of. I think that should cover your usecase, thought I might need help in testing it since the cgroup logic has worked on all machines that I've used jci with.
5. Over the past weekend I did realize that the section on secrets and environment variables was missing in the docs. I've added a stub for it in the docs for now but properly writing that would take me some time.

[macmacal]

You have made a great tool for very small (~1 person) development teams 😃 .

Answering the all points:

1. Great idea, I was too focused on just adding hosts to job containers. I made changes and prepared PR for that mini-feature: Feature: docker_kwargs for job containers #26 .
2. I agree with you. Mounting (read-only) volumes with secrets (SSL certs) is worth of consideration, especially from the perspective of scaling up JCI to multiple repositories. I will tinker with that idea in new repositories.
3. Oh I see it now. Good to hear that I didn't miss anything important during read of the codebase. I prepared PR with my mini ssh-support: Feature: Tests for http, https & ssh remotes #25
4. --cidfile sounds like a better approach. I don't have variety of machines, but I think I can help in covering tests in two setups mentioned earlier: Ubuntu 22.04 & Arch. Indeed, more volunteers are needed. It is also worth mentioning that I really miss some kind of build-in functions to cleanup stopped containers (and their images) periodically. Grouping jobs with pipelines looks like a good place to start for developing such functionality.
5. All right, I keep your fingers crossed on that 😉 .

[theSage21]

I've made a few changes to Jaypore CI over the past weeks.

• Installations are now pinned so that your repo will not break with new releases.
• You can upgrade your installation to something new with a simple version change.

Do check out the latest version / docs at jayporeci.in and let me know if something is still broken for you and requires a change.

[macmacal]

Sounds good! I will test it in following days.

I appreciate the change of documentation theme to ReadTheDocs 👍 .