Make integration tests to build outside of OBS #2746
Conversation
|
This is just a start to make sure the integration tests can also build outside of OBS. There are a number of other tests not yet covered:
|
schaefi
force-pushed
the
fix_build_tests_to_run_outside_of_obs
branch
2 times, most recently
from
6059454 to
595ffa6
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
build-tests/x86/tumbleweed/test-image-custom-partitions/custom_partitions_create.sh
Fixed
Show fixed
Hide fixed
build-tests/x86/tumbleweed/test-image-custom-partitions/custom_partitions_create.sh
Fixed
Show fixed
Hide fixed
build-tests/x86/tumbleweed/test-image-custom-partitions/custom_partitions_create.sh
Fixed
Show fixed
Hide fixed
build-tests/x86/tumbleweed/test-image-custom-partitions/custom_partitions_setup.sh
Fixed
Show fixed
Hide fixed
|
@Conan-Kudo finally I updated all integration tests for all archs and distributions and built them outside of obs. Thanks to the fedora infrastructure people who allowed me to access a ppc machine (I have none). As you can see I added a little helper script and also documentation how to use it. As I'm using boxbuild in container mode and together with the universal box built as a container, you can actually build the tests on any host. For x86 and aarch64 I used an AWS instance (os doesn't matter), for s390 I used IBM cloud as I have an account there. For ppc64 I used the fedora infrastructure. So in theory we can have a conversation how hard it would be to distribute integration test builds according to specific constraints to "some" server. At the moment it's all connected to OBS, but with this PR it becomes at least possible to do it everywhere. Thoughts ? |
schaefi
force-pushed
the
fix_build_tests_to_run_outside_of_obs
branch
from
f23358e to
7c2d776
Compare
schaefi
changed the title
schaefi
force-pushed
the
fix_build_tests_to_run_outside_of_obs
branch
from
7c2d776 to
9e0ec47
Compare
I think this is a great idea. I think we can probably pull together some way to run these consistently and finally be able to put them into pull request validation too. |
Update and extend all integration tests such that they also build outside of the Open Build Service. Along with the changes on the descriptions a simple build-tests.sh script was added to drive the build process. The build is based on the kiwi boxbuild plugin in container mode to build the tests from a given build-tests directory. A new chapter to document how to Build the Build Tests is also provided and referenced on the github main page.
schaefi
force-pushed
the
fix_build_tests_to_run_outside_of_obs
branch
from
daa3201 to
15470b5
Compare
|
@Conan-Kudo thanks much for updating the repo locations 👍 I applied your suggestions |
Yes it would also be my plan to be able to do this. However, we might have a conversation about it as I can think of some hurdles. @Conan-Kudo @davidcassany @dcermak First of all there are three triggers that for building the integration test images:
The setup in OBS automates the rebuild for 2. For 1. it automatically rebuilds if the change lands in main or someone manually points the service file to another branch. A change to kiwi automatically (3.) rebuilds if a new kiwi package is placed in the Staging project. If we take the integration tests out of the buildservice and with this PR in place the trigger mechanics changes.
So outside of OBS we would have to know:
I don't see an easy way to connect this rebuild task with a PR on the kiwi github project. I also believe at least one of the above will always have a change. So my idea was more going to the following direction:
As long as the PR is open we would watch the build results. There is for sure a relatively long delay until all builds have finished with the new staging container. We still have no dedicated trigger for a change in the distribution but I believe for those we have no handle anyway. On an individual reviewer based decision we can merge the PR or wait or debug failed integration tests. I don't want to put much implementation effort in this setup. With this PR we would handle the actual building of integration tests. I would be open to implement the REST based API gateway as a cloud service and two paths to send and receive data. It would be good to have support from someone else writing the github action. And of course I would be open for better ideas and looking forward to your feedback Thanks |
schaefi
added
Discuss at the Community Meeting
and removed
Discuss at the Community Meeting
labels
|
Marcus Schäfer ***@***.***> writes:
schaefi left a comment (OSInside/kiwi#2746)
> I think this is a great idea. I think we can probably pull together some way to run these consistently and finally be able to put them into pull request validation too.
Yes it would also be my plan to be able to do this. However, we might have a conversation about it as I can think of some hurdles. @Conan-Kudo @davidcassany @dcermak
First of all there are three triggers that should turn into building these integration test images:
1. The integration test description itself has changed. This happens very rarely
2. The list of packages from which the integration test builds the image has changed. This happens very often, actually always because we have tests that builds against rolling releases like rawhide or TW and even on slow rolling distros there are constantly changes on the packages or their dependencies
3. kiwi has changed. This happens less often than packages changes in two and more often than test description changes in one
The setup in OBS automates the rebuild for 2. For 1. it automatically rebuilds if the change lands in main or someone manually points the service file to another branch. A change to kiwi automatically (3.) rebuilds if a new kiwi package is placed in the Staging project.
If we take the integration tests out of the buildservice and with this PR in place the trigger mechanics changes.
1. Outside of OBS some system would call `build-tests.sh --test-dir some --box-name staging`. Calling it that way will take a kiwi change into account when a new kiwi lands in the Staging project (still a manual submission) and the staging container has rebuilt (the rebuild happens automatically).
2. Outside of OBS a change of a package that is used in the test image builds will not automatically trigger a rebuild
3. Outside of OBS a change to the test image description will not automatically trigger a rebuild
So outside of OBS we would have to know:
* did the distro change (dependency resolution of the test image descriptions)
* did kiwi change (pull request and new container available)
* did the test image description change (pull request)
I don't see an easy way to connect this rebuild task with a PR on the kiwi github project. I also believe at least one of the above will always have a change.
So my idea was more going to the following direction:
* We setup/or use systems that builds our integration tests on a time schedule depending on how powerful the machine is. A systemd timer service that runs `build-tests.sh --test-dir some --box-name staging`. We need systems for each architecture, s390, ppc, x86_64, aarch64. We can start small with whatever server we can afford. I could setup that in the cloud for e.g. x86_64 and aarch64.
* we write a little flask based service connected to an open API gateway (REST) that every worker can send the build results to.
* when we open a pull request against kiwi we write the github action in a way that it does the following:
a) submit a new kiwi to Staging
b) wait until the staging container has finished its rebuild (it now includes the Staging kiwi), we would test the package build as a nice side effect
c) continuously reach out to the API gateway showing the current build results
As long as the PR is open we would watch the build results. There is for sure a relatively long delay until all builds have finished with the new staging container. We still have no dedicated trigger for a change in the distribution but I believe for those we have no handle anyway. On an individual reviewer based decision we can merge the PR or wait or debug failed integration tests.
I don't want to put much implementation effort in this setup. With this PR we would handle the actual building of integration tests. I would be open to implement the REST based API gateway as a cloud service and two paths to send and receive data. It would be good to have support from someone else writing the github action.
And of course I would be open for better ideas and looking forward to your feedback
No offense, but this is way too complicated. This "simple" flask
application would duplicate a lot of the existing functionality of the
Open Build Service.
Let's please take a step back and reconsider what we're trying to solve
here.
The current setup of building the integration tests on OBS already
solves cases 1. & 2. (i.e. build descriptions change or distro
changes). We could implement better reporting for the first case by
using OBS SCM CI, but tbh I'm not sure if it's worth it for the hassle.
So that leaves us with the third issue: kiwi changes and that requires a
rebuild of all images and preferably a re-test. IMHO, this can be
performed via a bit of OBS or pbuild automation:
1. create a branch of kiwi and commit the sources of the current
PR/release branch into it
2. branch all integration tests and configure the project from 1. in
their path before the distro's kiwi
3. wait for the builds to finish
4. (eventually) run the openQA integration tests
5. Profit 💰️
Sure, this is all very hand-wavy, but to be honest it sounds much more
achievable than hand writing a service that duplicates a good chunk of
OBS. Additionally, I have already implemented a comparable workflow for
BCI, so I think it is not only a theoretical possibility.
|
I agree which is also the reason why I'm not taking an immediate action and wanted to have this conversation
yep and I also agree that this can be done with OBS and the proposed way sounds like a good option. But my motivation covers the following:
From the two aspects above I wanted to have a conversation how "easy" it would be to let the integration tests build without the hard coupling to OBS. I believe this PR made sense because now you can quickly build any integration test on your local machine. The bits to trigger and automatically build them and the machine resources to actually do this are not so easy and I agree with you that this duplicates functionality of OBS. So I myself are unclear if we should do anything else other than making the tests at least build'able outside of OBS And I appreciated anyone's feedback in this regard. Thanks |
|
Marcus Schäfer ***@***.***> writes:
schaefi left a comment (OSInside/kiwi#2746)
@dcermak
> No offense, but this is way too complicated...
I agree which is also the reason why I'm not taking an immediate action and wanted to have this conversation
> So that leaves us with the third issue: kiwi changes and that requires a rebuild of all images and preferably a re-test.
yep and I also agree that this can be done with OBS and the proposed way sounds like a good option.
I also never planned to delete/replace the integration tests from OBS.
But my motivation covers the following:
1. Our integration tests should be able to be built outside of OBS. If you look at the changes from this PR you already see that I needed to change not only the repos but also many other parts of the image description to allow building with the native package managers. OBS has its own OBS specific way of resolving and almost no image did built without modifications. This hides several issues in integration tests which never gets uncovered by building in OBS
I am aware of the problem of OBS' dependency resolver being "hard to
work with", so being able to run them in and outside of OBS is
definitely a win.
2. OBS changes its source backend to git. That sounds awesome but the way it is done is not really an integration with the git API, it's still sort of a copy but the copied data is now hidden and no longer accessible. Our project holding kiwi and also the integration test projects are scheduled to be moved into this concept. Our integration tests are using the old model of the scm service to fetch data from git and there is also some automation scripting around using project tokens. I have a very big concern that all of this will be broken once we get moved by the OBS team. I want to make sure that there is also another opportunity to build the integration tests.
No, the backend does not change, the OBS backend is still using the same
version control that it always did, the only difference is the
`<scmsync>` element in combination with the git-scm-bridge which commits
git sources into the obs backend. For us, things will actually become
simpler due to this. All we have to do is change the integration test
projects to use scmsync with the correct ?subdir query and we'll be
set. Well and then we'll still need to do a osc service remoterun to
ensure that the scm bridge actually updates the sources on OBS.
|
Ok my understanding was different here. But if it should be that straight forward may I ask you to perform this change once we got migrated ?
This part is already automated through a project token. See the https://github.com/OSInside/kiwi/blob/main/.github/workflows/ci-update-build-tests.yml github action |
|
Marcus Schäfer ***@***.***> writes:
schaefi left a comment (OSInside/kiwi#2746)
@dcermak
> No, the backend does not change, the OBS backend is still using the same
version control that it always did, the only difference is the
`<scmsync>` element in combination with the git-scm-bridge which commits
git sources into the obs backend. For us, things will actually become
simpler due to this. All we have to do is change the integration test
projects to use scmsync with the correct ?subdir query and we'll be
set.
Ok my understanding was different here. But if it should be that straight forward may I ask you to perform this change once we got migrated ?
Sure, but in theory we can already migrate the sources now.
> Well and then we'll still need to do a osc service remoterun to ensure that the scm bridge actually updates the sources on OBS.
This part is already automated through a project token. See the https://github.com/OSInside/kiwi/blob/main/.github/workflows/ci-update-build-tests.yml github action
I meant to use OBS SCM CI for that: https://openbuildservice.org/help/manuals/obs-user-guide/cha-obs-scm-ci-workflow-integration
|
Fix build-tests/x86/tumbleweed to build outside of OBS. In addition I added a simple build-tests.sh script which makes use of boxbuild in container mode to build the tests from a given build-tests directory.