Parse and import both rpm and deb packages metadata #9101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
waterflow80
force-pushed
the
parse-and-import-debian-packages-md
branch
3 times, most recently
from
9cf2fd5 to
d12d51a
Compare
waterflow80
changed the title
waterflow80
force-pushed
the
parse-and-import-debian-packages-md
branch
from
4e0c10e to
9740fbe
Compare
Contributor
|
👋 Hello! Thanks for contributing to our project. If you are unsure the failing tests are related to your code, you can check the "reference jobs". These are jobs that run on a scheduled time with code from master. If they fail for the same reason as your build, it means the tests or the infrastructure are broken. If they do not fail, but yours do, it means it is related to your code. Reference tests: KNOWN ISSUES Sometimes the build can fail when pulling new jar files from download.opensuse.org . This is a known limitation. Given this happens rarely, when it does, all you need to do is rerun the test. Sorry for the inconvenience. For more tips on troubleshooting, see the troubleshooting guide. Happy hacking! |
github-actions
bot
added
python3
java
java_lint_checkstyle
java_pgsql_tests
proxy
API
susemanager_unittests
ui
javascript_lint
secondary-features
ci-pipelines
testing
ruby_rubocop
webapp
salt-actions-implementations
data-model
xmlrpc
test-framework
old-ui
build-validation-features
webui-templates
labels
Parsed the update-info.xml file and imported the parsed patches/updates to the database. We used pretty much the same code from the old Reposync class.
Import the parsed rpm and debian packages to the database in batche, and associate each pacakge with the corresponding channel
Parsed the debian Packages metadata file in a lazy way and yield the metadata of each package separately.
Parsed the debian's Translation file that contains the full description of packages, grouped by description-md5, and cache the parsed descriptions in a cache directory.
Using both packages_parser and translation_parser, return the full packages' metadata, pacakge by package, using lazing parsing Also set the debian repository's information in a DebRepo class
Given the channel label, fetch important repository's information form the database, and store it in a temporary object RepoDTO
Added the necessary command line arguments. Identify the target repositories, prepare the datastructures, and execute the lazy synchronization of repositories/packages.
Added a new dependency python-gnupg used to verify repo signature.
Ignored two linting complains about rasing exceptions floowing the approach in the old reposync. We can enhance the code instead of doing this though.
This commit completes almost all the logic and use cases of the new lazy reposync. **Note** that this commit will be restructured and possibly divided into smaller and more convenient commits. This commit is for review purposes.
Seemingly this error happened because we reached the maximum number of unclosed db connections. And thought that this might be due to the fact that the close() method in the Database class was not implemented, and the rhnSQL.closeDB() was not closing any connection. However, we're still hesitating about whether this is the root cause of the problem, because the old(current) reposync is was using it without any error.
This is the latest and almost the final version of the lzreposync service. (gpg sig check not complete) It contains pretty much all the necessary tests, including the ones for updates/patches import. Some of the remaining 'todos' are either for code enhancements or some unclear concepts that will be discussed with the team. Of course, this commit will be split into smaller ones later after rebase.
- Removed some todos. - Changed some sql queries with equivalent ones using JOIN...ON. - Some other minor cleanup
Optimized some code by changing classes and methods in some logics with free functions. Consolidated the debian repo parsing.
Completed the gpg signature check for rpm repositories, mainly for the repomd.xml file. This is done by downloading the signature file from the remote rpm repo, and executing 'gpg verify' to verify the repomd.xml file against its signature using the already added gpg keys on the filesystem. So, if you haven't already added the required gpg keyring on your system, you'll not be able to verify the repo. You should ideally run this version directly on the uyuni- server, because the gpg keyring will probably be present there.
makedirs() in uyuni.common.fileutils now accepts relative paths that consist of only a directory name or paths with trailing slashes.
Completed the gpg signature check for debian repositories. If you haven't already added the required gpg keyring on your system, you'll not be able to verify the repo, and you'll normally get a GeneralRepoException. You should ideally run this version directly on the uyuni- server, because the gpg keyring will probably be present there.
Mocked the SPACEWALK_GPG_HOMEDIR value to `~/.gnupg/`, which is the default directory for gpg, in order to execute the gpg tests outside the uyuni-server
waterflow80
force-pushed
the
parse-and-import-debian-packages-md
branch
from
574e43e to
9e651bc
Compare
Contributor
|
This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR change?
Parse both rpm and deb packages/repositories' metadata files and import the data to the database.
GUI diff
No difference.
Documentation
No documentation needed: only internal and user invisible changes
DONE
Test coverage
No tests: still in progress
DONE
Links
Issue(s): #
Port(s): # add downstream PR(s), if any
Changelogs
Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository
If you don't need a changelog check, please mark this checkbox:
If you uncheck the checkbox after the PR is created, you will need to re-run
changelog_test(see below)Re-run a test
If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run:
Before you merge
Check How to branch and merge properly!