Issues with LCM promotions since migration to containerized release

[ppanon2022]

We've had a few database-related issues since migrating to the containerized version of Uyuni. Some of which I've now figured out how to resolve, some of which are possibly more due to concomitant application changes (i.e. not a result of containerization), and some of which are still a problem.

1. pg_hba.conf and postgresql logging. PG used to create a few KB's worth of information/error logs per day before migration. Afterwards it's been a few megabytes. A huge part of it has to do with ident permission entries like host all all 127.0.0.1/32 ident. Before containerization we were running an ident daemon but haven't been able to after since I think there is no option or files for that service in the server container, and there probably isn't any in MicroOS either. The log was filling with periodic errors (watchdog checks by podman?). Editing the conf file is challenging in the context of the post-2025.05, bare-bones postgresql container. Wound up using sed, remembering that the separation between columns is using tabs, \t. You might also need to add a rule to allow specifically for the polling using the postgres account to the postgres db on the 127.0.0.1 loopback.

2. There was still old pg14 data post-migration under /var/lib/pgsql/data/data-pg14. It takes up a lot of room. I've kept the config files and the (quite small) log files for history and recycled the rest to recover a lot of space.

3. The above is important because our LCM project version builds and promotions have started using a lot of temp storage under /var/lib/pgsql/data/base/pgsql_tmp/, so our LCM project version builds were failing. Clearing up all the above space and adding a little more seems to have helped.

4. Those LCM builds are also taking way longer. Most of the time seems to be running various queries populating the rhnServerNeededCache table. I mean many, many hours for each project build. Are these new, or has there been some changes to the way they are populated which decreases their efficiency? Some of these queries seem to be at least partially dynamically generated and yet also parameterized. For example,

INSERT INTO rhnServerNeededCache (server_id, package_id, channel_id)
                  ( WITH retracted_packages AS (
                    SELECT DISTINCT ep.package_id AS pid, sc.server_id AS sid
                    FROM rhnserverchannel sc
                      JOIN rhnchannel c ON c.id = sc.channel_id
                      JOIN rhnchannelerrata ce ON ce.channel_id = c.id
                      JOIN rhnerrata e ON e.id = ce.errata_id
                      JOIN rhnerratapackage ep ON ep.errata_id = e.id
                    WHERE e.advisory_status::text = 'retracted'::text
                      AND ep.package_id in (767866,767779,765279,773035,765317,776344,765067,773202,767775,765220,765098,776315,773171,765306,773226,769898,773074,770629,769963,776243,765430,770578,773083
,773182,773280,769885,763958,763728,770619,767785,770651,765242,773129,770656,773348,773387,765167,773190,773005,776351,771209,773462,765152,770622,773450,773284,776334,770624,770628,769883,776338,773172,
765195,767797,773075,773208,776296,773236,773238,770654,773106,765044,773261,763946,773166,773354,776314,770582,765156,773337,773430,773330,773444,767811,763948,773000,773014,773026,769884,773068,773139,7
70572,773306,773092,776294,767815,773101,773049,773290,767802,773241,763961,769910,765260,770604,776265,773277,773215,770560,765373,763969,773343,773036,776305,773115,769909,773345,773309,769882,770554,77
3299,765403,765350,776320,773441,773163,770623,763962,765086,773224,770621,770585,763954,769902,763945,770625,773066,773244,773282,773112,773310,773318,773264,765178,769952,765331,765410,773037,773483,776
260,776349,773401,773225,765404,773144,776327,765265,762570,773313,773315,770601,765164,773484,773379,776311,776301,773394,770576,773094,764324,770664,769900,776276,773429,773378,773279,767848,773141,7706
60,773055,772995,769896,773353,773127,773149,773360,769911,770587,773448,770557,767870,773118,773183,765115,770569,773213,765099,763725,770608,765141,776304,773437,776288,770620,770558,765370,773408,77332
1,765381,773167,776295,773069,773327,763976,763928,762585,773050,776343,773432,770556,767839,773229,776299,770597,770650,773351,776309,773195,773045,773294,767846,767791,776281,773227,773105,773031,767786
,770672,765052,776300,773481,765375,773012,773438,765289,773211,773201,773479,770561,765119,765142,770565,772997,763970,776336,765303,773085,765294,769968,776249,776342,765347,765101,773089,773178,765429,
765356,773328,776248,773029,773452,764319,773021,770644,773344,770616,767796,764323,769914,769920,769908,765193,765338,773114,769933,769951,773376,773052,765036,767868,773007,770583,765374,773024,773386,7
73147,773270,773266,776298,763981,773097,767828,770598,773303,773198,773333,773460,765305,765329,765096,770659,763936,773246,773373,779443,767812,765268,773342,769921,773240,776289,763947,767806,776263,76
5364,765120,773446,773396,767832,765072,767858,767865,773103,770570,765109,773275,765129,776268,765269,767807,776346,773016,773426,773419,773312,763952,765184,773028,773044,770671,773400,773235,770617,773
057,773475,764322,773233,765170,773385,773286,770669,773082,773015,770555,770606,773142,763938,763968,773084,773135,773177,769907,773165,770564,773146,765092,770602,765124,773216,765412,773288,773063,7762
64,776347,765079,776328,773439,776326,765424,765224,767818,773304,769904,770562,773025,770648,776241,772999,765319,773368,770640,763940,773453,773099,773388,765163,763979,767841,773064,773001,776255,76509
7,773218,776293,770600,773251,773352,773137,773205,773155,765077,765128,773023,763957,773051,770596,773065,773164,773042,765179,765212,765222,765346,773404,776244,770580,770568,762568,776341,765033,770662
,769931,776318,769926,773363,773148,770610,773223,773193,773234,765054,770594,773267,765149,763960,763971,770667,765108,773179,773173,765334,776330,773472,765236,773341,773281,773187,773151,773428,770579,
771211,773043,765122,773297,776325,769927,773130,776321,773285,773476,763943,773407,765249,776262,773461,776261,765144,773038,773324,773466,773006,773323,770592,776339,773221,769890,776270,773255,765211,7
65095,773459,773322,773176,765312,773048,769944,773259,773203,773056,773293))
                    SELECT DISTINCT
                     S.id as server_id,
                     P.id as package_id,
                     $1 as channel_id
                        FROM
                             susePackageExcludingPartOfPtf P,
                             rhnServerPackageArchCompat SPAC,
                             rhnPackageEVR P_EVR,
                             rhnPackageEVR SP_EVR,
                             rhnServerPackage SP,
                             rhnServerChannel SC,
                             rhnServer S

                        WHERE
                                 SC.channel_id = $2
                          AND    SC.server_id = S.id
                          AND    p.id in (767866,767779,765279,773035,765317,776344,765067,773202,767775,765220,765098,776315,773171,765306,773226,769898,773074,770629,769963,776243,765430,770578,773083,7
73182,773280,769885,763958,763728,770619,767785,770651,765242,773129,770656,773348,773387,765167,773190,773005,776351,771209,773462,765152,770622,773450,773284,776334,770624,770628,769883,776338,773172,76
5195,767797,773075,773208,776296,773236,773238,770654,773106,765044,773261,763946,773166,773354,776314,770582,765156,773337,773430,773330,773444,767811,763948,773000,773014,773026,769884,773068,773139,770
572,773306,773092,776294,767815,773101,773049,773290,767802,773241,763961,769910,765260,770604,776265,773277,773215,770560,765373,763969,773343,773036,776305,773115,769909,773345,773309,769882,770554,7732
99,765403,765350,776320,773441,773163,770623,763962,765086,773224,770621,770585,763954,769902,763945,770625,773066,773244,773282,773112,773310,773318,773264,765178,769952,765331,765410,773037,773483,77626
0,776349,773401,773225,765404,773144,776327,765265,762570,773313,773315,770601,765164,773484,773379,776311,776301,773394,770576,773094,764324,770664,769900,776276,773429,773378,773279,767848,773141,770660
,773055,772995,769896,773353,773127,773149,773360,769911,770587,773448,770557,767870,773118,773183,765115,770569,773213,765099,763725,770608,765141,776304,773437,776288,770620,770558,765370,773408,773321,
765381,773167,776295,773069,773327,763976,763928,762585,773050,776343,773432,770556,767839,773229,776299,770597,770650,773351,776309,773195,773045,773294,767846,767791,776281,773227,773105,773031,767786,7
70672,765052,776300,773481,765375,773012,773438,765289,773211,773201,773479,770561,765119,765142,770565,772997,763970,776336,765303,773085,765294,769968,776249,776342,765347,765101,773089,773178,765429,76
5356,773328,776248,773029,773452,764319,773021,770644,773344,770616,767796,764323,769914,769920,769908,765193,765338,773114,769933,769951,773376,773052,765036,767868,773007,770583,765374,773024,773386,773
147,773270,773266,776298,763981,773097,767828,770598,773303,773198,773333,773460,765305,765329,765096,770659,763936,773246,773373,779443,767812,765268,773342,769921,773240,776289,763947,767806,776263,7653
64,765120,773446,773396,767832,765072,767858,767865,773103,770570,765109,773275,765129,776268,765269,767807,776346,773016,773426,773419,773312,763952,765184,773028,773044,770671,773400,773235,770617,77305
7,773475,764322,773233,765170,773385,773286,770669,773082,773015,770555,770606,773142,763938,763968,773084,773135,773177,769907,773165,770564,773146,765092,770602,765124,773216,765412,773288,773063,776264
,776347,765079,776328,773439,776326,765424,765224,767818,773304,769904,770562,773025,770648,776241,772999,765319,773368,770640,763940,773453,773099,773388,765163,763979,767841,773064,773001,776255,765097,
773218,776293,770600,773251,773352,773137,773205,773155,765077,765128,773023,763957,773051,770596,773065,773164,773042,765179,765212,765222,765346,773404,776244,770580,770568,762568,776341,765033,770662,7
69931,776318,769926,773363,773148,770610,773223,773193,773234,765054,770594,773267,765149,763960,763971,770667,765108,773179,773173,765334,776330,773472,765236,773341,773281,773187,773151,773428,770579,77
1211,773043,765122,773297,776325,769927,773130,776321,773285,773476,763943,773407,765249,776262,773461,776261,765144,773038,773324,773466,773006,773323,770592,776339,773221,769890,776270,773255,765211,765
095,773459,773322,773176,765312,773048,769944,773259,773203,773056,773293)
                          AND    NOT EXISTS (SELECT 1 FROM retracted_packages WHERE sid = S.id AND pid = P.id)
                          AND    NOT EXISTS (SELECT 1 FROM suseServerAppStreamHiddenPackagesView   WHERE sid = S.id AND pid = P.id)
                          AND    p.package_arch_id = spac.package_arch_id
                          AND    spac.server_arch_id = s.server_arch_id
                          AND    SP_EVR.id = SP.evr_id
                          AND    P_EVR.id = P.evr_id
                          AND    SP.server_id = S.id
                          AND    SP.name_id = P.name_id
                          AND    SP.evr_id != P.evr_id
                          AND    (SP_EVR.evr).type = (P_EVR.evr).type
                          AND    SP_EVR.evr <  P_EVR.evr
                          AND    SP_EVR.evr = (SELECT MAX(PE.evr) FROM
                                          rhnServerPackage SP2, rhnPackageEvr PE WHERE PE.id = SP2.evr_id
                                          AND SP2.server_id = SP.server_id AND SP2.name_id = SP.name_id)
                          AND NOT EXISTS (SELECT 1 -- prevent inserting in case cache already contains the row (possibly with non-null errata_id)
                                          FROM rhnServerNeededCache EXISTING
                                          WHERE EXISTING.server_id = S.id
                                          AND EXISTING.package_id = p.id
                                          AND EXISTING.channel_id = $3))

Is there something that can be done to improve performance here? What is rhnServerNeededCache needed for anyways? Because I don't understand why it would be taking hours to populate for the very small handful of servers attached to our first-stage test environment channels. It's causing us way more pain than it likely helps. Something seems wrong with this approach. In addition to the core O/S channels, we also have a number of custom 3rd-party child channels and queries like the above seem like massive overkill for those channels which wouldn't have anything to do with revoked packages, among other things.

5. How to clear weird orphaned draft LCM versions? We've got some orphaned draft versions. For instance on one of or projects if you look at the version history in the project details, one of the versions is

Version 10: (draft - not built) - Check the changes below
Version 10: (draft - not built) - Check the changes below

but there also is an actual real version 10

Version 10: June 2025 ubuntu22 updates
Version 10: June 2025 ubuntu22 updates

and the current version number is much higher (due to failures caused by the above-mentioned space issues). If we try to build a new version, the dialog box shows 10 in the [greyed-out] Version field. Where is it picking that up from and how do I clear it up? I couldn't find a PG sequence that would be responsible and there seems to be no row for it in susecontentprojecthistoryentry like there is for the other versions. So how do I get rid of it - where is it coming from? Almost all of our projects appear to have these. We're using the REST API to schedule a lot of LCM project version/env builds, except when one fails and we need to manually fill the gap with the GUI. Maybe the REST API isn't updating something the GUI uses for tracking the next version?

[ppanon2022]

We just did our first LCM build cycle since upgrading to 2025.10 and the LCM builds appeared to run much faster.

[admd]

very good. Thank you for feedback. I will close the issue then.

[admd]

a) the Ungrouped system list doesn't filter for systems that aren't a member of any group. It just lists them all

This is also fixed but I am not sure if it was part of 2026.01

[ppanon2022]

Some spacecmd commands also no longer seem to work properly anymore.

system_schedulepackagerefresh search:name:[partialname]
Now returns a whole bunch of warnings
WARNING: Can't find system ID for 1000011264
WARNING: Can't find system ID for 1000011268
WARNING: Can't find system ID for 1000010375
WARNING: Can't find system ID for 1000011656
...
and doesn't do anything

Running a shell expansion instead, like
spacecmd -y system_schedulepackagerefresh $(spacecmd system_list | grep -E '^')
will return the same warnings followed by
ERROR: com.redhat.rhn.common.translation.TranslationException: Could not find translator for class java.lang.String to class java.lang.Integer

Sorry, just found that one today.

[admd]

Please open an issue for this spacecmd bug. If it was working before, it should be working now.

[ppanon2022]

To be honest, I've never tried using that particular function until recently, so I don't know how recently it stopped working. Usually, we have just used the web portal/GUI to trigger a package list update for one or two systems, but recently I wound up needing to do that for a large number of systems and thought to try to do it with spacecmd and leverage the system name pattern matching to speed up the process.

However, I already opened Issue 11482