httpd failure due to postgresql service error

[rhclopes]

I am seeing an error in several hosts in UK running toolkit 5.2.3-1.

The journalctl for httpd shows the error attached.

errors-20251119.md

The problem seems to be solved by restarting postgresql service followed by a restart in httpd.

Raul

[simpsonst]

Hi! This error was occurring every minute or so in httpd journal until postgresql was restarted:

Exception: psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'importlib_resources'
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/var/www/pscheduler-server/pschedulerapiserver/tools.py", line 29, in tools
    cursor = dbcursor_query("SELECT * FROM api_tools_for_test(%s)",
  File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 117, in dbcursor_query
    raise ex
  File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 102, in dbcursor_query
    cursor.execute(query, args)
psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'importlib_resources'
Internal error xxxxxx GET https://yyyyyy/pscheduler/tools: Internal problem; see system logs.

(These lines were prefixed with pscheduler-api[68071] ERROR.)

[rhclopes]

This error seems to have affected all UK hosts running the toolkit 5.2.3.1.

Hosts that have between 32GB ad 192GB RAM.

[rhclopes]

The problem causing havoc in most UK hosts

https://ps-mesh.perf.ja.net/grafana/d/aa15bb77-25d6-53d1-b3c7-bf51b6417afa/jisc-uk-tests?orgId=1&from=now-24h&to=now&timezone=browser

[mfeit-internet2]

The missing module is part of python3-libs, which is a core piece of Python present even on minimally-installed EL systems. It's several layers of dependencies down from anything we tinker with. The module is imported by the netaddr module, which is part of the Alma and Rocky distributions

I just did a build on a fresh Alma 9.7 and had no difficulties

If you have root access to one of the misbehaving systems, I'd like to have a look around. Get in touch with me first thing tomorrow (9:00 EST or later) and we'll jump on Zoom and have a look at it.

[timchown]

Hi Mark,
Raul, I and others here should be free around then - after 2pm GMT as we have a netperf team call before that. I'm not sure if a system Raul has root on is currently affected, but there's 8 hosts not getting results to our mesh right now:
https://ps-mesh.perf.ja.net/grafana/d/aa15bb77-25d6-53d1-b3c7-bf51b6417afa/jisc-uk-tests?orgId=1&from=now-24h&to=now&timezone=browser
We have ps-london and ps-slough there as our servers.
Every other one is a different admin at a different site.
Tim

[rhclopes]

This error has been showing in hosts that were healthy until a few weeks ago.

The admins did not touch them.

[drwhitehouse]

As far as I can tell we had this issue, and simply rebooting the server was sufficient to get things working again.

[timchown]

Sure, but that's not getting to the root cause, so you don't have to keep fixing it.

[drwhitehouse]

Sure, but that's not getting to the root cause, so you don't have to keep fixing it.

Well, I'm happy to leave root cause analysis to the developers, but I think that the breakage occurred when we updated Rocky to 9.7 which I believe is a fairly substantial upgrade.
I suspect that rebooting the machine simply removed something that was resident in memory from prior to the upgrade, that was not playing nicely with something else read from the disk that the upgrade had installed.

[timchown]

Thanks, that is possible, Raul is still digging, but working on some other priorities as well. On 11/12/2025, 22:28, "drwhitehouse" ***@***.***> wrote: [https://avatars.githubusercontent.com/u/8874233?s=20&v=4]drwhitehouse left a comment (perfsonar/pscheduler#1604)<#1604 (comment)> Sure, but that's not getting to the root cause, so you don't have to keep fixing it. Well, I'm happy to leave root cause analysis to the developers, but I think that the breakage occurred when we updated Rocky to 9.7 which I believe is a fairly substantial upgrade. I suspect that rebooting the machine simply removed something that was resident in memory from prior to the upgrade, that was not playing nicely with something else read from the disk that the upgrade had installed. — Reply to this email directly, view it on GitHub<#1604 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABMV24BNYV6B2OZGRQLNJJ34BHVXFAVCNFSM6AAAAACMTNDPEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNBUGA2TOMRUGA>. You are receiving this because you commented.

[laeti-tia]

Should we maybe add a plot for the number of concurrently opened files or available file handles to the Host Metrics dashboard?

[mfeit-internet2]

@laeti-tia Couldn't hurt.