Pangolin laggy with 0.12.0/0.12.1 (long page load times)

[v1rusnl]

Describe the Bug

After upgrading to 1.12.0, I rolled back to 1.11.1 today on both my LAN Pangolin instance + my VPS instance with Gerbil/Newt connected to my Homelab. I'm using Traefik 3.5.4 on both. Tried also the new 1.12.1 update, but it's the same. Migration throws no error in logs.

Something is really off with 1.12.0 (at least both of my instances). The loading times of webpages (even in my LAN locally) multiplied by min. 5 times. After rollback, everything is blazing fast again.

Comparison with the exact same Service (Emby) using Pangolin/Traefik locally as my LAN Reverse Proxy without Gerbil/Newt over Gigabit:

1.12.0/1.12.1

1.11.1

Some of the same assets load up to 10 times slower than with 1.11.1. I've sorted the elements by loading time. In 1.11.1, the slowest element (the backdrop) is 300ms, in 1.12.0/1.12.1 it's 4 seconds. Man, you feel that when opening a webpage... and it's reproduceable. Tried to migrate 3 times from my 1.11.1 backup to 1.12.0 and then to 1.12.1, but it's the same everytime. 1.12.0/1.12.1=slow, back to 1.11.1=fast.

Environment

• OS Type & Version: LAN Instance: Debian 12 / VPS Instance: Ubuntu 24.04
• Pangolin Version: 0.12.0/0.12.1 (tried both)
• Gerbil Version: 1.2.2
• Badger Version: 1.2.1
• Traefik Version: 3.5.4
• Newt Version: 1.6.0
• Olm Version: not used

LAN Instance just uses Pangolin+Traefik/Badger
VPS instance uses Pangolin+Gerbil+Traefik/Badger connected to 3 Newt in my Home LAN

To Reproduce

1. Open served ressource with 1.11.1 -> fast load times
2. Update to 1.12.0/1.12.1 -> long load time/laggy behaviour of ressource
3. Downgrade to 1.11.1 -> fast load times again

Expected Behavior

Update to 1.12.0/1.12.1 -> page load times are as fast as in 1.11.1

[Re4mstr]

I am also experiencing this, but I would say almost 10x slowdown.

[oschwartz10612]

Hi I am sorry you are experiencing this! @v1rusnl do you see anything in the pangolin logs when you load the resource?

It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up?

[v1rusnl]

Hi I am sorry you are experiencing this! @v1rusnl do you see anything in the pangolin logs when you load the resource?

It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up?

Owen, that did the trick. Everything back to normal...
Are there any disadvantages in disabling it?

[oschwartz10612]

Hum that's concerning but good we narrowed it down.

Are you using a sqlite or postgres database?

Do you see anything in the logs when it is enabled?

Do you serve a lot of traffic on this resource?

[v1rusnl]

sqlite

I have to say, in my Home LAN I run Pangolin on a microserver with a system-HDD, no SSD. Could the HDD be the culprit because it cannot keep up with the log writing cycles parallel to the ressource calls? I disabled also access logs on my home LAN from day one on, since I don't need them. On the VPS, I don't know what hardware is behind it. But there I use Traefik Log dashboard from HHF nevertheless.

When it' enabled, logs are shown

Edit: I can live with having no logs in my LAN, enabling logs -> slow; diabling logs -> fast. Perhaps I try to move Pangolin to another machine with a SSD the next days. Also I try to upgrade again on the VPS and check that one out.

[oschwartz10612]

Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster.

Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working!

[v1rusnl]

Ok, went on the VPS again, rechecked traefik_config.yml, dynamic_config.yml and config.yml, aligned them all to look exactly like the actual repo ones, then updated from 1.11.1 to 1.12.0 to 1.12.1 and set badger to 1.2.1, and everythings working now with fast loading times. I guess Hetzner is using SSD storage on their VPS plans.

So it seems Pangolin logs should not be used on HDDs for now 😆

[v1rusnl]

With the new logging, it would be overkill to also use Traefik log viewer on the VPS, right?

[v1rusnl]

Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster.

Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working!

Spun up a Pangolin on my SSD machine with a copy of the HDD pangolin folder and it's blazing fast even with logging.

So it's definitely a problem with the logging with parallel resource serving when using a HDD. It doesnt matter if the resource is traffic heavy like Emby with a lot of images or if it's just Portainer with small data chunks

[oschwartz10612]

Thanks for all of the info. Will look into making it async for the next release!

[v1rusnl]

Thank you for this awesome peace of Software!

[roadkingvrod]

Just to add further data to this, I experienced the exact same issue. Glad it's a somewhat easy fix!