Severe Puppetserver Degradation on Consecutive Runs in High-Volume Environments

[cukal]

In a high-volume setup (~10k vault lookups), a fresh Puppetserver run completes in under 10 minutes without any error whatsoever. Restart puppetserver, run again: same result, clean finish in <10 minutes. But run it a second time without restarting puppetserver, and it consistently degrades: errors pile up and execution drags out to ~25 minutes. It’s fully repeatable.

Load testing on the HashiCorp Vault cluster confirms it can handle significantly more requests per second than Puppet ever throws at it — Vault’s not the bottleneck.

To mitigate the timeouts in the module, a try/catch/retry workaround mechanism was added to avoid failing the run for the agent entirely.

Is this module not suited for high volume lookups?

2025-05-07T15:07:00.206+02:00 WARN  [qtp945893216-56] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request
java.net.ConnectException: Timeout connecting to [host-vault-01/1x.x.x.x:8200]
        at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
        at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632)
        at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898)
        at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.base/java.lang.Thread.run(Thread.java:829)