Changing max_threads does not take effect

[sirhc-chris]

We have a large number of rules so I want to increase the number of threads available to process them. Increasing max_threads from the default of 10 to 30 (or any other value, including 1), and restarting the container, does not result in a change of behavior - I continue to see the same warning periodically logged suggesting the thread pool size is unchanged:

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: foo.bar.m. Connection pool size: 10

max_threads is being set in the main config.yaml mapped to /opt/elastalert/config.yaml within the container (not within the rule config) - an example config:

rules_folder: /opt/elastalert/rules

run_every:
  seconds: 10

buffer_time:
  minutes: 15

es_host: foo.bar.m
es_port: 9200

writeback_index: elastalert_status

alert_time_limit:
  days: 1
  
max_threads: 30

ElastAlert2 version: 2.23.0
Elasticsearch version: 8.15.2
Environment: Kubernetes (direct, not Helm) + reproduced in local Docker

[jertel]

max_threads is used for the worker thread pool. The log message above is referencing a connection pool. Without more log data I can't be sure which connection pool it is.

[sirhc-chris]

I created a small app using the requests lib to reproduce this locally. The connection pool size we're seeing in the log is defined by the pool_maxsize parameter on the urllib3 HTTPAdapter, and that parameter is set by connections_per_node here - so we have the correct parameter, we're just not overriding it in the correct place...

[sirhc-chris]

the only change I've had success with is to override DEFAULT_POOLSIZE here...

[jertel]

Based on what you've found it seems to indicate the issue is in the Elastic driver not passing that connections_per_node value all way to the connection pool class.

[jertel]

I see that Elastic 8.x has this change:

Changed Urllib3HttpNode(connections_per_host) parameter to connections_per_node

ElastAlert 2 is still using the 7.x lib due to compatibility reasons (search the PR/issues/discussions if interested in details on that) so try using that connections_per_host to see if that makes a difference.

[sirhc-chris]

thanks, unfortunately connections_per_host doesn't make a difference