Document the ulimit changes needed for running fleet-server in larger environments!

[jdixon-86]

Describe the enhancement:
Please document the required changes for the ulimit settings, especially on Ubuntu in order to run the fleet server in larger environment. If people do not do this then their fleet servers can stop responding due to having too many open files

Describe a specific use case for the enhancement or feature:
What I have ran into with nearly 4,000 agents and two fleet servers behind a load balancer is hitting a open file limit in Ubuntu with the fleet server. Currently there are no documented ulimit changes for fleet server but there is for Elasticsearch, however the Elasticsearch changes do not apply because Elasticsearch runs under a different user (named elasticsearch) and fleet-server will run under the user root.

To see the current limitation find the PID of the fleet-server process:
pidof fleet-server

Then run the following to see the output of the SOFT limitation:
prlimit -n --pid=[pid]

What you will notice is the soft limitation will still remain at 1024 even if you modify the /etc/systemd/system.conf file.

RESOURCE DESCRIPTION                SOFT   HARD UNITS
NOFILE   max number of open files 1024 524288 files

In order to fix this you must edit /etc/systemd/system.conf and change the following line to something higher:
DefaultLimitNOFILE=262144:524288

Editing the /etc/pam.d/login file to include the pam_limits.so does not fix the SOFT limit problem. This should be properly documented so time is not wasted. Ticket 00975556

[AndrewMcQuerry]

👍 Thanks @jdixon-86. This information just saved our bacon in our "large" environment.

Would love if the documentation could be updated to account for this (or at least someone from Elastic reply with an alternative work-around editing /etc/systemd/system.conf is not required.

[jlind23]

@kilfoyle isn't this something we should rather transfer to the ingest-doc repository?
cc @nimarezainia

[kilfoyle]

@jlind23 Certainly, this can be transferred to the ingest-docs repo.

One request: Can we please have a developer assigned to it along with me? I'd really need more detail on exactly what to change in the docs. I don't want to accidentally give incorrect advice, and it seems like something we may need to test before documenting.

[cmacknz]

This might be something we can add to Fleet Server so it does it automatically using https://pkg.go.dev/syscall#Setrlimit, which should work as long as Fleet Server is privileged/root or has CAP_SYS_RESOURCE.