lmdb: mdb_env_set_maxreaders support

[mengfp]

Is your feature request related to a problem? Please describe.
By default, lmdb alows 126 readers. This number seems a little too small.

Describe the solution you'd like
Just like map_size:

ac = adb.Arctic('lmdb://path/to/desired/database?map_size=2GB')

ArcticDB should support maxreaders to be set in some way like:

ac = adb.Arctic('lmdb://path/to/desired/database?maxreaders=1024')

[phoebusm]

Hi @mengfp greetings from the team!
Would you please illustrate the scenario which you find the setting as the bottleneck?

[mengfp]

@phoebusm Thanks for the reply.

As a researcher, I'm doing backtest for quantitative trading strategies using python multiprocessing pool. My computer has 64 CPU cores with 128 threads. If I create 128 processes reading Arctic at the same time, I get error messages and the program hangs:

[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] epoll_create: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] epoll_create: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
[warn] epoll_create: Too many open files
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files

Even when I create 64 processes, it works the first time and fails when running again. When I set the number of processes to 56, it works fine.

[phoebusm]

@mengfp thanks for the test. I'll open a ticket on our internal kanban.

Would you please try fork by spawn? It's a long shot but it's worthing trying I think

[mengfp]

@phoebusm Yes, 'spawn' works. Thanks a lot.