scripts/biorxiv_occurences.py takes too long to run

[cruddybanana1102]

Hi, I'm Parth,

I have been trying to reproduce the results of your paper, but the scripts/biorxiv_occurrences.py seeems to take infinity to process.

A major bottleneck is the for loop(in line 18), which calls upon scripts/biorxiv-scraper.R (as seen from line 21).
For perspective: the for loop(in line 18) does ~1M iterations, and I have only processed <1000 iterations in a span of 48 hours. I want to know if there is a serious issue in how I'm running the scripts?(I'm only following the README.md) or is it just a very slow server.

[seasonyao]

Hi Parth,

Unfortunately, the processing speed of this step is really slow at present, maybe you can process it into multiple threads or divide the phrases into multiple groups to run separately.

[borisveytsman]

I suspect the bottleneck is the biorxiv server; it might be slow...