Download latest Wikidata dump

[thewillyhuman]

When downloading the latest Wikidata dumps we find that the download speed is very slow. After analyzing in detail this aspect and even running parallel download we see that the smokes are downloaded at an average speed 4-100Mbps. Depending on the time of day the speed is one or another. This implies that downloading a dump takes between 5-7h.

Running the same program to perform a concurrent download of a file from another source we get the full bandwidth, ~1.000Mbps.

Therefore we will continue to explore alternatives.

P.S: The program used for downloading is the docker image wesogroup/parallel-download-job.

[thewillyhuman]

According to Addam Shore: "After some testing, I found that the dumps.wikimedia.your.org mirror appears to have the fastest download speed." And it is still completely true. This mirror is downloading at about 800Mbps.