Inflation of the latest bz2 database

[laem]

Hi, as it can be seen on the following link, the database archive gained multiple Go lately.

https://download1.graphhopper.com/public/

Is it normal ? Does it result from the addition of OSM attributes to the set ?

[laem]

The download takes around 50 hours 😅

[lonvia]

This will now be the new normal. I've just enabled heavy rate limiting on the server. We usually had about 10TB of download volume per month. Since beginning of this year this has increased to >150TB! That's more than the server can handle, so rate limiting it is to ensure a level playing field for everybody.

Please think carefully before deciding to download a new dump every week. For most use cases it is not necessary to be up to date with the latest changes.

If you want weekly updates faster, try the new json dumps at https://download1.graphhopper.com/public/raw/ . They will download in only 4 hours and a json import then takes about half a day. Altogether much less time than waiting for the full dump to download.

[laem]

Alright, I understand. Last week, I asked the question on Graphhopper's forum about another way to download the data using a less costly solution, what do you think ?

https://discuss.graphhopper.com/t/download-only-incremental-changes-of-photon-db-file/9266/3?u=cartesapp

If you want weekly updates faster, try the new json dumps at https://download1.graphhopper.com/public/raw/ . They will download in only 4 hours and a json import then takes about half a day. Altogether much less time than waiting for the full dump to download.

I did not know about that ! Yes, for cartes.app I'd like at weekly updates at least. In the long term, we'll derive our own daily index but we're not there yet. I'll investigate using these json dumps.

[laem]

Also, my server could be used as a realy for downloads in case it helps :) Torrents could also be a solution but needs more setup.

Have you heard of https://source.coop ? It looks to me that it could be a cool relay.

[lonvia]

That is a fallout of the reorganisation of the address import in #897. There is more information saved in the context now. Needs some consideration what can be trimmed.

[laem]

Actually, after trying for the third time this week, I have the impression that something's wrong downloading the archive, decompressing it and launching Photon.

The most probable I believe is that the download fails. I've raised an issue on Graphhopper's forum https://discuss.graphhopper.com/t/download-only-incremental-changes-of-photon-db-file/9266/2

The archive has about the right size in Go, but the md5sum is different.

There must be a way of downloading the file in less than 50 hours, it makes iteration impossible for me to solve this matter.

[laem]

Trying wget -c mode to check if the download was interrupted, it looks like it did not.

As can be seen in this capture, the decompression fails.

[mtmail]

the md5sum is different
So the download broke. I can download the file in 3h (up to 14MB/sec) but maybe I (my server) is closer to the download server than yours.

[laem]

I've just tried from my local machine in Wifi, I get max 2Mb but fluctuating. This is strange, distance should not be a limitation I guess.
Scaleway is a big cloud provider.

[laem]

Thanks for your input