Optimized crawling for faster updates

[barryvdh]

Sorry, I know this issue is asked before and the main reasons are:

• updating is manual, takes time
• crawling is not 'official' so don't want to overload Magento with crawl requests.
• crawl script is not public to avoid above.

But without seeing what you already do, can't you do something similar to this:

• Get package list from https://connect20.magentocommerce.com/community/packages.xml
• Compare versions with last checked version, if newer, load package information from https://connect20.magentocommerce.com/community/AW_Blog/releases.xml
• store new versions in Satis, based on the URl download scheme: https://connect20.magentocommerce.com/community/AW_Blog/1.3.18/AW_Blog-1.3.18.tgz

You could run that daily without stressing the server for static xml files, when you just update the handful extensions that are updated, instead of all in one time.
(or you could even use the package list version directly, although that could miss versions..)

Don't mean to be rude, all respect for volunteering your time, but it could be a security issue, when packages are not updated here for months, when they are patched on Connect. Last update now was from June..

Also, to avoid questions, perhaps add some info about this on the Readme :)