@Roboshield update list of crawlers to block on robots.txt

[thepsalmist]

Comparing to other sites, we may be missing some crawlers
The Guardian, Washington Post, BBC, etc.

Originally posted by @kilemensi in CodeForAfrica/ckanext-openafrica#55 (comment)