Insane number of views

[jcm4atx]

I deployed Zipline and almost immediately had an insane number of views. I figured out it was webcrawlers.

I use Apache for my reverse proxy. I added the following lines to my vhosts:

	ProxyPass /robots.txt !
	Alias /robots.txt /var/www/robots.txt

And then created /var/www/robots.txt with the following text:

User-agent: *
Disallow: /

Now my percent views is at 0%. If you don't want anyone crawling your site I suggest doing this before you start uploading files.

[diced]

I think we could integrate this within Zipline, where it would disallow any file paths. This solution is fine for now though.

[diced]

Added in 39a8d52

FEATURES_ROBOTS_TXT=true, will automatically adapt to your uploader/url routes, etc.