Better main loop (in Scrapy?)

[gingerbeardman]

Currently the main loop runs in a shell script, a file of URLs is consumed by a loop that runs Scrapy once per URL.

A better approach, which I failed to get working, would be to run scrape once and manage the loop and consumption of URLs in there. This should increase performance/speed considerably.