deal with different web properties that share the same terms

[michielbdejong]

For instance when visiting cnet.com you agree to the cbsinteractive.com privacy policy. Should we crawl that as rules/cnet.com.xml or as rules/cbsinteractive.com.xml? I opted for the first now, but this issue deserves more attention.

[michielbdejong]

also for instance apple-icloud is a separate services, but on tosback its terms are crawled as part of apple.com