NUTCH-3087 BasicURLNormalizer to keep userinfo for protocols which might require it

[sebastian-nagel]

• strip the userinfo from the authority only for HTTP and HTTPS

[HiranChaudhuri]

Does it make sense to decide stripping authority data based on the protocol? I acknowledge most users want to scan the internet anonymously. But intranets or users interested to index 'their' content, be it on local or remote servers will need authority data to be preserved while they have no control over the protocol. Thus I suspect sometimes it may be required even though https is used.

How about making it configurable, maybe via regexp? This would allow Nutch users to define the protocol, or the site or ... where to preserve the authority.

[sebastian-nagel]

intranets or users interested to index 'their' content, be it on local or remote servers will need authority data to be preserved

@HiranChaudhuri, I understand your argument. Thanks!

However, let's keep it simple here. If the authority parts are really required, the simple solution would be to disable the basic URL normalizer by removing the plugin from the plugin.includes. When crawling the intranet or a specific site, strict URL normalization is less a requirement than for a broad web crawl. Of course, this means you need to have a separate configuration for the intranet crawl. But from my experience, this is often already necessary because of other specific configuration options, e.g. a different revisit schedule, etc.