Add a mechanism to allow package maintainers to opt-out of search engine indexing of documentation

[daveverwer]

The swift-testing documentation is now hosted at both https://swiftpackageindex.com/swiftlang/swift-testing/documentation/testing and also https://developer.apple.com/documentation/testing/. This is intentional as the developer.apple.com documentation is for Apple’s private version of the package and the SPI version covers the open-source package.

There are other examples too, like https://swiftpackageindex.com/apple/swift-system/documentation/systempackage and https://developer.apple.com/documentation/system. There are almost certainly more Apple packages in this situation.

There is value in keeping the open-source package documentation available in addition to Apple’s hosted version. However, the reality is that the differences between the two documentation archives will be subtle and the content will likely be flagged as duplicate by Google and other search engines, harming the SEO reputation of both SPI and Apple’s developer site.

In situations like this, most developers are probably looking for the Apple-hosted documentation. So, we should make a list of packages that have their documentation flagged to be noindex to avoid the issue.

We could implement a flag in .spi.yml manifests where package authors can opt their package’s documentation out of being visible to search engine crawlers. We already have the mechanism to collect that data easily and can check requests before serving the documentation. We can then submit a few pull requests for the Apple docs to take those cases out specifically.