Repository navigation
Support for multiple X-Robots-Tags聽#32102
Description
How to use GitHub
- Please use the 馃憤 reaction to show that you are interested into the same feature.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
Is your feature request related to a problem? Please describe.
The Nextcloud Overview page issues a warning when the X-Robots-Tag is not set exclusively to 'none'.
The problem is 'none' is not supported by Bing. This causes multiple Nextcloud instances to be indexed by Bing, even though this is probably not intended.
One example would be https://cloud.dresden.de/. The page is easily found by searching in Bing and it does have the X-Robots-Tag set to 'none'.
Describe the solution you'd like
One solution would be to keep the requirement for the 'none' tag but allow more tags. This way X-Robots-Tag could be set to 'none,noindex,nofollow'. This way the instance should not be indexed by Bing as well as other search engines.
Describe alternatives you've considered
The alternative currently is to ignore the warning, which is kind of annoying.
(A way to disable warnings would also be appreciated.)
Activity
ozboss commentedon Apr 23, 2022
Well this issue might have been created prematurely, after looking closer these pages also appear on the Google search results. But without any information ... just like on Bing. So apparently pages with a lot of access still appear in the search results.
Is the 'none' tag then supported by Bing? I found only information on it not being supported. But apparently the behavior is the same as on Google and it does not actually stop it from showing up in search results.
kesselb commentedon Apr 23, 2022
I guess it's fine to change
nonetonoindex,nofollow.https://developers.google.com/search/docs/advanced/robots/robots_meta_tag
https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240
Feel free to submit a pr.
MichaIng commentedon Feb 13, 2023
Wow, this explains a lot, never know that Bing does not support
none. Yes makes much sense then to change it.Btw, Nextcloud also sets the HTML meta tag
So actually one should be safe also for Bing. I'll open a report.
MichaIng commentedon Feb 13, 2023
PR up: #36689
It does not exactly address the request here to support multiple values, i.e. for not unsetting but appending the values via
.htaccess(?), but for a Nextcloud instance these are the only correct values I'd say, and for Google equivalent than before.