In trying to crawl my site in Google Search Console, I see the following error on each and every one of my pages:
I updated the X-Robots-Tag to have the following: < X-Robots-Tag: usasearch all; googlebot all; none and verified that this is live with Google's Robots Testing tool that Googlebot and Googlebot-Mobile are both allowed. Also confirmed here is what a server header checker tool shows:
Final response
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=300
< Vary: Accept-Encoding,
< Cache-Control: must-revalidate, no-cache, private
< Date: Fri, 27 Sep 2019 21:04:41 GMT
< X-Drupal-Dynamic-Cache: UNCACHEABLE
< Link: ; rel="shortlink", ; rel="canonical"
< X-UA-Compatible: IE=edge
< Content-language: en
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN, SAMEORIGIN
< Expires: Sun, 19 Nov 1978 05:00:00 GMT
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
< X-Robots-Tag: usasearch all; googlebot all; noneYet no matter how many times I use the URL Inspect tool I get the same error pictured above. Not sure what else I need to do. Should I just wait a while since the change was done recently?
Any suggestions?
2 Answers
I also faced the issue with my website once. At that time I was on blogger. I remember editing the theme and removing noindex code part from it. The problem was resolved. However, I am now on Wordpress.
It appears as though Google defaults to the most restrictive tag in X-Robots-Tag. In this case none. After removing that attribute, the search console was able to reach and index that page correctly.
