Google will no longer support Robots.txt Noindex directive

We have seen that many websites are not comfortable in indexing a link on Google Search or any other search engine. Now, there are various reasons for doing so such as bad links, negative SEO impact etc. Therefore, what site owners do is that they place a noindex tag after the link in their robots.txt file which means that the Googlebot or any other search engine crawler does not index that link.

However, Google has just made a big announcement and it might concern you if you are using noindex links on your website. Google says that GoogleBot will not obey a Robots.txt directive which is related to indexing any longer. This means that it does not matter if you place a noindex tag or not in your robots.txt file, GoogleBot will still crawl the links.

The company says that website owners have time until September 1, 2019 in order to remove the noindex tag and if they want to not index the links, they will have to use an alternate directive. Now, we also have the reason why this directive is cancelled by Google. Recently, Google officially submitted a proposal to make Robots.txt the official directive for website crawlers.

For this reason, the noindex directive has been dropped because it is unofficial. In the past, Google supported this directive because of Robots.txt was not the official method of crawling but since it is now an official method, any unofficial directive cannot be included.

Here is what Google announced:

Today we’re saying goodbye to undocumented and unsupported rules in robots.txt

If you were relying on these rules, learn about your options in our blog post.

In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.

Source