.Google's John Mueller answered an inquiry regarding why Google marks pages that are prohibited from crawling by robots.txt and why the it's risk-free to disregard the similar Browse Console files regarding those creeps.Bot Website Traffic To Query Specification URLs.The person asking the question recorded that crawlers were developing links to non-existent concern parameter Links (? q= xyz) to pages with noindex meta tags that are actually also blocked out in robots.txt. What cued the question is that Google.com is actually crawling the hyperlinks to those web pages, obtaining obstructed by robots.txt (without envisioning a noindex robots meta tag) after that getting turned up in Google Explore Console as "Indexed, though shut out through robots.txt.".The individual inquired the complying with inquiry:." Yet listed here's the significant concern: why will Google.com index pages when they can't also observe the content? What's the benefit because?".Google.com's John Mueller validated that if they can not crawl the webpage they can not observe the noindex meta tag. He additionally produces an interesting reference of the web site: hunt operator, advising to ignore the outcomes because the "normal" consumers will not find those end results.He composed:." Yes, you're appropriate: if we can't creep the page, we can not see the noindex. That mentioned, if our team can not crawl the pages, after that there's certainly not a lot for our company to mark. Thus while you might view several of those pages with a targeted web site:- question, the typical user won't observe all of them, so I definitely would not fuss over it. Noindex is also fine (without robots.txt disallow), it just means the Links will certainly wind up being actually crawled (and wind up in the Search Console report for crawled/not recorded-- neither of these standings induce concerns to the rest of the website). The essential part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the restrictions in operation the Site: hunt evolved search operator for diagnostic main reasons. Among those main reasons is given that it's not linked to the frequent hunt mark, it's a separate factor altogether.Google's John Mueller discussed the web site search operator in 2021:." The short response is actually that a web site: question is certainly not indicated to be full, neither made use of for diagnostics functions.A web site inquiry is actually a certain kind of search that limits the results to a certain website. It is actually primarily simply words website, a colon, and after that the web site's domain name.This question limits the outcomes to a details site. It is actually not implied to be a thorough collection of all the webpages coming from that web site.".2. Noindex tag without making use of a robots.txt is fine for these sort of circumstances where a bot is connecting to non-existent web pages that are actually getting uncovered through Googlebot.3. URLs with the noindex tag will generate a "crawled/not recorded" item in Explore Console and also those won't possess an unfavorable effect on the remainder of the web site.Review the inquiry as well as address on LinkedIn:.Why will Google.com mark pages when they can't even see the information?Featured Image through Shutterstock/Krakenimages. com.