.Google.com's John Mueller responded to a question regarding why Google marks webpages that are forbidden from creeping by robots.txt and why the it's safe to dismiss the related Explore Console files regarding those crawls.Robot Traffic To Question Criterion URLs.The individual inquiring the question recorded that crawlers were actually producing hyperlinks to non-existent inquiry criterion Links (? q= xyz) to web pages along with noindex meta tags that are actually also blocked out in robots.txt. What triggered the question is that Google is actually creeping the web links to those webpages, acquiring blocked out through robots.txt (without envisioning a noindex robots meta tag) at that point getting shown up in Google Look Console as "Indexed, though obstructed by robots.txt.".The person asked the observing concern:." But here's the large question: why would Google mark pages when they can't also see the material? What is actually the perk because?".Google.com's John Mueller confirmed that if they can not creep the webpage they can't view the noindex meta tag. He also makes an appealing acknowledgment of the web site: search driver, advising to overlook the end results considering that the "common" users won't find those outcomes.He wrote:." Yes, you're right: if our team can't creep the web page, we can't view the noindex. That claimed, if our company can't crawl the webpages, then there is actually not a whole lot for our company to index. Thus while you might see some of those pages along with a targeted site:- inquiry, the normal user won't observe them, so I would not fuss over it. Noindex is additionally alright (without robots.txt disallow), it just implies the URLs will definitely end up being crawled (as well as end up in the Look Console report for crawled/not indexed-- neither of these conditions cause problems to the rest of the web site). The integral part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the limitations in operation the Website: search accelerated hunt driver for diagnostic explanations. Among those explanations is considering that it's certainly not linked to the routine search mark, it's a separate thing entirely.Google.com's John Mueller talked about the web site search operator in 2021:." The brief response is that a web site: question is not suggested to become complete, neither used for diagnostics objectives.A web site inquiry is actually a specific sort of search that restricts the outcomes to a specific site. It is actually primarily only the word web site, a colon, and then the internet site's domain.This query limits the results to a details web site. It's not suggested to become a thorough compilation of all the webpages from that website.".2. Noindex tag without using a robots.txt is actually great for these kinds of circumstances where a robot is linking to non-existent pages that are actually receiving uncovered through Googlebot.3. Links along with the noindex tag will generate a "crawled/not recorded" item in Browse Console and also those won't have an unfavorable result on the remainder of the site.Go through the concern as well as address on LinkedIn:.Why would Google mark web pages when they can not also observe the information?Featured Photo through Shutterstock/Krakenimages. com.