You would use a 'noindex' meta tag instead of disallowing a URL in robots.txt when you need Googlebot to crawl a page to discover the 'noindex' tag. The robots.txt file prevents crawlers from accessing a page entirely. A 'noindex' meta tag allows crawlers to access the page but instructs them not to include it in Google's index. The key difference is that with robots.txt,....
Log in to view the answer
