This brief tutorial shows students and new users how to stop or prevent search engines and other web crawlers from crawling and indexing WordPress websites easily.
There are many reasons why one might want to stop search engines from crawling or indexing WordPress websites.
One might be to prevent indexing of website content while the site is still under development. You probably don’t want Google to index your under construction or maintenance mode page while the site is still under construction.
Another reason might be to prevent search engine from indexing or listing sensitive information that should only be hosted on your sites.
What ever the case, it’s easy to configure WordPress sites to instruct search engine not to crawl or index content from a site you don’t want to list online.
When you’re ready to prevent search engines from crawling WordPress, follow the steps below:
Discourage Search Engine from Indexing Site
WordPress comes with a built-in feature that allows this to be done easily. It instructs all search engines not to index the site and all pages produced on the site.
Although the instruction to these crawlers is not to index or list your content, it’s up to the crawlers and bots to respect and honor the site’s request. Some bots online will still crawl and list your content regardless of what you no index policy states.
To instruct all bots not to index your site, go to do is visit Settings » Reading and check the box next to Search Engine Visibility option as shown in the image below:
Click Save Changes button and you’re done.
When you enable the settings above, WordPress will add the line below into your website’s header file.
<meta name="robots" content="noindex,follow" />
WordPress also modifies your site’s robots.txt file and add these lines to it:
User-agent: * Disallow: /
These lines ask robots and web crawlers not to index your pages.
Again, it is totally up to search engines to accept this request or ignore it. Most search engines respect will respect you request and this may not totally stop some of your pages to get indexed. It’s just how the web works.
Password Protection
A sure fire way to make sure search engines don’t index or crawl your website is to password protect the entire site at the server level. Password protecting your sites makes it so that another who wants to view the content will have to enter the correct password.
We have shown you how to use Apache2 or Nginx HTTP server to password protect directories and content.
You may also install Password Protected plugin to do that for you.
To view our posts on protecting Apache2 or Nginx directory, click the links below:
Apache2: Password protect directories using Apache2
Nginx: Password protect directories using Nginx
These steps may help you protecting your sites from search engines, bots and other crawlers from listing or indexing your content.
Hope this helps
You may also like the post below:
