Block Search Engines

Search engines are equipped with robots, also known as spiders or bots, that crawl and index webpages. If your site or page is under development or contains sensitive content, you may want to block bots from crawling and indexing your site. Learn how to block entire websites, pages, and links with robots.txt files and block specific pages and links with <meta> </meta> html tags. Read on to discover how to block specific bots from accessing your content.

Steps

Blocking Search Engines with robots.txt Files

  1. Understand robots.txt files. A robots.txt file is a plain or ASCII text file that informs search engine spiders what they are allowed to access on your site. Files and folders listed in a robots.txt file may not be crawled and indexed by a search engine spiders. You may need a robots.txt file if:
    • You want to block specific content from search engine spiders.
    • You are developing a live site and are not prepared to have search engine spiders crawl and index the site
    • You want to limit access to reputable bots.[1]
  2. Create and save and robots.txt file. To create the file, launch a plain text editor or a code editor. Save the file as: robots.txt. The file name must be all lowercase.[1]
    • Do not forget the “s.”
    • When you save the file, choose the extension “‘.txt”’. If you are using Word, select the “Plain Text” option.
  3. Write a full-disallow robots.txt file. It is possible to block every reputable search engine spider from crawling and indexing your site with a “full-disallow” robots.txt. Write the following lines in your text file:
    • Using a “full-disallow” robots.txt file is not strongly recommended. When a bot, such as Bingbot, reads this file, it will not index your site and the search engine will not display your website.
    • User-agents: this is another term for search engine spiders, or robots
    • *: the asterisk signifies that the code applies to all user-agents
    • Disallow: /: the forward slash indicates that the entire site is off-limits to bots[2]
  4. Write a conditional-allow robots.txt file. Instead of blocking all bots, consider blocking specific spiders from certain areas of your site.[2] Common conditional-allow commands include:
  5. Encourage bots to index and crawl your site. Many people want to welcome, instead of block, search engine spiders because they want their entire site indexed. To accomplish this, you have three options. First, you can opt out of creating a robots.txt file—when the robot does not find a robots.txt file, it will continue to crawl and index your entire site. Second, you can create an empty robots.txt file—the robot will find the robots.txt file, recognize that it is empty, and continue to crawl and index your site. Lastly, you can write a full-allow robots.txt file.[1] Use the code:
  6. When a bot, such as googlebot, reads this file, it will feel free to visit your entire site.
  7. User-agents: this is another term for search engine spiders, or robots
  8. *: the asterisk signifies that the code applies to all user-agents
  9. Disallow: the blank disallow command indicates that all files and folders are accessible
  10. Save the txt file to the root of your domain. After you have written the robots.txt file, save the changes. Upload the file to your site’s root directory. For example, if your domain is www.yourdomain.com, place the robots.txt file at www.yourdomain.com/robots.txt.

Blocking Search Engines with Meta Tags

  1. Understand HTML robots meta tags. The robots meta tag allows programmers to set parameters for bots, or search engine spiders. These tags are used to block bots from indexing and crawling an entire site or just parts of the site. You can also use these tags to block a specific search engine spider from indexing your content. These tags appear in the head of your HTML file.[4]
    • This method is commonly used by programmers that do not have access to a website’s root directory.
  2. Block bots from a single page. It is possible to block all bots from indexing a page and or from following a page’s links. This tag is commonly used when a live site is under development. Once the site is complete, it is strongly recommended that you remove this tag. If you do not remove the tag, your page will not be indexed or searchable via search engines.[4]
  3. Allow the bots to index a page, but not follow its links. If you allow the bots to index the page, the page will be indexed; if you prevent the spiders from following the links, the link path from this specific page to other pages will break.[4] Insert the following line of code into your header:
  4. Let the search engine spiders follow the links but not index the page. If you allow the bots to follow the links the link path from this specific page to other pages will remain in tact; if you restrict them from indexing the page, your web page will not appear in the index.[4] Insert the following line of code into your header:
  5. Block a single outgoing link. To hide a single link on a page, embed a rel tag within the <a href> </a> link tag. You may wish to use this tag to block links on other pages that lead to the specific page you want to block.[5]
  6. Block a specific search engine spider. Instead of blocking all bots from your web page, you may wish to prevent one bot from crawling and indexing the page. To accomplish this, replace “‘robot”’ within the meta tag with the name of a specific bot.[5] Examples include: googlebot, googlebot-news, googlebot-image, bingbot, and teoma.[3]
  7. Encourage bots to crawl and index your page. If you want to ensure that your page will be indexed and its links will be followed, you can insert a follow-allow meta “robot” tag into your header.[4] Use the following code:

Related Articles

Sources and Citations

  1. ↑ 1.0 1.1 1.2 https://varvy.com/robottxt.html
  2. ↑ 2.0 2.1 https://support.google.com/webmasters/answer/6062596?hl=en
  3. ↑ 3.0 3.1 https://www.elegantthemes.com/blog/tips-tricks/how-to-stop-search-engines-from-indexing-specific-posts-and-pages-in-wordpress
  4. ↑ 4.0 4.1 4.2 4.3 4.4 https://searchenginewatch.com/sew/how-to/2067564/how-to-use-html-meta-tags
  5. ↑ 5.0 5.1 https://css-tricks.com/snippets/html/meta-tag-to-prevent-search-engine-bots/
Retrieved from "https://kipkis.com/index.php?title=Block_Search_Engines&oldid=286347"