How to use of ROBOTS meta tag?
What's ROBOTS meta tag?
We know that search engines handle meta tags and the ROBOTS is a part of META tags. The HTML META ROBOTS Tag help and talk to search engines crawlers. The standard and valid values for the CONTENT attribute are just "INDEX", "NOINDEX", "FOLLOW" and "NOFOLLOW but the valid meta robots content values fo G-BOT are:
NOINDEX: prevents the page from being included in the index.
NOFOLLOW: prevents Googlebot from following any links on the page.
NOARCHIVE: prevents a cached copy of this page from being available in the search results.
NOSNIPPET: prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
NOODP: blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
NONE: equivalent to "NOINDEX, NOFOLLOW".
FOLLOW: a command for crawler to follow the links in that webpage.
INDEX: a command for crawler to index that webpage.
INDEX, FOLLOW: Follow The search engine robots will start crawling your website from the main/index page and then will continue to the rest of the pages.
<meta name="robots" content="index, follow">I know that "INDEX, FOLLOW" is unecessary content values and by default, Google-BOT will index a page and follow links to it and there's no need to have a tag with content values of INDEX or FOLLOW but i want to sure about G-BOT and other BOTs. You can find list of all BOTs from here.