HTML Meta Information Robots


The robots attribute, supported by several major search engines, controls whether search engine spiders are allowed to index a page or not and whether they should follow links from a page or not.

<meta name="robots" content="noindex">

This example instructs all search engines to not show the page in search results. Other allowed values are:

allDefault. Equivalent to index, follow. See note below.
noindexDo not index the page at all.
nofollowDo not follow the links on this page
followThe links on the page can be followed. See note below.
noneEquivalent to noindex, nofollow.
noarchiveDo not make a cached version of this page available in search results.
nocacheSynonym of noarchive used by some bots such as Bing.
nosnippetDo not show a snippet of this page in search results.
noodpDo not use metadata of this page from the Open Directory project for titles or snippets in search results.
notranslateDo not offer translations of this page in search results.
noimageindexDo not index images on this page.
unavailable_after [RFC-850 date/time]Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format.

Note: Explicitly defining index and/or follow, while valid values, is not necessary as pretty much all search engines will assume they are allowed to do so if not explicitly prevented from doing so. Similar to how the robots.txt file operates, search engines generally only look for things they are not allowed to do. Only stating things a search engine isn't allowed to do also prevents accidentally stating opposites (such as index, ..., noindex) which not all search engines will treat in the same way.