HTML Meta Information Robots
Example
The robots attribute, supported by several major search engines, controls whether search engine spiders are allowed to index a page or not and whether they should follow links from a page or not.
<meta name="robots" content="noindex">
This example instructs all search engines to not show the page in search results. Other allowed values are:
| Value/Directive | Meaning |
|---|---|
all | Default. Equivalent to index, follow. See note below. |
noindex | Do not index the page at all. |
nofollow | Do not follow the links on this page |
follow | The links on the page can be followed. See note below. |
none | Equivalent to noindex, nofollow. |
noarchive | Do not make a cached version of this page available in search results. |
nocache | Synonym of noarchive used by some bots such as Bing. |
nosnippet | Do not show a snippet of this page in search results. |
noodp | Do not use metadata of this page from the Open Directory project for titles or snippets in search results. |
notranslate | Do not offer translations of this page in search results. |
noimageindex | Do not index images on this page. |
unavailable_after [RFC-850 date/time] | Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format. |
Note: Explicitly defining index and/or follow, while valid values, is not necessary as pretty much all search engines will assume they are allowed to do so if not explicitly prevented from doing so. Similar to how the robots.txt file operates, search engines generally only look for things they are not allowed to do. Only stating things a search engine isn't allowed to do also prevents accidentally stating opposites (such as index, ..., noindex) which not all search engines will treat in the same way.
