How will the web pages that use robots meta tag or X-Robots-Tag HTTP header be processed?
If the corresponding options are active, the pages will be processed according to the set values of robots meta (noindex, nofollow).
If necessary, you can always apply the processing of only noindex or only nofollow independently of each other.
The tags intended for certain search crawlers (for example, Googlebot) are considered when choosing the appropriate crawler identification option as a search bot.
Also, you can use meta tags on webpages for our robot, which will be taken into account when choosing the direct identification of our robots.
Examples of using robots meta tag:
<meta name="robots" content="noindex" />
<meta name="robots" content="nofollow" />
<meta name="robots" content="noindex,nofollow" />Examples of using X-Robots-Tag HTTP headers:
X-Robots-Tag: noindex
X-Robots-Tag: nofollow
X-Robots-Tag: noindex, nofollow
