MySitemapGenerator's Bot

Mysitemapgenerator's bot is a web crawler that powers MySitemapGenerator web services. This software is used by our users to crawl their websites to generate special XML files that help improve websites' search engine visibility or set up ad campaigns.

To prevent excessive load on a web server, crawling intensity is strictly limited.

Mysitemapgenerator's bot respect Robots Exclusion Standard by default.

How to Сontrol MySitemapGenerator's Bot on Your Website?

To control how our crawler interact with your website, you have two options:

Robots.txt file
X-Robots-Tag HTTP header or HTML meta tag

Which user agent string should be used for this?

In robots.txt:

    User-agent: Mysitemapgenerator

In meta-data:

    <meta name="Mysitemapgenerator" />

In HTTP headers:

    X-Robots-Tag: Mysitemapgenerator