Mysitemapgenerator / Help center / Website Crawling

Does the crawler handle canonical links (rel=canonical)?

Yes, for this it is enough to mark the corresponding option "Follow & consolidate canonical URLs". When the appropriate option is activated, the crawler will take into account the instructions of canonical links, and noncanonical links will be excluded from crawl results.
Our crawler handles the instructions in the HTML code, as well as the HTTP headers in the same way. An example of specifying a canonical link in HTML (placed in the <head> section of the noncanonical version of the page): Canonical meta tag example:

    <link rel="canonical" href="http://www.website.tld/canonical_page.html"/>

An example of specifying a canonical reference using an HTTP header:

    Link: <http://www.website.tld/canonical_page.html>; rel="canonical"

Pay attention to the technical aspect of processing canonical links by our crawler: the reference to the canonical page is equal to a server-side redirect (HTTP 303 See Other) and is processed in accordance with the general rules of the redirection processing.