Customize Sitemap Generator
For websites that work on the CMS with embedded systems to restrict access
Please note that the indexing process, the scanner sends a large number of requests to your website. Some of the CMS with appropriate settings can block our crawler requests for reasons of safety or load on a Web server. Recommend that you remove such protection at the time indexing of your site. If such an opportunity you do not - use a minimum speed of indexing, which will minimize the load on your server.
Selecting the optimum speed of indexing and the level of burden on your web server
In the options of the scanner are three levels of speed indexing, creating appropriate levels of load on the servers to be indexed:
- High - This level of load using a scanner by default. If you have a quality hosting, chances are you need not worry about the stress created in the process of indexing your site. Recommend using a load value, which allows the bot to crawl your site for maximum speed.
- Average – Use this load level, if your server requires a gentle treatment of indexation.
- Low – The level of load limiting, which allows you to index your site by creating a minimal load on the server. This load level recommends to index sites that are on free hosting, or for sites requiring restriction of traffic flows.
However, please note that this level will slow down the process of indexing your site.
Indexing hidden pages (Deep Web)
Deep Web – Web pages indexable by search engines due to the lack of hyperlinks on them with the available pages. For example - this page was generated through HTML-forms interface or the contents of frames.
If you want to find and include these pages in a Sitemap, check the options:
- Index forms - to crawl website forms (submit occurs without filling).
- Index frames - to crawl <frameset> and <iframe>.
- Index Javascript - to crawl some types of Javascript links.