Lipperhey uses its own developed spider to analyze how websites are put together, what techniques they use, how well optimized they are for search engines, their popularity in search and what keywords they are optimized for. Our spiders download and analyze tens of thousands of websites every day!
The Lipperhey spider is a specially developed software robot that works like this:
Lipperhey spiders identify themselves with a user agent, making them visible in logs and site statistics programs. Our spiders can be recognized by the following user agent:
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)
Lipperhey obeys the robot exclusion protocol. The purpose of the robot exclusion protocol is to protect portions of a site from (certain) spiders. In particular, the robot exclusion protocol is used to prevent a site from being downloaded without a request for it to be downloaded. It's a built-in precaution that helps protect a site from unwanted activities.
The robot exclusion protocol uses the robots.txt file, which is required to be placed in the root directory of a site. As an alternative to this file, META attribute robots can also be used. With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.
<meta name="robots" content="index,nofollow" />
With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.
To minimize the load from a site, the Lipperhey spider calls up every 300 ms. The spider downloads not more than 250 pages per site. To save bandwidth, illustrations and files are ignored. Pages exceeding 125 Kb are not downloaded.