Lipperhey spider
We use the Lipperhey Spider to analyze how websites are put together, what techniques they use, how well optimized they are for search engines, their popularity in search, and what keywords they are optimized for. Our various spiders download and analyze tens of thousands of websites every day!
The Lipperhey spider is a specially developed software robot that works like this:
- when the spider fires up its engine it connects with the Lipperhey server
- the spider receives instructions and a list of websites to be analyzed
- the spider analyzes the sites on its list
- the results of the analyses are reported back to Lipperhey
Lipperhey spider User Agent
Lipperhey spiders identify themselves by means of a special user agent, making them visible in site statistics programs as robots. Our spiders can be recognized by means of the following user agent:
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)
Robot Exclusion Protocol
Lipperhey observes robot exclusion protocol. The purpose of the robot exclusion protocol is to protect portions of a site from (certain) spiders. In particular, the robot exclusion protocol is used to prevent a site from being downloaded without a request for it to be downloaded. It's a built-in precaution that helps protect a site from unwanted activities.
The robot exclusion protocol uses the robots.txt file, which is required to be placed in the root directory of a site. As an alternative to this file, META attribute robots can also be used. With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.
<meta name="robots" content="index,nofollow" />
With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.
Speed and Load
To minimize the load from a site, the Lipperhey spider calls up every 300 ms. The spider downloads not more than 250 pages per site. To save bandwidth, illustrations and files are ignored. Pages exceeding 147 Kb are not read.
