Lipperhey Spider

Lipperhey spider

Lipperhey uses its own developed spider to analyze how websites are put together, what techniques they use, how well optimized they are for search engines, their popularity in search and what keywords they are optimized for. Our spiders download and analyze tens of thousands of websites every day!

The Lipperhey spider is a specially developed software robot that works like this:

  • when the spider fires up its engine, it connects with the Lipperhey server
  • the spider receives instructions and a list of websites to be analyzed
  • the spider analyzes the sites on its list
  • the results of the analyses are reported back to Lipperhey

Lipperhey spider User Agent

Lipperhey spiders identify themselves with a user agent, making them visible in logs and site statistics programs. Our spiders can be recognized by the following user agent:

								Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)
							

Robot Exclusion Protocol

Lipperhey obeys the robot exclusion protocol. The purpose of the robot exclusion protocol is to protect portions of a site from (certain) spiders. In particular, the robot exclusion protocol is used to prevent a site from being downloaded without a request for it to be downloaded. It's a built-in precaution that helps protect a site from unwanted activities.

The robot exclusion protocol uses the robots.txt file, which is required to be placed in the root directory of a site. As an alternative to this file, META attribute robots can also be used. With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.

								<meta name="robots" content="index,nofollow" />
							

With META attribute robots, a robot can be instructed whether or not to index a given webpage, and whether or not to follow the links to another page.

Speed and Load

To minimize the load from a site, the Lipperhey spider calls up every 300 ms. The spider downloads not more than 250 pages per site. To save bandwidth, illustrations and files are ignored. Pages exceeding 125 Kb are not downloaded.


Sign up today. It's free!

Last week, over websites were improved with Lipperhey.

Get started Or take a tour →