1

I'm currently working on a webscraper that is going to be processing about 2 million pages in total. But I'm not entirely sure how to benchmark it.

Currently it processes about 100,000 pages in an hour at about 2mb/s. It isn't saturating the the bandwidth of the server but I've tried a lot of ways to make it quicker but to no avail. CPU is running 50% or so on average.

4 Answers4

5

Whose pages are you scraping, and for what reason?

You're talking about roughly thirty page requests per second, or 2 MB/s. You say it's not saturating the bandwidth of "the" server - is that your server or theirs? Saturating the bandwidth of somebody else's server is generally considered to be a denial of service attack.

Two megabytes a second is a lot of somebody else's bandwidth to be using. I couldn't do that from home, as my broadband connection is simply not that fast. Some web hosting services have bandwidth caps, meaning you'd be shutting people's sites down for a certain period of time.

If the pages you're scraping belong to somebody big enough not to mind this, check to see if they've got an API you can use instead. If they don't, consider cutting back so you're not perceived as an attacker.

David Thornley
  • 20,270
  • 2
  • 56
  • 83
3

What is a good speed for a webscraper?

That's impossible to answer. It depends on many things; e.g.

  • How rate at which the site you are scraping can (or is willing to) handle your requests.
  • The effective network latency and bandwidth when fetching web pages.
  • How much work your machine has to do to process each scraped page and record the information you extracted.

If you are running at 50% CPU, you may be able to do better, or you may not. For instance, if your requests are being throttled / limited by the remote server, or if that's as fast as it can go, then you are stuck. If you are running into network latency / bandwidth issues then you may need to upgrade your network connection ... or move to a data centre that is nearer the site you are trying to scrape.

Stephen C
  • 25,178
2

That means you can get a fresh copy of every page roughly every day? Is that fast enough for you? If it's not, profile and improve.

dan_waterworth
  • 7,297
2

Seems like you need to go to the next level and do "Distributed" scraping.

You are going to need to build your application to be able to scale out to multiple servers.

The challenge will be somehow managing the the separate notes in your scraping "Grid".

You could look at centralised head server that dispatches work out to the nodes.

Other things to avoid would be two or more servers scraping the same site (this is duplication that is simply not needed).

Darknight
  • 12,199
  • 1
  • 39
  • 58
  • 1
    +1: After the software's been profiled for bottlenecks and there really is no obvious way to make it go faster, this will be an excellent solution. – doppelgreener Dec 19 '10 at 08:02
  • you could use consistent hashing to work out which node should crawl which pages. – dan_waterworth Dec 19 '10 at 08:51
  • 1
    Be careful of the fine line between distributed scraping and distributed denial of service. – David Thornley Mar 29 '11 at 13:58
  • @David Thornley: Which line ? When you are talking about distributed scraping and distributed is between "", it's pretty clear it's not scraping anymore. Even if it is, the server getting scraped will be on it's knees for all pratical purposes. – Gilles Apr 05 '11 at 12:56