cobweb

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Rubygem cobweb

Total Downloads
70165
Releases
81
Current Version
1.0.19
Released
2013-11-26 00:00:00 UTC
First Release
2010-11-10 00:00:00 UTC
Depending Gems
2

Github stewartmckee/cobweb

Watchers
74
Forks
19
Development activity
Less active
Last commit
2014-01-17 11:32:48 UTC
Top contributors
Contributors
5
Issues
Wiki pages

Resources

There are no resources yet, be the first to submit one!

Comments

There are no comments yet, be the first to write one!

Post a comment

Markdown supported
or
×

In order to continue, you must be signed in using your Github account.

If you're signing in using this account for the first time Github will ask for your permission to give access to your public user data to the Ruby Toolbox.

Although the Github Authorization page does not mention it, the request includes read-only access to your verified email address (user:email OAuth scope). This is neccessary so there's a way to notify you about comments, information about your accepted project edits and the like. You can review your notification settings on your account page once you're signed in.