Rails Analyzer Tools contains Bench, a simple web page benchmarker, Crawler, a tool for beating up on web sites, RailsStat, a tool for monitoring Rails web sites, and IOTail, a tail(1) method for Ruby IOs.

Website crawler and fulltext indexer.

A web crawler written in ruby

livedoor-feeddiscover performs feed autodiscovery using the livedoor Feed Discover API. livedoor Feed Discover API find a Atom/RSS feed(s) from the livedoor Reader crawler database. So, livedoor-feeddiscover do not access the target URL.

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

A simple movie converter, queue and directory crawler, all in one!

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

Ruby Web crawler

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

BFS webcrawler that implements Observable

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

A little website crawler.

A simple directory crawler DSL.

Crawl multiple torrent sites.

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

Crawl websites

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.