Crawl multiple torrent sites.

A little website crawler.

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

BFS webcrawler that implements Observable

Ruby Web crawler

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

Website crawler and fulltext indexer.

A simple movie converter, queue and directory crawler, all in one!

livedoor-feeddiscover performs feed autodiscovery using the livedoor Feed Discover API. livedoor Feed Discover API find a Atom/RSS feed(s) from the livedoor Reader crawler database. So, livedoor-feeddiscover do not access the target URL.

Rails Analyzer Tools contains Bench, a simple web page benchmarker, Crawler, a tool for beating up on web sites, RailsStat, a tool for monitoring Rails web sites, and IOTail, a tail(1) method for Ruby IOs.

A web crawler written in ruby