webget gem - a web (go get) crawler incl. web cache

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

A crappy crawler for a crappy bank interface

s7o crawler optimized for programmer happiness and sustainable productivity.

This plugin need set the initial url.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

Crawl the senegalese web, looking for jobs using the excellent wombat gem

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

A simple movie converter, queue and directory crawler, all in one!

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered