Voight-Kampff detects bots, spiders, crawlers and replicants

CrawlerDetect is a library to detect bots/crawlers via the user agent

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Gem for crawling data from external sources

Generic Web crawler with a DSL that parses structured data from web pages

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

This gem helps Crawler Writers to interact with the PromoQui REST API

Ruby web crawler using PhantomJS

Asynchronous web crawler, scraper and file harvester

Cangrejo lets you consume crabfarm crawlers using a simple DSL

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

Simple little website crawler.

Show DMM and DMM.R18's crawled data. e.g. ranking

An easy to use distributed web-crawler framework based on Redis

Ruby web crawler to access omelete informations

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Website crawler and fulltext indexer.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.