Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

Gem for crawling data from external sources

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

This gem helps Crawler Writers to interact with the PromoQui REST API

Generic Web crawler with a DSL that parses structured data from web pages

Show DMM and DMM.R18's crawled data. e.g. ranking

Asynchronous web crawler, scraper and file harvester

CrawlerDetect is a library to detect bots/crawlers via the user agent

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

Cangrejo lets you consume crabfarm crawlers using a simple DSL

Ruby web crawler using PhantomJS

This is a crawler framework.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Simple little website crawler.

An easy to use distributed web-crawler framework based on Redis

Voight-Kampff detects bots, spiders, crawlers and replicants

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

Ruby web crawler to access omelete informations

A simple, fast web crawler