Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Simple web crawler to crawl a domain and generate sitemap

plugin-container totally sucks..

A simple web crawler gem

Empower World Travel Information Technology

Empower World Travel Information Technology

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

The Baidu Crawler is to crawl data with your demmand

株価情報を取得してあれこれするライブラリ

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

A ruby based zhihu content crawler.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

s7o crawler optimized for programmer happiness and sustainable productivity.

Generic Web crawler with a DSL that parses event-related data from web pages

A simply web crawler.

Crawler for http://legendas.tv to see the most dowloaded subtitles

check how many links are available inside the website

A web crawler using Ruby and Redis.