Ruby Web crawler

A ruby social media stat crawler

A web crawler using Ruby and Redis.

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Crawl multiple torrent sites.

A ruby based zhihu content crawler.

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-char-detector gem provides charset detection support.

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.