A web crawler using Ruby and Redis.

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

Rack middleware that executes javascript before serving pages to crawlers.

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Simple Web Crawler

Simple Twitter crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

check how many links are available inside the website

livedoor-feeddiscover performs feed autodiscovery using the livedoor Feed Discover API. livedoor Feed Discover API find a Atom/RSS feed(s) from the livedoor Reader crawler database. So, livedoor-feeddiscover do not access the target URL.

image download from instagram

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

Multithreaded web crawler with transparent DSL and requests caching.

A client for the PageMunch web crawler API

Pantopoda is a web crawler that visits all links on a given domain that's fast and effective.

Bulbasaur is a helper for crawler operations used in Pread.ly