Add support for Cassandra in Polipus crawler

A web crawler written in ruby

Friendly, neighborhood web crawler for quality assurance.

A simple crawler to list SESC events schedule on Terminal

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Crawl multiple torrent sites.

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

Rack middleware that executes javascript before serving pages to crawlers.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Simple Web Crawler

check how many links are available inside the website