Add support for Cassandra in Polipus crawler

Block crawlers who spam your site with fake HTTP referers

Automatically protects your staging app from web crawlers and casual visitors.

The wrapper of capybara for crawler

Friendly, neighborhood web crawler for quality assurance.

Crawler that auto generates SitePrism Page Objects

Ruby Web crawler

A ruby social media stat crawler

A web crawler using Ruby and Redis.

A simple crawler to list SESC events schedule on Terminal

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Gem para acessar os dados do TIA Mackenzista

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-char-detector gem provides charset detection support.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.

Simple Web Crawler

check how many links are available inside the website