Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http gem contains and http client agnostic abstraction layer.

Headless HTTP crawler/scraper

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-async-httpclient based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http-test gem contains a HTTP test server for testing HTTP client implementations.

Simple little website crawler.

crawler of niconico-douga

Add support for ElasticSearch in Polipus crawler

This is web comic crawler that Akihabara otaku.

Block crawlers who spam your site with fake HTTP referers

Automatically protects your staging app from web crawlers and casual visitors.

The wrapper of capybara for crawler

Crawler that auto generates SitePrism Page Objects

Ruby Web crawler

A ruby social media stat crawler

A web crawler using Ruby and Redis.

A simple crawler to list SESC events schedule on Terminal

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

A crawler for such as mailing lists.

CIA World Factbook crawler

Gem para acessar os dados do TIA Mackenzista