SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

A web scrawler to get a Marmiton's recipe

JavaScript enabled web crawler kit

Website crawler and fulltext indexer.

your friendly neighborhood web crawler

BFS webcrawler that implements Observable

A simple, fast web crawler

This is a crawler framework.

A simple directory crawler DSL.

Gem for crawling data from external sources

A web crawler using Ruby and Redis.

A ruby social media stat crawler

Crawler that auto generates SitePrism Page Objects

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

A crawler for such as mailing lists.

Crawl multiple torrent sites.

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

The wrapper of capybara for crawler

Add support for Cassandra in Polipus crawler