SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Automatically protects your staging app from web crawlers and casual visitors.

BFS webcrawler that implements Observable

A Crawler for NewRank

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Add support for ElasticSearch in Polipus crawler

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Bulbasaur is a helper for crawler operations used in Pread.ly

A simple, fast web crawler

JavaScript enabled web crawler kit

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

webget gem - a web (go get) crawler incl. web cache

A ruby social media stat crawler

A flexible, modular web crawler

your friendly neighborhood web crawler

Simple site crawler using Capybara

The Baidu Crawler is to crawl data with your demmand

Empower World Travel Information Technology

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.