ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

Allows your rails application to be spiderable by crawlers

A client for the PageMunch web crawler API

A gem that collects mangas from websites

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A web crawler dsl.

This gem is a web crawler sample code.So I don't reccmmend that you use.

The wrapper of capybara for crawler

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

A flexible, modular web crawler

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

crawler of niconico-douga

crawler of niconico

Lightweight web crawler

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

Rack middleware that executes javascript before serving pages to crawlers.

Crawler for downloading comics from komiks.gildia.pl

An easy to use distributed web-crawler framework based on Redis