render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Allows to crawl bandcamp sites, including release and track information

Crawler Guru provides all basic functionalities to extract data from web pages

A ruby based zhihu content crawler.

Simple ruby web crawler

A simple web crawler gem

株価情報を取得してあれこれするライブラリ

DSL to build crawlers easily

Crawler for http://legendas.tv to see the most dowloaded subtitles

Simple Twitter crawler

The Baidu Crawler is to crawl data with your demmand