SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Web crawler help you with parse and collect data from the web

Checks a user agent for a web crawler

Bulbasaur is a helper for crawler operations used in Pread.ly

This gem helps Crawler Writers to interact with the PromoQui REST API

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

A ruby social media stat crawler

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Simple Web Crawler

Rack middleware that executes javascript before serving pages to crawlers.

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

Add support for ElasticSearch in Polipus crawler

This gem allows to crawl news articles from RSS feeds.

The wrapper of capybara for crawler

Friendly, neighborhood web crawler for quality assurance.

Simple Twitter crawler

A ruby based zhihu content crawler.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)