Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

A watir based web crawler

Simple web crawler to crawl a domain and generate sitemap

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

株価情報を取得してあれこれするライブラリ

A crappy crawler for a crappy bank interface

A crawler and parser to the Currículo Lattes.

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A crappy crawler for crashlytics

A crawler for a single domain web application

Simple ruby web crawler

Asynchronous web crawler, scraper and file harvester

check how many links are available inside the website

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

plugin-container totally sucks..

Checks a user agent for a web crawler

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

A simple news crawler. You can specify the structure of your xml or rss feeds.