Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Web crawler to get courses from XML pages at mymav.uta

Simple web crawler to crawl a domain and generate sitemap

A simple news crawler. You can specify the structure of your xml or rss feeds.

A flexible, modular web crawler

A simply web crawler.

This gem is a web crawler sample code.So I don't reccmmend that you use.

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run them periodically. Thanks to consistent Twitter API and twitter gem (http://twitter.rubyforge.org/), it is quite easy to track various types of timelines (such as user_timeline, home_timeline, search...)

Jobs crawler

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

A simple web crawler gem

Simple ruby web crawler

a crawler toolkit

crawler of niconico

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

A crappy crawler for crashlytics

This file crawler helps to decect if there are new files in a directory.

Asynchronous web crawler, scraper and file harvester