Crawler for http://legendas.tv to see the most dowloaded subtitles

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run them periodically. Thanks to consistent Twitter API and twitter gem (http://twitter.rubyforge.org/), it is quite easy to track various types of timelines (such as user_timeline, home_timeline, search...)

Jobs crawler

Crawler Guru provides all basic functionalities to extract data from web pages

A watir based web crawler

A simple web crawler gem

crawler of niconico

a simple web crawler using DSL of capybara

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Crawler for downloading comics from komiks.gildia.pl

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

Simple site crawler using Capybara

A simply web crawler.

Web page crawler.

A simple web crawler gem

Simple ruby web crawler

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

This gem is a web crawler sample code.So I don't reccmmend that you use.