A crappy crawler for crashlytics

it starts a crawler for Middleman sites

A demo of Web Crawler using arb-crawler

Web crawler to get courses from XML pages at mymav.uta

A flexible, modular web crawler

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run them periodically. Thanks to consistent Twitter API and twitter gem (http://twitter.rubyforge.org/), it is quite easy to track various types of timelines (such as user_timeline, home_timeline, search...)

a simple web crawler using DSL of capybara

Web crawler help you with parse and collect data from the web

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

The Baidu Crawler is to crawl data with your demmand

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

crawler of niconico

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

This gem is a web crawler sample code.So I don't reccmmend that you use.

A simple web crawler gem

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A simple web crawler gem

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Simple web crawler to crawl a domain and generate sitemap

A simple news crawler. You can specify the structure of your xml or rss feeds.