Web page crawler.

it starts a crawler for Middleman sites

Simple Gem Using Watir For Phantom Crawler

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

This gem is a web crawler sample code.So I don't reccmmend that you use.

A demo of Web Crawler using arb-crawler

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A flexible, modular web crawler

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

Web crawler to get courses from XML pages at mymav.uta

Crawl instagram photos, posts and videos for download.

Asynchronous web crawler, scraper and file harvester

An easy to use distributed web-crawler framework based on Redis

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Ruby web crawler using PhantomJS

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Gem for crawling data from external sources