Simple Gem Using Watir For Phantom Crawler

This gem is a web crawler sample code.So I don't reccmmend that you use.

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

A generic web crawler that doesn't crawl outside URLs.

A watir based web crawler

Crawler Guru provides all basic functionalities to extract data from web pages

Simple ruby web crawler

A simple web crawler gem

A simple web crawler gem

Asynchronous web crawler, scraper and file harvester

Crawl instagram photos, posts and videos for download.

An easy to use distributed web-crawler framework based on Redis

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Ruby web crawler using PhantomJS

JavaScript enabled web crawler kit