A simply web crawler.

Generic Web crawler with a DSL that parses event-related data from web pages

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A simple web crawler gem

a crawler toolkit

A crappy crawler for crashlytics

A simple web crawler gem

Web crawler help you with parse and collect data from the web

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

Asynchronous web crawler, scraper and file harvester

Crawl instagram photos, posts and videos for download.

An easy to use distributed web-crawler framework based on Redis

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Ruby web crawler using PhantomJS

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

your friendly neighborhood web crawler

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)