This gem is a web crawler sample code.So I don't reccmmend that you use.

Jobs crawler

A watir based web crawler

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A simple web crawler gem

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

Simple Gem Using Watir For Phantom Crawler

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

Simple site crawler using Capybara

Crawl instagram photos, posts and videos for download.

Asynchronous web crawler, scraper and file harvester

An easy to use distributed web-crawler framework based on Redis

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Ruby web crawler using PhantomJS

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

A simple, fast web crawler

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)