render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

A simple directory crawler DSL.

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

This is a crawler framework.

JavaScript enabled web crawler kit

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

check how many links are available inside the website

livedoor-feeddiscover performs feed autodiscovery using the livedoor Feed Discover API. livedoor Feed Discover API find a Atom/RSS feed(s) from the livedoor Reader crawler database. So, livedoor-feeddiscover do not access the target URL.

Simple Twitter crawler

image download from instagram

Ruby web crawler to access omelete informations

Multithreaded web crawler with transparent DSL and requests caching.

Simple async HTTP crawler based on em-synchrony

Bulbasaur is a helper for crawler operations used in Pread.ly

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file