Website crawler and fulltext indexer.

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

A simple directory crawler DSL.

A simple, fast web crawler

JavaScript enabled web crawler kit

your friendly neighborhood web crawler

Simple async HTTP crawler based on em-synchrony

Bulbasaur is a helper for crawler operations used in Pread.ly

Rails Analyzer Tools contains Bench, a simple web page benchmarker, Crawler, a tool for beating up on web sites, RailsStat, a tool for monitoring Rails web sites, and IOTail, a tail(1) method for Ruby IOs.

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file

ActiveRecord adapter for kudzu crawler

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

plugin-container totally sucks..

This gem allows to crawl news articles from RSS feeds.

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!