Website crawler and fulltext indexer.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

A simple, fast web crawler

A simple directory crawler DSL.

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

Multithreaded web crawler with transparent DSL and requests caching.

Pantopoda is a web crawler that visits all links on a given domain that's fast and effective.

Simple async HTTP crawler based on em-synchrony

Rails Analyzer Tools contains Bench, a simple web page benchmarker, Crawler, a tool for beating up on web sites, RailsStat, a tool for monitoring Rails web sites, and IOTail, a tail(1) method for Ruby IOs.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

your friendly neighborhood web crawler

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file

ActiveRecord adapter for kudzu crawler

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

plugin-container totally sucks..