Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

Simple Web Crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-filter gem contains a fundamental filtering/chain of responsbility sub-system.

Simple async HTTP crawler based on em-synchrony

Bulbasaur is a helper for crawler operations used in Pread.ly

a demo for study ruby

Allows your rails application to be spiderable by crawlers

your friendly neighborhood web crawler

Show DMM and DMM.R18's crawled data. e.g. ranking

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

plugin-container totally sucks..

DSL to build crawlers easily

This gem allows to crawl news articles from RSS feeds.

Cangrejo lets you consume crabfarm crawlers using a simple DSL

Crawls the mails in a IMAP Folder

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

A crappy crawler for a crappy bank interface

A simple crawler that gets posts and pages from wordpress websites that have an exposed api

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.