Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-filter gem contains a fundamental filtering/chain of responsbility sub-system.

ActiveRecord adapter for kudzu crawler

Show DMM and DMM.R18's crawled data. e.g. ranking

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A ruby stock price & option chain crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

DSL to build crawlers easily

Cangrejo lets you consume crabfarm crawlers using a simple DSL

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!

The Taiwan VSCinema crawler to get latest film list.

FileCrawler searches and controls files in local directory

Crawls the mails in a IMAP Folder

A crappy crawler for a crappy bank interface

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

BFS webcrawler that implements Observable

s7o crawler optimized for programmer happiness and sustainable productivity.