Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-char-detector gem provides charset detection support.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Asynchronous web crawler, scraper and file harvester

Simple Twitter crawler

A simple web crawler for ruby

JavaScript enabled web crawler kit

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

Generic Web crawler with a DSL that parses event-related data from web pages

Mobile App Review Crawler

A little website crawler.

Cangrejo lets you consume crabfarm crawlers using a simple DSL

A crawler for a single domain web application

Crawler4J filter plugin for Embulk

Simple crawler using Redis as backend

Lightweight web crawler

A generic web crawler that doesn't crawl outside URLs.

株価情報を取得してあれこれするライブラリ

Simple little website crawler.

A gem that collects mangas from websites