Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Asynchronous web crawler, scraper and file harvester

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.

Show DMM and DMM.R18's crawled data. e.g. ranking

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

JavaScript enabled web crawler kit

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

A simple web crawler for ruby

Crawls the mails in a IMAP Folder

This is a crawler framework.

The Baidu Crawler is to crawl data with your demmand

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A ruby stock price & option chain crawler

BFS webcrawler that implements Observable

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

A crappy crawler for crashlytics

A crawler for a single domain web application

Lightweight web crawler

Crawler Engine provides function of crawl all news from the customized website

Headless HTTP crawler/scraper