A web crawler dsl.

Crawler Engine provides function of crawl all news from the customized website

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

Jobs crawler

A simple web crawler gem

Headless HTTP crawler/scraper

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

book's price crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.