A simple news crawler. You can specify the structure of your xml or rss feeds.

A simple movie converter, queue and directory crawler, all in one!

A watir based web crawler

A web crawler dsl.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

Crawler for downloading comics from komiks.gildia.pl

A simple crawler that gets posts and pages from wordpress websites that have an exposed api

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.