Add support for Cassandra in Polipus crawler

A ruby social media stat crawler

A web crawler using Ruby and Redis.

A crawler for such as mailing lists.

CIA World Factbook crawler

Gem para acessar os dados do TIA Mackenzista

Crawl multiple torrent sites.

A ruby based zhihu content crawler.

Rack middleware that executes javascript before serving pages to crawlers.

Simple Web Crawler

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

A client for the PageMunch web crawler API

Pantopoda is a web crawler that visits all links on a given domain that's fast and effective.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-service provides a fuzzy simhash lookup index as a distributed service.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

check how many links are available inside the website