A crawler for such as mailing lists.

Crawl multiple torrent sites.

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-service provides a fuzzy simhash lookup index as a distributed service.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

Simple Web Crawler

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

check how many links are available inside the website

image download from instagram

Ruby web crawler to access omelete informations

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

Multithreaded web crawler with transparent DSL and requests caching.

A client for the PageMunch web crawler API