A simple, fast web crawler

Bulbasaur is a helper for crawler operations used in Pread.ly

FileCrawler searches and controls files in local directory

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Website crawler and fulltext indexer.

JavaScript enabled web crawler kit

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

A demo of Web Crawler using arb-crawler

A ruby social media stat crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Crawl the senegalese web, looking for jobs using the excellent wombat gem

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.