A simple web crawler for ruby

Bulbasaur is a helper for crawler operations used in Pread.ly

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

FileCrawler searches and controls files in local directory

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

JavaScript enabled web crawler kit

Website crawler and fulltext indexer.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGeoData are supported. Additional the GeoNames module allows to get Factbook and FAO country information from GPS coordinates.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

Checks a user agent for a web crawler

A demo of Web Crawler using arb-crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

A ruby social media stat crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

A web crawler written in ruby