Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

JavaScript enabled web crawler kit

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

This is a crawler framework.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGeoData are supported. Additional the GeoNames module allows to get Factbook and FAO country information from GPS coordinates.

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Bulbasaur is a helper for crawler operations used in Pread.ly

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

A web crawler written in ruby

Checks a user agent for a web crawler

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

A simple, fast web crawler

FileCrawler searches and controls files in local directory

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

A simple web crawler for ruby

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.