Allows to crawl bandcamp sites, including release and track information

The Taiwan VSCinema crawler to get latest film list.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

webget gem - a web (go get) crawler incl. web cache

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

Simple crawler using Redis as backend

Lightweight web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http gem contains and http client agnostic abstraction layer.

Headless HTTP crawler/scraper

Ruby web crawler to access omelete informations

A set of classes for dealing with options. It includes a crawler for Yahoo!Finance.

Multithreaded web crawler with transparent DSL and requests caching.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

your friendly neighborhood web crawler

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.