This plugin need set the initial url.

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

The wrapper of capybara for crawler

Crawler that auto generates SitePrism Page Objects

Web crawler with JSON-based DSL and EventMachine-powered page fetching

Simple web crawler to crawl a domain and generate sitemap

DSL to build crawlers easily

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A ruby stock price & option chain crawler

A crappy crawler for a crappy bank interface

Block crawlers who spam your site with fake HTTP referers

Web page crawler.

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

A simple crawler demo crawler

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

A crappy crawler for a crappy bank interface