web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

Web crawler to get courses from XML pages at mymav.uta

A flexible, modular web crawler

Web crawler with JSON-based DSL and EventMachine-powered page fetching

a simple web crawler using DSL of capybara

A simple web crawler gem

This gem is a web crawler sample code.So I don't reccmmend that you use.

Web page crawler.

A watir based web crawler

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

A generic web crawler that doesn't crawl outside URLs.

a crawler toolkit

A simple web crawler gem

A demo of Web Crawler using arb-crawler

Crawler for downloading comics from komiks.gildia.pl

Simple web crawler to crawl a domain and generate sitemap

A simple news crawler. You can specify the structure of your xml or rss feeds.