web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A simple web crawler gem

Crawler for downloading comics from komiks.gildia.pl

it starts a crawler for Middleman sites

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A simple web crawler gem

Simple site crawler using Capybara

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

A demo of Web Crawler using arb-crawler

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

A simple web crawler gem

crawler of niconico

Web crawler help you with parse and collect data from the web

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run them periodically. Thanks to consistent Twitter API and twitter gem (http://twitter.rubyforge.org/), it is quite easy to track various types of timelines (such as user_timeline, home_timeline, search...)

Generic Web crawler with a DSL that parses event-related data from web pages

A watir based web crawler

A simple news crawler. You can specify the structure of your xml or rss feeds.

a crawler toolkit

Simple web crawler to crawl a domain and generate sitemap

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.