a crawler toolkit

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

A crappy crawler for crashlytics

The Baidu Crawler is to crawl data with your demmand

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

Web crawler help you with parse and collect data from the web

Crawler for downloading comics from komiks.gildia.pl

This file crawler helps to decect if there are new files in a directory.

Crawler for http://legendas.tv to see the most dowloaded subtitles

Simple site crawler using Capybara

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Jobs crawler

it starts a crawler for Middleman sites

A simple web crawler gem

Web page crawler.

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A generic web crawler that doesn't crawl outside URLs.

A simple web crawler gem