web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Simple site crawler using Capybara

This file crawler helps to decect if there are new files in a directory.

A simply web crawler.

Simple Gem Using Watir For Phantom Crawler

a crawler toolkit

a simple web crawler using DSL of capybara

Web crawler help you with parse and collect data from the web

A generic web crawler that doesn't crawl outside URLs.

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

An easy to use distributed web-crawler framework based on Redis

Asynchronous web crawler, scraper and file harvester

Crawl instagram photos, posts and videos for download.

Ruby web crawler using PhantomJS

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.