Jobs crawler

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A flexible, modular web crawler

crawler of niconico

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Web crawler with JSON-based DSL and EventMachine-powered page fetching

The Baidu Crawler is to crawl data with your demmand

Simple site crawler using Capybara

Crawler for http://legendas.tv to see the most dowloaded subtitles

Web crawler help you with parse and collect data from the web

Asynchronous web crawler, scraper and file harvester

Crawl instagram photos, posts and videos for download.

An easy to use distributed web-crawler framework based on Redis

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Ruby web crawler using PhantomJS

A web scrawler to get a Marmiton's recipe