Generic Web crawler with a DSL that parses structured data from web pages

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Crawl instagram photos, posts and videos for download.

Voight-Kampff detects bots, spiders, crawlers and replicants

Asynchronous web crawler, scraper and file harvester

CrawlerDetect is a library to detect bots/crawlers via the user agent

An easy to use distributed web-crawler framework based on Redis

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Ruby web crawler using PhantomJS

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A simple directory crawler DSL.

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

your friendly neighborhood web crawler

A simple, fast web crawler

Website crawler and fulltext indexer.

JavaScript enabled web crawler kit