Simple web crawler to crawl a domain and generate sitemap

a simple web crawler using DSL of capybara

A generic web crawler that doesn't crawl outside URLs.

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Crawler for downloading comics from komiks.gildia.pl

This file crawler helps to decect if there are new files in a directory.

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

A flexible, modular web crawler

A simply web crawler.

Asynchronous web crawler, scraper and file harvester

Crawl instagram photos, posts and videos for download.

An easy to use distributed web-crawler framework based on Redis

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Ruby web crawler using PhantomJS

A simple, fast web crawler