Pantopoda is a web crawler that visits all links on a given domain that's fast and effective.

Simple async HTTP crawler based on em-synchrony

Bulbasaur is a helper for crawler operations used in Pread.ly

a demo for study ruby

Allows your rails application to be spiderable by crawlers

your friendly neighborhood web crawler

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

This gem allows to crawl news articles from RSS feeds.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

Lightweight interface to communicate with the Marvel Comics API

Empower World Travel Information Technology

webget gem - a web (go get) crawler incl. web cache

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

s7o crawler optimized for programmer happiness and sustainable productivity.

This plugin need set the initial url.

This gem crawls the latest CircleCI artifact file you specified. For Example, you can get the result JSON of simplecov.gem etc.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A simple movie converter, queue and directory crawler, all in one!

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.