Web crawler help you with parse and collect data from the web

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

A client for the PageMunch web crawler API

A ruby social media stat crawler

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

This gem helps Crawler Writers to interact with the PromoQui REST API

Rack middleware that executes javascript before serving pages to crawlers.

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Bulbasaur is a helper for crawler operations used in Pread.ly

Simple Web Crawler

Checks a user agent for a web crawler

A ruby stock price & option chain crawler

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

This is a crawler framework.

DSL to build crawlers easily

Show DMM and DMM.R18's crawled data. e.g. ranking

A generic web crawler that doesn't crawl outside URLs.

A simple web crawler for ruby

A gem that collects mangas from websites