A client for the PageMunch web crawler API

Pantopoda is a web crawler that visits all links on a given domain that's fast and effective.

Bulbasaur is a helper for crawler operations used in Pread.ly

a demo for study ruby

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Samao is a web crawler written in ruby.

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

The Taiwan VSCinema crawler to get latest film list.

s7o crawler optimized for programmer happiness and sustainable productivity.

This plugin need set the initial url.

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A simple movie converter, queue and directory crawler, all in one!

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.