The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

plugin-container totally sucks..

This gem allows to crawl news articles from RSS feeds.

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Lightweight interface to communicate with the Marvel Comics API

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

This plugin need set the initial url.

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

This gem helps Crawler Writers to interact with the PromoQui REST API

Empower World Travel Information Technology

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A simple movie converter, queue and directory crawler, all in one!

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.