This gem allows to crawl news articles from RSS feeds.

FileCrawler searches and controls files in local directory

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

Crawls the mails in a IMAP Folder

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

A crappy crawler for a crappy bank interface

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Lightweight interface to communicate with the Marvel Comics API

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

The Taiwan VSCinema crawler to get latest film list.

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

This gem crawls the latest CircleCI artifact file you specified. For Example, you can get the result JSON of simplecov.gem etc.

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success