Crawls the mails in a IMAP Folder

Simple crawler using Redis as backend

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

Mobile App Review Crawler

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.

Crawler4J filter plugin for Embulk

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!

Crawler for downloading comics from komiks.gildia.pl

Web page crawler.

A little website crawler.

Allows your rails application to be spiderable by crawlers

The wrapper of capybara for crawler

CIA World Factbook crawler

livedoor-feeddiscover performs feed autodiscovery using the livedoor Feed Discover API. livedoor Feed Discover API find a Atom/RSS feed(s) from the livedoor Reader crawler database. So, livedoor-feeddiscover do not access the target URL.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Grab the movies information from the atmovies.com

A simple movie converter, queue and directory crawler, all in one!

A simple news crawler. You can specify the structure of your xml or rss feeds.

A Crawler for NewRank

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.