A periodic crawler that fetches the latest CVE additions, parses them, and filters them

Crawler for downloading comics from komiks.gildia.pl

A flexible, modular web crawler

Simple site crawler using Capybara

Web crawler with JSON-based DSL and EventMachine-powered page fetching

a simple web crawler using DSL of capybara

This gem is a web crawler sample code.So I don't reccmmend that you use.

it starts a crawler for Middleman sites

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

Crawler for http://legendas.tv to see the most dowloaded subtitles

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A simply web crawler.

Generic Web crawler with a DSL that parses event-related data from web pages

a crawler toolkit

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

A crappy crawler for crashlytics

Simple Gem Using Watir For Phantom Crawler

A generic web crawler that doesn't crawl outside URLs.

Jobs crawler