Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

a simple web crawler using DSL of capybara

A simple web crawler gem

Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Web crawler help you with parse and collect data from the web

A simple web crawler gem

Crawler Guru provides all basic functionalities to extract data from web pages

A simple web crawler gem

Web crawler with JSON-based DSL and EventMachine-powered page fetching

it starts a crawler for Middleman sites

Web crawler to get courses from XML pages at mymav.uta

Crawler for http://legendas.tv to see the most dowloaded subtitles

A flexible, modular web crawler

This file crawler helps to decect if there are new files in a directory.

The Baidu Crawler is to crawl data with your demmand

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz