Simple Gem Using Watir For Phantom Crawler

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

a demo for study ruby

This gem is a web crawler sample code.So I don't reccmmend that you use.

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

A crappy crawler for a crappy bank interface

This file crawler helps to decect if there are new files in a directory.

FileCrawler searches and controls files in local directory

A simply web crawler.

A ruby stock price & option chain crawler

Crawler4J filter plugin for Embulk

Block crawlers who spam your site with fake HTTP referers

Friendly, neighborhood web crawler for quality assurance.

Simple ruby web crawler

A web crawler using Ruby and Redis.

Simple web crawler to crawl a domain and generate sitemap

CIA World Factbook crawler

A ruby based zhihu content crawler.