Ruby Web crawler

A web crawler using Ruby and Redis.

A simple crawler to list SESC events schedule on Terminal

Lightweight web crawler

Simple crawler using Redis as backend

Website crawler harvesting e-mails. Uses Sidekiq and Typhoeus.

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A ruby based zhihu content crawler.

Simple Web Crawler

Simple Twitter crawler

image download from instagram

Simple async HTTP crawler based on em-synchrony

Bulbasaur is a helper for crawler operations used in Pread.ly

Rails Analyzer Tools contains Bench, a simple web page benchmarker, Crawler, a tool for beating up on web sites, RailsStat, a tool for monitoring Rails web sites, and IOTail, a tail(1) method for Ruby IOs.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-filter gem contains a fundamental filtering/chain of responsbility sub-system.

Samao is a web crawler written in ruby.

Show DMM and DMM.R18's crawled data. e.g. ranking

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file