Simple Web Crawler

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

Gem para acessar os dados do TIA Mackenzista

A crappy crawler for crashlytics

Block crawlers who spam your site with fake HTTP referers

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

A web crawler dsl.

Allows to crawl bandcamp sites, including release and track information

crawler of niconico

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A simple web crawler gem

plugin-container totally sucks..

A crawler and parser to the Currículo Lattes.

The Taiwan VSCinema crawler to get latest film list.

株価情報を取得してあれこれするライブラリ

Simple web crawler to crawl a domain and generate sitemap

Simple ruby web crawler

check how many links are available inside the website

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.