JavaScript enabled web crawler kit

A simple, fast web crawler

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGeoData are supported. Additional the GeoNames module allows to get Factbook and FAO country information from GPS coordinates.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Bulbasaur is a helper for crawler operations used in Pread.ly

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

BFS webcrawler that implements Observable

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Add support for ElasticSearch in Polipus crawler

Automatically protects your staging app from web crawlers and casual visitors.

The Baidu Crawler is to crawl data with your demmand

A simple web crawler for ruby

株価情報を取得してあれこれするライブラリ

Show DMM and DMM.R18's crawled data. e.g. ranking

A ruby social media stat crawler

Add support for Cassandra in Polipus crawler

Cangrejo lets you consume crabfarm crawlers using a simple DSL

A flexible, modular web crawler