Website crawler and fulltext indexer.

A simple directory crawler DSL.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

This is a crawler framework.

JavaScript enabled web crawler kit

A simple, fast web crawler

Friendly, neighborhood web crawler for quality assurance.

Add support for Cassandra in Polipus crawler

Ruby Web crawler

A ruby social media stat crawler

Crawl websites

This is web comic crawler that Akihabara otaku.

A simple crawler to list SESC events schedule on Terminal

Add support for ElasticSearch in Polipus crawler

A Crawler for NewRank

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

A gem that collects mangas from websites

A web crawler dsl.

A ruby based zhihu content crawler.