Crawl websites

Add support for Cassandra in Polipus crawler

Ruby Web crawler

A ruby social media stat crawler

A web crawler using Ruby and Redis.

A crawler for such as mailing lists.

CIA World Factbook crawler

Gem para acessar os dados do TIA Mackenzista

Crawl multiple torrent sites.

A ruby based zhihu content crawler.

Rack middleware that executes javascript before serving pages to crawlers.

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-barc gem contains support for the BARC Basic ARChive format.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

ActiveRecord adapter for kudzu crawler

plugin-container totally sucks..

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology