Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http gem contains and http client agnostic abstraction layer.

Headless HTTP crawler/scraper

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http-test gem contains a HTTP test server for testing HTTP client implementations.

株価情報を取得してあれこれするライブラリ

A simple web crawler for ruby

Grab the movies information from the atmovies.com

Add support for ElasticSearch in Polipus crawler

Automatically protects your staging app from web crawlers and casual visitors.

A web crawler written in ruby

Friendly, neighborhood web crawler for quality assurance.

Ruby Web crawler

A ruby social media stat crawler

Gem para acessar os dados do TIA Mackenzista

Crawl multiple torrent sites.

A web crawler dsl.

A ruby based zhihu content crawler.

Rack middleware that executes javascript before serving pages to crawlers.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.