Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

A web crawler dsl.

A ruby based zhihu content crawler.

Another Web crawler running with Amazon SQS and ElastiCache(Redis)

Rack middleware that executes javascript before serving pages to crawlers.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-char-detector gem provides charset detection support.

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-service provides a fuzzy simhash lookup index as a distributed service.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Samao is a web crawler written in ruby.

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, siblings etc. Furthermore, you are able to download images, web pages, and store all content in the database. Please visit my Github account for more details.

ActiveRecord adapter for kudzu crawler

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Lightweight interface to communicate with the Marvel Comics API

Empower World Travel Information Technology