Simple ruby web crawler

A web crawler using Ruby and Redis.

A simple crawler to list SESC events schedule on Terminal

Simple web crawler to crawl a domain and generate sitemap

CIA World Factbook crawler

Gem para acessar os dados do TIA Mackenzista

Rack middleware that executes javascript before serving pages to crawlers.

Checks a user agent for a web crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-html gem contains filters for HTML parsing, filtering, exracting text and links.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-da gem provides a PostgreSQL-based content meta-data store and work priority queue.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

Crawler for http://legendas.tv to see the most dowloaded subtitles

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

Allows your rails application to be spiderable by crawlers

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file