Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-async-httpclient based implementation of the iudex-http interfaces.

Crawl websites

Crawls the mails in a IMAP Folder

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http-test gem contains a HTTP test server for testing HTTP client implementations.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-char-detector gem provides charset detection support.

a crawler toolkit

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

A gem that collects mangas from websites

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

Headless HTTP crawler/scraper

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Simple crawler using Redis as backend

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

Add support for ElasticSearch in Polipus crawler

A Crawler for NewRank

Rack middleware that executes javascript before serving pages to crawlers.

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

This gem offers: classes to subclass and create a manga site crawler; a dowloader to use with these classes; some site-specific scripts.