is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A flexible, modular web crawler

Crawl the senegalese web, looking for jobs using the excellent wombat gem

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

webget gem - a web (go get) crawler incl. web cache

Simple Web Crawler

This file crawler helps to decect if there are new files in a directory.

Simple async HTTP crawler based on em-synchrony

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-simhash gem contains support for generation and searching over simhash fingerprints

A generic web crawler that doesn't crawl outside URLs.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-http gem contains and http client agnostic abstraction layer.

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is an rjack-httpclient-3 based implementation of the iudex-http interfaces.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-rome gems is an adaption of rjack-rome for feed parsing in Iudex.

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-filter gem contains a fundamental filtering/chain of responsbility sub-system.

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

Crawl websites