A web scrawler to get a Marmiton's recipe

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

BFS webcrawler that implements Observable

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Add support for ElasticSearch in Polipus crawler

JavaScript enabled web crawler kit

A simple, fast web crawler

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

A Crawler for NewRank

Automatically protects your staging app from web crawlers and casual visitors.

Bulbasaur is a helper for crawler operations used in Pread.ly

A simple directory crawler DSL.

A simple web crawler for ruby

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

Simple site crawler using Capybara

Cangrejo lets you consume crabfarm crawlers using a simple DSL

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz