Web page crawler.

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

This is a crawler framework.

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

Ruby web crawler using PhantomJS

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

A Crawler for NewRank

it starts a crawler for Middleman sites

A ruby based zhihu content crawler.

A ruby social media stat crawler

Friendly, neighborhood web crawler for quality assurance.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

A ruby stock price & option chain crawler

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A web crawler using Ruby and Redis.

This gem helps Crawler Writers to interact with the PromoQui REST API

Simple Twitter crawler

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Crawler4J filter plugin for Embulk