A web scrawler to get a Marmiton's recipe

This is a crawler framework.

== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

A simple, fast web crawler

Website crawler and fulltext indexer.

A simple directory crawler DSL.

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

Crawler Engine provides function of crawl all news from the customized website

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

A demo of Web Crawler using arb-crawler

a crawler toolkit

Allows to crawl bandcamp sites, including release and track information

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

Web crawler with JSON-based DSL and EventMachine-powered page fetching

Mobile App Review Crawler

Cangrejo lets you consume crabfarm crawlers using a simple DSL

FileCrawler searches and controls files in local directory

A crappy crawler for crashlytics

A ruby stock price & option chain crawler

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-filter gem contains a fundamental filtering/chain of responsbility sub-system.