== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Generic Web crawler with a DSL that parses structured data from web pages

CrawlerDetect is a library to detect bots/crawlers via the user agent

Crawler Engine provides function of crawl all news from the customized website

Crawler4J filter plugin for Embulk

A crappy crawler for a crappy bank interface

Voight-Kampff detects bots, spiders, crawlers and replicants

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A simple crawler demo crawler

a crawler toolkit

This gem is a web crawler sample code.So I don't reccmmend that you use.

A generic web crawler that doesn't crawl outside URLs.

This crawler will use my personnal scraper named 'RecipeScraper' to dowload recipes data from Marmiton, 750g or cuisineaz

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

Web crawler help you with parse and collect data from the web

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A simple web crawler gem