== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

CrawlerDetect is a library to detect bots/crawlers via the user agent

Crawler Engine provides function of crawl all news from the customized website

Crawler4J filter plugin for Embulk

Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.

A crappy crawler for a crappy bank interface

Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.

Generic Web crawler with a DSL that parses structured data from web pages

Voight-Kampff detects bots, spiders, crawlers and replicants

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A simple crawler demo crawler

A simple web crawler gem

Web page crawler.

Simple web crawler to crawl a domain and generate sitemap

A simple news crawler. You can specify the structure of your xml or rss feeds.

a simple web crawler using DSL of capybara

This file crawler helps to decect if there are new files in a directory.

A watir based web crawler

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.