== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

CrawlerDetect is a library to detect bots/crawlers via the user agent

Crawler Engine provides function of crawl all news from the customized website

Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.

Crawler4J filter plugin for Embulk

Protect your content from AI crawlers and monetize every request with Senthor. Real-time detection, crawler control, and detailed analytics.

A crappy crawler for a crappy bank interface

Generic Web crawler with a DSL that parses structured data from web pages

Voight-Kampff detects bots, spiders, crawlers and replicants

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A simple crawler demo crawler

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

A periodic crawler that fetches the latest CVE additions, parses them, and filters them

A generic web crawler that doesn't crawl outside URLs.

Crawler for downloading comics from komiks.gildia.pl

Simple ruby web crawler

A simple web crawler gem

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

A demo of Web Crawler using arb-crawler