== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

CrawlerDetect is a library to detect bots/crawlers via the user agent

Crawler Engine provides function of crawl all news from the customized website

Generic Web crawler with a DSL that parses structured data from web pages

A crappy crawler for a crappy bank interface

Crawler4J filter plugin for Embulk

Voight-Kampff detects bots, spiders, crawlers and replicants

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

A simple crawler demo crawler

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

Web crawler help you with parse and collect data from the web

Web crawler to get courses from XML pages at mymav.uta

A watir based web crawler

a simple web crawler using DSL of capybara

A simple web crawler gem

crawler of niconico

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

A demo of Web Crawler using arb-crawler

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...