SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

BFS webcrawler that implements Observable

Dead simple yet powerful Ruby crawler for easy parallel crawling with support for an anonymity.

Gem para acessar os dados do TIA Mackenzista

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

A web scrawler to get a Marmiton's recipe

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

webget gem - a web (go get) crawler incl. web cache

Cangrejo lets you consume crabfarm crawlers using a simple DSL

Automatically protects your staging app from web crawlers and casual visitors.

== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect usefu...

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

it starts a crawler for Middleman sites

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

A crappy crawler for a crappy bank interface

A crappy crawler for a crappy bank interface

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

FileCrawler searches and controls files in local directory

SuperCrawler allows you to easily crawl full web sites or web pages (extracting internal links and assets) in few seconds.

Web crawler help you with parse and collect data from the web