The Taiwan VSCinema crawler to get latest film list.

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

This plugin need set the initial url.

This gem crawls the latest CircleCI artifact file you specified. For Example, you can get the result JSON of simplecov.gem etc.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

This gem helps Crawler Writers to interact with the PromoQui REST API

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the pro...

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A simple movie converter, queue and directory crawler, all in one!

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

Iudex is a general purpose web crawler and feed processor in ruby/java. This gem is a Jetty HTTP Client based implementation of the iudex-http interfaces.

Allows to crawl bandcamp sites, including release and track information

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

BFS webcrawler that implements Observable

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

FileCrawler searches and controls files in local directory

RegexpCrawler is a Ruby library for crawl data from website using regular expression.

Gem for crawling data from external sources