Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-worker gem provides a worker deamon for feed/page processing.

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Lightweight interface to communicate with the Marvel Comics API

Empower World Travel Information Technology

The Taiwan VSCinema crawler to get latest film list.

webget gem - a web (go get) crawler incl. web cache

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

rails_angular_seo allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

s7o crawler optimized for programmer happiness and sustainable productivity.

This plugin need set the initial url.

This gem crawls the latest CircleCI artifact file you specified. For Example, you can get the result JSON of simplecov.gem etc.

This gem helps Crawler Writers to interact with the PromoQui REST API

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the pro...

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

A simple ruby gem to recursively traverse all URLs on a Root URL. It returns all the URLs it encountered

A simple movie converter, queue and directory crawler, all in one!

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.