DSL to build crawlers easily

A simple crawler that gets posts and pages from wordpress websites that have an exposed api

Discovery Mission is an easy-to-use website crawler. Use it for generating sitemaps.

plugin-container totally sucks..

A simple web crawler gem

A simple web crawler gem

A simply web crawler.

A simple web crawler gem

Empower World Travel Information Technology

Empower World Travel Information Technology

Lightweight interface to communicate with the Marvel Comics API

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the pro...

Crawler Engine provides function of crawl all news from the customized website

This gem is a web crawler sample code.So I don't reccmmend that you use.

Crawler Guru provides all basic functionalities to extract data from web pages

A web crawler using Ruby and Redis.

Simple web crawler to crawl a domain and generate sitemap

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Crawl multiple torrent sites.