A simply web crawler.

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

Ruby Cheerio is a jQuery style HTML parser, which take selectors as input. This is a Ruby version NodeJS package named 'Cheerio', which is extensively used by crawlers. Please visit the home page for usage details.

Crawler4J filter plugin for Embulk

Botch is a DSL for quickly creating web crawlers. Inspired by Sinatra.

FileCrawler searches and controls files in local directory

Gem for crawling data from external sources

A simple web crawler for ruby

it starts a crawler for Middleman sites

Add support for ElasticSearch in Polipus crawler

CIA World Factbook crawler

Simple Twitter crawler

ActiveRecord adapter for kudzu crawler

The Taiwan VSCinema crawler to get latest film list.

A generic web crawler that doesn't crawl outside URLs.

A ruby social media stat crawler

Simple ruby web crawler

Fake User-Agents of about %80 of real devices to use in headers of web crawlers. It keeps your script away from being nested by many UA strings.

ITunesCrawler provides an easy way to download the requested iTunes data through Apple's Search API.

A simple crawler to list SESC events schedule on Terminal