CrawlerDetect is a library to detect bots/crawlers via the user agent

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

ActiveRecord adapter for kudzu crawler

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the pro...

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

A simple web crawler for ruby

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Voight-Kampff detects bots, spiders, crawlers and replicants

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Simple Web Crawler

Generic Web crawler with a DSL that parses structured data from web pages

A simple crawler demo crawler

The Taiwan VSCinema crawler to get latest film list.

Crawler Guru provides all basic functionalities to extract data from web pages

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

webget gem - a web (go get) crawler incl. web cache

Ruby web crawler using PhantomJS

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Block crawlers who spam your site with fake HTTP referers

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!