CrawlerDetect is a library to detect bots/crawlers via the user agent

初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

ActiveRecord adapter for kudzu crawler

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the process, providing a clean and efficient solution.

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

A simple web crawler for ruby

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Voight-Kampff detects bots, spiders, crawlers and replicants

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Simple Web Crawler

A simple web crawler gem

Generic Web crawler with a DSL that parses structured data from web pages

This rubygem does not have a description or summary.

This rubygem does not have a description or summary.

This rubygem does not have a description or summary.

This rubygem does not have a description or summary.

A simple crawler demo crawler

The Taiwan VSCinema crawler to get latest film list.

This rubygem does not have a description or summary.

This rubygem does not have a description or summary.