初级开发工程师，基于 http 写的爬虫扩展包。请不要随意下载里面有很多坑。

Retrieves a list of URLs to seed the crawler by publishing them to a RabbitMQ exchange.

Simple Web Crawler

Hushes worthless Rails exceptions & logs, such as those caused by bots and crawlers.

A simple crawler demo crawler

The Taiwan VSCinema crawler to get latest film list.

CrawlerDetect is a library to detect bots/crawlers via the user agent

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

your friendly neighborhood web crawler

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

Empower World Travel Information Technology

a demo for study ruby

== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push Medusa is a framework for the ruby language to crawl and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily. === Features * Choose the links to follow on each page with +focus_crawl+ * Multi-threaded design for high performance * Tracks +301+ HTTP redirects * Allows exclusion of URLs based on regular expressions * Records response time for each page * Obey _robots.txt_ directives (optional, but recommended) * In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta] * Inherits OpenURI behavior (redirects, automatic charset and encoding detection, proxy configuration options). <b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b> === Examples Medusa is versatile and to be used programatically, you can start with one or multiple URIs: require 'medusa' Medusa.crawl('https://www.example.com', depth_limit: 2) Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus: require 'medusa' Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler| crawler.discard_page_bodies = some_flag # Persist all the pages state across crawl-runs. crawler.clear_on_startup = false crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0') crawler.skip_links_like(/private/) crawler.on_pages_like(/public/) do |page| logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}" end # Use an arbitrary logic, page by page, to continue customize the crawling. crawler.focus_crawl(/public/) do |page| page.links.first end end

A simple web crawler for ruby

Simple Gem Using Watir For Phantom Crawler

Generic Web crawler with a DSL that parses structured data from web pages

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.