BFS webcrawler that implements Observable

Website crawler and fulltext indexer.

Gem for crawling data from external sources

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

This is a crawler framework.

your friendly neighborhood web crawler

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

A simple, fast web crawler

A web scrawler to get a Marmiton's recipe

JavaScript enabled web crawler kit

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGeoData are supported. Additional the GeoNames module allows to get Factbook and FAO country information from GPS coordinates.

plugin-container totally sucks..

This gem allows to crawl news articles from RSS feeds.

The Taiwan VSCinema crawler to get latest film list.

Crawl the senegalese web, looking for jobs using the excellent wombat gem

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by Demotix, to protect development and staging servers from search engine crawlers and other unwanted traffic.

Arachnidish is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the process, providing a clean and efficient solution.

RegexpCrawler is a Ruby library for crawl data from website using regular expression.