A working website link checker

Various scrapers for the Socializer application.

A simple and speedy web spider for pages downloading

This gem provides a simple tool to make a full online website into a static one. This can be useful for companies that needs to take system off from a client, but want to keep it online in static manner.

Crawl a site with 'clean-URLs' and generate a files and folders from it. Example: the URL /page will become /page/index.html instead of /page.html so you can serve it straight from Apache and all the links are still working.

Find out the total no. of shares of a website or webpage on all major social networks.

Quick way to crawl and screenshot your site.

Rails plugin for crawling domains for detecting differences in the UI

Scrape a website as plain text.

This library # This class contains all the necessary methods to do the following: -Crawl most websites and return an array of their URLS. -Convert all HTML and most web linked PDF documents to one large string given an array of urls. -Parse out all non words and non human sensible markup. -St...