Jobs crawler

Web crawler with JSON-based DSL and EventMachine-powered page fetching

A demo of Web Crawler using arb-crawler

Web page crawler.

Simple site crawler using Capybara

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better target...

This gem is a web crawler sample code.So I don't reccmmend that you use.

Simple Gem Using Watir For Phantom Crawler

web crawler that generates a sitemap to a neo4j database. It will also store broken_links and total number of pages on site

Email crawler: crawls the top ten Google search results looking for email addresses and exports them to CSV.

A simply web crawler.

Generic Web crawler with a DSL that parses event-related data from web pages

Asynchronous web crawler, scraper and file harvester

An easy to use distributed web-crawler framework based on Redis

Crawl instagram photos, posts and videos for download.

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

A simple, fast web crawler