Headless HTTP crawler/scraper

Server browser and Crawler for many games (L4D2, TF2, CS:S, KZMOD, The Ship)

Simple async HTTP crawler based on em-synchrony

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-service provides a fuzzy simhash lookup index as a distributed service.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

Mobile App Review Crawler

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

DSL to build crawlers easily

Ruby web crawler to access omelete informations

Simple little website crawler.

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

The Baidu Crawler is to crawl data with your demmand

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success

JavaScript enabled web crawler kit

CIA World Factbook crawler

Gem for crawling data from external sources

is_crawler does exactly what you might think it does: determine if the supplied string matches a known crawler or bot.

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.

Rack Middleware adhering to the Google Ajax Crawling Scheme, using a headless browser to render JS heavy pages and serve a dom snapshot of the rendered state to a requesting search engine.