Crawl the senegalese web, looking for jobs using the excellent wombat gem

A web crawler written in ruby

This will be crawling data from websites. Need to give the xpaths clearly. Will be updating with new functionalities in future

Allows to crawl bandcamp sites, including release and track information

MurmuringSpider is a concise Twitter crawler. When we write a data-mining / text-mining application based on twitter timeline, we have to collect and store tweets first. I am irritated with writing such crawler repeatedly, so I wrote this. What you have to do is only to add query and to run th...

crawler of niconico-douga

Web crawler to get courses from XML pages at mymav.uta

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, si...

Allows your rails application to be spiderable by crawlers

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-core gem contains core facilities and notably, does not contain such facilities as database-backed state management.

Iudex is a general purpose web crawler and feed processor in ruby/java. The iudex-brutefuzzy-protobuf gem contains the protocol buffer generated java classes for the iudex-brutefuzzy-service.