Multithreaded web crawler with transparent DSL and requests caching.

A very simple crawler for RubyGems.org used to demo the power of ElasticSearch at RubyConf 2013

Easy to use DSL that helps scraping data from websites. Thanks to it, writing web crawlers would be very fast and intuitive. Traversing through html nodes and fetching all of the HTML attributes, would be possible. Just like in jQuery - you will find methods like parent, children, first, find, siblings etc. Furthermore, you are able to download images, web pages, and store all content in the database. Please visit my Github account for more details.

Allows your rails application to be spiderable by crawlers

The application crawls a URL and extracts links, tags and sequences. These features are written to an output file

plugin-container totally sucks..

A simple web crawler gem

a simple web crawler using DSL of capybara

A simple web crawler gem

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!

s7o crawler optimized for programmer happiness and sustainable productivity.