Project

rwspider

0.0
No commit activity in last 3 years
No release in over 3 years
rwspiderrinzi/rwspiderHomepageDocumentationSource CodeBug TrackerWiki
RW Spider is an multithreading spider client written in Ruby designed to make easy the development of programs that spider the web.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
8,001
Stars
1
Forks
0
Watchers
1
 Releases
Current version
0.4.3
Total releases
2
First release
2010-02-05
Latest release
2010-02-06
 Development
Primary Language
Ruby
Average date of last 50 commits
2010-01-18
Reverse Dependencies
0

 Dependencies

Development

echoe
~> 3.1
hpricot
~> 0.8.2
rake
~> 0.8
robotstxt
~> 0.5.2
 Project Readme

RW Spider¶ ↑

RW Spider is an multithreading spider client written in Ruby. The library was designed to make easy the development of programs that spider the web.

RW Spider design comes from the direct experiences of the development of another PHP library that is currently used as the engine for an freeware SEO tool.

Features¶ ↑

  • Multithreading spider

  • Customizable options for the spider job

  • Robots.txt support

  • Indexing of web pages and others files (images, CSS, JavaScript, PDF and more)

  • Following redirects

Requirements¶ ↑

  • Ruby >= 1.8.7

  • Hpricot >= 0.8.2

  • Robotstxt >= 0.5.2

Installation¶ ↑

This library is intended to be installed via the Gemcutter system.

$ gem install rwspider

You might need administrator privileges on your system to install it.

Author¶ ↑

Author

Simone Rinzivillo <srinzivillo@gmail.com>

Resources¶ ↑

Changelog¶ ↑

See the CHANGELOG.rdoc file for details.

License¶ ↑

Copyright © 2009 Simone Rinzivillo, RW Spider is released under the MIT license.