0.0
No commit activity in last 3 years
No release in over 3 years
Web crawler with JSON-based DSL and EventMachine-powered page fetching
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

~> 1.3
>= 0
>= 0

Runtime

~> 1.8.0
~> 1.5.9
 Project Readme

Rumba::Crawler

Dependency Status Build Status Code Climate Gem Version

Web crawler with JSON-based DSL and EventMachine-powered page fetching

Installation

Add this line to your application's Gemfile:

gem 'rumba-crawler'

And then execute:

$ bundle

Or install it yourself as:

$ gem install rumba-crawler

Usage

Gem supports "css", "root" and "regexp" service keys:

  • css: CSS locator for node
  • root: custom root for current node (parent node is used by default )
  • regexp: regular expression to extract data

If you have multiple objects of the same type, put object description into [ ].

Besides it supports shortcut for "css" attribute, so if you don't have any nested nodes(leaf node) and you don't need "root" and "regexp" keys, you can omit "css" key and provide locator as string value ("name": "span.name" in example).

Map your models to page structure like this:

[
  {
    "game": {
      "css":".games",
      "teams":[
        { 
          "css": ".teams",
          "name": "span.name",
          "scores": { 
            "css":".score",
            "regexp":"[0-9]+"
          },
          "sport": {
            "css":"#sport",
            "root":"body"
          }
        }
	  ]	
	}
  }
]

This will allow you to extract data from page for current model structure:

class Game
  # has_many :teams
  def teams=(array_value)
    array_value.each do |value|
      Team.find_or_create(value)
    end
  end
end
class Team
  # belongs_to :game
  # belongs_to :sport
  attr_writer :name, :scores
  def sport=(value)
    Sport.find_or_create(value)
  end
end
class Sport
  # has_many :teams
end

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request