Project

scraper_rb

0.0
No commit activity in last 3 years
No release in over 3 years
scraper_rbpromptapi/scraper_rbHomepageDocumentationSource CodeBug TrackerWiki
Ruby wrapper for Prompt API's Scraper Checker API
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
5,823
Stars
0
Forks
0
Watchers
4
 Releases
Current version
0.1.2
Total releases
4
First release
2020-09-04
Latest release
2020-10-06
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2020-09-11
Reverse Dependencies
0

 Dependencies

Development

awesome_print
~> 1.8
bond
~> 0.5.1
colorize
~> 0.8.1
wirble
~> 0.1.3

Runtime

faraday
~> 1.0, >= 1.0.1
 Project Readme

Ruby Gem Version Build Status

Prompt API - Scraper API - Ruby Package

scraper_rb is a simple python wrapper for scraper-api.

Requirements

  1. You need to signup for Prompt API
  2. You need to subscribe scraper-api, test drive is free!!!
  3. You need to set PROMPTAPI_TOKEN environment variable after subscription.

then;

$ gem install scraper_rb

or; install from GitHub:

$ gem install scraper_rb --version "0.1.2" --source "https://rubygems.pkg.github.com/promptapi"

Example Usage

Basic scraper:

require "scraper_rb"

s = ScraperRb.new('https://pypi.org/classifiers/') # no params
s.get
s.response
# {
#     :headers=>{:"Content-Length"=>...}, 
#     :url=>"https://pypi.org/classifiers/",
#     :data=>"<!DOCTYPE html>\n<html> ...",
# }

s.response[:headers]     # => return response headers
s.response[:data]        # => return scraped html
s.save('/tmp/data.html') # => {:file=>"/tmp/data.html", :size=>321322}

# or

save_result = s.save('/tmp/data.html')
puts save_result[:error] if save_result.key?(:error) # we have a file error

You can add url parameters for extra operations. Valid parameters are:

  • auth_password: for HTTP Realm auth password
  • auth_username: for HTTP Realm auth username
  • cookie: URL Encoded cookie header.
  • country: 2 character country code. If you wish to scrape from an IP address of a specific country.
  • referer: HTTP referer header
  • selector: CSS style selector path such as a.btn div li. If selector is enabled, returning result will be collection of data and saved file will be in .json format.

Here is an example with using url parameters and selector:

require "scraper_rb"

params = {country: 'EE', selector: 'ul li button[data-clipboard-text]'}
s = ScraperRb.new('https://pypi.org/classifiers/', params)
s.get
s.response[:headers]       # => return response headers
s.response[:data]          # => return an array, collection of given selector
s.response[:data].length   # => 734 
s.save('/tmp/test.json')   # => {:file=>"/tmp/test.json", :size=>174449}

# or

save_result = s.save('/tmp/test.json')
puts save_result[:error] if save_result.key?(:error) # we have a file error

Default timeout value is set to 10 seconds. You can change this while initializing the instance:

s = ScraperRb.new('https://pypi.org/classifiers/', params={}, timeout=50) 
# => 50 seconds timeout w/o params

s = ScraperRb.new('https://pypi.org/classifiers/', params={country: 'EE'}, timeout=50) 
# => 50 seconds timeout

You can add extra X- headers:

s = ScraperRb.new('https://pypi.org/classifiers/', headers={'X-Referer': 'https://www.google.com'}) 

# or
s = ScraperRb.new('https://pypi.org/classifiers/', params={country: 'EE'}, headers={'X-Referer': 'https://www.google.com'}, timeout=50) 
# => 50 seconds timeout

headers param is a Hash, you can add key/value data. Header keys must star with X- prefix. More detail can found at Mozilla site.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org

$ rake -T

rake build            # Build bin_checker_rb-X.X.X.gem into the pkg directory
rake clean            # Remove any temporary products
rake clobber          # Remove any generated files
rake install          # Build and install bin_checker_rb-X.X.X.gem into system gems
rake install:local    # Build and install bin_checker_rb-X.X.X.gem into system gems without network access
rake release[remote]  # Create tag v0.0.0 and build and push bin_checker_rb-X.X.X.gem to rubygems.org
rake test             # Run tests
  • If you have PROMPTAPI_TOKEN you’ll have real http request based tests available.
  • Set RUBY_DEVELOPMENT to 1 for more verbose test results

License

This project is licensed under MIT

Contributer(s)

Contribute

Bug reports and pull requests are welcome on GitHub:

  1. fork (https://github.com/promptapi/scraper_rb/fork)
  2. Create your branch (git checkout -b my-feature)
  3. commit yours (git commit -am 'Add awesome features...')
  4. push your branch (git push origin my-feature)
  5. Than create a new Pull Request!

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.