WebInspector

Ruby gem to inspect web pages. It scrapes a given URL and returns its title, description, meta tags, links, images, and more.

Installation

Add this line to your application's Gemfile:

gem 'webinspector'

And then execute:

$ bundle

Or install it yourself as:

$ gem install webinspector

Usage

Initialize a WebInspector instance

page = WebInspector.new('http://example.com')

With options

page = WebInspector.new('http://example.com', {
  timeout: 30,                         # Request timeout in seconds (default: 30)
  retries: 3,                          # Number of retries (default: 3) 
  headers: {'User-Agent': 'Custom UA'} # Custom HTTP headers
})

Accessing response status and headers

page.response.status  # 200
page.response.headers # { "server"=>"apache", "content-type"=>"text/html; charset=utf-8", ... }
page.status_code      # 200
page.success?         # true if the page was loaded successfully
page.error_message    # returns the error message if any

Accessing page data

page.url           # URL of the page
page.scheme        # Scheme of the page (http, https)
page.host          # Hostname of the page (like, example.com, without the scheme)
page.port          # Port of the page
page.title         # title of the page from the head section
page.description   # description of the page
page.links         # array of all links found on the page (absolute URLs)
page.images        # array of all images found on the page (absolute URLs)
page.meta          # meta tags of the page
page.favicon       # favicon URL if available

Working with meta tags

page.meta                 # all meta tags
page.meta['description']  # meta description
page.meta['keywords']     # meta keywords
page.meta['og:title']     # OpenGraph title

Filtering links and images by domain

page.domain_links('example.com')  # returns only links pointing to example.com
page.domain_images('example.com') # returns only images hosted on example.com

Searching for words

page.find(["ruby", "rails"]) # returns [{"ruby"=>3}, {"rails"=>1}]

JavaScript and Stylesheets

page.javascripts  # array of all JavaScript files (absolute URLs)
page.stylesheets  # array of all CSS stylesheets (absolute URLs)

Language Detection

page.language  # detected language code (e.g., "en", "es", "fr")

Structured Data

page.structured_data  # array of JSON-LD structured data objects
page.microdata        # array of microdata items
page.json_ld          # alias for structured_data

Security Information

page.security_info  # hash with security details: { secure: true, hsts: true, ... }

Performance Metrics

page.load_time  # page load time in seconds
page.size       # page size in bytes

Content Type

page.content_type  # content type header (e.g., "text/html; charset=utf-8")

Technology Detection

page.technologies  # hash of detected technologies: { jquery: true, react: true, ... }

HTML Tag Statistics

page.tag_count  # hash with counts of each HTML tag: { "div" => 45, "p" => 12, ... }

Export all data to JSON

page.to_hash # returns a hash with all page data

Contributors

Steven Shelby (@stevenshelby)
Sam Nissen (@samnissen)

License

The WebInspector gem is released under the MIT License.

Contributing

Fork it ( https://github.com/davidesantangelo/webinspector/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

webinspector

Development

Runtime