ScraperWikiMorph

A simple compatibility layer so that you can use something that looks almost the same as the ScraperWiki gem - the only real difference is that it writes to an SQLite database called data.sqlite (rather than ScraperWiki's default scraperwiki.sqlite) and a table called data (rather than ScraperWiki's default swdata).

This makes it very easy to transition scraper code from ScraperWiki to morph.io.

Ruby 3.2 or later.

Add this line to your application's Gemfile:

gem 'scraperwiki-morph'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install scraperwiki-morph

Use ScraperWikiMorph exactly as you would use ScraperWiki - every method is delegated to the scraperwiki gem, with the database preconfigured for morph.io:

require 'scraperwiki-morph'

# Save a record (upserted on the unique keys) to the "data" table
# in data.sqlite
ScraperWikiMorph.save_sqlite([:id], { id: 1, name: "Alice" })

# Save to a different table
ScraperWikiMorph.save_sqlite([:id], { id: 1, name: "Alice" }, "people")

# Query the data back
ScraperWikiMorph.select("* from data")
# => [{"id" => 1, "name" => "Alice"}]

# Persist scraper state between runs
ScraperWikiMorph.save_var("last_run", "2026-08-12")
ScraperWikiMorph.get_var("last_run")

# Fetch a page
html = ScraperWikiMorph.scrape("https://example.com/")

After checking out the repo, install dependencies and run the tests:

$ bundle install
$ bundle exec rspec

The test suite runs with SimpleCov coverage reporting and uses WebMock so it never touches the network.

Releases are published to RubyGems.org automatically by the Release workflow using RubyGems trusted publishing (OIDC) - no API keys are stored in the repository.

To release a new version:

1. Create a branch (e.g. release/v0.2.0) and bump the version number in lib/scraperwiki-morph/version.rb following Semantic Versioning
2. Move the entries under "Unreleased" in CHANGELOG.md into a new section for the version, dated today, and update the comparison links at the bottom
3. Open a pull request and get it reviewed and merged
4. When the version bump lands on main, the Release workflow builds the gem, pushes the vX.Y.Z git tag and publishes to RubyGems.org

The workflow checks RubyGems.org first and skips publishing if the version already exists, so re-runs and merges that don't change the version are safe. If it can't get an answer from RubyGems.org the job fails rather than skipping quietly, so a red Release job may mean the check couldn't reach RubyGems.org rather than that publishing failed. Re-run it once the API is reachable again.

Publishing will fail until a gem owner performs this one-time setup:

1. On rubygems.org, sign in as an owner of scraperwiki-morph, go to the gem's Settings → Trusted publishers (see the RubyGems guide) and add a GitHub Actions publisher with:
   • Repository owner: openaustralia
   • Repository name: scraperwiki-morph
   • Workflow filename: release.yml
   • Environment: rubygems
2. On GitHub, in this repository go to Settings → Environments and create an environment named rubygems. Under Deployment branches and tags, restrict deployments to the main branch - the workflow also refuses to run off main, but the environment rule enforces it at the platform level. Optionally add further protection rules (e.g. required reviewers) to control who can trigger a release.

1. Fork it
2. Create your feature branch (git checkout -b feature/my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin feature/my-new-feature)
5. Create new Pull Request

The gem is available as open source under the terms of the MIT License.