Consyncful

Contentful -> MongoDB synchronisation for Rails.

Requesting complicated models from the Contentful Delivery API in Rails applications is often too slow, and makes testing applications painful. Consyncful uses Contentful's synchronisation API to keep a local, up-to-date copy of the entire content in a Mongo database.

Once the content is available locally, finding and interact with contentful data is as easy as using Mongoid ODM.

This gem doesn't provide any integration with the management API, or any way to update Contentful models from the local store. It is strictly read only.

flowchart TD
  CF["Contentful (Delivery API / Sync API)"]
  SY["rake consyncful:sync"]
  DB["MongoDB (single collection)"]
  APP["Rails models (Mongoid, subclass Consyncful::Base)"]

  CF -->|sync| SY
  SY -->|writes| DB
  APP -->|queries| DB

• Setup
  • Installation
  • Configuration options
• Creating contentful models in your Rails app
• Synchronizing contentful data
  • Continuous sync
  • Refresh from scratch
  • Enabling webhook mode
  • Sync callbacks
  • Sync specific contents using contentful tags
• Finding and interacting with models
  • Querying
  • References
  • Finding entries from different content types
  • Using Locales for specific fields
  • Preserving Contentful timestamps
• MongoDB Configuration
  • Choosing the Mongo Database
  • Why MongoDB?
• Development
• Contributing
• License

Add this line to your application's Gemfile:

gem 'consyncful'

And then execute:

$ bundle

If you don't already use Mongoid, generate a mongoid.yml by running:

$ rake g mongoid:config

Create config/initializers/consyncful.rb. An example with common configuration is:

Consyncful.configure do |config|
  config.locale = 'en-NZ'
  config.contentful_client_options = {
    api_url: 'cdn.contentful.com',
    space: 'space_id',
    access_token: 'ACCESS_TOKEN',
    environment: 'master',        # optional
    logger: Logger.new(STDOUT)    # optional for debugging
  }

  # optional to override the Sync API defaults
  config.contentful_sync_options = { 
    limit: 50
  }
end

Create models by inheriting from Consyncful::Base

class ModelName < Consyncful::Base
  contentful_model_name 'contentfulTypeName'
end

Model fields will be dynamically assigned, but Mongoid dynamic fields are not accessible if the entry has an empty field. If you want the accessor methods to be reliably available for fields it is recommended to define the fields in the model:

class ModelName < Consyncful::Base
  contentful_model_name 'contentfulTypeName'

  field :title
  field :is_awesome, type: Boolean
end

Contentful reference fields are a bit special compared with standard Mongoid associations. Consyncful provides the following helpers to set up the correct relationships:

class ModelWithReferences < Consyncful::Base
  contentful_model_name 'contentfulTypeName'

  references_one :thing
  references_many :other_things
end

Consyncful supports two sync modes:

• Polling (default) — checks Contentful on an interval and syncs changes.
• Webhook — Contentful calls your app; the worker syncs when a webhook arrives.

Run the same task in both modes — the behaviour depends on your configuration:

$ rake consyncful:sync

• Polling mode: after the initial full sync, the worker polls every 15s (configurable) and applies changes it finds.
• Webhook mode: after the initial full sync, the worker does not poll. It waits for a webhook signal and then runs a sync.

If you want to resynchronize everything (e.g., after model/content type renames), run:

$ rake consyncful:refresh

This performs a full rebuild of data from contentful.

It is recommended to refresh your data if you change model names.

Now you've synced your data, it is all available via your Rails models.

# e.g. config/initializers/consyncful.rb
Consyncful.configure do |c|
  c.sync_mode = :webhook
end

Expose the engine so Contentful can POST to it

# config/routes.rb
mount Consyncful::Engine, at: "/consyncful"

The webhook endpoint lives under this mount (e.g. /consyncful/trigger_sync).

Webhook authentication is on by default:

Consyncful.configure do |c|
  c.webhook_authentication_required = true   # default
  c.webhook_user     = ENV["CONSYNCFUL_WEBHOOK_USER"]
  c.webhook_password = ENV["CONSYNCFUL_WEBHOOK_PASSWORD"]
end

To accept webhooks without auth (not recommended), explicitly disable it:

c.webhook_authentication_required = false

In your Contentful space/environment, add a webhook that points to your mounted route (e.g. https://your-app.example.com/consyncful/trigger_sync) and select which events should trigger a sync (publish/unpublish, entries, assets, etc.). See Contentful documents here for information on setting up a webhook: Configuring a webhook

You may want to attach some application logic to happen before or after a sync run, for example to update caches.

Callbacks can be registered using:

Consyncful::Sync.before_run do
  # do something before the run
end

Consyncful::Sync.after_run do |updated_ids|
  # invalidate cache for updated_ids, or something
end

You can configure Consyncful to sync or ignore specific contents using Contentful Tag.

Consyncful.configure do |config|
  # Any contents tagged with 'myTag' will be stored in the database.
  # Other contents without 'myTag' would be ignored.
  config.content_tags = ['myTag'] # defaults to []
end

Also, you can ignore contents with specific Tags.

Consyncful.configure do |config|
  # Any contents tagged with 'ignoreTag' won't be stored in the database.
  config.ignore_content_tags = ['ignoreTag'] # defaults to []
end

Models are available using standard Mongoid queries.

instance = ModelName.find_by(instance: 'foo')

instance.is_awesome # true

References work like you would expect:

instance = ModelWithReferences.find('contentfulID')

instance.thing # returns the referenced thing
instance.other_things # all the referenced things, polymorphic, so might be different types

Except: references_many associations return objects in a different order from how they are ordered in Contentful. If you want them in the order they appear in Contentful, use the .in_order helper:

instance.other_things.in_order # ordered the same as in Contentful

Because all Contentful models are stored as polymorphic subtypes of Consyncful::Base, you can query all entries without knowing what type you are looking for:

Consyncful::Base.where(title: 'a title') # [ #<ModelName>, #<OtherModelName> ]

If fields have multiple locales then the default locale will be mapped to the field name. Additional locales will have a suffix (lower snake case) on the field name. e.g title (default), title_mi_nz (New Zealand Maori mi-NZ)

If you need to access the timestamps from Contentful, you can enable it by setting preserve_contentful_timestamps to true.

Consyncful.configure do |config|
  # Consyncful models will have two extra fields that contains the value of timestamps in Contentful.
  # contentful_created_at
  # contentful_updated_at
  config.preserve_contentful_timestamps = true # defaults to false
end

You can configure which Mongoid client Consyncful uses, as well as the name of the collection where entries are stored. This is useful if you want Consyncful data to live in a separate MongoDB database from your application-specific database.

Consyncful.configure do |config|
  config.mongo_client = :consyncful # defaults to :default (referencing the clients in mongoid.yml)
  config.mongo_collection = 'contentful_models' # this is the default
end

Consyncful currently only supports the Mongoid ODM because models require dynamic schemas. Extending support to ActiveRecord could be possible in the future, but it would also require maintaining database migrations alongside Contentful content type changes—which adds complexity we wanted to avoid.

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/boost/consyncful.

The gem is available as open source under the terms of the MIT License.