elastic-backup-2s3 – Snapshot, Backup, and Restore Elasicsearch Indicies to and from S3
When you have an itch to scratch…
I simply want to be able to control the backup and restoration of the Elasticsearch cluster to S3 without any fuss or having to dilly around with curl -XPUTS and friends.
I have finally added Shared Filesystem Support! Please see the docs below on this.
Elastic Backup simply allows you to specifiy some or all of the indicies, some target bucket, and run the backup / snapshot.
It hides all the sordid details about “registration” and the like, and will optionally clean up behind itself and remove the registration if you did not specify a name for the registration.
The New 1.x version is not backards-compatable with the 0.x versions. Especially for the File System-style backups, the repo name is not appended to the directory path, to allow for multiple repo backups on the file system. Please be aware of this moving forward.
To install from command line:
gem install elastic-backup-2s3
For a Gemfile entry:
Documentation is comming. Basically, just running the command without any parameters will give you some basic usage.
We adopt a convention to make it easy for you to specify the S3 bucket, path, and snapshot name stored in the bucket. The S3 URL format is:
In some cases, :SNAPSHOT is optional, but usually expected to be there.
Support for Shared Filesystem Snapshots
To do S3 backups requires no additional configurations outside of being in the AWS clould (or at least being connected to one.) Shared File System (FS) backups require that you have each and every node configured to permit the backup path on all your nodes in the cluster, and thus not as simple as merely writing it through the JSON REST interface.
So presumably you either have this setup in place, and in any rate, it would be a bit messy to have to specify the “local” path in the repo here.
For this reason we will not (over)write the repo settings for FS backups by default. We provide a new option that will allow you to write out the FS repo, specifying all the parameters you need to at that time.
The default repository name should be configurable in any case. we could do a configuration file, but for this I think it’s overkill. so we will use environment variables instead.
|ESB_DEFAULT_REPO||Changes the default repository from ‘elastic-backup’ to what is specified here.|
|ESB_SHARED_VOLUME||For Shared Volume backups, default local path to use. Note that this path must be configured on all nodes.|
|ESB_TIMEOUT||Default for –timeout option|
|ESB_SNAPSHOT_MAX_BYTES_SEC||Shared Volume max_snapshot_bytes_per_sec|
|ESB_RESTORE_MAX_BYTES_SEC||Shared Volume max_restore_bytes_per_sec|
To backup/snapshot your ES to S3 and wait until the backup is completed:
es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w
The default repository of ‘elastic-backup’ is used. To specify your own repository name:
es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --repo your-repo-name
To backup/snapshot two indices index1 and index2:
es-snapshot snapshot localhost s3://your-backup-bucket/your-backup-path:snap_1 -v2 -w --indices index1 index2
To restore your snapshot
es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w
To restore your snapshot using your-repo-name:
es-snapshot restore s3://your-backup-bucket/your-backup-path:snap_1 localhost -v2 -w --repo your-repo-name
You may specify –dryrun (-u) to see the JSON that will be sent without actually executing the commands. Also -v2 will show you the same while allowing your commands to execute.
To see the status of your snapshots:
es-snapshot list status localhost
To see a list of your indices:
es-snapshot list indices localhost
|2016-07-21||v1.0.0||Breaking changes in how the subdirectory paths are created. Please see Breaking Changes|
|2016-07-12||v0.1.5||Fixed single index restore bug.|
|2016-06-10||v0.1.3||Added a new –nv flag to switch off shared FS verification.|
|2016-05-02||v0.1.2||Fixed a couple of minor issues with the restore from shared volume.|
|2016-04-22||v0.1.0||Support for FS volume backups added|
|2016-04-17||v0.0.3||Added a delete snapshot option to allow deletion of a running snapshot.|
|2061-04-17||v0.0.3||‘delete’ options not fully implemented.|
|Error return codes not fully fleshed out|
|‘delete snapshot’ requires the full S3URL, but only needs the snapshot part.|
|2061-04-21||v0.0.4||Some features remain unimplemented, though there are options for them. Please bear with me.|
|What shall we do with configurations? I am thinking we save them in ES, but also allow for config files. The ES settings will override the config file settings. Config files in Yaml format.|
My own personal notes on this project. These are not gauranteed to be useful to you nor up-to-date.
Contributing to elastic-backup-2s3
- Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
- Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
- Fork the project.
- Start a feature/bugfix branch.
- Commit and push until you are happy with your contribution.
- Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
- Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
Copyright (c) 2016 Fred Mitchell. See LICENSE.txt for further details.