Rocket Job

Process millions of records across thousands of workers.

Rocket Job is a distributed, priority-based, MongoDB-backed batch processing system for Ruby. Run conventional background jobs, or split a single job's input into slices and process it concurrently across thousands of workers, spilling from memory to disk so very large files never fall over.

Full documentation is at rocketjob.reidmorrison.com.

• Introduction — what Rocket Job is and why it exists
• Installation
• Programmer's Guide — simple jobs
• Batch Guide — parallel batch jobs
• Included Jobs and Directory Monitor
• Events
• Web UI (Mission Control)
• Deployment
• Architecture and Internals
• API Reference

• Ask questions in Rocket Job Discussions
• Report bugs

See the Upgrading guide for the code and data changes needed between major versions. Per-release notes are in the GitHub Releases.

Contributions are welcome. See CONTRIBUTING.md for the development setup and how to run the test suite, and the Architecture and Internals page for how Rocket Job is put together.

The documentation site lives in docs/ as Jekyll markdown. To preview changes locally:

cd docs
bundle update
jekyll serve

Then open http://127.0.0.1:4000 and edit the files under docs/.

This project uses Semantic Versioning.

Apache License v2.0. See LICENSE.txt.

Reid Morrison

Contributors