Project

gem

0.0
No commit activity in last 3 years
No release in over 3 years
gemsj26/gemHomepageDocumentationSource CodeBug TrackerWiki
Just enough not-Rubygems to index a collection of gems for download... and maybe more.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
26,384
Stars
4
Forks
2
Watchers
4
 Releases
Current version
0.0.1.alpha
Total releases
1
First release
2012-03-30
Latest release
2012-03-30
 Development
Primary Language
Ruby
Average date of last 50 commits
2012-06-17
Reverse Dependencies
0

 Dependencies
 Project Readme

Gem

Just enough not-Rubygems to index a collection of gems for download... and maybe more.

There are some severe caveats. Some are by design, others will be addressed later.

Created by Samuel Cochran for Railscamp X.

Thoughts

Rubygems is a bit... horrible. It was a good fit for the problem it solved for a long time, but we've outgrown it. Here are a list of my thoughts on what's wrong and how we could fix it. Many of these ideas are or will permeate this gem gem as a (mostly backwards-compatible) drop-in replacement for rubygems.

Local index should be more granular and allow stream processing. Same with specs lists. Marshalling and constantly re-downloading a whole index, or even just a list of specs, is ludicrous at current sizes.

Break index up into more files for efficient lookup?

Learn from PIP, have quick lookups for common cases.

JSON is a better choice than YAML—it's just plain faster and becoming ubiquitous. Built-in (usually reasonably fast) support in 1.9, including streaming, which is cross-platform friendly (much moreso than YAML).

Auto-negotiated transport compression should counteract any size difference and work for partial updates.

Compression is purely a transport/storage concern.

Index journal for efficient partial updates via HTTP.

Use plain old HTTP (but the full functionality of 1.1) and static files to allow serving at edges by cloudfront, easy proxying/caching, etc.

Same directory structure and index format locally and remotely, backwards compatible:

Given the following definitions:

  • <basename>: <name>-<version>[-<platform>]
    • <name>: gem name, i.e. "rails"
    • <version>: gem version, i.e. "3.2.2"
    • <platform>: gem platform, omitted if platform is "ruby", the default

Proposed directory structure:

  • metadata.index — newline separated list of all names, append only.
  • metadata/ — index of gems.
    • <name>.index — newline separated list of all basenames for a name, append only.
    • <name>/ -
      • <version>[-<platform>].json – metadata for the gemball.
  • source.json — metadata about this source. May contain canonical URL.
  • sources/ — cached source indexes.
    • <source-sha>/ — SHA of canonical source URL (i.e. http://rubygems.org)
      • source.json — metadata about the source.
      • metadata.index/ — a cache of the top-level metadata.index for the source
      • metadata/ — a cache of the top-level metadata/ for the source
  • gems/ — gemballs and unpacked trees.
    • <basename>.gem — the gemball.
    • <basename>/ — the unpacked gem tree. Optional if it allows zip loading.

Use net/http/persistent per-source for gem operations like rubygems-mirror, respecting HTTP content-type, transport-encoding (compression), range and freshness controls, and negating overhead of many small files.

Multiplex mirroring and potentially other operations over a pool of threads like rubygems-mirror.

Break old marshalling support into modules, only include them when backwards-compatible behaviour required (i.e. during upgrade, when compat is requested/configured).

Add more checks and guards to make sure the index/gemspecs/gems can't get into an invalid state.

License

MIT (see LICENSE). Some parts adapted from Rubygems, which is under the Ruby or MIT license.