dl-core
Shared foundation for the <site>-dl gems that build offline archives: arxiv-dl, jstor-dl, and more.
What's in it
| Class | Purpose |
|---|---|
DL::Core::Client |
Polite HTTP: User-Agent naming the gem, minimum interval between requests, timeouts, retries on 429/503 (honoring Retry-After) |
DL::Core::Error, DL::Core::HTTPError
|
Base error for every gem; HTTPError for non-success responses (status, url) |
DL::Core::Author |
name and affiliations
|
DL::Core::Slug |
Title → URL-safe slug, truncated at a word boundary |
DL::Core::PaperFolder |
A paper's folder: flat while one version is archived, v<N>/ folders once there are several; override moved_into to fix up files moved one level deeper |
DL::Core::Sidecar::YAML, JSON
|
metadata.yaml / metadata.json from any metadata Data object |
DL::Core::Sidecar::Markdown |
metadata.md: YAML frontmatter from the metadata, plus a body each gem writes |
DL::Core::CLI |
The shared command line: targets as args or --input FILE|-, -p, --rate-limit, -v, -q, per-target errors, exit 1 on any failure |
Building a <site>-dl gem on it
Each gem's errors subclass DL::Core::Error, and its CLI subclasses DL::Core::CLI:
module Example
module Downloader
class Error < DL::Core::Error; end
class CLI < DL::Core::CLI
def program = 'example-dl'
def target_name = 'EXAMPLE_ID_OR_URL'
# EXAMPLE_DOWNLOAD_PATH, EXAMPLE_RATE_LIMIT
def env_prefix = 'EXAMPLE'
def default_path = File.join(Dir.home, 'Downloads', 'Example_Papers')
def version = VERSION
def user_agent = "example-dl/#{VERSION} (+https://github.com/xoengineering/example-dl)"
def identifier_for(target) = Identifier.new(target)
def archive_for(identifier, root:, client:) = Archive.new(identifier, root:, client:)
end
end
endidentifier_for raises a DL::Core::Error subclass for input the gem doesn't recognize; the object it returns responds to id. archive_for returns an object whose run archives the paper and returns its folder.
Development
script/setup # install dependencies
script/test # run specs and rubocop
script/console # interactive promptLicense
MIT — see LICENSE.md.
Code of Conduct
This project follows the Contributor Covenant 3.0 — see CODE_OF_CONDUCT.md.