arxiv-dl

Ruby gem to download papers and metadata from arxiv.org for offline archives.

For each paper it produces a self-contained per-paper folder with:

• The PDF
• The single-page abstract HTML
• The full HTML version (with images and a deduplicated, cross-paper shared assets cache)
• The TeX/LaTeX source tarball, extracted
• Four sidecar metadata files: metadata.md, metadata.yaml, metadata.json, metadata.bib

gem install arxiv-dl

Or in a Gemfile:

gem 'arxiv-dl'

arxiv-dl <ARXIV_ID_OR_URL> [<ARXIV_ID_OR_URL>...]

Accepted input forms:

-v and -q are mutually exclusive.

Precedence: CLI flag > ENV var > default.

A target that fails (unrecognized ID, no such paper, HTTP error, network failure) is reported on stderr as <target>: <message>, and the remaining targets still download. Exit status is 0 when every target succeeds and 1 when any fails.

Download a single paper to the default location:

arxiv-dl 2508.16190

Download to a custom directory:

arxiv-dl --path ~/Archives/arxiv 2508.16190

Download multiple papers, verbose:

arxiv-dl -v 2508.16190 1207.7214 cs/0002001

Download every paper listed in a file, or piped in:

arxiv-dl --input reading-list.txt
cat reading-list.txt | arxiv-dl --input -

Disable rate limiting (when running against a local mirror, etc):

arxiv-dl --rate-limit 0 2508.16190

$ARXIV_DOWNLOAD_PATH/                   # default: $HOME/Downloads/ArXiv_Papers
  _shared/                              # cross-paper deduplicated static assets
    arxiv.org/static/...
    cdn.jsdelivr.net/...
  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
    <arxiv-id>v<N>.pdf
    <arxiv-id>v<N>-abstract.html
    metadata.md                         # YAML frontmatter + Markdown body
    metadata.yaml
    metadata.json
    metadata.bib                        # upstream BibTeX, falls back to synthesized
    html/                               # absent when the paper has no HTML version
      <arxiv-id>v<N>.html               # path-rewritten to local assets
      x1.png, x2.png, ...               # paper-specific images
    src/                                # absent for PDF-only submissions
      *.tex, *.bbl, ...                 # extracted from /src/<id>v<N>

An unversioned ID (2508.16190) archives the latest version. A versioned ID (2508.16190v1) archives that version.

A paper with a single archived version v1 is kept flat, as above. When a paper has more than one version, each version gets its own v<N>/ folder with the same contents:

  YYYY/MM/DD/<primary_category>/<arxiv-id>-<slug>/
    v1/
      <arxiv-id>v1.pdf
      ...
    v2/
      <arxiv-id>v2.pdf
      ...

Archiving a second version of a flat paper first moves the existing files into v<N>/ (rewriting html/ links into _shared/ for the new depth). A paper whose latest version is v2 or later starts out in v<N>/ folders.

Each version downloads into a sibling .partial folder and is renamed into place only when every file succeeded. Re-running skips versions already archived and retries interrupted ones from scratch.

YYYY/MM/DD is the original submission date. <primary_category> is from the paper's metadata (cs.CL, math.NT, etc). <slug> is derived from the paper title (Unicode → ASCII, hyphenated, truncated to 80 chars at a word boundary).

For legacy IDs containing / (e.g. cs/0002001), the slash is replaced with - in the directory name and file names (cs-0002001-.../cs-0002001v1.pdf).

require 'arxiv/downloader'

identifier = Arxiv::Downloader::Identifier.new '2508.16190'
client     = Arxiv::Downloader::Client.new                # 3-second rate limit by default
path       = Arxiv::Downloader::Archive.new(identifier, root: '/tmp/papers', client: client).run
# => "/tmp/papers/2025/08/22/cs.CL/2508.16190-comicscene154-a-scene-dataset-for-comic-analysis"

script/setup    # install dependencies
script/test     # run specs and rubocop
script/console  # interactive prompt

Specs run offline against recorded fixtures in spec/fixtures/http/. To check those fixtures against the live arxiv API, run:

ARXIV_LIVE=1 script/test

MIT — see LICENSE.md.

This project follows the Contributor Covenant 3.0 — see CODE_OF_CONDUCT.md.