upload_inspector
Fail-closed structural inspection of untrusted uploaded files for any Ruby or Rails app. Checks a path, a Rails upload, an Active Storage attachment, any IO, or an S3 object uploaded through a presigned URL.
A file is accepted only when its structure is proven to match a known-safe shape. Anything unknown,
unparsable, encrypted, oversized, or erroring is rejected. Pure Ruby (marcel, rubyzip), no AV engine,
no native parsers, nothing is rendered or executed.
Not an antivirus. It blocks structural attack vectors: disguised executables, macros, PDF JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG and CSV injection. It does not detect malware signatures or exploits hidden inside otherwise valid content. Pair it with an AV scanner (e.g. Amazon GuardDuty Malware Protection for S3, endpoint AV) when you need that.
What it does and does not defend against: THREAT_MODEL.md. Report vulnerabilities privately: SECURITY.md.
What it checks
| Format | Checks |
|---|---|
| All | real type by magic bytes (Marcel) must be allowlisted; file extension must agree with content; size limit; file name without bidi overrides (invoice\u202Efdp.exe), control characters or an executable extension before the last one (shell.php.jpg, invoice.exe.pdf) |
PDF (and PDF-based .ai) |
rejects /JavaScript /JS /Launch /EmbeddedFile(s) /EF /FileAttachment /Filespec /Collection /RichMedia /XFA /SubmitForm /ImportData /GoToR /GoToE /Rendition, hex-escaped spellings included, also inside compressed object streams (located by /ObjStm or /First and parsed with a real PDF lexer, so comments, strings and fake headers cannot hide them); rejects risky parsers (/JBIG2Decode, 3D/U3D/PRC, PostScript XObjects), URI links other than http(s)/mailto/relative (file:, UNC, javascript:, indirect values), encrypted PDFs and unsupported object-stream filters |
| docx / pptx / xlsx | every part name must match an allowlist, and no check trusts a part's path (Office finds parts through content types and relationships): every part is inspected as XML or proven not to be XML, every XML part gets every check, and only relationship types known to be passive are accepted. Rejects VBA, embeddings/, activeX/, macro sheets, external data (connections, query tables, external/DDE/OLE links, Power Query) by part name, content type, relationship type and element, macro content types, DTDs, external relationships other than http(s)/mailto hyperlinks, Word DDE/INCLUDE(TEXT/PICTURE)/IMPORT/LINK/QUOTE/DATABASE/MACROBUTTON fields, PowerPoint program/macro actions, Excel DDE formulas and defined names, malformed XML, XML that is not UTF-8/ASCII/Latin-1; checks run on entity-decoded text and CDATA |
| PNG / JPEG / GIF / WebP | container walked block by block (nothing is decoded): every length must stay inside the file, declared dimensions within max_image_pixels, and nothing may follow the end of the image (appended scripts or archives). JPEG files holding several images back to back (HDR gain maps, MPF) are accepted; other trailers (motion-photo video, vendor data) are rejected, re-encode those |
| HEIC / HEIF / AVIF / MP4 / QuickTime | box structure walked (nothing is decoded): every box length must stay inside its parent and the boxes must cover the file exactly (no appended data), ftyp must come first and name a brand of the detected type, images need a meta box and video a moov box, declared image (ispe) and track (tkhd) dimensions within max_image_pixels. Files whose brand Marcel does not know are reported as QuickTime and rejected unless the brand is qt ; data hidden inside a box (e.g. a zip in a trailing free box) is not detected |
| SVG | static only: rejects script elements, on* handlers, javascript:/data: links (also set through animation), XHTML/MathML content, stylesheet processing instructions, DTDs other than the standard W3C DOCTYPE, CSS @import/escapes, and any reference outside the file (href, src, CSS url()) except #fragment, embedded `data:image/png |
| CSV / TSV / TXT | valid UTF-8 (BOM allowed; UTF-16 and legacy code pages rejected), no NUL or control characters other than tab/CR/LF, at most max_inflate_size. No HTML, SVG or server-side tags (<script, <html, <svg, <img, <iframe, <?php, <%, ..., any tag with an on*= handler, and at the start the signatures browsers sniff as HTML such as <p>, <div>, <!--; a<b and <b> mid-text are fine), which a sniffing browser would render as a page; allow_text_markup: true turns this off. CSV/TSV also reject cells a spreadsheet would run as a function call or DDE (=HYPERLINK(...), +cmd|..., @SUM(...)), found after any comma, semicolon or tab and inside quotes; negative numbers, phone numbers, @handles and plain arithmetic are accepted. allow_csv_formulas: true turns this off |
| zip | path traversal, symlinks, encrypted entries, duplicates, entry/total size and compression-ratio limits, nested archives; every member extracted (bounded) and inspected with the same policy |
| Others | rejected unless listed in allow_unverifiable (type + extension check only) |
Large PDFs can be rejected by chance
PDF names are searched in the raw bytes, compressed image data included, so random bytes that
happen to spell a short blocked name (/JS, /3D, /EF) reject a clean file as
:pdf_active_content or :pdf_risky_feature. The estimated chance is about 1.2% per MB of
compressed stream data: roughly 1 in 8 for a 10 MB PDF, 1 in 2 for 50 MB, 7 in 10 for 100 MB.
Text-only PDFs are unaffected. For image-heavy PDFs (scans, .ai artwork) keep uploads to a few
MB, or expect to ask users to re-export. Skipping stream data is not an option without
following the cross-reference table the way a reader does, because objects can hide in it.
Images, SVG, media and text are verified when their MIME type is in allowed_types
(image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime text/csv text/tab-separated-values text/plain). Listing them in allow_unverifiable
instead skips these checks.
Install
gem 'upload_inspector'Requires Ruby >= 3.2.
Usage
- Quick start
- What you can pass in
- Rails validator and job
- Presets
- Shadow mode and instrumentation
- Serving uploads safely
- Error messages
- S3 (presigned URL uploads)
- Protecting the server from large files
- Policy options
Quick start
ALLOWED_TYPES = %w[application/pdf image/png image/jpeg text/csv].freeze
result = UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
if result.safe?
# store the file
else
render json: { error: result.message }, status: :unprocessable_entity
endEvery call returns a Result and never raises on a hostile file:
| Method | Value |
|---|---|
safe? |
true when the file is accepted |
reason |
why it was rejected, e.g. :pdf_active_content, :extension_mismatch (nil when safe) |
message |
end-user message for the reason, e.g. "is password-protected" (nil when safe) |
detail |
technical hint for logs, not for end users |
detected_type |
the real MIME type, e.g. "application/pdf"
|
would_reject |
in mode: :report, the reason the file would have been rejected (nil otherwise) |
Only types in allowed_types are accepted (default: PDF only). See What it checks
for the types that get a structural check.
What you can pass in
check takes the upload object itself. Rails/Rack uploads and Active Storage carry the
client-supplied name; anything else needs filename: (its extension must match the content).
# Controller upload (ActionDispatch::Http::UploadedFile)
UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
# Active Storage attachment or blob (downloaded to a temp file first)
UploadInspector.check(contract.document, allowed_types: ALLOWED_TYPES)
# Path, File, Tempfile, StringIO or any IO
UploadInspector.check("/tmp/upload.bin", filename: "report.pdf", allowed_types: ALLOWED_TYPES)
UploadInspector.check(io, filename: "report.pdf", allowed_types: ALLOWED_TYPES)Inspection takes up to timeout (60 s by default): check large files in a background job.
Rails validator and job
Both load on first use; the gem does not depend on Rails.
class Contract < ApplicationRecord
has_one_attached :document
# Checks files uploaded with this request, before save. Keep it to small files.
validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES }
endThe error type is the rejection reason (errors.details shows { error: :pdf_active_content }),
the message comes from result.message. The validator covers UploadedFiles and
attach(io:, filename:). Blobs already in storage (direct uploads, signed blob ids) are skipped,
and form strings are never read as server paths. Check those with the job:
# After save, the file is in storage. Unsafe attachments are purged.
UploadInspector::InspectJob.perform_later(contract, "document", allowed_types: ALLOWED_TYPES)
# React to a rejection: subclass and override on_unsafe (called after the purge).
class InspectUploadJob < UploadInspector::InspectJob
def on_unsafe(record, name, attachment, result)
UploadMailer.rejected(record, attachment.filename.to_s, result.message).deliver_later
end
endPass purge: false to keep unsafe files. Options are the same keywords as check.
Presets
UploadInspector::Presets holds frozen allowed_types lists of verified types, to combine freely.
The default stays PDF only.
| Preset | Types |
|---|---|
PDF |
|
OFFICE |
docx, xlsx, pptx |
DOCUMENTS |
PDF + OFFICE
|
IMAGES |
PNG, JPEG, GIF, WebP, HEIC, HEIF, AVIF |
SVG |
SVG |
MEDIA |
MP4, QuickTime |
TEXT |
plain text, CSV, TSV |
UploadInspector.check(file, allowed_types: UploadInspector::Presets::IMAGES + UploadInspector::Presets::DOCUMENTS)Shadow mode and instrumentation
To roll out on a live app, measure first: mode: :report runs every check but always accepts,
and sets result.would_reject to the reason it would have rejected. The validator never adds an
error and the job never purges.
validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES, mode: :report }When ActiveSupport is loaded, every check / check_s3 publishes check.upload_inspector
(the event's duration is the check time):
ActiveSupport::Notifications.subscribe("check.upload_inspector") do |event|
# event.payload: { mode:, reason:, would_reject:, detected_type:, filename_ext:, size: }
rejected = event.payload[:reason] || event.payload[:would_reject]
Rails.logger.info("upload_inspector #{rejected || 'safe'} #{event.payload[:filename_ext]} #{event.duration.round}ms")
endServing uploads safely
A checked file can still be dangerous if the app serves it so a browser renders it as a page.
- Serve user files from a separate domain (e.g.
usercontent.example.comor the S3/CDN domain), never from the app's own origin, so a rendered file cannot read the app's cookies. - Send
Content-Disposition: attachment(download, not render) except for raster images and video, and alwaysX-Content-Type-Options: nosniff. A strict CSP withsandboxblocks scripts if a file is rendered anyway. Keep SVG and PDF as attachments: browsers run SVG script when the SVG is opened directly, and Chrome refuses to show PDFs under asandboxCSP. - Never reuse the client's name as is.
UploadInspector.safe_filename(name, detected_type:)drops the path, control and bidi characters, inner dots (x.php.jpg→x_php.jpg) and Windows reserved names, caps it at 255 bytes, and takes the extension from the detected type.
safe_headers builds those headers (inline: true is honoured for raster images and video only):
result = UploadInspector.check(path, filename: name, allowed_types: ALLOWED_TYPES)
response.headers.merge!(UploadInspector.safe_headers(result, filename: name, inline: true))
# Content-Type (application/octet-stream unless safe), Content-Disposition,
# X-Content-Type-Options: nosniff, Content-Security-Policy: default-src 'none'; ...; sandboxActive Storage serves only content_types_allowed_inline inline and forces
content_types_to_serve_as_binary (HTML, SVG, ...) to download; keep SVG and text in the latter.
For S3, set the headers on the presigned GET:
s3_object.presigned_url(:get, response_content_disposition: "attachment; filename=\"#{safe_name}\"",
response_content_type: result.detected_type)Error messages
result.message gives an end-user message for the reason ("is password-protected", "does not
match its file extension", ...). Translate or override any of them under
upload_inspector.errors.<reason>, with upload_inspector.errors.default as the fallback:
vi:
upload_inspector:
errors:
default: "chứa nội dung không được phép"
too_large: "quá lớn"S3 (presigned URL uploads)
Files uploaded straight to S3 never pass through the app, and the browser chose their
Content-Type and size. check_s3 reads the size with HeadObject (oversized objects are
rejected without downloading), downloads the object pinned to that ETag, and inspects the bytes.
client is any Aws::S3::Client (aws-sdk-s3 is not a dependency of this gem).
s3 = Aws::S3::Client.new
result = UploadInspector.check_s3(
client: s3,
bucket: "uploads",
key: "incoming/abc/report.pdf",
filename: "report.pdf", # optional, defaults to the key basename
version_id: nil, # optional, pin a version on versioned buckets
allowed_types: ALLOWED_TYPES,
)
s3.delete_object(bucket: "uploads", key: "incoming/abc/report.pdf") unless result.safe?S3 errors (missing key, access denied, network) raise, so a job retries rather than
deleting a file over an outage. Upload to a quarantine prefix nothing serves, and move the object
to its final key only once it is safe: a presigned URL can overwrite the object after the check
until it expires (use copy_object with copy_source_if_match and the checked ETag).
Protecting the server from large files
No input is read into Ruby memory: S3 objects and IO are streamed to a temp file, Rails uploads are already on disk (Rack buffers them), and inspection runs in the memory-limited sandbox. Formats like zip, Office and PDF need random access, so the whole file must be on disk. What is left to bound is disk and time:
-
Size before download.
check_s3rejects fromHeadObject, Active Storage frombyte_size, IO stops atmax_file_size + 1bytes. Setmax_file_sizeto what you actually accept, not the 500 MB default. -
Refuse oversized uploads at the source. Rails: cap the request body in the web server (nginx
client_max_body_size). S3: hand out a presigned POST with a size range; a presigned PUT cannot enforce one.post = Aws::S3::Bucket.new("uploads").presigned_post( key: "incoming/#{SecureRandom.uuid}/${filename}", content_length_range: 1..(20 * 1024 * 1024), expires: Time.now + 300, )
-
Download time.
timeoutcovers inspection only, not the S3 download. Bound the download on the client you pass in, e.g.Aws::S3::Client.new(http_open_timeout: 5, http_read_timeout: 30, retry_limit: 2). -
Disk under load. Each concurrent check can use up to
max_file_sizeof temp disk (plus extracted zip members, up tomax_zip_total_size). Run checks in a job queue with capped concurrency and size the temp volume forconcurrency × max_file_size.
Policy options
| Option | Default |
|---|---|
allowed_types |
['application/pdf'] |
allow_unverifiable |
[] |
max_file_size |
500 MB |
max_zip_entries / max_zip_entry_size / max_zip_total_size
|
5 000 / 200 MB / 1 GB |
max_compression_ratio |
100 (entries ≥ 1 MB) |
max_inflate_size |
50 MB per PDF object stream / OOXML XML part / SVG or text file |
max_image_pixels |
100 000 000 (declared width × height) |
allow_csv_formulas |
false |
allow_text_markup |
false |
timeout / cpu_limit
|
60 s / 60 s |
memory_limit |
1 GB (RLIMIT_DATA, Linux only) |
Sandbox
By default (sandbox: true) inspection runs in a child Ruby process with CPU, memory (Linux) and
wall-clock limits. A crash, hang or memory blow-up kills only the child and yields a rejection
(:timeout, :resource_limit, :inspection_failed), so web/job workers are unaffected.
The child's temp files live in a directory the parent removes, also when the child is killed.
It is resource isolation, not a security boundary (same OS user).
Development
Tests build hostile-shaped samples in memory (no real malware, no AV test signatures). Run them in a container or CI rather than on a workstation with endpoint AV:
docker run --rm -v "$PWD":/src:ro ruby:3.3 \
bash -c 'cp -r /src /app && cd /app && rm -rf vendor .bundle && bundle install && bundle exec rake test'CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x. The Rails helper tests
use activemodel / activejob from the Gemfile (development only).
test/sample_virus_malware.rb is the hostile-sample corpus: every attack shape with its expected
verdict, plus known gaps (skipped until fixed). bundle exec rake samples[DIR] writes the inert
samples to disk (default tmp/samples) to cross-check against real readers or other scanners.
License
MIT