Iriq — IRI Query

Iriq finds the shape of a URL — the structural template you get when you erase the parts that vary and keep the parts that don't. …/users/123 and …/users/999 are the same shape: /users/{user_id}. Feed iriq a pile of messy URLs — a log file, a column of links, free-text prose — and it collapses them into a small set of stable, deterministic route templates. Fifty thousand distinct URLs become twelve shapes.

What's an IRI? Internationalized Resource Identifiers cover everyday URLs https://…, plus URNs like urn:isbn:0451450523, other schemes like mailto:, and internationalized addresses with non-ASCII characters like https://例え.jp/パス. Formally it's the Unicode superset of URI/URL. The name is IRI Query: iriq queries an IRI for its structure.

Everything iriq does — parsing, normalizing, classifying path and query components, clustering, learning new patterns — exists to derive, render, or group by that shape.

And it gets sharper the more you feed it. A corpus — on by default — records what it sees and improves classifications as data flows in: high-churn slots get promoted to placeholders, and whole types emerge that no single URL can reveal (a position that's always 100–599 is an HTTP status; one bounded to a dozen values is an enum).

$ iriq -n https://foo.com/users/123
https://foo.com/users/{user_id}

It answers questions like:

• "What routes does this service actually expose?" (cluster a log file)
• "Which params are stable identifiers vs. churning IDs vs. enums?" (--stats)
• "Are these 50,000 distinct URLs really just 12 templates?" (clustering)
• "What does /api/v1/users/abc-123-def become as a route shape?" (/api/{version}/users/{user_id})

Iriq ships as a command-line tool (iriq) and a Rust library.

$ iriq https://foo.com/users/123
# parse
original:      https://foo.com/users/123
kind:          url
scheme:        https
host:          foo.com
path_segments: ["users", "123"]
canonical:     https://foo.com/users/123

# normalize
https://foo.com/users/{user_id}

$ iriq -n https://foo.com/users/123
https://foo.com/users/{user_id}

$ iriq -n https://shop.com/pricing/usd?currency=eur
https://shop.com/pricing/USD?currency=EUR     # currency upcased

$ cat access.log | iriq                       # ≥ 10 IRIs → cluster view
[190] docs.example.com  /users/{user_id}
[186] app.example.com   /users/{user_id}
...

$ cat access.log | iriq --stats               # rolling aggregates
$ iriq ./access.log -n                        # auto-detect file → normalize each
$ iriq -J < access.log                        # newline-delimited JSON
$ iriq --corpus team.db < access.log          # use a specific corpus file

Per-IRI output (-n, -p, -c, -J) streams — each line is read, classified, and flushed as it arrives, so iriq works on an unbounded live feed:

$ tail -f access.log | iriq -n                # one shape per line, as logs land
$ tail -f access.log | iriq -J                # same, as newline-delimited JSON

Every invocation observes into a persistent corpus by default, so iriq gets smarter the more you run it. The corpus-only types (e.g. enum / http_status) emerge from the distribution of values observed.

# Feed a stream where ?status only ever holds a couple of words:
$ for n in $(seq 1 20); do
    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=open"   >/dev/null
    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=closed" >/dev/null
  done

# Ask what it learned. ?status is now an enum — a verdict no single URL
# could support, since one URL shows only one value:
$ iriq --corpus demo.db cluster
[40] api.foo.com  /orders/{order_id}
    https://api.foo.com/orders/1?status=open
    https://api.foo.com/orders/1?status=closed
    https://api.foo.com/orders/2?status=open
    + 37 more
    status  enum  (2 distinct, 100%)

These learned types also flow into normalized output: once the corpus has pegged ?status as an enum, iriq -n …?status=open renders ?status={enum}.

The default corpus lives at $XDG_DATA_HOME/iriq/default.db (Linux), ~/Library/Application Support/iriq/default.db (macOS), or %LOCALAPPDATA%/iriq/default.db (Windows). First-run creation prints a one-line stderr notice. Three knobs control it:

$ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
$ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
$ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
$ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
$ iriq --reset                                     # delete the corpus DB and exit

Pick by the question you're asking:

• --canonical — clean up this URL, keeping the specifics. HTTP://Foo.com:80/pull/42 → http://foo.com/pull/42 (scheme/host lowercased, default port dropped; path and query left alone). Handy, but table stakes — plenty of libraries do it.
• --normalize (the default) — find the URL's shape, erasing the specifics into placeholders. …/pull/42 → …/pull/{id}. This is the part you came to iriq for.

Same input, two questions: "what's the clean form of this URL?" vs "what kind of URL is this?" The second is iriq's reason to exist.

# Homebrew (recommended)
brew install dpep/tools/iriq

# Cargo, from crates.io
cargo install iriq

One crate ships both the library and the iriq binary. Corpora persist to SQLite (bundled, WAL) out of the box — nothing to flag, install, or rebuild.

Iriq classifies each path/query segment into one of ~25 types — the first matching rule wins, and heuristics are deterministic:

• literal — plain word (users, orders, Profile, こんにちは)
• integer — pure digits below the timestamp range
• float — decimal with digits on both sides (3.14, -2.5, 1.0)
• boolean — true / false (any case)
• version — semver-ish with v prefix (v1, v2.0.1, v1.2.3-beta)
• locale — BCP 47-ish (en-US, fr_CA, zh-Hant, bare en/fr/ja)
• currency — ISO 4217 codes (USD, EUR, JPY)
• uuid — f47ac10b-58cc-4372-a567-0e02b2c3d479
• date — 2024-05-23, 2024/05/23, 20240523, 05/23/2024. Canonicalized to ISO in --normalize output.
• timestamp — ISO 8601, or 10/13-digit UNIX epoch
• hash — 32+ hex chars (md5 / sha)
• slug — my-cool-post, my_cool_post
• ipv4 / ipv6 — collapsed to {ip} in normalized output
• url — https://..., ftp://..., also scheme-less foo.com/path
• email — local@host.tld
• phone — E.164 (+15551234567) or NANP (555-666-7777, (555) 666-7777)
• jwt — three base64url segments separated by dots
• mime — image/png, application/vnd.api+json
• file — name.ext for known extensions; per-kind grouping (image/document/data/...)
• color — hex form (#fff, #ffffff, #ffffff80)
• coordinate — lat,lng pair with plausible-range validation
• country — ISO 3166-1 alpha-2 codes (US, JP, GB)
• base64 — standard base64 blobs with disambiguating +///=
• opaque_id — short alphanumeric mix that doesn't fit elsewhere

When a variable segment follows a literal one, iriq derives a hint by singularizing the literal and suffixing _id (or _uuid for UUIDs). That's what produces {user_id} from /users/123 and {order_id} from /orders/456. Semantic types (version, locale, currency, date, boolean) skip the hint and surface as {type} — /api/v1/status renders as /api/{version}/status, not the misleading /api/{api_id}/status. Pass -N / --no-hints for mechanical placeholders ({integer} instead of {user_id}).

Four types emerge from the distribution of values across many observations:

Mechanically, 200 is just an integer. Across ten thousand URLs where that slot is always 100–599, it's likely an HTTP status.

The corpus maintains rolling aggregates and per-(host, prefix) frequency stats, so classification improves as more data comes in — handy for an unbounded stream of identifiers. The default corpus already persists; --corpus PATH points iriq at a specific file instead, to keep separate corpora or share one across runs.

A .db / .sqlite / .sqlite3 path is stored in SQLite (WAL journaling, incremental UPSERTs — multiple iriq --corpus processes can write concurrently); a .json path writes a plain JSON file instead.

A corpus persists the source-IRI log alongside the materialized views. --reinfer drops every view and replays the log through the current classifier and reducers. Tune a threshold, swap in a different classifier, or activate new recognizers (below) — then reinfer to see the new results without re-feeding URLs.

$ iriq --corpus c.db --reinfer

Iriq doesn't just classify against a fixed list — it watches the stream and proposes new recognizers for patterns it keeps seeing. Notice ghp_… or cus_… recurring at a slug position and iriq will suggest a recognizer for it, with evidence: coverage, host count, confidence.

# Print proposals (human-readable, or --json)
$ iriq --corpus c.db --propose-recognizers

# Auto-activate every proposal with confidence ≥ 0.9, then reinfer
$ iriq --corpus c.db --propose-recognizers --activate-above 0.9

A route shape that recurs across multiple hosts is independent evidence of a semantic pattern — two unrelated hosts inventing the same /users/{integer} structure by accident is unlikely.

$ iriq --corpus c.db --cross-host-shapes [--min-hosts N]

The same signal feeds back into proposal confidence: each additional host beyond the first adds 0.05 to the score (capped at 1.0), so a prefix proposed on 5 hosts is meaningfully stronger than the same coverage seen on 1 host.

Pipe-mode extraction picks up explicit-scheme URLs (http, https, ftp, ws, wss, urn) and foo.com/path-style scheme-less URLs (small TLD allow-list, required path). It trims trailing sentence punctuation and preserves balanced parens (https://en.wikipedia.org/wiki/Ruby_(programming_language) stays intact; (see https://foo.com) drops the outer paren).

Known limitations (intentional):

• Comma is a URL boundary, so query strings like ?q=37.7,-122.4 truncate. Trade-off picked to keep CSV-shaped text working.
• No HTML entity decoding (& stays as-is).
• Scheme-less mode skips bare hostnames without a path (too noisy in prose).

Disable scheme-less extraction with --no-scheme-less.

Under the shape sits one idea: Position + Evidence. A Position is a slot in a host's structure — a typed path prefix, or a query-param name. Evidence is everything the corpus has observed about that slot: which values, how often, across how many hosts. Strings are observations; types are inferences drawn from the pile. Shape is the surface you see; Position + Evidence is the engine underneath. See docs/ARCHITECTURE.md for the full model.

Single input — combined parse + normalize summary; trim with section flags (-p, -n).

Piped stdin — extraction runs by default. Output auto-switches: small inputs get a deduplicated URL list, larger inputs (≥ 10 IRIs) get the cluster view via an ephemeral corpus.

Environment variables:

A positional argument that doesn't parse as an IRI but IS an existing file is read and extracted from automatically — iriq ./access.log and iriq /var/log/foo.log Just Work. (pipe with cat to disambiguate)

Exit codes: 0 success, 1 usage error, 2 parse error.

cargo add iriq

use iriq::{parse, normalize, Corpus};

let iri = parse("https://foo.com/users/123")?;
iri.host;             // "foo.com"
iri.path_segments;    // ["users", "123"]
iri.canonical();      // "https://foo.com/users/123"

normalize("https://foo.com/users/123")?;   // "https://foo.com/users/{user_id}"

// Streaming clustering against a persistent corpus.
let mut corpus = Corpus::open("c.db")?;
corpus.observe("https://foo.com/users/1")?;
corpus.save("c.db")?;

Full API on docs.rs/iriq; see the crate README for the library tour.

Iriq does not:

• Implement RFC 3986, RFC 3987, or the WHATWG URL standard fully.
• Convert between Unicode (IRI) and punycode (URI) — the display form is preserved as-is.
• Percent-encode or decode path/query bytes. Bytes are kept as written.
• Validate scheme-specific structure beyond URL vs. URN.
• Resolve relative references against a base URL.
• Round-trip canonical back to the exact original byte-for-byte (whitespace is stripped, default ports are dropped, dot segments are collapsed).

Iriq's focus is the analysis side: classification, normalization, and clustering — not a complete URL implementation.