twfilter

Provenance analysis for Chinese text: is this span Taiwan Mandarin (臺灣華語), and if not, on what evidence. Pure Ruby, no runtime dependencies, no native extensions, no network, deterministic output.

Traditional Chinese is written in Taiwan, Hong Kong and Macau, and by mainland publishers converting from simplified. Script alone does not identify origin. Every string below is written in traditional characters, and only three of them are Taiwan Mandarin.

TWFilter.keep?("這個政策有沒有經過完整評估？")    # => true
TWFilter.keep?("请把这个信息转发给同事")         # => false (simplified script)
TWFilter.keep?("請把這個信息轉發給同事")         # => false (lexicon: 信息, Taiwan writes 資訊)
TWFilter.keep?("他站在門裏面等了很久")           # => false (orthography: 裏, Taiwan writes 裡)
TWFilter.keep?("佢哋唔知道發生咗咩事")           # => false (Cantonese particles)
TWFilter.keep?("這事兒辦得挺漂亮的")             # => false (erhua suffix on 事)
TWFilter.keep?("他的女兒今年考上大學")           # => true (兒 here is a root morpheme)

Depth matters as much as breadth. Conversion tables are not provenance signals, and subject matter is not provenance either:

# 程序 is ordinary Taiwanese legal vocabulary, not a mainland form
TWFilter.keep?("行政程序法所定之程序應由主管機關進行")    # => true

# 了嗎 is attested in every native Taiwanese source; it is not a marker
TWFilter.keep?("你吃飯了嗎？我們等一下再出發。")    # => true

# Ambiguous vocabulary is flagged for review, not discarded
TWFilter.examine("這個視頻的畫質很不錯").marks.map(&:code)    # => [:mainland_soft]

# Taiwanese reporting about China is Taiwan Mandarin; the topic is tagged, not rejected
TWFilter.examine("行政院昨天回應了北京的最新聲明").marks.map(&:code)    # => [:foreign_topic]

# 南京 is also a Taipei street; an address suffix suppresses the topic tag
TWFilter.examine("臺北市南京東路五段的捷運站出口").findings    # => []

Provenance is decided on orthography, character inventory, lexical choice, morphology and a positive evidence signal, and every criterion that fired is reported.

Does — decide provenance of a span of text; normalize punctuation to the Taiwan standard; segment into sentences; report a character-difficulty tier; report every reason for every decision.

Does not — segment words, tag parts of speech, detect language (assume Han script input), translate, convert between simplified and traditional, or judge grammaticality. A span rejected here is not ungrammatical; it is not Taiwanese.

1. Learner-facing corpora. Extracted from TaiwanCards, where it gates every sentence, gloss and collocation before it can reach a card. The publishable policy encodes the boundary of the pedagogical inventory: MOE 常用 4 808 only, no ambiguous vocabulary, MOE punctuation only.
2. Web corpus construction. Filtering crawl output where the language identifier reports zho_Hant but the provenance is unknown. Traditional-script crawls conflate four written standards; this separates them at 10⁴ sentences per second per core.
3. Training-set curation. Building or auditing a monolingual set for a model intended to produce Taiwan Mandarin rather than converted mainland text. Report#findings is machine-readable, so exclusions stay attributable.
4. Editorial and localization QA. Detecting mainland vocabulary, orthography and punctuation in copy that is supposed to be Taiwanese — the mainland_soft band is designed for review queues rather than automatic rejection.

flowchart TD
  T["input span"] --> N["Punctuation.normalize<br/>教育部《重訂標點符號手冊》"]
  N --> SH

  subgraph SH["Checks::Shape"]
    A1["han count ∈ policy.han_range"]
    A2["han ratio ≥ policy.min_han_ratio"]
    A3["inventory ⊆ Han ∪ Latin ∪ Bopomofo ∪ punctuation"]
  end

  SH --> SC
  subgraph SC["Checks::Script — character inventory"]
    B1["simplified-only set<br/>OpenCC STCharacters ∖ TSCharacters ∖ Taiwan variants"]
    B2["converted orthography 裏着衞爲説麽綫墻眞啓衆産囯麪"]
    B3["MOE tier ≤ policy.max_tier"]
  end

  SC --> LX
  subgraph LX["Checks::Lexicon, Erhua, Wenyan"]
    C1["mainland hard ×149 → reject"]
    C2["mainland soft ×38 → mark"]
    C3["Cantonese characters ×17, HK/SEA lexis ×19 → reject"]
    C4["foreign topics ×80 → mark"]
    C5["erhua: 兒 outside a headed or tailed window → reject"]
    C6["literary density > 0.05 → reject"]
  end

  LX --> EV["Evidence — Taiwan lexicon, particles, institutions,<br/>有 + V perfective outside negation"]
  EV --> R["Report{ok?, findings[], tier, han, evidence}"]

Positive evidence never rejects. 了嗎 is not a defect: it is attested in every native source measured. 有 as a perfective marker counts in favour, never against — but only outside negation, since 沒有看 and 有沒有看 are pan-Mandarin and carry no Taiwan signal.

Input — anything responding to to_s. Empty, whitespace-only, Latin-only and supplementary-plane input are reported, not raised on. Invalid byte sequences must be scrubbed by the caller. There is no length limit; segmentation and normalization are linear in input length.

Output — TWFilter::Report:

A Finding is {check, code, severity, detail}. Codes are stable across releases; severity depends on the policy. Nothing is ever discarded silently — keep? is a convenience over examine, not a separate path.

Immutable value objects. Derive with #with.

TWFilter::Policy.corpus.with(max_tier: :secondary, han_range: (4..80))

require "twfilter"

# Provenance, with reasons
report = TWFilter.examine("请把这个信息转发给同事")
report.ok?                    # => false
report.reasons                # => ["simplified: 请这个转", "unlisted: 请这个转", "mainland: 信息"]
report.findings.map(&:code)   # => [:simplified, :unlisted, :mainland]

# Editorial review rather than rejection
report = TWFilter.examine("這個程式的運行狀況非常穩定")
report.ok?                    # => true
report.marks.map(&:detail)    # => ["運行"]
TWFilter::Checks::Lexicon.taiwan_form("運行")   # => "執行"

# Pedagogical gate
TWFilter.keep?("他每天都會去公園散步運動。", policy: TWFilter::Policy.publishable)  # => true
TWFilter.keep?("饕餮紋飾在青銅器上十分常見。", policy: TWFilter::Policy.publishable)  # => false

# Punctuation to the Taiwan standard
TWFilter.normalize("他說“好”...")        # => "他說「好」……"
TWFilter.normalize("馬丁·路德ＰＤＦ")     # => "馬丁‧路德PDF"

# Segmentation: 。！？…； terminate, closers stay attached, 、 does not break
TWFilter.sentences("他說：「好。」我走了。")   # => ["他說：「好。」", "我走了。"]

# Character difficulty
TWFilter.tier("散步")   # => 0
TWFilter.tier("饕餮")   # => 1

# Block judgment — a paragraph is accepted as a unit or not at all
TWFilter::Block.judge(paragraph_sentences)

# Table fingerprint, to record beside derived measurements
TWFilter.provenance
# => {version: "1.0.0", dir: "…/lib/twfilter/data", tables: 19, digest: "14cd9fd7fc97dc9a"}

Normalize before segmenting: the ellipsis and dash rules change sentence boundaries. Segment before examining: the shape thresholds are defined on sentences. Examine before deduplicating and counting, so rejected material never reaches the statistics.

acquire → normalize → segment → examine → deduplicate → tokenize → count
             │           │         │
       Punctuation   Sentences   Report

Block.judge is an optional stage between segmentation and examination for sources whose documents are topically coherent: it rejects a whole window when any member fails, and requires a minimum density of positive evidence across the window. Use it on crawls, not on dictionaries.

Existing Taiwan Mandarin resources control provenance at acquisition time: TaiwanWaC (260 M words) by collecting traditional-script web text, the Sinica TMC and NCCU spoken corpora by recording speakers in Taiwan. None of them publishes "is this span Taiwan Mandarin" as a reusable decision, and acquisition-time control does not transfer to new text.

The gap is measurable. Run over 37.8 M sentences that had already passed a .tw-host / html_lang: zh-TW admission gate (the twngrams build), these checks still removed 1.72 % on character inventory, a further 1.80 % on lexicon — mainland pairs, written Cantonese, Hong Kong and Singapore forms — and the block-evidence policy another 20.20 %: in total, 23 % of what URL metadata alone would have admitted. That residue is what a domain-gated corpus of traditional-script text contains unless a text-level instrument removes it.

Single core, Apple M5 Max, on 199 936 sentences of Taiwanese web text (6.2 M characters):

Suitable for corpora of arbitrary size. Cost is linear in input length and dominated by Checks::Lexicon, which scans 264 union-compiled patterns per span; the other four checks together account for under a third of the total. Memory is constant — tables load once (≈ 3 MB resident) and every entry point is a pure function over a string, so the work parallelizes by fork or by thread without coordination.

A 37.7 M-sentence corpus takes ≈ 3.5 minutes of wall time across 18 cores. In practice I/O, JSON parsing and deduplication dominate; this library is not usually the bottleneck.

lib/twfilter/data/ — plain text, one record per line, tab-separated where a record has fields. Diffable, reviewable and versioned with the gem.

Grading. Candidate pairs are graded by attestation ratio against a reference corpus of ten natively Taiwanese sources, 11 923 110 characters: hard when the mainland form is unattested and the Taiwan form occurs at least three times; soft when the Taiwan form dominates by 20 : 1 or the pair is hand-graded ambiguous; excluded otherwise. Most candidates are excluded — conversion tables built for orthographic mapping contain many pairs where both forms are ordinary Taiwanese.

The reference corpus admits no source with an open authorship gate. Open web comments on Taiwanese channels are Taiwan-hosted but not Taiwan-authored: every mainland form attested anywhere in the corpus occurs there and only there, so counting them would license the mainland vocabulary of mainland commenters as Taiwanese.

Membership. Each table states the rule that admits a row, and spec/twfilter/membership_spec.rb enforces it against the shipped data:

The rules exclude more than they admit. Subject matter is not provenance, so Taiwanese food names are not evidence; A-not-A interrogatives are pan-Mandarin, so 好不好 is not evidence; and a character in everyday Taiwanese use cannot mark another variety, however characteristic of it that character may be.

Topics are not provenance. foreign_topics.txt is a subject-matter tag. Taiwanese media covers the mainland constantly, several entries are ordinary ROC administrative vocabulary, and Taipei streets are named after mainland cities. It marks under corpus and rejects only under publishable, where excluding foreign subject matter is an editorial decision.

Preferred: contribute to the published dataset. The tables are maintained at huggingface.co/datasets/taiwan-corpora/twfilter-tables, where each release carries its grading evidence, and the gem vendors a copy of a specific version. A correction accepted there reaches every consumer at the next release, with a record of what changed and why — which a local edit does not.

Three table sources are supported. All three are reported by provenance.

Bundled — the default, versioned with the gem, the only configuration whose results are comparable with anyone else's.

Replacement — a directory holding the full set, e.g. a checkout of the dataset newer than the gem:

TWFilter::Tables.dir = "path/to/twfilter-tables"   # or TWFILTER_DATA_DIR

Overlay — the bundled set plus your additions. Filename suffixes select the operation:

TWFilter::Tables.overlay = "path/to/house-style"    # or TWFILTER_OVERLAY_DIR

Row format is validated on load and a violation raises TWFilter::InvalidTableError:

Files named *.remove.* carry keys only and are not shape-checked. UTF-8, one record per line, no header, no comments; blank lines ignored.

Record provenance alongside any measurement derived this way. Two corpora filtered with different tables are not comparable, and the digest is what makes that detectable.

Policy thresholds are the supported knob and need no table changes:

lenient = TWFilter::Policy.corpus.with(soft_lexicon_rejects: false, foreign_topics_reject: false)

bundle install
bundle exec rake          # specs and RBS validation
bundle exec rake tables   # regenerate the tables and their manifest

Specs require no data, no network and no credentials.

@software{patin_twfilter,
  title     = {twfilter: Taiwan Mandarin variety identification},
  author    = {Patin, Den},
  year      = {2026},
  version   = {0.1.0},
  doi       = {10.5281/zenodo.21761465},
  publisher = {Zenodo},
  url       = {https://github.com/taiwan-corpora/twfilter}
}

10.5281/zenodo.21761465 is the concept DOI and always resolves to the latest release; cite 10.5281/zenodo.21761466 when the exact version matters. The reference tables are published separately as taiwan-corpora/twfilter-tables.

Code: MIT. Tables: mixed, all permitting commercial use. See NOTICES.md for per-table provenance and terms.

Two attribution strings must be reproduced by anything shipping this gem:

數位發展部，CNS11643中文標準交換碼全字庫網站
OpenCC — https://github.com/BYVoid/OpenCC, Apache License 2.0

No table carries a NonCommercial, ShareAlike or copyleft term. Verdicts and counts produced by the gem are the caller's, not derivatives of the tables.