Commenter: ISO Comment Sheet Processor

Purpose

Commenter is a Ruby gem for working with ISO comment sheets in DOCX format.

It provides utilities for parsing, manipulating, and serializing ISO comment data, converting between DOCX and structured YAML with schema validation.

The format is taken from:

"ISO/IEC/CEN/CENELEC electronic balloting commenting template/version 2012-03"

This gem only supports plain text comment extraction and filling. Only use this to handle plain text comments and resolutions.

Note	Mathematical formulas, images, and complex formatting are not supported due to limitations in the underlying docx gem. Comments containing such elements will have their text content extracted, but formatting and embedded objects will be lost.

Installation

Add this line to your application’s Gemfile:

gem 'commenter'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install commenter

Usage

Importing comments from DOCX

Convert an ISO comment sheet DOCX file to structured YAML:

$ commenter import "ISO 80000-2 review comments.docx" -o comments.yaml

This will create two files:

comments.yaml: The structured comment data
schema/iso_comment_2012-03.yaml: The YAML schema for validation

Import options

$ commenter import input.docx -o comments.yaml --exclude-observations --schema-dir schemas/

Options:

-o, --output FILE: Output YAML file (default: comments.yaml)
-e, --exclude-observations: Skip the observations column
--schema-dir DIR: Directory for schema file (default: schema)

Metadata (extraction limitation)

Warning

Due to a limitation in the underlying docx gem, metadata fields (Date, Document, Project) from the DOCX header cannot be automatically extracted. This is because the required functionality is not yet merged (see docx gem PR #73).

The generated YAML will have empty metadata fields that you can manually populate:

# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

version: "2012-03"
date: ""           # Manually add: e.g., "2023-04-25"
document: ""       # Manually add: e.g., "ISO 80000-2:2019"
project: ""        # Manually add: e.g., "Project name"
comments:

Alternatively, you can provide metadata during import using CLI options (planned for future release).

Example YAML output

# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

version: "2012-03"
date: "2023-04-25"
document: "ISO 80000-2:2019"
project: "Mathematics review"
comments:
  - id: DE-001
    body: DE
    locality:
      line_number:
      clause: "_whole document"
      element:
    type: ge
    comments: |
      The document should include more examples
      to clarify the implementation requirements.
    proposed_change: |
      Add section 4.5 with practical examples
      showing typical use cases.
    observations: |
      Accepted. Examples will be added in the
      next revision.
  - id: US-002
    body: US
    locality:
      line_number: "45"
      clause: "5.2.1"
      element: "Table 3"
    type: te
    comments: |
      The values in Table 3 appear to be
      inconsistent with the formula in 5.1.
    proposed_change: |
      Correct the values in column 2 of Table 3
      to match the calculation method.
    observations:

Filling DOCX templates from YAML

This gem contains a command-line utility to fill a DOCX template with comments from a YAML file. It generates a filled comment sheet that can be used for review and resolution tracking.

The base template is the ISO comment sheet template located at data/iso_comment_template_2012-03.docx. You can also provide a custom template file using the --template option.

Syntax:

$ commenter fill comments.yaml -o filled_comments.docx

Fill options

Options:

-o, --output FILE: Output DOCX file (default: filled_comments.docx)
-t, --template FILE: Custom template file
-s, --shading: Apply status-based cell shading

GitHub Integration

Overview

The commenter gem provides comprehensive GitHub integration for collaborative comment review and tracking. The workflow consists of two main commands:

github-create - Creates GitHub issues from comments and tracks them in YAML
github-retrieve - Retrieves final observations from closed GitHub issues

Creating GitHub issues from comments

Basic usage

$ commenter github-create --config github_config.yaml comments.yaml

This command creates GitHub issues for each comment and automatically updates the YAML file with GitHub issue information (issue numbers, URLs, status, timestamps).

Configuration file

Create a GitHub configuration file to specify repository, authentication, and issue settings:

github:
  repository: "owner/repo-name"
  token: "ghp_xxxxxxxxxxxx"
  milestone:
    name: "ISO 80000-2 DIS Review"
  default_labels: ["comment-review", "iso-standard"]
  stage_labels:
    WD: ["working-draft"]
    DIS: ["draft-international-standard"]

A sample configuration file is provided at data/github_config_sample.yaml.

Issue templates

General

The gem uses Liquid templates to format GitHub issue titles and bodies. Default templates are provided, but you can customize them.

Title template variables

The title template (data/github_issue_title_template.liquid) supports these variables:

stage - Approval stage (WD/CD/DIS/FDIS/PRF/PUB)
document - Document identifier
comment_id - Comment identifier
brief_summary - Generated summary combining locality and description
body - Member body abbreviation
type - Comment type code
clause, element, line_number - Location information

Body template variables

The body template (data/github_issue_body_template.liquid) supports all title variables plus:

comments - Full comment text
proposed_change - Proposed change text
observations - Secretariat observations
has_observations - Boolean for conditional rendering
has_proposed_change - Boolean for conditional rendering
project - Project name
date - Comment sheet date
version - Template version
type_full_name - Full comment type name (General/Technical/Editorial)
locality_summary - Formatted locality string

Custom templates

Create custom templates and reference them in your configuration:

github:
  templates:
    title: "custom_title.liquid"
    body: "custom_body.liquid"

Command options

-c, --config FILE - GitHub configuration YAML file (required)
--stage STAGE - Override approval stage
--milestone NAME - Override milestone name
--assignee HANDLE - Override assignee GitHub handle
--title-template FILE - Custom title template
--body-template FILE - Custom body template
--dry-run - Preview issues without creating them

Examples

Create issues with custom stage:

$ commenter github-create --config github_config.yaml --stage DIS comments.yaml

Preview issues before creation:

$ commenter github-create --config github_config.yaml --dry-run comments.yaml

Security considerations

For security reasons, it is recommended to use the GITHUB_TOKEN environment variable instead of storing the token in the configuration file.

$ export GITHUB_TOKEN=ghp_xxxxxxxxxxxx
$ commenter github comments.yaml --config github_config.yaml

Duplicate detection

The gem automatically checks for existing issues to avoid duplicates by searching for the comment ID in issue titles. If an issue already exists, it will be skipped.

Retrieving observations from GitHub issues

Basic usage

After GitHub issues have been created and reviewed, use the github-retrieve command to extract final observations from closed issues:

$ commenter github-retrieve --config github_config.yaml comments.yaml

This command:

Reads GitHub issue information from the YAML file (no searching required)
Fetches observations from closed GitHub issues only
Updates the same YAML file with extracted observations
Preserves all existing comment data

Magic comment syntax

To provide official observations in GitHub issues, use markdown blockquotes with special markers:

> **OBSERVATION:**
> Accepted. The proposed change will be included in clause 5.2.1 of the next revision.
> Technical details have been reviewed and approved by the working group.

Alternative shorter syntax:

> **COMMENTER OBSERVATION:**
> Noted. This will be considered for future revisions.

Retrieval options

-c, --config FILE - GitHub configuration YAML file (required)
-o, --output FILE - Output YAML file (default: update original)
--include-open - Include observations from open issues (not recommended)
--dry-run - Preview observations without updating YAML

Examples

Preview observations before updating:

$ commenter github-retrieve --config github_config.yaml --dry-run comments.yaml

Save to new file instead of updating original:

$ commenter github-retrieve --config github_config.yaml -o final_comments.yaml comments.yaml

Include observations from open issues:

$ commenter github-retrieve --config github_config.yaml --include-open comments.yaml

Enhanced YAML structure

After using github-create, your YAML file will include GitHub integration information:

comments:
  - id: US-001
    body: US
    locality:
      clause: "5.2.1"
      element: "Table 3"
    type: te
    comments: "The values in Table 3 appear inconsistent..."
    proposed_change: "Correct the values in column 2..."
    observations: ""
    github:
      issue_number: 123
      issue_url: "https://github.com/owner/repo/issues/123"
      status: "open"
      created_at: "2024-01-15T10:30:00Z"

After using github-retrieve (when the issue is closed):

comments:
  - id: US-001
    # ... other fields unchanged ...
    observations: "Accepted. The proposed change will be included in clause 5.2.1."
    github:
      issue_number: 123
      issue_url: "https://github.com/owner/repo/issues/123"
      status: "closed"
      created_at: "2024-01-15T10:30:00Z"
      updated_at: "2024-01-20T14:45:00Z"

Configuration for retrieval

The GitHub configuration file supports retrieval-specific settings:

github:
  repository: "owner/repo-name"
  retrieval:
    # Magic comment markers to look for
    observation_markers:
      - "**OBSERVATION:**"
      - "**COMMENTER OBSERVATION:**"

    # Fallback to last comment if no magic comment found
    fallback_to_last_comment: true

    # Only retrieve from closed issues (recommended)
    closed_issues_only: true

Comment ID format

Typical comment IDs follow the pattern: {MB/NC}-{number} or {MB/NC}-{org_id}-{seq_id}

US-001 - First comment from ANSI (US)
DE-01-002 - Second comment from organization 01 within DIN (DE)
**-001 - First comment from ISO secretariat

Where:

US = ANSI (American National Standards Institute)
DE = DIN (Deutsches Institut für Normung)
** = ISO Secretariat
CC = CalConnect

Comment types

The comment types are defined as follows:

ge: General comment
te: Technical comment
ed: Editorial comment

Workflow integration

flowchart LR
    A[ISO Comment Sheet DOCX] --> B[commenter import]
    B --> C[YAML + Schema]
    C --> D[commenter github-create]
    D --> E[YAML + GitHub Info]
    E --> F[GitHub Issues + Review Process]
    F --> G[commenter github-retrieve]
    G --> H[YAML + Observations]
    H --> I[commenter fill]
    I --> J[Final DOCX]
    J --> K[ISO Secretariat]

Shading rules

When the --shading option is used, the following status patterns are recognized and applied to the observations column:

Status Pattern	Intended Color	Hex Code	Example
`accept(ed)?`	Green	#92D050	"Accepted"
`awm\|accept with modifications`	Olive Green	#C4D79B	"Accept with modifications"
`noted`	Blue	#8DB4E2	"Noted"
`reject(ed)?`	Pink	#FF99CC	"Rejected"
`todo`	Diagonal stripes	#D9D9D9	"TODO: Review"

Data model

The comment structure follows this schema:

version: "2012-03"  # Template version
date: string | null # Comment sheet date (manually populated)
document: string | null # Document being reviewed (manually populated)
project: string | null  # Project name (manually populated)
comments:           # Array of comment objects
  - id: string      # Comment identifier
    body: string    # Member body abbreviation
    locality:       # Location information
      line_number: string | null
      clause: string
      element: string | null
    type: "ge" | "te" | "ed"  # Comment type
    comments: string          # Comment text
    proposed_change: string   # Proposed solution
    observations: string | null  # Secretariat observations (optional)
    github:         # GitHub integration information (optional)
      issue_number: integer   # GitHub issue number
      issue_url: string       # GitHub issue URL
      status: "open" | "closed"  # GitHub issue status
      created_at: string      # ISO 8601 timestamp
      updated_at: string      # ISO 8601 timestamp (optional)

Schema validation

Each exported YAML file includes a schema reference for IDE support:

# yaml-language-server: $schema=schema/iso_comment_2012-03.yaml

This enables:

Auto-completion in VS Code and other editors
Real-time validation
Inline documentation

Development

Getting started

After checking out the repo, run bin/setup to install dependencies:

$ git clone https://github.com/metanorma/commenter.git
$ cd commenter
$ bin/setup

Then, run rake spec to run the tests:

$ bundle exec rake spec

You can also run bin/console for an interactive prompt that will allow you to experiment:

$ bin/console

Running tests

The gem includes comprehensive test coverage for all major components:

# Run all tests
$ bundle exec rspec

# Run specific test files
$ bundle exec rspec spec/commenter/comment_spec.rb
$ bundle exec rspec spec/commenter/comment_sheet_spec.rb
$ bundle exec rspec spec/commenter/github_integration_spec.rb

# Run tests with coverage
$ bundle exec rspec --format documentation

Testing GitHub integration

To test the GitHub integration features:

Create a test repository on GitHub
Generate a personal access token with appropriate permissions
Create a test configuration file:

github:
  repository: "your-username/test-repo"
  default_labels: ["test-comment"]
  default_assignee: "your-username"

Test with dry-run mode first:

$ GITHUB_TOKEN=your_token bundle exec exe/commenter github test_comments.yaml --config test_config.yaml --dry-run

Note	For testing template rendering and dry-run functionality without a real GitHub token, you can use a dummy token:

$ GITHUB_TOKEN=dummy_token bundle exec exe/commenter github test_comments.yaml --config test_config.yaml --dry-run

This allows you to test the issue preview functionality, template rendering, and configuration parsing without making actual GitHub API calls.

Code structure

The gem is organized into several key components:

Core classes

Commenter::Comment: Represents individual comments with locality, type, and content
Commenter::CommentSheet: Container for multiple comments with metadata
Commenter::Parser: Handles DOCX parsing and YAML generation
Commenter::Filler: Fills DOCX templates with comment data
Commenter::GitHubIssueCreator: Creates GitHub issues from comments

CLI interface

Commenter::Cli

Thor-based command-line interface with subcommands:

import - Convert DOCX to YAML
fill - Fill DOCX template from YAML
github-create - Create GitHub issues from comments
github-retrieve - Retrieve observations from GitHub issues

Templates and configuration

data/iso_comment_template_2012-03.docx - Base DOCX template
data/github_issue_title_template.liquid - GitHub issue title template
data/github_issue_body_template.liquid - GitHub issue body template
data/github_config_sample.yaml - Sample GitHub configuration
schema/iso_comment_2012-03.yaml - YAML schema for validation

Debugging

Enable debug output for troubleshooting:

# Enable verbose output
$ bundle exec exe/commenter import input.docx --verbose

# Debug GitHub API calls
$ OCTOKIT_DEBUG=true bundle exec exe/commenter github comments.yaml --config config.yaml --dry-run

Troubleshooting

Common issues

DOCX parsing errors

Ensure the DOCX file follows the ISO comment template format
Check for corrupted or password-protected files
Verify table structure matches expected format

GitHub API errors

Verify your GitHub token has appropriate permissions
Check rate limiting if making many requests
Ensure repository exists and is accessible

Template rendering errors

Validate Liquid template syntax
Check that all referenced variables are available
Test templates with sample data first

Schema validation errors

Ensure YAML follows the required structure
Check for missing required fields
Validate comment ID format

Getting help

Check the issue tracker on GitHub
Review existing test cases for usage examples
Run commands with --help for detailed options
Use --dry-run mode to preview operations safely

Copyright

This gem is developed, maintained and funded by Ribose

License

The gem is available as open source under the terms of the 2-Clause BSD License.

commenter

Runtime

Commenter: ISO Comment Sheet Processor

Purpose

Installation

Usage

Importing comments from DOCX

Import options

Metadata (extraction limitation)

Example YAML output

Filling DOCX templates from YAML

Fill options

GitHub Integration

Overview

Creating GitHub issues from comments

Basic usage

Configuration file

Issue templates

General

Title template variables

Body template variables

Custom templates

Command options

Examples

Security considerations

Duplicate detection

Retrieving observations from GitHub issues

Basic usage

Magic comment syntax

Retrieval options

Examples

Enhanced YAML structure

Configuration for retrieval

Comment ID format

Comment types

Workflow integration

Shading rules

Data model

Schema validation

Development

Getting started

Running tests

Testing GitHub integration

Code structure

Core classes

CLI interface

Templates and configuration

Debugging

Troubleshooting

Common issues

Getting help

Copyright

License