Project

purehtml

0.0
There's a lot of open issues
PureHTML is a parsing specification and library for extracting JSON from HTML documents using simple YAML schemas.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
 Dependencies

Development

>= 5.0
>= 12.0

Runtime

>= 1.10.0
 Project Readme

PureHTML

Documentation & Playground License: MIT

PureHTML is a declarative, language-agnostic specification and toolkit for extracting structured JSON data from HTML documents using simple YAML or JSON schemas.

Instead of writing repetitive, error-prone DOM querying and scraping logic in application code, PureHTML allows you to describe what data to extract and how to transform it declaratively. The schema can be authored, validated, and shared across different services and programming languages.

Explore the documentation and interactive playground at https://purescraps.github.io/purehtml/.


Language Ports

This monorepo houses official implementations of the PureHTML specification across multiple languages, all validated against a shared compliance test suite:

Language Directory Package / Module Badges
TypeScript / JavaScript typescript/ @purescraps/purehtml npm
Go go/ github.com/purescraps/purehtml/go Go Reference
Python python/ purehtml-purescraps PyPI
Java java/ com.purescraps.purehtml:purehtml Maven Central
Ruby ruby/ purehtml Gem Version

How It Works

PureHTML separates data extraction into declarative schemas, an HTML document input, and an extraction engine.

1. Define a Schema (extract.yaml)

selector: .product-card
type: array
items:
  properties:
    title:
      selector: h2.title
      transform: trim
    price:
      selector: .price
      transform: [trim, number]
    link:
      selector: a.buy-btn
      transform:
        - attr(href)
        - resolve(https://example.com)

2. Run with Your Language of Choice

```typescript
import { ConfigFactory, extract } from '@purescraps/purehtml';

const config = ConfigFactory.fromYAML(yamlString);
const result = extract(htmlString, config, 'https://example.com');
```
<!-- slide -->
```go
import "github.com/purescraps/purehtml/go"

factory := purehtml.NewConfigFactory()
config, _ := factory.FromYAML(yamlString)
result, _ := purehtml.ExtractFromString(purehtml.DefaultBackend, htmlString, config, "https://example.com")
```
<!-- slide -->
```python
from purehtml import ConfigFactory, extract

config = ConfigFactory.from_yaml(yamlString)
result = extract(config, htmlString, "https://example.com")
```
<!-- slide -->
```java
import com.purescraps.purehtml.configs.ConfigFactory;
import static com.purescraps.purehtml.PureHTML.extract;

var config = ConfigFactory.fromYAML(yamlString);
var result = extract(config, htmlString, "https://example.com");
```
<!-- slide -->
```ruby
require 'purehtml'

config = PureHTML::ConfigFactory.from_yaml(yamlString)
result = PureHTML.extract(config, htmlString, "https://example.com")
```

Key Features

  • Declarative Schemas: YAML-based definitions for extracting primitives, nested objects, arrays, constants, unions, and default fallbacks.
  • Fallback Selectors: Supply an array of selectors (selector: ['.sale-price', '.price']) or selector objects with sampleHTMLs for resilient scraping when websites A/B test or change their DOM structure.
  • Chainable Transformers: 20+ built-in transformers including trim, number, boolean, attr, html, text, replace, split, join, json, jsonpath, resolve, urlQueryParam, and more with support for quoted arguments and regex patterns.
  • Cross-Language Compliance: All ports are continuously tested against shared test fixtures in specs/ to guarantee identical parsing and transformation behavior.
  • Zero Runtime Overhead Schemas: Pre-compile schemas once and execute them efficiently over millions of documents.

Specification Highlights

Fallback Selectors (Resilient Scraping & A/B Testing)

Websites often vary their DOM structure across A/B test variants or responsive views. selector can be configured as an array of fallback selectors evaluated in order:

selector:
  - .flash-deal-price
  - .sale-price
  - .regular-price
transform: trim

The schema also supports selector objects with sampleHTMLs, allowing you to document and reference HTML fixtures alongside each selector variant:

selector:
  - selector: .promo-price
    sampleHTMLs:
      - tests/fixtures/promo.html
  - selector: .standard-price
    sampleHTMLs:
      - tests/fixtures/standard.html

Transformer Argument Quoting Rules

Transformers support parameter lists, but arguments containing spaces, commas, colons, or regex escape sequences must follow quoting rules:

  • Bare (unquoted) arguments: Simple alphanumeric identifiers matching [\w-]+ can be written without quotes (e.g. attr(href), trim, split(-)).
  • Quoted arguments ("..." or '...'): Required when arguments contain spaces, commas, colons, or regex metacharacters:
    • replace("\s+", " ") or 'replace(",", "")'
    • split(", ") or 'split("::")'
    • join(" - ") or join(", ")
    • urlQueryParam("utm_source") or urlQueryParam('ref')
  • Escaping inside quotes: Backslashes escape matching quote characters (\" or \') or literal backslashes (\\). Regex escape sequences like \s, \d, and \w pass through untouched to regex engines.
  • YAML tip: In YAML schemas, wrap transform expressions in outer quotes (e.g. transform: 'replace("\s+", " ")') or use YAML array syntax (- replace("\s+", " ")) to prevent YAML parsing conflicts.

Recipes & Real-World Patterns

1. Scraping Next.js / Nuxt / JSON-LD Data

Modern frameworks (Next.js, Nuxt, Remix) and SEO-rich web pages embed structured JSON data directly inside <script> tags (script#__NEXT_DATA__, script#__NUXT_DATA__, or script[type="application/ld+json"]). Extracting directly from these hydration scripts is vastly faster and more durable than querying volatile presentation CSS classes.

Combine json with jsonpath to pluck properties directly:

properties:
  productId:
    selector: script#__NEXT_DATA__
    transform:
      - json
      - 'jsonpath("$.props.pageProps.product.id")'
  productName:
    selector: script#__NEXT_DATA__
    transform:
      - json
      - 'jsonpath("$.props.pageProps.product.title")'
  price:
    selector: script#__NEXT_DATA__
    transform:
      - json
      - 'jsonpath("$.props.pageProps.product.price")'
  jsonLdPrice:
    selector: 'script[type="application/ld+json"]'
    transform:
      - json
      - 'jsonpath("$.offers.price")'
      - number

2. Handling Pagination & Relative Links

Scrapers and crawlers frequently encounter relative links in pagination navigation (<a class="next" href="/catalog/page-3">Next</a>) or listing item anchors (<a href="details/401">).

Combine attr(href) with resolve to automatically resolve relative URLs against the base URL provided to extract():

properties:
  currentPage:
    selector: .pagination .current
    transform:
      - 'replace("Page ", "")'
      - trim
      - number
  nextPageUrl:
    selector: .pagination a.next
    transform:
      - attr(href)
      - resolve
  items:
    selector: ul.items li a
    type: array
    items:
      properties:
        name:
          transform: trim
        url:
          transform:
            - attr(href)
            - resolve

3. Resilient Data Extraction: Combining Union and Default

Real-world pages frequently exhibit variable markup due to A/B testing, promo badges, or optional fields (e.g., discounted sale prices vs. standard prices, out-of-stock items missing prices, or promotional items marked "FREE").

Combine union with a fallback constant value at the end to gracefully handle variant structures with reliable defaults:

selector: .products .product
type: array
items:
  properties:
    name:
      selector: .name
      transform: trim
    price:
      union:
        - selector: .sale-price
          transform:
            - 'replace("\$", "")'
            - trim
            - number
        - selector: .price
          transform:
            - 'replace("\$", "")'
            - trim
            - number
        - selector: .free-text
          transform:
            - 'replace("FREE", "0")'
            - number
        - constant: 0
    status:
      union:
        - selector: .badge
          transform: trim
        - constant: "Standard"

4. Real-World Case Study: Hacker News Scraper

See docs/src/examples/advanced.hackernews-listing.ts for a production-ready Hacker News listing scraper that extracts post IDs, titles, URLs, timestamps, scores, and comment links:

selector: tr.athing.submission
items:
  properties:
    id: { transform: attr(id) }
    title: { selector: span.titleline }
    url: { selector: 'span.titleline a:first-child', transform: [attr(href), resolve] }
    date: { selector: '+tr span.age', transform: attr(title) }
    score: { selector: '+tr span.score' }
    comments: { selector: '+tr span.age a', transform: [attr(href), resolve] }

Explore and test this example interactively in the Playground.


Specification & Testing

The PureHTML schema definition is located at spec.schema.yaml and typescript/src/config-schema.json.

Shared cross-language test suites are maintained in specs/. Each language implementation runs these specifications to ensure complete compatibility:

  • Go: cd go && go test -v
  • Python: cd python && python3 run_tests.py
  • Java: cd java && mvn test
  • TypeScript: cd typescript && pnpm test
  • Ruby: cd ruby && rake test

IDE Support & Autocompletions

You can use yaml-language-server for schema validation and autocompletions in YAML files.

Add this modeline to the top of your YAML schema:

# yaml-language-server: $schema=typescript/src/config-schema.json

For VS Code, install the Red Hat YAML extension.


License

MIT